Slides March 25th, 2015

Communication Patterns for Embedded
System Design
Johan Philips
University of Leuven, Belgium
http://people.mech.kuleuven.be/~jphilips/
March 25, 2015
Communication Patterns for Embedded System Design
Johan Philips
March 25, 2015
1
Objectives of this talk
I
discuss different communication methods and their design
implications
I
give a brief introduction to computer networking: network
models, protocols and topologies
I
define a set of communication patterns to address common
issues (ZeroMQ)
I
apply lessons learned to currently ongoing EU-projects
SHERPA and Pick-n-Pack
We will NOT cover messaging formats and serialisers
Communication Patterns for Embedded System Design
Johan Philips
March 25, 2015
2
Communication Examples
Analog world
I telephone call
I conversation
I oral examination
I leave a note
I passing in team sports
I using the TV remote
I sending a letter
I filling in an evaluation report
I writing on blackboard
I handshaking
Communication Patterns for Embedded System Design
Johan Philips
March 25, 2015
3
Communication Examples
Digital world
I sending an email
I writing on electronic blackboard
I writing data to file
I posting a message on Facebook
I chatting with your girlfriend
I publishing sensor data on data bus
I playing WoW with your friends
I clicking on a news item online
I opening a folder on your computer
I tweeting about your latest achievement
Communication Patterns for Embedded System Design
Johan Philips
March 25, 2015
4
Asynchronous vs synchronous
Asynchronous
I takes place outside of real time → time lag
I no need to wait for reply → flexible
I data is transmitted intermittently → notification required
I lack of immediacy → information can be out of date
Synchronous
I similar to a conversation → no time lag
I real time responses → involved parties need to be ready and
willing
I immediacy → no need to wait for reply
I more interactive than asynchronous communication
Communication Patterns for Embedded System Design
Johan Philips
March 25, 2015
5
Communication in software systems
Depending on the deployment of your system (of systems),
communication can occur
I within one process
I
between processes on the same machine
I
between processes on different machines in same network
I
between processes on different machines in different networks
Communication Patterns for Embedded System Design
Johan Philips
March 25, 2015
6
Specific challenges in embedded systems
I
Concurrency: how to interconnect many nodes?
I
High performance: how to ensure communication does not
block computations?
I
Small footprint: how to provide communication with a
minimal footprint (memory, CPU)?
→ We need design patterns for communication! But first some
more Computer Networking 101...
Communication Patterns for Embedded System Design
Johan Philips
March 25, 2015
7
The OSI model
Open Systems Interconnection model
I application layer: user-facing application
I presentation layer: crypto, conversion between
representations
I session layer: tie together multiple streams
I transport layer: multiplexing applications
I network layer: move packets to any other node in the
network
I data link layer: move frames to other node across link
I physical layer: move bits to other node across link
→ On which levels have we been focussing during this course?
Communication Patterns for Embedded System Design
Johan Philips
March 25, 2015
8
Internet (TCP/IP) model
Presentation and session layer mixed with application and
transport layer and no physical layer
I
application layer
I
I
I
I
provide user services
formatting specifics and data presentation part of libraries and
APIs
treat transport layer as black box which provides stable
network connection
transport layer
I
I
unconcerned with specifics of higher level protocols
no examination of encapsulated traffic
internet layer
link layer
In embedded systems, lightweight IP (lwIP) is widely used open
source TCP/IP stack
I
I
Communication Patterns for Embedded System Design
Johan Philips
March 25, 2015
9
IP stack connections
Network Topology
Host
A
Router
Host
B
Router
Data Flow
Application
process-to-process
Application
Transport
host-to-host
Transport
Internet
Internet
Internet
Internet
Link
Link
Link
Link
Fiber,
Satellite,
etc.
Ethernet
Ethernet
”IP stack connections” by en:User:Kbrose - Prior Wikipedia artwork by en:User:Cburnett. Licensed under CC BY-SA
3.0 via Wikimedia Commons http://commons.wikimedia.org/wiki/File:IP stack connections.svg#/media/File:IP stack connections.svg
Communication Patterns for Embedded System Design
Johan Philips
March 25, 2015
10
Data encapsulation
Application
Data
UDP UDP
header data
Transport
IP data
Internet
IP
header
Frame
header
Frame data
Frame
footer
Link
”UDP encapsulation” by en:User:Cburnett original work, colorization by en:User:Kbrose - Original artwork by
en:User:Cburnett. Licensed under CC BY-SA 3.0 via Wikimedia Commons http://commons.wikimedia.org/wiki/File:UDP encapsulation.svg#/media/File:UDP encapsulation.svg
Communication Patterns for Embedded System Design
Johan Philips
March 25, 2015
11
Challenges to layered model
I
What belongs to which layer?
I
How do you divide services?
I
Which abstractions do you make?
I
What do you leave out?
Communication Patterns for Embedded System Design
Johan Philips
March 25, 2015
12
Some common protocols
What do you need to communicate and who do you talk to?
I
I
I
I
I
I
I
application layer: HTTP, SMTP, Telnet, ...
presentation layer: MIME, ...
session layer: RTP, RTCP, ...
transport layer: TCP, UDP, ...
network layer: IP, ICMP, ...
data link layer: MAC, ARP, ...
physical layer: Ethernet, CAN bus, GSM, IEEE 802.11,
Bluetooth, RS-232, ...
Communication Patterns for Embedded System Design
Johan Philips
March 25, 2015
13
OSI model usage in embedded systems
I
Embedded systems typically operate on the lowest layers in
the OSI model
I
Trend to more complex embedded systems and systems of
systems also require higher layers!
Communication Patterns for Embedded System Design
Johan Philips
March 25, 2015
14
Meanwhile, on the physical layer...
... different topologies and hardware architectures exist!
I based on size: PAN, LAN, MAN, WAN
I based on purpose: SAN, EPN, VPN
I based on topology: P2P, bus, star, ring, mesh, tree, cloud
→ We need communication patterns to address some physical
requirements and/or limitations we might have!
Communication Patterns for Embedded System Design
Johan Philips
March 25, 2015
15
Intermezzo: programming models
Synchronous model
I single-threaded
I simple programming style
I each task is completely finished
before another starts
I particular order in which tasks are
executed → scheduling
I e.g. Arduino car
Figure from Dave Peticolas’ introduction on asynchronous programming
Communication Patterns for Embedded System Design
Johan Philips
March 25, 2015
16
Intermezzo: programming models (2)
Threaded model
I each task is executed in a separate thread of control
I OS manages threads
I tasks are implicitly scheduled by OS
I complex due to inter thread communication and coordination
I e.g. ODROID board
Figure from Dave Peticolas’ introduction on asynchronous programming
Communication Patterns for Embedded System Design
Johan Philips
March 25, 2015
17
Intermezzo: programming models (3)
Asynchronous model
I tasks are interleaved
I single thread of control
I explicitly schedule tasks
I performs well with large number of
tasks (less waiting time, no context
switches, ...)
I e.g. composition pattern
Figure from Dave Peticolas’ introduction on asynchronous programming
Communication Patterns for Embedded System Design
Johan Philips
March 25, 2015
18
Intermezzo: programming models (4)
Event-driven programming
I
I
I
I
I
no blocking for I/O
program flow is determined by events
callbacks to receive notifications
very common in GUI frameworks
Reactor pattern to separate event logic
from business logic
Communication Patterns for Embedded System Design
Johan Philips
March 25, 2015
19
Remember? Communication in an Activity
when triggered
do {
communicate() % get latest events
coordinate()
% react to them
configure()
% possibly requiring reconfiguration
schedule()
% now do one’s Behaviour
coordinate()
% execution could trigger new events
communicate() % that others might want to know about
log()
}
Communication Patterns for Embedded System Design
Johan Philips
March 25, 2015
20
Back to Communication: Sockets
I
I
I
endpoint of inter-process communication flow
follow the protocol specifications that define interfaces
typically interface between application layer and transport
layer
Communication Patterns for Embedded System Design
Johan Philips
March 25, 2015
21
ZeroMQ
I
I
I
socket library, but much more...
models for communication
patterns addressing typical networking issues
images copyright (c) 2014 iMatix Corporation and licensed under cc-by-sa 3.0
Communication Patterns for Embedded System Design
Johan Philips
March 25, 2015
22
ZeroMQ - zero-em-queue - MQ
I
Connect your code in any language, on any platform.
I
Support for many protocols: inproc, IPC, TCP, TIPC, multicast.
I
Smart sockets like pub-sub, push-pull, and router-dealer.
I
High-speed asynchronous I/O engines, in a tiny library.
I
Backed by a large and active open source community.
I
Build any architecture: centralised, distributed, small, or large.
I
Free software with full commercial support.
adopted from zeromq.org, copyright (c) 2015 iMatix Corporation and licensed under cc-by-sa 3.0
Communication Patterns for Embedded System Design
Johan Philips
March 25, 2015
23
SHERPA project
I
I
I
I
I
Search & rescue in alpine environments
Hetereogeneity of actors: human,
ground vehicle and aerial robotic platforms
Distributed cognitive world model
QoS for network infrastructure
http://sherpa-project.eu/
Communication Patterns for Embedded System Design
Johan Philips
March 25, 2015
24
Pick-n-Pack project
I
I
I
I
I
Flexible production line in food industry
Line controller and several modules with
one or more devices
Traceability of products, processes and
processors
Communication and interaction between
all actors
http://www.picknpack.eu/
Communication Patterns for Embedded System Design
Johan Philips
March 25, 2015
25
Communication challenges in these projects
I
I
I
I
I
I
I
I
I
Uncertain communication network
Dynamic configurations of multiple entities/robots
Synchronisation of mission execution/production
Bootstrapping and initialisation of coooperative robot
application
Tracing of data and intermediate (mission) states
Failure detection
Highly configurable communication protocol and
infrastructure
Deployment based communication and data optimisations
...
Communication Patterns for Embedded System Design
Johan Philips
March 25, 2015
26
REQ-REP
I
I
I
I
Example: GUI sending request to
robot controller, HTTP request
Typically used in synchronised
Client/Server model
Client sends request to Server and
(actively) waits for reply
Server (actively) waits for request
from Client and sends reply when
request is received
Client
REQ
Hello
World
REP
Server
images copyright (c) 2014 iMatix Corporation and licensed under cc-by-sa 3.0
Communication Patterns for Embedded System Design
Johan Philips
March 25, 2015
27
PUB-SUB
I
I
I
I
Example: Sensor broadcasting data
measurements
Typically used in asynchronised p2p
model
Publisher write data to PUB socket
Subscriber subscribes to SUB
socket and receives a notification if
new data arrives
Publisher
PUB
bind
updates
updates
updates
updates
connect
connect
SUB
SUB
Subscriber
Subscriber
images copyright (c) 2014 iMatix Corporation and licensed under cc-by-sa 3.0
Communication Patterns for Embedded System Design
Johan Philips
March 25, 2015
28
PUSH-PULL & PAIR-PAIR
Ventilator
Step 1
PUSH
PAIR
tasks
Ready!
PULL
PULL
PAIR
Worker
Worker
Step 2
PUSH
PUSH
PAIR
Ready!
results
I
I
I
PULL
PAIR
Sink
Step 3
Synchronise work flow between different computations within
one process (PAIR) or between processes (PUSH-PULL)
PAIR: optimise by using shared memory
PUSH-PULL: divide work (ventilator) load and collect results
(sink)
images copyright (c) 2014 iMatix Corporation and licensed under cc-by-sa 3.0
Communication Patterns for Embedded System Design
Johan Philips
March 25, 2015
29
ROUTER-DEALER
Client
I
DEALER
I
I
ROUTER
Server
I
Asynchronous request-reply
N to N communication
Router sockets adds addressee in
message header to route replies
correctly
Dealer sockets enforce explicit
bookkeeping in application code
images copyright (c) 2014 iMatix Corporation and licensed under cc-by-sa 3.0
Communication Patterns for Embedded System Design
Johan Philips
March 25, 2015
30
Patterns on http://zguide.zeromq.org
I
I
I
I
I
I
I
I
Binary Star
Last value caching
Relay race
Freelance
Simple Pirate
Node coordination
Suicidal snail
I
I
I
I
I
Espresso
Lazy Pirate
Parallel task
distribution and
collection
Bridge
RPC and task
distribution
Shared queue
I
I
I
I
I
I
I
Communication Patterns for Embedded System Design
Johan Philips
March 25, 2015
Parallel pipeline w/
kill signalling
Load balancing
Majordomo
Paranoid Pirate
One-way data
distribution
Titanic
Black box
31
Patterns on http://zguide.zeromq.org
I
I
I
I
I
I
I
I
Binary Star
Last value caching
Relay race
Freelance
Simple Pirate
Node coordination
Suicidal snail
I
I
I
I
I
Espresso
Lazy Pirate
Parallel task
distribution and
collection
Bridge
RPC and task
distribution
Shared queue
I
I
I
I
I
I
I
Communication Patterns for Embedded System Design
Johan Philips
March 25, 2015
Parallel pipeline w/
kill signalling
Load balancing
Majordomo
Paranoid Pirate
One-way data
distribution
Titanic
Black box
32
Node coordination
Publisher
PUB
REP
I
(3)
(2)
I
(1)
I
SUB
Synchronise initialisation
Separate channel to
bootstrap communication
Solves late joiner problem
REQ
Subscriber
images copyright (c) 2014 iMatix Corporation and licensed under cc-by-sa 3.0
Communication Patterns for Embedded System Design
Johan Philips
March 25, 2015
33
Paranoid pirate
Client
Client
Retry
Retry
REQ
REQ
I
I
ROUTER
Queue
Heartbeat
I
ROUTER
DEALER
DEALER
Heartbeat
Heartbeat
Worker
Worker
Heartbeating between
peers: e.g. server and its
workers
(Basic) discovery of modules
ready to provide particular
service
Asynchronous
communication between
requester (client), provider
(server) and executor
(worker)
images copyright (c) 2014 iMatix Corporation and licensed under cc-by-sa 3.0
Communication Patterns for Embedded System Design
Johan Philips
March 25, 2015
34
Espresso
Publisher
Publisher
PUB
PUB
connect
connect
I
connect
I
bind
XSUB
Proxy
XPUB
bind
I
connect
connect
SUB
SUB
Subscriber
Subscriber
I
Plugin a proxy between
PUB-SUB
Log all or some
communication
(configurable)
Limit data transfer
Convert data formats
between peers
images copyright (c) 2014 iMatix Corporation and licensed under cc-by-sa 3.0
Communication Patterns for Embedded System Design
Johan Philips
March 25, 2015
35
SHERPA relevant patterns
I
Node coordination (Synchronised PUB-SUB & REQ-REP)
I
I
Paranoid Pirate (DEALER-ROUTER-ROUTER-DEALER)
I
I
I
I
initialisation between base stations and robots assigned to a
mission
discovery of new UAVs by base station
detecting out-of-range issues in large missions
reliable & robust communication with heart beating
Espresso and/or Last value caching
I
I
I
local and/or remote data logging
allows distributed world model sharing
enables mission tracking
Communication Patterns for Embedded System Design
Johan Philips
March 25, 2015
36
Pick-n-Pack relevant patterns
I
Node coordination (Synchronised PUB-SUB & REQ-REP)
I
I
Paranoid Pirate (DEALER-ROUTER-ROUTER-DEALER)
I
I
I
I
initialisation of Line Controller and PnP Modules
discovery of new Modules by Line Controller
reliable & robust communication with heart beating
failure detection
Espresso (PUB-XSUB, XPUB-SUB & PAIR-PAIR)
I
I
data logging in Modules
enables traceability
Communication Patterns for Embedded System Design
Johan Philips
March 25, 2015
37
Real world (Embedded Systems) use cases
I
I
I
I
I
I
I
I
Discovery - how do we detect peers in a network?
Presence - how do we track dynamic nodes?
Connectivity - how do we connect nodes?
Point-to-point messaging - how do we send messages?
Group messaging - how do we multicast messages?
Testing and simulation - how do we simulate large
networks and test performance?
Distributed logging - how do we log data in a cloud of
nodes?
Content distribution - how do we send content around?
→ Centralise everything or create a distributed solution?
adopted from chapter 8 of the ZeroMQ guide
Communication Patterns for Embedded System Design
Johan Philips
March 25, 2015
38
Further Reading
Communication Patterns for Embedded System Design
Johan Philips
March 25, 2015
39
RFC and STD
RFC → http://ietf.org/rfc
I Request For Comments by Internet Engineering Task Force
(IETF) and Internet Society
I Describes novel concepts submitted for peer review
I Some RFCs become standards
STD
I Special case of RFC or set of RFCs which has been turned
into an Internet Standard
I http://tools.ietf.org/html/rfc5000 contains (long) list of
STDs
All information is publicly available!
Communication Patterns for Embedded System Design
Johan Philips
March 25, 2015
40
IP
Internet Protocol
I
I
I
I
Hourglass architecture of the Internet: IP works over many
network types with many (application) protocols on top
Used by most computer networks today
IPv4 (32-bit address) vs IPv6 (128-bit address) → allows
3.4x1038 addresses
Does not provide much service → encapsulate another
protocol within IP
Communication Patterns for Embedded System Design
Johan Philips
March 25, 2015
41
UDP & TCP
User Datagram Protocol
I
I
I
packet-switching
adds multiplexing on top of IP
unreliable: packets can be dropped, reordered, duplicated
Transmission Control Protocol
I
I
I
provides illusion of reliable pipe between two processes
handles congestion and flow control
requires connection and handshaking
Communication Patterns for Embedded System Design
Johan Philips
March 25, 2015
42
HTTP
HyperText Transfer Protocol
I
Application layer protocol typically on top of TCP/IP
I
Request / Reply communication in Client / Server model
I
Used in the World Wide Web but not exclusively
I
Many embedded devices have server running that supports
HTTP!
Communication Patterns for Embedded System Design
Johan Philips
March 25, 2015
43
ROUTER-DEALER extended example
Client
Broker
statebe
connect
bind
I
I
I
request
request
state
connect
connect
Worker
Broker
statefe
Local request-reply flow between
broker and its clients and workers.
Cloud request-reply flow between
broker and its peer brokers.
State flow between broker and its
peer brokers.
images copyright (c) 2014 iMatix Corporation and licensed under cc-by-sa 3.0
Communication Patterns for Embedded System Design
Johan Philips
March 25, 2015
44
Why real world application need distributed
solutions?
I
I
I
I
I
Dawn of Internet of Things (IoT) → exponential increase of
devices over time
Using wires is so 20th century...
Control shifts away from the center: star topology is being
replaced by the cloud of clouds (intercloud).
Networks are increasingly collaborative, less controlled: BYOD
The cost of connecting a new node must fall proportionally →
reduce configuration
Communication Patterns for Embedded System Design
Johan Philips
March 25, 2015
45
ZRE and ZOCP
ZeroMQ Realtime Exchange Protocol (ZRE) http://rfc.zeromq.org/spec:36/ZRE
I Auto discovery of group of peers in a network
I Uses ZMTP, a transport layer protocol for exchanging
messages
I Goals: robust on poor networks, work without centralised
service
Z25 Orchestration Control Protocol (ZOCP) https://github.com/z25/pyZOCP
I Goal: control multiple computers in live performances → low
latency, reliability
I Zero configuration, open standards, KISS
I Declare capability models to allow data exchange
Communication Patterns for Embedded System Design
Johan Philips
March 25, 2015
46
Concluding remarks
I
I
I
I
I
Communication is everywhere and at many levels!
Existing standards in computer networks offer many
solutions at lower OSI layers
Communication patterns are essential to model and design
applications
Embedded systems will become (if not already) omnipresent
in our daily life
Distributed solutions are required to tackle communication
challenges in system of systems
Communication Patterns for Embedded System Design
Johan Philips
March 25, 2015
47
Concluding remarks (cont’d)
I
I
ZeroMQ offers many mechanisms (socket pairs) & policies
(patterns) and has a large community
Existing & tested patterns can be adopted to robotics use
cases:
I
I
I
I
I
I
for
for
for
for
initialising multiple cooperative robots
logging & tracing data
setting up QoS monitoring for communication
detecting failures (in the network or of robots)
Sockets promote flexibility by shielding off legacy and
facilitating integration
Advanced protocols such a ZRE and ZOCP offer solutions to
many real world communication challenges.
Communication Patterns for Embedded System Design
Johan Philips
March 25, 2015
48