notes.txt

Notes for The Guide

++ Heartbeating

You'll often hit the problem of knowing whether a peer is alive or not. This is not specifically a 0MQ issue. TCP has a long timeout (30 minutes or so) that means that it's impossible sometimes to know whether a peer has died, been disconnected, or gone on a long weekend to Prague with a case of vodka, a redhead, and a large expense account.

Heartbeating is not trivial and worth getting right, since it can make the difference between a working, and failing architecture. So using our standard approach, let's start with the simplest possible heartbeat design, and develop better and better designs until we have one with no visible faults.

+++ Model one - the Shruggery Pattern

Shruggery is the simplest possible design: no heartbeating at all. This is the starting point for all 0MQ applications, of course. For PUSH-PULL sockets, this is the only model that works. For other socket types, this has at least the following problems:

* If you have ROUTER-based broker devices that manage some resources attached to the identities of peers, as peers disconnect and reconnect, the devices will accumulate entropy and eventually run out of resources.

* If you have SUB or DEALER-based data recipients, you can't tell the difference between good silence (there's no data) and bad silence (the other end died).

* If you have a TCP connection that stays silent for a long while, it will, in some networks, die. Sending something (technically, a "keep-alive" more than a heartbeat), will keep the network alive.

+++ Model two - the Heartbeat Pattern

So, we create the Heartbeat pattern by sending a "heartbeat" message from each node to its peers, every second or so. When one node hears nothing from another, within some timeout (several seconds, typically), it will treat that peer as dead.

For PUB-SUB, this is the only model that works. SUB sockets cannot talk back to PUB sockets, but PUB sockets can happily send "I'm alive" messages to their subscribers.

As an optimization to model 2, the sender can start heartbeating only when there is no data to send. The recipient should then treat ANY incoming data as a sign of life. This might be a little inaccurate, since receiving data does not actually mean the sender is (still) alive. Light from a dying sun, and all that.

So, problems with this design:

* It can be inaccurate when large amounts of data are sent, since heartbeats will be delayed behind that data. If heartbeats are delayed, you can get false timeouts and disconnections due to network congestion.

* While the PUB-SUB pattern will drop messages for disappeared recipients, ROUTER and DEALER sockets will queue them. So, if you send heartbeats to a dead peer, and it comes back, it'll get all the heartbeats you sent. Which can be thousands. Whoa, whoa!

* It assumes that heartbeat timeouts are the same across the whole network. But that won't be accurate. Some peers will want very aggressive heartbeating, to detect faults rapidly, some will want very relaxed heartbeating, to leave sleeping networks lie, and save power.

+++ Model three - the Belts and Braces pattern

Belts and Braces consists of a ping-pong dialog. One peer sends a ping command to the other, which replies with a pong command. Neither command has any payload. Pings and pongs are not correlated. Since the roles of "client" and "server" are sometimes arbitrary, we specify that either peer can in fact send a ping and expect a pong in response. However, since the timeouts depend on network topologies known best to dynamic clients, it is usually the client which pings the server.

This works for all ROUTER-based brokers. Two optimizations make this work even better:

* Treat any incoming data as a pong.
* Only send a ping when not otherwise sending data.





- pubsub ordering
    - N-to-N ordering guarantees
    - single-stream using a broker


- collection of patterns
    Least Recently Used
    Asynchronous Client-Server
    Suicidal Snail
    Lazy Pirate
    Simple Pirate
    Paranoid Pirate
    Clone

- how to make an error logging console

- how to make a TCP-to-0MQ bridge, with examples
http://lists.zeromq.org/pipermail/zeromq-dev/2011-March/010186.html
- FD_EVENT integration


- router trees
 (federation)

- cross-over router
    - send to, send from
    - flips addresses, gives you reply model

    //  If socket is a ROUTER, get identity first
    int socket_type;
    size_t type_size = sizeof (socket_type);
    zmq_getsockopt (socket, ZMQ_TYPE, &socket_type, &type_size);
    if (socket_type == ZMQ_ROUTER) {
        zmq_msg_t mesg;
        zmq_msg_init (&mesg);
        zmq_recv (socket, &mesg, 0);
    }

- explain why callbacks are a pita
    - what thread do they run in?
    - how do you connect them to main app thread
    - how do you wait, do you sleep, call something, etc.?

- integrating 0MQ into TCP loops: http://lists.zeromq.org/pipermail/zeromq-dev/2011-March/010186.html

Then we can use this to start building some reusable pieces:

- timer device
- name service
- custom load-balancing
- custom publish-subscribe
- stateful publish-subscribe
- reliable request-reply
- logging device


+++ Presence Detection

- peers come and go
- if we want to route to them explicitly, we need to know if they are present
- heartbeating, liveness, etc.
- fresh data, failover, etc.
- purging old identities from routing tables
- example of eight robots and console
- robots come and go...


+++ A ZeroMQ Name Service

- name service
    translate logical name into connect or bind string
    service runs locally, connect via ipc://zns
    gets name updates asynchronously from central server
    also local zns lookup file
    using zpl syntax
    pubsub state / refresh example

how to map names?
    - XXX -> tcp://lo:5050 if I'm on server 1
    - XXX -> tcp://somename:5050
    -> does ZMQ do host lookup?  Doesn't seem like it...
    -> resolve host...


+++ Pipelines

- does PUSH block if there are no PULL sockets ready?
- how do maintain a queue?

+++ File Transfer

example of file transfer

+++ Generating Identities


+++ Setting Queue Limits

- setting queue limits prevents nodes from overflowing memory
- by default 0MQ does not set any limits
- example of application that will overflow memory
    - publish to subscribe socket but don't read
- now fix this by setting queue limit, ZMQ_HWM
- actual behaviour with automagic LWM
- how this works on different socket types
    - 'exception' on each socket type, from man page
- adding capacity for disk offload, ZMQ_SWAP
- creating persistent queues using identity
    - example of HWM 1 plus massive SWAP


+++ Reliable Request-Reply

We'll create a reliable request-reply application that uses XREQ and XREP and a simple resend mechanism.  When this works between two peers we'll show how it scales across a request-reply broker to effectively create edge-to-edge reliability.  We'll also open up the message format for request-reply and explore identities in horrible detail.

+++ Configuration Distribution

We'll look at how to dynamically configure a network of devices using a central configuration broker.

+++ Logging Subsystem

many applications
many subscribers
broker in the middle
persistent logfiles
replay via subscribe socket


+++ Failover and Recovery

We'll look at how to handle crashes in a distributed architecture.  For each pattern there is an ideal architecture, and we'll explore each of these with worked examples.

+++ Encrypted Publish-Subscribe

We'll look at how to secure pubsub data against snooping.  The actual technique involves out-of-band exchange of keys, symmetric encryption, and a broker that helps the thing along.  Hopefully all fairly easy to make, as usual, using 0MQ.

+++ Building a Multicast Bus

We'll now look at how the pgm: and epgm: protocols work.  With pgm:, the network switch basically acts as a hardware FORWARDER device.

++++ Customized Publish-Subscribe

- use identity to route message explicitly to A or B
- not using PUBSUB at all but XREP/????
    - limitations: no multicast, only TCP
    - how to scale with devices...

When a client activates, it chooses a random port that is not in use and creates a SUB socket listening for all traffic on it. The client then sends a message via REQ to the publisher containing the port that it is listening on. The publisher receives this message, acknowledges it, and creates a new pub socket specific to that client. All published events specific to this client go out that socket.

When the client deactivates, it sends a message to the publisher with the port to deactivate and close.

You end up creating a lot more PUB sockets on your server end and doing all of the filtering at the server. This sounds acceptable to you.

I didn't need to do this to avoid network bandwidth bottlenecks; I created this to enforce some security and entitlements.

+++ A Clock Device

We'll look at various ways of building a timer into a network.   A clock device sends out a signal (a message) at more or less precise intervals so that other nodes can use these signals for internal timing.

+++ Serializing Data

Examples of using Protocol Buffers and other options.

[!--
- ipc://name
 - connects two processes on a single box
 - supports all messaging patterns
 - typical use case is for multithreading apps
 - runs on Unix domain sockets (not available on Windows, OpenVMS)
 - permissions issues:

> Since I want to work under /tmp, this all had to be done programatically. My
> server now mkdir -p's a socket subdirectory and chmod 777's it. The server
> creates and binds the socket in that folder, and then chmod 777's it. The
> server must be run as root (which is fine for my project luckily). If it is
> run as a normal user, the client's still timeout.

- tcp://ipaddress:port
 - bind to address:port
 - bind to *:5555
 - localhost
  - also bind to interface: lo:port, eth1:port, etc.
  - Linux: eth1, eth2, eth3
  - Mac OS X: en1, en2, en3
  - Solaris: e1000g, etc.
 - connect to remote address: host:port

- pgm://address;multicastgroup:port
 - address can be interface name
 - requires decent hardware support
 - means enterprise level switches with IGNP snooping
 - some routers also support PGM
 - runs over IP, requires root privileges
 - more standard
 - rate-limited protocol, sender must define bandwidth
 - pgm is currently broken

- epgm://address;multicastgroup:port
 - encapsulated in UDP packets
 - requires decent hardware support
 - does not require root access
 - non-standard to pgm

- add peer example
   - exclusive lock on peer
   - for owning other party
   - solve reverse connection
 - e.g. to cross firewall
 - you need to add a bind to allow the client to accept a connection
 - could be usecase for EXCLUSIVE socket

XREQ is like PUSH+PULL, XREP is like PUSH+PULL+routing

--]


* How do we tunnel 0MQ connections over 0MQ connections?
    - e.g. to get 3-4 ports into DMZ via one single port
    - two devices, one reads, one writes
    - over other protocols: HTTP, SSH, etc...?
    - acts as split device, ...

I highly recommend that you try out the simpler topology and *verifying* that 0mq cannot keep up with your message rates when publishing all data to all clients. With smart topics the client can reject the data *very* fast. A 1 GB or 10 Gb switched network can also move quite a bit of data without a lot of blocking behavior in the hardware too. You may be able to save yourself a lot of unnecessary work, so just try it.



- explain in
- debugging message flow
    - using a queue device that prints out message parts
    - debugging versions of devices...

- heartbeating
    - set HWM to 1 message only

[!--

[!--
+++ The Wire-Level Protocol

- writing a thin client in JavaScript

+++ Building a Language Binding

+++ Tuning 0MQ

High performance multiple i/o threads
 - for HP applications, several threads
 - then, set affinity on sockets

+++ Contributing to 0MQ


+++ What's Missing from 0MQ



- handling crashing peers
- non-blocking send/recv
    EAGAIN
- reliable pub
    - pub connects to all subs
    - subs send request to pub
- reliable messaging over xreq / xrep
- how to do timer events
- there is no way to disconnect; second connection creates two
    - need to destroy the socket
- how to send multipart job and execute / cancel




Here is, I think, how to build an authenticated pubsub service using ØMQ.  The design handles authentication, arbitrary routing criteria, and flushing of dead client connections.

You will use ZMQ_XREQ at the client side and ZMQ_XREP at the service side.  There is a simple protocol between client and server that is loosely modelled on HTTP:

* Client sends messages containing: identity (ØMQ adds this automatically), authentication credentials, and subscription criteria.
* Server sends messages containing: identity (automatic), error/success code, and content.

The server has two main data structures:

* A connection table that stores authenticated identities and timestamps.  Each identity corresponds to a client connection.
* A routing table that maps message keys (e.g. topic value) to identities.  There are techniques for doing [http://www.zeromq.org/whitepapers:message-matching high speed message matching].



0MQ Quickstarter

- building & installing
- performance for language
- basic examples
- socket types
- transports
- main patterns
- problem solving
-> translated into different programming languages