MQ @ NodeJS

MQ: Small Pieces for Scalable Software

Fernando D. Alonso

1. Facts about MQ
[ ]

Facts about MQ

Low level messaging library

It is written in C++. A thin layer between application and transport layers. Message-passing using in-memory queues. Concise API which talks over the ZMTP protocol. Works on top of TCP, IPC, in-memory, and PGM/EPGM communication protocols.

Facts about MQ

A broker is not mandatory

Philosophy: horizontal scalability to reduce points of failure. Network topology aware. Point to point communication reduce latencies. Communication is reduced to a few patterns, which combined are powerful. No need to have a multitenant service.

Facts about MQ

Messaged oriented communication

Messages as first-class citizen. No need to fight with framing or buffering. You can send multipart messages. Messages are atomic. Asynchronous messaging. A single background I/O thread does the job for MQ sockets.
"A MQ message is a discrete unit of data passed between applications or components of the same application. MQ messages have no internal structure and from the point of view of MQ itself they are considered to be opaque binary data."

Facts about MQ

Fast for development

Interchangeable transports: e.g, scaling from one IPC server to a bunch of TCP. Automatic reconnections. Multicore made easy. Provides zmq_poll to read BSD sockets.

Facts about MQ

Aims for simplicity

It doesn't provide message persistence. It doesn't provide data serialization. It doesn't provide data compression. It doesn't provide message encryption. It doesn't provide security mechanisms. Not in the next version of the ZMTP protocol !

(*) ZTMP 3.0 draft protocol: http://hintjens.com/blog:39

Facts about MQ

Platforms & Languages

Bindings for 30+ languages including NodeJS, Ruby, Python, Java, C, C++, Scala, Erlang, Perl, PHP, .NET. Linux & Windows. Raspberry PI. Android OS.

2. Install
[ , ]

Install

On Linux from source

sudo apt-get install build-essential libtool autoconf automake uuid-dev wget http://download.zeromq.org/zeromq-3.2.3.tar.gz tar xvzf zeromq-3.2.3.tar.gz ./configure make sudo make install sudo ldconfig

NOTES: (*) Check http://download.zeromq.org/ for more versions. (**) Avoid 3.0.x, 3.1.x, 3.2.0, 3.2.1 versions. They lack backwards compatibility. (***) Check ./configure --help for more options on build. (****) ./configure --with-pgm

Install

On Linux using packages

Libzmq 3.2:

Debian Sid: sudo apt-get install libzmq3-dev

Libzmq 2.2:

Debian Wheezy, Jessie, Sid: sudo apt-get install libzmq-dev Debian Squeeze backport: http://packages.debian.org/squeezebackports/libzmq1

Install

On Mac
brew install zmq

Install

NodeJS bindings
npm install zmq

node > var zmq = require('zmq');

NPM Packages:

zmq (https://github.com/JustinTulloss/zeromq.node) [RECOMMENDED] zmq-stream (https://github.com/Schoonology/zmq-stream)

Install

Bindings in other languages

Ruby: https://github.com/chuckremes/ffi-rzmq [RECOMMENDED] Python: https://github.com/zeromq/pyzmq Erlang: https://github.com/zeromq/erlzmq2 Java JZMQ (uses JNI): https://github.com/zeromq/jzmq Java JeroMQ: https://github.com/zeromq/jeromq Scala ZeroMQ: https://github.com/mDialog/scala-zeromq Scala ZeroMQ (official): https://github.com/valotrading/zeromq-scala-binding ...

3. Sockets

Sockets

Creation
NodeJS ZMQ req rep
var sock = zmq.socket ("<SOCKET_TYPE>");

MQ C API ZMQ_REQ ZMQ_REP ZMQ_DEALER ZMQ_ROUTER ZMQ_PUSH ZMQ_PULL ZMQ_PUB ZMQ_SUB ZMQ_XSUB ZMQ_XPUB
Request Reply Dealer (async Request) Router (async Reply) Push Pull Publisher Subscriber XSubscriber XPublisher

NodeJS ZMQ

dealer router

MQ C API
zmq_socket(context, <SOCKET_TYPE>);

push pull pub sub xsub xpub

(*) http://api.zeromq.org/3-2:zmq-socket

Sockets

Bind & Connect

NodeJS ZMQ MQ C API

Bind
sock.bind("tcp://10.0.0.1:5555", function(err) { .. }); zmq_bind(socket, "tcp://10.0.0.1:5555");

Connect
sock.connect("tcp://10.0.0.1:5555"); zmq_connect(socket, "tcp://10.0.0.1: 5555");

Close
sock.close(); zmq_close(socket);

(*) http://api.zeromq.org/3-2:zmq-bind (**) http://api.zeromq.org/3-2:zmq-connect (***) http://api.zeromq.org/3-2:zmq-close

Sockets

Bind & Connect

NodeJS ZMQ

BindSync
sock.bindSync("tcp://10.0.0.1:5555"); // This line gets executed once the socket // is binded, but it breaks on error.

Sockets

Transports
TCP

usage:

tcp://<address>:<port>
NodeJS ZMQ MQ C API
zmq_bind(socket, "tcp://192.168.1.100: 5555");

sock.bind("tcp://192.168.1.100:5555");

sock.bind("tcp://eth0:5555"); // binds to the first eth0 interface sock.bind("tcp://*:5555"); // binds to all interfaces

zmq_bind(socket, "tcp://eth0:5555");

zmq_bind(socket, "tcp://*:5555");

(*) http://api.zeromq.org/3-2:zmq-tcp

Sockets

Transports
Inter-Process Communication (IPC)

usage:

ipc://<address>
NodeJS ZMQ MQ C API

sock.bind("ipc:///tmp/mysocket");

zmq_bind(socket, "ipc:///tmp/mysocket");

Requires RW permissions on the specified path.

(*) http://api.zeromq.org/3-2:zmq-ipc

Sockets

Transports
In-process Communication (in-memory)

usage:

inproc://<address>
NodeJS ZMQ MQ C API

sock.bind("inproc://queue");

zmq_bind(socket, "ipc://queue");

Requires to bind before connect. Max. address name length is 256.

(*) http://api.zeromq.org/3-2:zmq-inproc (**) Buggy on node-zmq package.

Sockets

Transports
PGM (multicast)

usage:

pgm://<address>;<multicast_address>:<port>
NodeJS ZMQ MQ C API

sock.bind( "pgm://192.168.1.100;239.192.1.1:3055" );

zmq_bind(socket, "pgm://192.168.1.100; 239.192.1.1:3055");

Requires sudo privileges to access raw IP sockets. Needs MQ built with PGM extension (./configure --with-pgm).

(*) http://api.zeromq.org/3-2:zmq-pgm

Sockets

Transports
Encapsulated PGM (multicast)

usage:

epgm://<address>;<multicast_address>:<port>
NodeJS ZMQ MQ C API

sock.bind( "epgm://192.168.1.100;239.192.1.1:3055" );

zmq_bind(socket, "epgm://192.168.1.100; 239.192.1.1:3055");

Needs MQ built with PGM extension (./configure --with-pgm).

(*) http://api.zeromq.org/3-2:zmq-pgm

Sockets

Send & Receive

Send
sock.send("My message");

Receive
sock.on("message", function(msg) { console.log(msg.toString()); }); sock.on("message", function(msg) { console.log("Received " + msg); });

// <msg> is a Buffer object.

(*) http://api.zeromq.org/3-2:zmq-send (**) http://api.zeromq.org/3-2:zmq-recv

Sockets

Multipart messages
Messages are atomic
A MQ message is composed of 1 or more message parts.
MQ Queue
zmq_send(socket, "Como", 4, ZMQ_SNDMORE);

Part 1
zmq_send(socket, "andas", 5, 0);

4 5

Como andas

Part 2

MQ ensures that peers receive either all message parts of a message or none at all. The total number of message parts is unlimited except by available memory.

Sockets

Multipart messages
Send
sock.send("First part", zmq.ZMQ_SNDMORE); sock.send("Second part");

sock.send(["First part", "Second part"]);

Check the outgoing buffer and socket state:

var r = sock.send("Como", zmq. ZMQ_SNDMORE); console.log(r);
running this will output -> > { type: 'req', _zmq: { state: 0, onReady: [Function] }, _outgoing: [ [ <Buffer 43 6f 6d 6f>, 2 ] ], _shouldFlush: true, _events: { message: [Function] } }

Sockets

Multipart messages
Receive
sock.on("message", function(first_part, second_part) { console.log(first_part.toString()); console.log(second_part.toString()); }); sock.on("message", function() { for(var key in arguments) { console.log("Part " + key + ": " + arguments[key]); }; }); sock.on("message", function() { var messages = Array.prototype.slice.call(arguments); messages.forEach(function(msg) { console.log("Part:" + msg); }); });

Sockets

Batched messaging
Client Thread MQ I/O Thread

MQ Queue send
sock.send("Msg"); sock.send("Part A", zmq.ZMQ_SNDMORE); sock.send("Part B");

MQ Socket <write busy>

3 send 6 send 6

Msg Part A Part B send batch <write ready> <write busy>

"MQ batches messages in opportunistic manner. It sends all the messages available at the moment in one go. Latency of subsequent messages will be improved because sending single batch to the card is faster then sending lot of small messages."

Patterns

Plugging multiple transports

Example
var zmq = require('zmq'), pub = zmq.socket('pub'); pub.bindSync('tcp://127.0.0.1:5555'); pub.bindSync('ipc:///tmp/zmq.sock'); setInterval(function() { pub.send("I am polyglot!"); }, 1000); sub = zmq.socket('sub'); sub.connect('ipc:///tmp/zmq.sock'); sub.subscribe(''); sub.on('message', function(msg) { console.log("Received: " + msg); }); var zmq = require('zmq'), sub = zmq.socket(sub'); sub.connect('tcp://127.0.0.1:5555'); sub.subscribe(''); sub.on('message', function(msg) { console.log(Received: ' + msg); });

Sockets

Setting Socket Options

Setsockopt

usage (NodeJS ZMQ API): sock.setsockopt(<option>, <value>); or sock.<option> = <value>;

Identity:
sock.identity = 'monitor-1'; sock.setsockopt('identity', 'monitor-1');

// value can be any string up to 255 length. // identity is required on sockets // connecting to a Router.

(*) http://api.zeromq.org/3-2:zmq-setsockopt

Sockets

Setting Socket Options

Setsockopt
Subscribe:
subscriber.subscribe('MUSIC'); subscriber.setsockopt('subscribe', 'MUSIC');

// sets a message filter for subscriber sockets.

Unsubscribe:
subscriber.unsubscribe('MUSIC'); susbscriber.setsockopt('subscribe', 'POP');

// sets off a message filter for subscriber sockets.

4. Patterns
[ ]

Patterns

Request / Reply
Synchronous task distribution
Used when each message needs to be matched with a response. Handles only one message at time. Strict send-recv cycle.

REQ
< synchronous > < round robin >

REP #1 REP # 2

...

Patterns

Request / Reply
REQ
send recv (wait response)

REP #1

REP # 2

send recv (dont wait response)

recv (dont wait response)

Patterns

Request / Reply
REQ
send recv

REP #1

REP # 2

send recv

send -> REP #1 is down

FAILURE REQ #1 will not recover !

Patterns

Request / Reply
Basic Request / Reply
var zmq = require('zmq'); reply = zmq.socket('rep'); reply.bind('tcp://127.0.0.1:5555', function(err) { if (err) throw err; } ); reply.on('message', function(msg) { console.log('Received: ' + msg); reply.send('Pong!'); }); var zmq = require('zmq'); request = zmq.socket('req'); request.connect('tcp://127.0.0.1:5555'); request.send('Ping'); request.on('message', function(msg) { console.log('Response: ' + msg); });

Patterns

Request / Reply
Dealer socket message delivery
Messages have to be multipart, consisting on: an empty delimiter header, followed by the content body parts.
Outgoing Queue

sock.send(new Buffer([]), zmq. ZMQ_SNDMORE); sock.send("Body");

send 0 send 6 Body

Patterns

Request / Reply
Dealer socket message delivery
Outgoing messages are round-robined among all connected peers. However, sending depends on the availability of the receiver.
Outgoing Queue
sock.send( ["", "Msg"]);

Peer Socket A

Peer Socket B <write ready>

send
To A

0 3 0 6 0 4

Msg send Part A <write busy> <write ready>

sock.send( ["", "Bla"); sock.send( ["", "Msg2"]);

send
To B

send
To A

Msg2

Patterns

Request / Reply
Dealer socket message delivery
Reconnections are handled automatically: messages are asynchronously delivered. Response time does not block sending more messages.
Outgoing Queue
sock.send( ["", "Msg"]);

Peer Socket A
-> Up again

Peer Socket B

send
To A

0 3

Msg send

sock.send( ["", "Msg2"]);

send
To A

0 4

Msg2

Patterns

Request / Reply
Dealer socket handles peer reconnection automatically. It will send messages that queued for that peer once it has established connection. DEALER
enqueue < Msg A > -> REP #1 is down send < Msg B >

REP #1

REP # 2

enqueue < Msg C > send < Msg A, Msg C > send < Msg D >

Patterns

Request / Reply
Dealer example
var dealer = zmq.socket('dealer'); dealer.connect('tcp://127.0.0.1:5001'); dealer.connect('tcp://127.0.0.1:5002'); setInterval(function() { dealer.send(["", "Msg"]); }, 1000); dealer.on('message', function(h, msg) { console.log('Response from: ' + msg); });

running this 'might' output ->

> > > > > > >

Response Response Response Response Response Response ...

from: from: from: from: from: from:

5001 5001 5002 5001 5001 5002

Patterns

Request / Reply
Router socket messaging
Messages have to be multipart, consisting on: an identity frame, an empty delimiter frame, followed by the content body parts. This applies to both incoming and outgoing messages.
Outgoing Queue
sock.send( "worker-1", zmq. ZMQ_SNDMORE); sock.send("", zmq. ZMQ_SNDMORE); sock.send("Body");

send 8 send send 0 6 Body worker-1

Patterns

Request / Reply
Router is asynchronous
Incoming messages are fair-queued among all connected and available peers.
Peer Socket A Peer Socket B MQ Router

send send send

Socket Incoming Queue

Patterns

Request / Reply
Router is asynchronous
Receiving depends on the availability of the sender.
MQ Router

Peer Socket A

Peer Socket B

send

Socket Incoming Queue

send

Patterns

Push / Pull
Unidirectional data distribution
The Push / Pull pattern fits well on pipelined communication, i.e., when no response is required for a given message.

...

Patterns

Push / Pull
Binding the Push socket
Outgoing messages are round-robined among all connected and available peers.
Outgoing Queue
push.send("Msg"); push.send("Part A", zmq.ZMQ_SNDMORE); push.send("Part B"); push.send("Msg2"); send send send

Peer Socket A

Peer Socket B

3 6 6

Msg Part A Part B

send

Msg2

Patterns

Push / Pull
Binding the Push socket
Outgoing messages are round-robined among all connected and available peers.
Outgoing Queue
push.send("Msg"); push.send("Part A", zmq.ZMQ_SNDMORE); push.send("Part B"); push.send("Msg2"); send send send send

Peer Socket A

Peer Socket B

3 6 6

Msg Part A Part B

send send

-> Down

send

Msg2

Patterns

Push / Pull
Binding the Push socket
Disconnections and reconnections are handled automatically.

Outgoing Queue
push.send("More"); send

Peer Socket A

Peer Socket B

4
push.send("Other"); push.send("Part 1", zmq.ZMQ_SNDMORE); push.send("Part 2"); send

More Other Part 1 Part 2

send

5
send send

send

6 6

send

Patterns

Push / Pull
Parallel task distribution

PUSH
< round robin >

PULL #1 PULL # 2

...

Patterns

Push / Pull
Passive agents / Monitor

PUSH PUSH

PULL

...

Patterns

Push / Pull
Binding the Pull socket
Push's outgoing messages are round-robined among all connected peers.

Outgoing Queue
push.send("Msg"); push.send("Part A", zmq.ZMQ_SNDMORE); push.send("Part B"); push.send("Msg2"); send send To B send

Peer Socket A

Peer Socket B

To A

3 6 6

Msg Part A Part B

send

To A

Msg2

Patterns

Push / Pull
Binding the Pull socket
Push's outgoing messages are round-robined among all connected peers.

Outgoing Queue
push.send("Msg"); push.send("Part A", zmq.ZMQ_SNDMORE); push.send("Part B"); push.send("Msg2"); send send To B send

Peer Socket A

Peer Socket B

To A

3 6 6

Msg Part A Part B

send

send

send

To A

Msg2

send

Patterns

Push / Pull
Binding the Pull socket
Disconnections and reconnections are handled automatically.

Outgoing Queue
push.send("More"); send send To B send

Peer Socket A

Peer Socket B

To A

4 5

More Other

-> Down

push.send("Other"); push.send("Part 1", zmq.ZMQ_SNDMORE); push.send("Part 2");

send send

To A

6 6

Part 1 Part 2

send

Patterns

Push / Pull
Binding the Pull socket
Sending is asynchronous, it depends on the availability of the receiver.

Outgoing Queue
push.send("More"); send send To B send

Peer Socket A

Peer Socket B
-> Up again

push.send("Other"); push.send("Part 1", zmq.ZMQ_SNDMORE); push.send("Part 2");

Other

send send

Patterns

Push / Pull
Active agents / collector pattern

PUSH #1 PUSH #2

PULL

...

Patterns

Publish / Subscribe
Broadcast data distribution

SUB #1 PUB #1 SUB #2

...

SUB #2

Patterns

Publish / Subscribe
Message distribution
Outgoing messages are send to each connected and available peers.

Outgoing Queue Subscriber A

sock.send("Msg"); send

Msg

send

Subscriber B

Subscriber C

Patterns

Publish / Subscribe
Message distribution
Outgoing messages are send to each connected and available peers.

Outgoing Queue Subscriber A

sock.send("Msg"); send

3
sock.send("Other"); send

Msg
send

Subscriber B

Other

Subscriber C

Patterns

Publish / Subscribe
Data filtering
Subscribers can register to a specific topic.
MQ Publisher

e AR subscrib e VE subscrib
subscr ibe VE

Subscriber A

Socket Subscriber B

Outgoing Queue Subscriber C

Patterns

Publish / Subscribe
Data filtering
Subscribers can register to a specific topic.
MQ Publisher

e AR subscrib e VE subscrib
subscr ibe VE

Subscriber A

Socket Subscriber B

se

nd

Outgoing Queue

sock.send("AR news");

send

Subscriber C 7 AR news

Patterns

Publish / Subscribe
Data filtering
Subscribers can register to a specific topic.
MQ Publisher

e AR subscrib e VE subscrib
subscr ibe VE

Subscriber A

Socket Subscriber B

se
nd

nd

Outgoing Queue

sock.send("VE news");

send

se

Subscriber C

VE news

Patterns

Publish / Subscribe
Data filtering example
pub = zmq.socket('pub'); pub.bindSync("tcp://10.0.0.12:3055"); count = 0 setInterval(function() { pub.send("TEST " + count++); }, 1000); // older messages won't be // received > > > > > running this 'might' output -> TEST TEST TEST TEST TEST 6 7 8 9 10

sub = zmq.socket('sub'); sub.connect("tcp://10.0.0.12:3055"); sub.subscribe("TEST"); sub.on("message", function(msg) { console.log("Received: " + msg); });

Patterns

Publish / Subscribe
Multicast example
var zmq = require('zmq'), pub = zmq.socket('pub'); pub.bind( "epgm://10.0.0.12;239.192.1.1:3055", function(err) { if(err) throw err; } ) setInterval(function() { pub.send("From pid: " + process.pid); }, 1000); var zmq = require('zmq'), sub = zmq.socket('sub'); sub.connect( "epgm://10.0.0.13;239.192.1.1:3055" ); sub.subscribe(""); sub.on('message', function(msg) { console.log("Received " + msg); });

(*) See running demo at http://youtu.be/NQrH0SATPk0

Patterns

XSUB / XPUB
Data distribution proxy
"XSUB and XPUB are exactly like SUB and PUB except they expose subscriptions as special messages."
XSUB/XPUB Proxy
cr ib e PY su

Publisher A XSUB Socket Publisher B

forward subscription

Subscriber A

bs

Subscriber B

Publisher C

XPUB Socket

Subscriber C

Patterns

XSUB / XPUB
Data distribution proxy
Messages are forwarded by the proxy to the registered subscribers.
XSUB/XPUB Proxy Subscriber A XSUB Socket
nd

Publisher A

sen d <ms "PY g>"

Publisher B

forward message

Subscriber B

Publisher C

XPUB Socket

se

Subscriber C

5. Useful links
[ ]

Useful Links

Workshop examples
http://github.com/krakatoa/node_zmq_workshop

Useful Links

The Guide

http://zguide.zeromq.org/page:all

Useful Links

Pieter Hintjens personal blog

http://hintjens.com/

Useful Links

API References

http://api.zeromq.org/3-2:zmq-socket

http://api.zeromq.org/3-2:zmq-setsockopt

Gracias !

Preguntas ? fedario@gmail.com github.com/krakatoa Fernando Dario Alonso