Before I get too far
into this one I finally found the documentation for PyZMQ API at http://zeromq.github.com/pyzmq/api/index.html [1].
This has been very
helpful to me in figuring some of this stuff out.
… Now onto the problem
at hand, how do you do round-robin with ZeroMQ?
For those unfamiliar with the concept of round robin you can see the
Wikipedia page about it here http://en.wikipedia.org/wiki/Round-robin_scheduling [2]. In
short you can have a group of workers and a foreman handing out work. Think of your machines as the workers, let’s
say there are 10. The foreman gets a job
and hands it out to the first worker, the second job to the second, and so on
until he gets the last worker (the 10th). After he has run out of workers to hand jobs
to he starts the process over again handing the next job to the first worker…
Now beyond this simple
example you can expand the idea out.
What if a worker leaves? What if
he comes back? What if you have a job
and the next worker is too busy to grab it?
ZeroMQ handles a lot of these ideas right out of the box.
In ZeroMQ they use the
term PUSH/PULL which is round-robin if
you go to this page http://api.zeromq.org/2-1:zmq-socket [3] and search for ZMQ_PUSH and ZMQ_PULL you will
get a lot more details then I will show here.
To quote the page
The
pipeline pattern is used for distributing data to nodes arranged
in a pipeline. Data always flows down the pipeline, and each stage of the
pipeline is connected to at least one node. When a pipeline stage is connected to multiple nodes data is
round-robined among
all connected nodes.
A
socket of type ZMQ_PUSH is used
by a pipeline node to send
messages to downstream pipeline nodes.
This guide assumes you
already have ZeroMQ and related libraries installed and running on your
machine. If you have not installed any
ZeroMQ refer to this guide Quick ZeroMQ with Python
This code was all written and tested on an
Ubuntu 12.04 server installation.
Push
First let’s write the
push program
I am going to run this
on a local virtual Ubuntu machine that already has all the necessary zeroMQ
libraries installed. The machines
address is 192.168.0.161
Open the document for editing
> vi zeroMQ_PUSH.py
|
#!/usr/bin/python
import zmq
import time
ctx = zmq.Context()
push_socket= ctx.socket(zmq.PUSH)
push_socket.bind('tcp://*:1234')
for x in range(20):
msg = str(x) + " , " + str(time.time())
print x
push_socket.send(msg)
#Pause 1 second
time.sleep(1)
|
This is a simple program
it binds a push to the port 1234.
It starts sending out
messages via push (round robin)
Pull
I am going to run this
on a local virtual Ubuntu machine that already has all the necessary zeroMQ
libraries installed. The machines
address is 192.168.0.162 and
192.168.0.163
Open the document for editing
> vi zeroMQ_PULL.py
|
#!/usr/bin/python
import zmq
ctx = zmq.Context()
pull_socket = ctx.socket(zmq.PULL)
pull_socket.connect('tcp://192.168.0.161:1234')
while True:
msg = pull_socket.recv()
print msg
|
Now run this program
from 2 machines 192.168.0.162 and 163
_with_Python_001.png){kind=small}
They both sit there and
wait for messages to come in.
Now, once they are both
running, start up the PUSH program
Then you will see the
work getting divided out between the two.
On the first machine
_with_Python_002.png)
You can see it gets the first two messages in a row (probably because the second machine has not yet connected up with the PUSH machine)
Here is the output from the second machine.
If I run this again and
kill one of the PULL machines midway through….
You can see the second
machine now gets all the messages, since the first machine was turned off after
message 7.
If I turn it off then bring it back…
You can after message 7 the second machine came online and the work was distributed between the two.
ZeroMQ does a lot of the
heavy lifting, getting back connections so you do not have to. You can focus on programming new features
rather than doing all this heavy lifting maintaining connections, dropouts and
reconnections.
References
[1] The PyZQM API
Visited 08/2012
[2] Wikipedia:
Round-robin scheduling
Visited 08/2012
[3] zmq socket
Visited 08/2012
No comments:
Post a Comment