zmq pipeline pattern can not work in multiprocess? - queue

Question

I think I do not use zmq in right pattern, what I want to do is:
send message by zmq in multiprocess
accept message in multiple client, but one message should be accept only once
according the second requirements I thought a pipeline should be ok ( PUSH/PULL ), but this mode can not work in multiprocess:
def foo(i):
return i
def producer():
context = zmq.Context()
zmq_socket = context.socket(zmq.PUSH)
zmq_socket.bind("tcp://127.0.0.1:5559")
with concurrent.futures.ProcessPoolExecutor(max_workers=10) as executor:
futs = [executor.submit(foo, i) for i in range(10)]
for fut in concurrent.futures.as_completed(futs):
work_message = { 'num' : fut.result() }
zmq_socket.send("test")
producer()
so, maybe I should use PUB/SUB pattern, but this can not meet the second requirement.
in fact what I want is something like this:
PUSH|-----| | PULL
PUSH|-----| | PULL
PUSH|-----|----- DEVICE -----| PULL
PUSH|-----| | PULL
PUSH|-----| | PULL

Answer 1

Not exactly, roger, ZeroMQ PUSH/PULL pattern works
it just does something else than you would like it to do.
ZeroMQ is a wonderfull toolbox kit of pre-baked behaviours with immense potential to assemble more complex behaviour models, where needed.
Start with understanding the primitive actors, than design your functional requirements.
PUSH/PULL Formal Communication Scenario gets two players:
1st: PUSH picks the phone and calls. Whom? The connected PULL-side. PUSH leaves a voice-mail message to be listened by PULL, once it decides to pick it from the voicemail.
2nd: PULL side ( at some point in time, not necessarily right upon 1.) hears the bell ringing and picks the phone.
3rd: PULL, if instructed to, processes the received message from PUSH
Nothing more, per se.
Assemblies of ZeroMQ primitive components:
yes, there is the way to step farther towards your goal:
Just complement your functional requirements with appropriate ZeroMQ primitives inter-connected so as to meet your additional requirements.
The most trivial-one, from Chapter 1 of the "Code Connected, Volume I." - a round-robin based forwarding of messages towards a pool-of-"worker"-processes.
More specialised assemblies may create additional functionalities for smart distributed complex behavioral models:
Typically a "control + signalling" SIG_PLANE is being implemented in parallel to the primary, functional processing.
Self-diagnostics service, hearbeat/self-healing signalling for non-stop processing scenarios, remote non-blocking logging service, front-end MVC/GUI-plane being the most typical layered design goals.
Your task?
If not interested in more features, just .connect() your Futures calculating PUSH-ers towards a PULL-ing endpoint.
This middle-step ( a performance / failure analyses singularity ) can collect on it's PULL-ing entry-side and right go and PUSH on it's other endpoint, here - on a round-robin basis, towards the pool of actual workers ( who are crowd-waiting for a task with their .connect()-ed PULL-er entry endpoints for an incoming task from the "sink"-collector entity ) - a task, from collected FIFO ( beware the buffering capacity and performance overheads you pay for this ), just goes to the "next" one, down the line.
________________________________ _____________________________
process(0)|....[PUSH].connect(A)|- ________________________________ -|.connect(B)[PULL] process(-1)
process(1)|....[PUSH].connect(A)|-- | | --|.connect(B)[PULL] process(-2)
process(2)|....[PUSH].connect(A)|--- ---.bind(A)[PULL]:NOP:[PUSH].bind(B)|--- ---|.connect(B)[PULL] process(-3)
process(3)|....[PUSH].connect(A)|-- |________________________________| --|.connect(B)[PULL] process(-4)
... ...
process(n)|....[PUSH].connect(A)|- -|.connect(B)[PULL] process(-m)
It's that simple.
Enjoy the cool ZeroMQ toolkit.