I have trouble with establishing asynchronous point to point channel using ZeroMQ.
My approach to build point to point channel was that it generates as many ZMQ_PAIR sockets as possible up to the number of peers in the network. Because ZMQ_PAIR socket ensures an exclusive connection between two peers, it needs the same number of peers. My first attempt is realized as the following diagram that represents paring connections between two peers.
But the problem of the above approach is the fact that each pairing socket needs a distinct bind address. For example, if four peers are in the network, then each peer should have at least three ( TCP ) address to bind the rest of peers, which is very unrealistic and inefficient.
( I assume that peer has exactly one unique address among others. Ex. tcp://*:5555 )
It seems that there is no way other than using different patterns, which contain some set of message brokers, such as XREQ/XREP.
( I intentionally avoid broker based approach, because my application will heavily exchange message between peers, which it will often result in performance bottleneck at the broker processes. )
But I wonder that if there is anybody who uses ZMQ_PAIR socket to efficiently build point to point channel? Or is there a way to bypass to have distinct host IP addresses for multiple ZMQ_PAIR sockets to bind?
Q: How to effectively establish ... well,
Given the above narrative, the story of "How to effectively ..." ( where a metric of what and how actually measures the desired effectivity may get some further clarification later ), turns into another question - "Can we re-factor the ZeroMQ Signalling / Messaging infrastructure, so as to work without using as many IP-addresses:port#-s as would the tcp://-transport-class based topology actually need?"
Upon an explicitly expressed limit of having not more than a just one IP:PORT# per host/node ( being thus the architecture's / desing's the very, if not the most expensive resource ) one will have to overcome a lot troubles on such a way forward.
It is fair to note, that any such attempt will come at an extra cost to be paid. There will not be any magic wand to "bypass" such a principal limit expressed above. So get ready to indeed pay the costs.
It reminds me one Project in TELCO, where a distributed-system was operated in a similar manner with a similar original motivation. Each node had an ssh/sshd service setup, where local-port forwarding enabled to expose a just one publicly accessible IP:PORT# access-point and all the rest was implemented "inside" a mesh of all the topological links going through ssh-tunnels not just because the encryption service, but right due to the comfort of having the ability to maintain all the local-port-forwarding towards specific remote-ports as a means of how to setup and operate such exclusive peer-to-peer links between all the service-nodes, yet having just a single public access IP:PORT# per node.
If no other approach will seem feasible ( PUB/SUB being evicted for either traffic actually flowing to each terminal node in cases of older ZeroMQ/API versions, where Topic-filtering gets processed but on the SUB-side, which both security and network Departments will not like to support, or for concentrated workloads and immense resources needs on PUB-side, in cases of newer ZeroMQ/API versions, where Topic-filter is being processed on the sender's side. Adressing, dynamic network peer (re-)discovery, maintenance, resources planning, fault resilience, ..., yes, not any easy shortcut seems to be anywhere near to just grab and (re-)use ) the above mentioned "stone-age" ssh/sshd-port-forwarding with ZeroMQ, running against such local-ports only, may save you.
Anyway - Good Luck on the hunt!
Related
My current system takes input stream from cameras, each camera in a separate instance, and apply Computer Vision models on each camera (Object Detection, Object Tracking and Personnel Recognition), and then pass the results to a sink/master process that performs the rest of functionality over those results and I'm using ZMQ as an inter-process communication.
What I implemented now is that each worker connects to a different port, and then the sink subscribes to these ports independently, but this solution is not scalable, as we might have 3 or 4 cameras/worker, and I felt that it won't be efficient to keep opening ports like that.
Multiple Ports Implementation
That's when I tried to implement Multi-Pub/Single-Sub module, where all workers will connect to one port and the sink will subscribe to that port only.
Single Port Implementation
The problem I faced is that I no longer can distinguish between different cameras since I'm receiving different footages in the same port which causes a problem in streaming them later, that's why I'm thinking about the possibility of having multiple sockets for each context, while each socket subscribes to a different IP, is that possible?
Note: I've seen this answer but it has different ports for different sockets which does not really serve my case.
Q : " ... I no longer can distinguish between different cameras ... "
A :Yet, there are ZeroMQ tools to do so - check details about :
.setsockopt( zmq.METADATA, "X-key:value" )
.setsockopt( zmq.ROUTING_ID, Id )
As you see, PUB/SUB-archetype is the worst one to be used here ( you pay all the costs of TOPIC-filter based subscription-management, yet receive nothing for doing that ).
Using better matching archetypes is the way to go.
Given not performance details were posted, the capacity may soon get over-saturated, so may use more specific steps to flatten the workload and protect smooth-flow of the service :
.setsockopt( zmq.TOS, aTransportPath_TOS )
.setsockopt( zmq.MAXMSGSIZE, aBLOB_limit_to_save_RAM )
Given a streaming could block on many "old"-frames not having got through the e2e-pipeline in due time, it might make sense to also set this :
.setsockopt( zmq.CONFLATE, 1 )
As you can see, there are many smart details in the configuration space of the ZeroMQ, plus once scaling is to grow larger and larger, your design shall also fine-tune the Context()-engine performance once instantiating :
.Context( aNumOfContextIOthreads2use )
I'm struggling with the following problem:
I'd like to make bidirectional asynchronous requests/replies between N clients and 1 server with ZeroMQ.
This means that any client can make a request to the server and the server must reply to the client.
In the other way, the server must be able to make a request to any of the identified clients and the client to reply to the server.
I think I must use routers/dealers, but I'm not sure I need them in both ways.
Moreover, is there a way to have this whole paradigm using only one port on the server side and on each client side?
Q: is there a way to have this using only one port on the server / client side?
Well, this is the simpler part. No, this is not achievable. Having just one telephone box in the college will ring in the hallway, but will never help to address a call to the required department ( it would not correctly reach the intended professor of Quantum Mechanics instead of the one in Fine Arts ).
Using but one port means having a chance to expose for public access a one and only one ZeroMQ Scalable Formal Communication Pattern Archetype AccessPoint and ( except the very specific PAIR/PAIR distributed behaviour Archetype ) there is always some kind of hard-wired distributed-behaviour of the interconnected agents' AccessPoint-s.
This means, using one port gives just one and only one kind of such distributed-behaviour, not a mix of them.
This also answers your first part. If a REQ/REP distributed-behaviour Archetype was used in a direction from client-nodes towards the server, and another REQ/REP distributed-behaviour Archetype was to be used in an opposite direction from server towards the client-nodes, these ( directed ) services cannot co-exist on the same address:port.
BONUS PART : A kind of Life-Saving, but a bit dirty Trick( Not to be used for Medical and/or Emergency Systems )one may sort of supersample one and only one of the REQ/REP messaging directions and add a tricky "quasi-protocol" so as to serve this same channel for the both intended signalling directions. If sending sufficiently enough protocol-messages from one side, be it { client | server } the REQ/REP message initiator will simply send NOP-messages often enough to permit the REP-side replying party to "quasi-initiate" its "quasi-REQ-message" when still being the authentic-REP-AccessPoint in the single REQ/REP distributed-behaivour Archetype.Yes, performance and resources use are a cost for doing this, but a careful soft-real-time system design practices will help to make this work, if your needs are extremely dependent on using but one port and if your consciousness and your deployment ecosystem may tolerate the increased traffic patterns of such supersampled-REQ/REP data-flows
You might also like posts, here on Stack Overflow, about unavoidable mutual deadlocks, the distributed REQ/REQ FSA-s will fall into.
ZeroMQ hierarchy explained in less than a five seconds
UN-AVOIDABLE DEADLOCKS
I have a project that needs to be written in Perl so I've chosen ZeroMQ.
There is a single client program, generating work for a variable number of workers. The workers are real human operators who will complete a task then request a new task. The job of the client program is keep all available workers busy all day. It's a call center.
So each worker can only process one task at time, and there may be some time before requesting a new task. And the number of workers may vary during the day.
The client needs to keep a queue of tasks ready to give to workers as and when they request them. Whenever the client queue gets low the client can generate more tasks to top-up the queue.
What design pattern (i.e. what ZeroMQ Socket combination) should I use for this? I've skimmed through all the patterns in the 0MQ Guide and can't find anything that matches this.
Thanks
Sure. ... there is not a single, solo Archetype to match the Requirement List use several ZeroMQ Scalable Formal Communication Patterns
Typical software Project uses many ZeroMQ sockets ( with various Archetypes ) as a certain form of node-node signalisation and message-passing platform.
It is fair to note, that automated Load-Balancers may work fine for automated processes, but not always so for processes, executed by Humans or interacting with Humans.
Humans ( both the Call centre Agents and their Line-Supervisors ) introduce another layer of requirements - sometimes with a need to introduce non-just-Round-Robin workload distribution logic, sometimes need to switch a call from Agent A to another Agent B ( which a trivial archetype will simply not be capable of and might get into troubles, if it's hardwired-logic runs into a collision ( mutually blocked REQ-REP stale-mate being one such example ).
So simply forget to wait for one super-powered archetype, but rather create a smart network of behaviours, that will cover your distributed-computing problem desired event-handling.
There are many other aspects, one ought learn before taking the first ZeroMQ socket into service.
failure resillience
performance scaling
latency-profiling ( high-priority voice-traffic, vs. low-priority logging )
watchdog acknowledgements and timeout situations handling
cross-compatibility issues ( version 2.1x vs 3.x vs 4.+ API )
processing robustness against a malfunctioning agent / malicious attack / deadly spurious traffic storms ... to name just a few of problems
all of which has some built-ins in the ZeroMQ toolbox, some of which may need some advanced thinking, so as to handle known constraints.
The Best Next Step?
A would advocate for a fabulous Pieter HINTJENS' book "Code Connected, Volume 1" -- for everyone, who is serious into distributed processing, this is a must-read -- do not hesitate to check other my posts to find a direct URL to a PDF-version of this ZeroMQ Bible.
Worth time and one's tears and sweat.
Why does this limitation exist? What is the technical reason for it?
AFAIU, ports were introduced to distinguish between facilities (services, connections, etc.) of the same host, so logically the limitation is reasonable. However, SO_REUSEADDR exists to allow one-port-to-many-sockets binding, but not the other way round. It seems practical, because it would spare a system call wasted on multiplexing; many SO questions seek (fruitlessly) a way to do it. But the lack of implementation suggests there are some obstacles I cannot figure.
The reason is that UDP and TCP connections are keyed based on the IP-Port Pair. This is how the stack figures out what goes with what internally.
If we had many ports to one it would require some other mechanism to key the connection so that the proper data would be delivered to the proper application thread/session.
I have a setup with two Erlang nodes on the same physical machine, and I wanna be able to send large files between the nodes.
From the symptoms I see it looks like there is only one Tcp connection between the nodes, and sending the large binary across stops all other traffic, is this the case?
And even more interesting is there a way of making the vm use several connections between the nodes?
Yeah, you only get 1 connection, according to the manual
The handshake will continue, but A is informed that B has another
ongoing connection attempt that will be shut down (simultaneous
connect where A's name is greater than B's name, compared literally).
Not sure what "big" means in the question, but generally speaking (and imho), it might be good to setup a separate tcp port to handle the payloads, and only use the standard erlang messages as a signaling method (to negotiate ports, setup a listener, etc), like advising there's a new incoming payload and negotiate anything needed.
Btw, there's an interesting thread on the same subject, and you might try tunning the net_* variables to see if they help with the issues.
hope it helps!
It is not recommended to send large messages between erlang nodes,
http://learnyousomeerlang.com/distribunomicon
Refer to "bandwidth is infinite" section, I would recommend use something else like GFS so that you don't lose the distribution feature of erlang.