Why does this limitation exist? What is the technical reason for it?
AFAIU, ports were introduced to distinguish between facilities (services, connections, etc.) of the same host, so logically the limitation is reasonable. However, SO_REUSEADDR exists to allow one-port-to-many-sockets binding, but not the other way round. It seems practical, because it would spare a system call wasted on multiplexing; many SO questions seek (fruitlessly) a way to do it. But the lack of implementation suggests there are some obstacles I cannot figure.
The reason is that UDP and TCP connections are keyed based on the IP-Port Pair. This is how the stack figures out what goes with what internally.
If we had many ports to one it would require some other mechanism to key the connection so that the proper data would be delivered to the proper application thread/session.
Related
I have trouble with establishing asynchronous point to point channel using ZeroMQ.
My approach to build point to point channel was that it generates as many ZMQ_PAIR sockets as possible up to the number of peers in the network. Because ZMQ_PAIR socket ensures an exclusive connection between two peers, it needs the same number of peers. My first attempt is realized as the following diagram that represents paring connections between two peers.
But the problem of the above approach is the fact that each pairing socket needs a distinct bind address. For example, if four peers are in the network, then each peer should have at least three ( TCP ) address to bind the rest of peers, which is very unrealistic and inefficient.
( I assume that peer has exactly one unique address among others. Ex. tcp://*:5555 )
It seems that there is no way other than using different patterns, which contain some set of message brokers, such as XREQ/XREP.
( I intentionally avoid broker based approach, because my application will heavily exchange message between peers, which it will often result in performance bottleneck at the broker processes. )
But I wonder that if there is anybody who uses ZMQ_PAIR socket to efficiently build point to point channel? Or is there a way to bypass to have distinct host IP addresses for multiple ZMQ_PAIR sockets to bind?
Q: How to effectively establish ... well,
Given the above narrative, the story of "How to effectively ..." ( where a metric of what and how actually measures the desired effectivity may get some further clarification later ), turns into another question - "Can we re-factor the ZeroMQ Signalling / Messaging infrastructure, so as to work without using as many IP-addresses:port#-s as would the tcp://-transport-class based topology actually need?"
Upon an explicitly expressed limit of having not more than a just one IP:PORT# per host/node ( being thus the architecture's / desing's the very, if not the most expensive resource ) one will have to overcome a lot troubles on such a way forward.
It is fair to note, that any such attempt will come at an extra cost to be paid. There will not be any magic wand to "bypass" such a principal limit expressed above. So get ready to indeed pay the costs.
It reminds me one Project in TELCO, where a distributed-system was operated in a similar manner with a similar original motivation. Each node had an ssh/sshd service setup, where local-port forwarding enabled to expose a just one publicly accessible IP:PORT# access-point and all the rest was implemented "inside" a mesh of all the topological links going through ssh-tunnels not just because the encryption service, but right due to the comfort of having the ability to maintain all the local-port-forwarding towards specific remote-ports as a means of how to setup and operate such exclusive peer-to-peer links between all the service-nodes, yet having just a single public access IP:PORT# per node.
If no other approach will seem feasible ( PUB/SUB being evicted for either traffic actually flowing to each terminal node in cases of older ZeroMQ/API versions, where Topic-filtering gets processed but on the SUB-side, which both security and network Departments will not like to support, or for concentrated workloads and immense resources needs on PUB-side, in cases of newer ZeroMQ/API versions, where Topic-filter is being processed on the sender's side. Adressing, dynamic network peer (re-)discovery, maintenance, resources planning, fault resilience, ..., yes, not any easy shortcut seems to be anywhere near to just grab and (re-)use ) the above mentioned "stone-age" ssh/sshd-port-forwarding with ZeroMQ, running against such local-ports only, may save you.
Anyway - Good Luck on the hunt!
If I want to use (UDP) sockets as an inter-process communication mechanism on a single PC, are there restrictions on what I can set up due to the two endpoints having the same IP address?
I imagine that in order to have two processes A and B both listening on the same IP/port address, SO_REUSADDR would be necessary - correct? And even though that might conceptually allow for full duplex comms over a single socket, there are other questions I have if I try to go full duplex:
would I end up receiving my own transmissions, and have to filter them out?
would I be exposing myself to other processes injecting spurious or malicious data into my sockets due to the use of SO_REUSEADDR... or do I face this possibility simply by using (connectionless) UDP?
how would things be different (in an addressing/security/restrictions sense) if I chose to use TCP instead?
I'm confident that there is a viable solution using two sockets at each end (one for A -> B data, one for B ->A data)... but is there a viable solution using a single socket at each end? Would there be any clear advantages to using one full-duplex socket per process if it is possible?
The question arises from a misunderstanding. The misunderstanding arises from reading variable names like receivePort and sendPort with different values, and reading them as if they have an implicit link to the socket at the local end. This might make one (mistakenly) believe that two sockets are being used, or must be used - one for send, one for receive. This is wrong - a single socket (at each end) is all that is required.
If using variables to refer to ports on a single host, it is preferable to name them such that it is clear that one is local or pertaining to "this" process, and the other is remote or peer and pertains to the address of a different process, despite being on the same local host. Then it should be clearer that, like any socket, it is entirely possibly to support both send and receive from the single socket with its single port number.
In this scenario (inter-process communication on the same host necessarily using different port numbers for the single socket at each end) all the other questions (SO_REUSEADDR, TCP vs UDP and receiving one's own transmissions) are distractions arising from the misunderstanding.
I was wondering,
1st question What are the pros and cons of using one socket (full duplex) vs. two socket (simplex) per peer: one for read and other write? Specially in terms of performance and resource utilization.
2nd question In case, if i choose to use more than 1 sockets per peer, on all i do read and write. Then will it helps me scale out in handling no of messages handled?
3rd question: what should help me determine the number of sockets per peer? Network Bandwidth? No. of message in and out?
All questions are different and do not have any inter-relation.
What are the pros and cons of using one socket (full duplex) vs. two socket one for read and other write? Specially in terms of performance and resource utilization.
Pro one socket: resource utilization. Contra one socket: nil. Performance: identical, except that you save on connect and close handshakes if you only use one socket.
In case, I choose to take two socket approach, then will not be useful to use both of them full duplex, that way it helps me scale out in terms of data flowing in and out?
Now you're comparing apples and oranges. You can't compare one full-duplex socket with two full-duplex sockets. I don't know why you think you might need two inbound and two outbound flows, but you don't. Every protocol I can think of except FTP uses only one.
what impact does network bandwidth has on it?
Nil.
or it has on network utilization?
Nil, apart from the connect and close handshakes. But it wastes resources at both ends.
We've added --full-duplex to iperf 2.0.14 which will test a full-duplex socket. One can dompare it to two sockets per the -d or --dualtest option. We've found "your mileage will vary" and there is no universal to answer of having equal performance or not. In theory, it seems they should be equal but, in practice, maybe not.
-d, --dualtest
Do a bidirectional test simultanous test using two unidirectional sockets
--fq-rate n[kmgKMG]
Set a rate to be used with fair-queueing based socket-level pacing, in bytes or bits per second. Only available on platforms supporting the SO_MAX_PACING_RATE socket option.
(Note: Here the suffixes indicate bytes/sec or bits/sec per use of uppercase or lowercase, respectively)
--full-duplex
run a full duplex test, i.e. traffic in both transmit and receive directions using the same socket
Bob
I'm looking for a ZMQ-capable solution for a communication between an bound endpoint that is connected to 0 or 1 peers and no more than that. The communication is two-way, and the connection can be ended or severed at any point in time; and the connection can be re-establish with either a new peer or the same peer. It doesn't matter if the bound endpoint blocks or doesn't block if it doesn't have a peer on the other side.
What ZMQ socket pair would suit this use case the best? I was initially thinking REP/REQ, but the socket pair allows for multiple REQs to connect to one REP, which I don't want; it also will need to handle the "I'm waiting for a recv/I'm going to send something" lockstep paradigm when someone disconnects. PAIR also seems bad because it doesn't naturally handle the reconnect, but it has the "0 or 1 peer" restriction I want.
Any suggestions?
The short answer: Unfortunately there isn't a pattern that exactly fits your needs out of the box.
The closest pattern is the ZMQ PAIR to PAIR pattern. However it has some limitations in the following ways:
ZMQ_PAIR sockets are designed for inter-thread communication across
the zmq_inproc(7) transport and do not implement functionality such as
auto-reconnection. ZMQ_PAIR sockets are considered experimental and
may have other missing or broken aspects.
ROUTER and DEALER is the most flexible pattern. You can control it to set the restrictions you need.
I'm writing a test program that needs to emulate several connections between virtual machines, and it seems like the best way to do that is to use Unix domain sockets, for various reasons. It doesn't really matter whether I use SOCK_STREAM or SOCK_DGRAM, but it seems like SOCK_STREAM is easier/simpler for my usage.
My problem seems to be a little backwards from the typical scenario. I want to have a single client communicating with the server over 4 distinct sockets. (I could have 4 clients with one socket each, but that distinction shouldn't matter.) Now, the thing I'm emulating doesn't have multiple threads and gets an interrupt whenever a data packet is received over one of the "sockets". Is there some easy way to emulate this with Unix sockets?
I believe that I have to do the socket(), bind(), and listen() for all 4 sockets first, then do an accept() for all 4, and do fcntl( fd, F_SETFF, FNDELAY ) for each one to make them nonblocking, so that I can check each one for data with read() in a round-robin fashion. Is there any way to make it interrupt-driven or event-driven, so that my main loop only checks for data in the socket if there's data there? Or is it better to poll them all like this?
Yes. Handling multiple connections is almost synonymous with "server", and they are often single threaded -- but please not this way:
check each one for data with read() in a round-robin fashion
That would require, as you mention, non-blocking sockets and some kind of delay to prevent your "round-robin" from becoming a system killing busy loop.
A major problem with that is the granularity of the delay. You can't make it too small, or the loop will still hog too much CPU time when nothing is happening. But what about when something is happening, and that something is data incoming simultaneously on multiple connections? Now your delay can produce a snowballing backlog of tish leading to refused connections, etc.
It just is not feasible, and no one writes a server that way, although I am sure anyone would give it serious thought if they were unaware of the library functions intended to tackle the problem. Note that networking is a platform specific issue, so these are not actually part of the C standard (which does not deal with sockets at all).
The functions are select(), poll(), and epoll(); the last one is linux specific and the other two are POSIX. The basic idea is that the call blocks, waiting until one or more of any number of active connections is ready to read or write. Waiting for a socket to be ready to write only meaningfully applies to NON_BLOCK sockets. You don't have to use NON_BLOCK, however, and the select() call blocks regardless. Using NON_BLOCK on the individual sockets makes the implementation more complex, but increases performance potential in a single threaded server -- this is the idea behind asynchronous servers (such as nginx), a paradigm which contrasts with the more traditional threaded synchronous model.
However, I would recommend that you not use NON_BLOCK initially because of the added complexity. When/if it ends up being called for, you'll know. You still do not need threads.
There are many, many, many examples and tutorials around about how to use select() in particular.