I am interested in making communications of commands between different MQTT clients and to perform the regarding actions on both end. Is it possible to have two way communication using MQTT? I am working on Raspberry PI.
Yes its possible by using different publisher and subscriber topic for same client. Also you need some handler on client side to act.
Not sure what you mean by two way communication. You have subscribers and publishers in Mqtt. You could have a subscriber sitting out there listening to a particular topic and have it react to certain messages.
The way you would interact with that subscriber is by a publisher. Have it send a message to that subscriber's topic it's listening on.
A client that you have subscribing and publishing is actually initiating the communication in both cases. Since the broker is ack and responding, the broker need not know the IP address of the client since it responds back through the client's TCP path to the socket. This, to an extent assists in security of the client that is behind a firewall since the client does not need port forwarding for the reason mentioned.
Related
I'm trying to build a entirely contained trading simulator using quickfix/J. The systems ought to consist of 2 client applications (a market/exchange and a broker) as well as a router (server/acceptor). In particular I'd like to know:
Client-Client communication
How the two clients can communicate to each other, but the server handling all the messaging logic, ie. messages should go through server and it should decide where and how to forward messages. I ought to be able to pass a targetID in FIX message, and the server app should handle routing to desired client.
Multiple clients on same port
Have multiple clients connected on same port but messages should only go to a particular sender comp Id ie. clients should not be privy of communication from other clients.
I've already set up the acceptor, and 2 clients. I know I could do this programmaticaly using plain old Java but I'd like to leverage the quickfix library and would like a relativly out of the box solution.
MVP: client (broker) sends fix message through the acceptor(router), message is processed and forwarded to a particular market, market recieves message through server and does some business logic, market sends fix message back to client through acceptor.
ps: I like the quickfix library but I'm very flexible if there any other library/languages you'd recommend
Short answer: QuickFIX/J (as far as I can tell similarly QuickFIX or quickfix/n) will not route messages based on tags. This has to be implemented in your application code.
Edit: with regard to your second point. There is no problem having your FIX server listening for multiple FIX connections on the same port (This applies for QuickFIX/J and I guess also the other language variants.) Sessions are addressed via the SessionID so it is ensured that only the correct FIX Session gets its messages.
I need to do the following:
multiple clients connecting to the SAME remote port
each of the clients open 2 different sockets, one is a PUB/SUB, the
other is a ROUTER/DEALER ( the server can occasionally send back to client heartbeats, different server related information ).
I am completely lost whether it can be done in ZeroMQ or not.
Obviously if I can use 2 remote ports, that is not an issue, but I fail
to understand if my setup can be achieved with some kind of envelope
usage in ZeroMQ.
Can it be done?
Thanks,
Update:
To clarify what I wish to achieve.
Multiple clients can communicate with the server
Clients operate on request-response basis mostly(on one socket)
Clients create a session socket, which means that whenever this
type of socket is created, a separate worker thread needs to be created
and from that time on the client communicates with this worker thread
with regards to requests processing, e.g. server thread must not block
the connection of other clients when dealing with the request of one client
However clients can receive occasional messages from the worker thread with regards to heartbeats of the worker.
Update2:
Actually I could sort it out. What I did:
identify clients obviously, so ROUTER/DEALER is used, e.g. clients
are indeed dealers, hence async processing is provided
clients send messages to the one and only local port, where the router sits
router peeks into messages (kinda the lazy pirate example), checks whether a new client comes in; if yes it offloads to a separate thread, and connects the separate thread with an internal "inproc:" socket
router obviously polls for the frontend and all connected clients' backends and sends messages back and forth.
What bugs me is that it is an overkill if I compare this with a "regular" socket solution, where I could have connected the client with the worker thread DIRECTLY (e.g. worker thread could recv from the socket opened by the client directly), hence I could spare the routing completely.
What am I missing?
There was a discussion on the ZeroMQ mailing list recently about multiplexing multiple services on one TCP socket. The proposed solutions is essentially what you implemented.
The discussion also mentions Malamute with its brokers which essentially provides a framework based on ZeroMQ which also provides the functionality you need. I haven't had the time to look into it myself, but it looks promising.
I have a client and a server application that is communicating just fine, there is a TIdCmdTCPServer in the server and a TIdTCPClient in the client.
The client has to authenticate in the server, the client asks the server for the newest version information and downloads any updates, and other communications. All this communication with TIdTCPClient.SendCmd() and TIdTCPClient.LastCmdResult.Text.Text.
The way it is, the server receives commands and replies, the clients only receives replies, never commands, and I would like to implement a way to make the client receives commands. But as I heard, if the client uses SendCmd it should never be listening for data like ReadLn() as it would interfere with the reply expected in SendCmd.
I thought of making a command to check for commands, for example, the client would send a command like "IsThereCommandForMe" and the server would have a pool of commands to each client and when the client asks, the server send it in the reply, but I think it would not be a good approach as there would be a big delay between the commands being available and the client asking for it. I also thought of making a new connection with new components, for example a TIdCmdTcpClient, but then there would be 2 connections for each client, I don't like that idea as I think it could easily give problems in the communication.
The reason I want this, is that I want to implement a chat functionality in the client, and it should be receiving messages from the server without asking for it all the time, imagine all clients continually asking the server if there is message for them. And I would like to be able to inform the client when there is an update available instead the client being asking if there is any. And with this I could send more commands to the client too.
what are your thoughts about this ? how can I make the server receiving commands from the clients, but also sends them ?
TCP sockets are bidirectional by design. Once the connection between 'client' and 'server' has been established, they are symmetric and data can be sent at any time from any side over the same socket.
It only depends on the protocol (which is just written 'contract' for the communication) which communication model is used. HTTP for example uses a request/reply model. With Telnet for example, both sides can initate data transmissions. (If you take a look at the Indy implementation for Telnet, you will see that it uses a background thread to listen for server data, but it uses the same socket connection in the main thread to send data from client to server).
A "full duplex" protocol which supports both request/response and server push, and also is firewall-friendly, is WebSockets. With WebSockets (a HTTP upgrade), the server can send data to the connected client(s) any time. This would meet your 'chat' requirement.
If you use TIdTCPClient / TIdCmdTCPServer, corporate firewalls might block the communication.
When using ØMQ socket of type SUB, one may use
sub_socket.setsockopt_string(zmq.SUBSCRIBE, 'topic')
Is the same possible also with REP sockets, allowing a worker to only handle specific topics, leaving other topics to different workers?
I'm very afraid that it is impossible, quoting http://learning-0mq-with-pyzmq.readthedocs.org/en/latest/pyzmq/patterns/pubsub.html:
In the current versions of ØMQ, filtering happens at the subscriber side, not the publisher side.
But still, I'm asking if there is some trick to achieve that, because such a functionality would have a huge impact on my infrastructure.
Nope. Can I assume that you've got a REQ or DEALER server socket that sends work to REP workers, that then respond with the completed work back to the server? And that you're looking for a way to make your server communicate to specific clients rather than just pass out tasks in a round-robin fashion?
Can't do it. See here, those sockets are only, always, round-robin. If you want to communicate to a specific client, you must either have a socket that talks only to that client, or you must start the communication from the client (switch your socket pairing so the worker requests whatever work its ready for, and the server responds with it, and then the worker creates a new request with the completed work). Doing anything else gets much more complicated.
I have clients that need to all connect to a single server process. I am using UDP discovery for the clients to find the server. I have the client and server exchange IP address and port number, so that a TCP/IP connection can be established after completion of the discovery. This way the packet size is kept small. I see that this could be done in one of two ways using UDP:
Each client sends out its own multicast message in search of the server, which the server then responds to. The client can repeat sending this multicast message in regular intervals (in the case that the server is down) until the server responds.
The server sends out a multicast message beacon at regular intervals. The clients subscribe to the multicast group and in this way receives the server's multicast message and complete the discovery.
In 1. if there are many clients then initially there would be many multicast messages transmitted (one from each client). Only the server would subscribe and receive the multicast messages from the clients. Once the server has responded to the client, the client ceases to send out the multicast message. Once all clients have completed their discovery of the server no further multicast messages are transmitted on the network. If however, the server is down, then each client would be sending out a multicast message beacon in intervals until the server is back up and can respond.
In 2. only the server would submit a multicast message beacon in regular intervals. This message would end up getting routed to all clients that are subscribed to the multicast group. Once the clients receive the packet the client's UDP listening socket gets closed and they are no longer subscribed to the multicast group. However, the server must continue to send the multicast beacon, so that new clients can discover it. It would continue sending out the beacon at regular intervals regardless of whether any clients are out their requiring discovery or not.
So, I see pros and cons either way. It seems to me that #1 would result in heavier load initially, but this load eventually reduces down to zero. In #2 the server would continue sending out a beacon forever.
UDP and multicast is a fairly new topic to me, so I am interested in finding out which would be the preferred approach and which would result in less network load.
I've used option #2 in the past several times. It works well for simple network topologies. We did see some throughput problems when UDP datagrams exceeded the Ethernet MTU resulting in a large amount of fragmentation. The largest problem that we have seen is that multicast discovery breaks down in larger topologies since many routers are configured to block multicast traffic.
The issue that Greg alluded to is rather important to consider when you are designing your protocol suite. As soon as you move beyond simple network topologies, you will have to find solutions for address translation, IP spoofing, and a whole host of other issues related to the handoff from your discovery layer to your communications layer. Most of them have to do specifically with how your server identifies itself and ensuring that the identification is something that a client can make use of.
If I could do it over again (how many times have we uttered this phrase), I would look for standards-based discovery mechanisms that fit the bill and start solving the other protocol suite problems. The last thing that you really want to do is come up with a really good discovery scheme that breaks the week after you deploy it because of some unforeseen network topology. Google service discovery for a starting list. I personally tend towards DNS-SD but there are a lot of other options available.
I would recommend method #2, as it is likely (depending on the application) that you will have far more clients than you will servers. By having the server send out a beacon, you only send one packet every so often, rather than one packet for each client.
The other benefit of this method, is that it makes it easier for the clients to determine when a new server becomes available, or when an existing server leaves the network, as they don't have to maintain a connection to each server, or keep polling each server, to find out.
Both are equally viable methods.
The argument for method #1 would be that in normal principle, clients initiate requests, and servers listen and respond to them.
The argument for method #2 would be that the point of multicast is so that one host can send a packet and it can be received by many clients (one-to-many), so it's meant to be the reverse of #1.
OK, as I think about this I'm actually drawn to #2, server-initiated beacon. The problem with #1 is that let's say clients broadcast beacons, and they hook up with the server, but the server either goes offline or changes its IP address.
When the server is back up and sends its first beacon, all the clients will be notified at the same time to reconnect, and your entire system is back up immediately. With #1, all of the clients would have to individually realize that the server is gone, and they would all start multicasting at the same time, until connected back with the server. If you had 1000 clients and 1 server your network load would literally be 1000x greater than method #2.
I know these messages are most likely small, and 1000 packets at a time is nothing to a UDP network, but just from a design standpoint #2 feels better.
Edit: I feel like I'm developing a split-personality disorder here, but just thought of a powerful point of why #1 would be an advantage... If you ever wanted to implement some sort of natural load balancing or scaling with multiple servers, design #1 works well for this. That way the first "available" server can respond to the client's beacon and connect to it, as opposed to #2 where all the clients jump to the beaconing server.
Your option #2 has a big limitation in that it assumes that the server can communicate more or less directly with every possible client. Depending on the exact network architecture of your operational system, this may not be the case. For example, you may be depending that all routers and VPN software and WANs and NATs and whatever other things people connect networks together with, can actually handle the multicast beacon packets.
With #1, you are assuming that the clients can send a UDP packet to the server. This is an entirely reasonable expectation, especially considering the very next thing the client will do is make a TCP connection to the same server.
If the server goes down and the client wants to find out when it's back up, be sure to use exponential backoff otherwise you will take the network down with a packet storm someday!