Scenario
We are developing a cypto coin exchange platform.
The data feed is in real time and its from multiple sources, users can listen to global channel which transports everything or it can connect to a specific channel which will transport only data for that x coin. The platform should be able to handle two-way communication between client and server.
Question
Should we use a socket server and to relay every msg from the data feed to the users or should we use a message broker (ex: RabbitMQ, Kafka) ?
In case we use a socket server should we use 2 instances one for the data feed and one for the two-way communication ?
Related
Let's assume there are a several clients that must receive updated data from server. They connect to server and communicate using Server Sent Event push.
How SSE knows that concrete message should be addressed to concrete client like it works in sockets?
Does it support broadcast or private messages?
How SSE knows that concrete message should be addressed to concrete client like it works in sockets?
The client connects to an URL on the server. You can optionally add query parameters to the URL that can be used for logic.
Since the client have initiated the connection, the server must all the time hold a "handle" to this connection, so it can use it to send data. In this way, it is similar to sockets.
Does it support broadcast or private messages?
The server must iterate over all connection handles to send data to all clients. It can send data to only some clients, similar to private messages. How the connections is handled is up to the server.
Raspberry pi is acts as local host i'm trying to send data to raspbberry pi using mqtt with node mcu with two different topics.
eg:
if x>10 then i send 1 otherwise 0
same logic i have used in both node mcu.
if im communicate with only one nodemcu it getting good response but if i connect with both nodemcu's then sometimes not getting value in raspberry pi console.
This often depends on both the client and the broker used, and your configuration of each. The fact that two have problems where one does not suggests a client ID collision: every MQTT client device must have a different client ID. If a broker receives subscriptions from two clients with the same ID, the broker may disconnect one, usually the first. If each client is configured to reconnect, this can cause an endless series of disconnects for both, each of them connected half the time.
Any broker that does not disconnect duplicate clients could still fail to deliver to one, because it uses the client IDs to track which clients a message has been delivered to. The first client that pings for messages on its subscriptions will receive the latest message, and any later ones will miss that message because the message is already marked as delivered to that client ID.
Most clients avoid these problems with random IDs, yet let the developer set one manually. Does your identical logic set a client ID? You can verify what is actually set on each device through the broker's logs.
I'm trying to build instant messaging app. Clients will not only send messages but also often send audios. And I've decided to use websocket connection to communicate with clients. It is fast and allows to send binary data.
The main idea is to receive from client1 message and notify about it client2. But here's the thing. My app will be running on GAE. And what if client1's socket is opened on server1 and client2's is opened on server2. This servers don't know about each others clients.
I have one idea how to solve it, but I am sure it is shitty way. I am going to use some sort of communication between servers(for example JMS or open another websocket connection between servers, doesn't matter right now).
But it surely will lead to a disaster. I can't even imagine how often those servers will speak to each other. For each message server1 should notify server2, server2 should notify client2. But things become even worse when serverN comes into play.
Another way I see this to work is Firebase. But it restricts message size to 4KB. So I can't send audios via it. As a solution I can notify client about new audio and he goes to my server for it.
Hope I clearly explained the problem. Does anyone know how to solve it? Or maybe there are another ways to build such apps?
If you are building a messaging cluster and expect communicating clients to connect to different instances of the server then server-server communication is inevitable. Usually it's not a problem though.
First, if you don't use any load balancing your clients will connect to the same server 50% of time on average (in case of 2 servers).
Second, intra-datacenter links are fast and free in all known public clouds.
Third, you can often do something smart on the frontend to make sure two likely to communicate clients connect to the same server. For instance direct all clients from the same country to the same server using DNS load balancing.
The second part of the question is about passing large media files. It's a common best practice to send it out of band - store on the server and only pass the reference to it. Like someone suggested in the comment, save the audio on the server and just send a message like "audio is available, fetch it from here ...". You don't need to poll the server for that. Just fetch it once when the receiving client requests it.
In general, it seems like you are trying to reinvent the wheel. Just use something off the shelf.
Let all client get connected to multiple servers and each server keeps this metadata
A centralized system like zookeeper stores active servers details
When a client c1 sends a message to client c2:
the message is received by a server (say s1, we can add a load balancer to distribute incoming requests)
s1 will broadcast this information to all other servers to get which server the client c2 is connected to OR a better approach to use consistent hashing which decides which server the client can connect to & in this approach message broadcast is not required
the corresponding server responses to server s1 (say s2)
now s1 sends the message m to s2 and server s2 to client c2
Cons of the above approach:
Each server will have a connection with the n-1 servers, creating a mesh topology
Centralized system (zookeeper) becomes a single point of failures (which is solvable)
Apps like Whatsapp, G-Talk uses XMPP and TCP/IP.
I am interested in making communications of commands between different MQTT clients and to perform the regarding actions on both end. Is it possible to have two way communication using MQTT? I am working on Raspberry PI.
Yes its possible by using different publisher and subscriber topic for same client. Also you need some handler on client side to act.
Not sure what you mean by two way communication. You have subscribers and publishers in Mqtt. You could have a subscriber sitting out there listening to a particular topic and have it react to certain messages.
The way you would interact with that subscriber is by a publisher. Have it send a message to that subscriber's topic it's listening on.
A client that you have subscribing and publishing is actually initiating the communication in both cases. Since the broker is ack and responding, the broker need not know the IP address of the client since it responds back through the client's TCP path to the socket. This, to an extent assists in security of the client that is behind a firewall since the client does not need port forwarding for the reason mentioned.
I have a client and a server application that is communicating just fine, there is a TIdCmdTCPServer in the server and a TIdTCPClient in the client.
The client has to authenticate in the server, the client asks the server for the newest version information and downloads any updates, and other communications. All this communication with TIdTCPClient.SendCmd() and TIdTCPClient.LastCmdResult.Text.Text.
The way it is, the server receives commands and replies, the clients only receives replies, never commands, and I would like to implement a way to make the client receives commands. But as I heard, if the client uses SendCmd it should never be listening for data like ReadLn() as it would interfere with the reply expected in SendCmd.
I thought of making a command to check for commands, for example, the client would send a command like "IsThereCommandForMe" and the server would have a pool of commands to each client and when the client asks, the server send it in the reply, but I think it would not be a good approach as there would be a big delay between the commands being available and the client asking for it. I also thought of making a new connection with new components, for example a TIdCmdTcpClient, but then there would be 2 connections for each client, I don't like that idea as I think it could easily give problems in the communication.
The reason I want this, is that I want to implement a chat functionality in the client, and it should be receiving messages from the server without asking for it all the time, imagine all clients continually asking the server if there is message for them. And I would like to be able to inform the client when there is an update available instead the client being asking if there is any. And with this I could send more commands to the client too.
what are your thoughts about this ? how can I make the server receiving commands from the clients, but also sends them ?
TCP sockets are bidirectional by design. Once the connection between 'client' and 'server' has been established, they are symmetric and data can be sent at any time from any side over the same socket.
It only depends on the protocol (which is just written 'contract' for the communication) which communication model is used. HTTP for example uses a request/reply model. With Telnet for example, both sides can initate data transmissions. (If you take a look at the Indy implementation for Telnet, you will see that it uses a background thread to listen for server data, but it uses the same socket connection in the main thread to send data from client to server).
A "full duplex" protocol which supports both request/response and server push, and also is firewall-friendly, is WebSockets. With WebSockets (a HTTP upgrade), the server can send data to the connected client(s) any time. This would meet your 'chat' requirement.
If you use TIdTCPClient / TIdCmdTCPServer, corporate firewalls might block the communication.