I have an iPhone application that uses Moscapsule in order to establish MQTT connection to a Mosquitto broker (mosquitto version 1.4.4).
I create the connection like this:
mqttConfig = MQTTConfig(clientId: "iphone7_UI", host:"x.x..x.x", keepAlive:9999)
Other than the above I am basically relying on the library to keep the connection up and running and properly sub/publish MQTT messages.
The application mostly publishes messages and so far I have not had any issues with publications. Recently I added a feature where it is required for the application to subscribe to a topic and this is where I am getting issues.
The application successfully subscribes to the topic and receives messages properly in the beginning but then after a while (random amount of time, a couple of minutes usually) I can see at the mosquitto printouts that the iphone client disconnects and then reconnects and when that happens it no longer receives any MQTT messages (it can still publish properly).
What could be the reason why the application (probably Moscapsule library that I am using) decides to discard the old connection and create a new one? any ideas?
Related
I am new to ZMQ and am not sure if what I want is even possible or if I should use another technology.
I would like to have a socket that multiple servers can stream to.
It appears that a ZMQ socket can do this based on this documentation: http://api.zeromq.org/4-0:zmq-setsockopt
How would I implement a ZMQ socket on the receiving end that only grabs the latest message sent from each server?
You can do this with Zmq's PUB / SUB.
The first key thing is that a SUB socket can be connected to multiple PUBlishers. This is covered in Chapter 1 of the guide:
Some points about the publish-subscribe (pub-sub) pattern:
A subscriber can connect to more than one publisher, using one connect call each time. Data will then arrive and be interleaved “fair-queued” so that no single publisher drowns out the others.
If a publisher has no connected subscribers, then it will simply drop all messages.
If you’re using TCP and a subscriber is slow, messages will queue up on the publisher. We’ll look at how to protect publishers against this using the “high-water mark” later.
So, that means that you can have a single SUB socket on your client. This can be connected to several PUB sockets, one for each server from which the client needs to stream messages.
Latest Message
The "latest message" can be partially dealt with (as I suspect you'd started to find) using high water marks. The ZMQ_RCVHWM option allows the number to be received to be set to 1, though this is an imprecise control.
You also have to consider what it is that is meant by "latest" message; the PUB servers and SUB client will have different views of what this is. For example, when the zmq_send() function on a PUB server returns, the sent message is the one that the PUBlisher would regard as the "latest".
However, over in the client there is no knowledge of this as nothing has yet got down through the PUBlishing server's operating system network stack, nothing has yet touched the Ethernet, etc. So the SUBscribing client's view of the "latest" message at that point in time is whichever message is in ZMQ's internal buffers / queues waiting for the application to read it. This message could be quite old in comparison to the one the PUBlisher has just started sending.
In reality, the "latest" message seen by the client SUBscriber will be dependent on how fast the SUBscriber application runs.
Provided it's fast enough to keep up with all the PUBlishers, then every single message the SUBscriber gets will be as close to the "latest" message as it can get (the message will be only as old as the network propagation delays and the time taken to transit through ZMQ's internal protocols, buffers and queues).
If the SUBscriber isn't fast enough to keep up, then the "latest" messages it will see will be at least as old as the processing time per message multiplied by the number of PUBlishers. If you've set the receive HWM to 1, and the subscriber is not keeping up, the publishers will try publishing messages but the subscriber socket will keep rejecting them until the subscribed application has cleared out the old message that's caused the queue congestion, waiting for zmq_recv() to be called.
If the subscriber can't keep up, the best thing to do in the subscriber is:
have a receiving thread dedicated to receiving messages and dispose of them until processing becomes available
have a separate processing thread that does the processing.
Have the two threads communicate via ZMQ, using a REQ/REP pattern via an inproc connection.
The receiving thread can zmq_poll both the SUB socket connection to the PUBlishing servers and the REP socket connection to the processing thread.
If the receiving thread receives a message on the REP socket, it can reply with the next message read from the SUB socket.
If it receives a message from the SUB socket with no REPly due, it disposes of the message.
The processing thread sends 1 bytes messages (the content doesn't matter) to its REQ socket to request the latest message, and receives the latest message from the PUBlishers in reply.
Or, something like that. That'll keep the messages flowing from PUBlishers to the SUBscriber, thus the SUBscriber always has a message as close to possible as being "the latest" and is processing that as and when it can, disposing of messages it can't deal with.
Currently I have to sent events externally to the client which needs to subscribe to the these events. I have an endpoint that the client calls (subscribe) that follow the Server-Sent Events specifications. This open a HTTP connection, that is kept alive by the server that send "heartbeat" events.
The problem is that when this service need to be redeployed, or it goes down is the responsibility of the client to re-subscribe making a call to this endpoint, to receive again the events in real-time.
I was wondering, if I switch to technology like rabbitMQ or Kafka can I solve this problem? In other word, I would like to remove the responsibility of the client to re-subscribe if something goes wrong on the server side.
If you can attached article/resources to your answer would be great.
With RabbitMQ , the reconnection feature is dependant on the client library. For e.g., Java and .NET clients do provide this ( check here)
With Kafka I see there are configurations to support this. Also it's worth reading the excellent recommendations from Kakfa for surviving broker outages here.
Raspberry pi is acts as local host i'm trying to send data to raspbberry pi using mqtt with node mcu with two different topics.
eg:
if x>10 then i send 1 otherwise 0
same logic i have used in both node mcu.
if im communicate with only one nodemcu it getting good response but if i connect with both nodemcu's then sometimes not getting value in raspberry pi console.
This often depends on both the client and the broker used, and your configuration of each. The fact that two have problems where one does not suggests a client ID collision: every MQTT client device must have a different client ID. If a broker receives subscriptions from two clients with the same ID, the broker may disconnect one, usually the first. If each client is configured to reconnect, this can cause an endless series of disconnects for both, each of them connected half the time.
Any broker that does not disconnect duplicate clients could still fail to deliver to one, because it uses the client IDs to track which clients a message has been delivered to. The first client that pings for messages on its subscriptions will receive the latest message, and any later ones will miss that message because the message is already marked as delivered to that client ID.
Most clients avoid these problems with random IDs, yet let the developer set one manually. Does your identical logic set a client ID? You can verify what is actually set on each device through the broker's logs.
Building an instant chat app (native IOS and web). Exploring whether to use XMPP or MQTT as application protocols. Seemingly I can't have users editing old messages on XMPP. Can messages be edited on MQTT?
Example: I want to implement "Edit Message" like Slack offers, but upon clicking "(edited)" to allow the user to see the different versions of the message and their timestamps (like the edit history for comments you find in Facebook), enabling an "audit trail" of the conversation.
Follow-up: As it seems this can only be achieved through a "hack", would it be better to get the hack done on XMPP or MQTT or some other protocol/websockets/JSON, etc?
Once a MQTT message is published to the broker the publishing client has no more control over that message at all.
Most brokers will not allow you to edit the message either as they will just forward the message instantly to all clients subscribed to the relevant topics and queue the message for any offline clients with persistent subscriptions.
The only exception may be the mosca broker that has a call back for when messages reach the broker, but this would not allow a user to edit a message, only the system to possibly update the payload in the instant before it was forwarded to the subscribed clients.
Hardlib's advice is correct, editing messages in this way is not supported by most MQTT implementations and to implement it would break the loose coupling between publisher and subscriber that is the virtue of MQTT. In other words this should be implemented at a higher level or through other means.
That said, if I understand editing to mean the ability to change what the broker forwards to clients that were not online during the initial publication, you could implement this with retained messages. Consider this:
Client A is subscribed to topic clientb/# and Client B is subscribed to topic clienta/#.
Client A publishes a message to clienta/(unique message id) while Client B is not actively connected. The broker retains the message.
Client A decides to edit the message so (through some interface you devise) they publish an amended message to clienta/(unique message id) which replaces the message and, from a subscribers perspective, edits what is there.
Client B receives the amended message when they come online and (as long as there isn't a persistent session or something like that) has no knowledge of the change.
From this example you can probably tell why this is a bad idea as the server would retain every single message in a different topic and would likely need regular pruning... not mention that it would make a mess out of timestamps and require all sorts of other work arounds. However, if there is some reason that you have to implement it this way you could hack something usable together.
I'm not sure if this is possible or not. If I set a number of messages to be persisted under a topic for some period of time, can I later grab all of them?
I have an MQTT Broker (Mosquitto) set up already for communication between my services but I now also need some storage for several messages, ideally keeping the last 24 hours worth of messages and being able to pull them out later.
Message persistence is only for clients that have subscribed but are currently disconnected and when they do reconnect do so with the cleanSession flag set to false. In which case all the messages published while that client was disconnected.
You can not use a MQTT broker to store an arbitrary number of messages and retrieve them later. If a client is connected then all messages for it's collection of subscribed topics will be delivered as soon as possible.
Of you want to log messages for later you will have to implement this separately, there are plenty of examples of applications that store messages in databases available