I am trying to do an IoT project where a node needs to receive some commands from cloud.
Previously, I made this with MQTT protocol. But searching about other protocols on internet I have found REST, which makes possible to communicate over HTTP. However I have seen that this protocol is more used for getting data (makeing a request to the node and receiving the data from it in the response).
I am very new with this protocol. So, I would like to know if it would also be possible to send commands to the node from the cloud, like in MQTT.
Thank you very much for your help.
The HTTP protocol and thus is based in a Request/Response model and using it for IoT device commands can have its drawbacks:
Your Devices will need to be accessible over the Internet and this can be a security compromise when you can have the Device with internet access using MQTT subscriptions but no incoming connections to it are allowed.
You will need to have some mechanisms for retrying and handling offline Devices in your cloud application sending the commands.
There is also more overhead on HTTP than MQTT given that MQTT has long lived connections. With HTTP you will waste more power on connecting and disconnection, also the network packets will be bigger than MQTT.
Can you use REST/HTTP for sending commands to Devices? Yes. Should you do it? Probably not, it all depends on your actual Device requirements and capabilities and why are you wanting to replace MQTT.
An IoT protocol that is similar to REST is CoAP. If you are thinking about using REST in the IoT context I recommend to have a look at it. If your nodes are always connected/reachable there is nothing that speaks against going in this direction. CoAP is a decentralized service protocol where each node might speak with another node.
MQTT is a publish/subscribe protocol with a central broker. As such your cloud could just send the commands to the broker and whenever the node connects to the broker it will receive the commands. So the node doesn't have to be online the whole time.
Related
The IBM IoT Foundation allows devices to submit events to the IBM cloud for consumption and recording. There appears to be two primary mechanisms to achieve the transmission of events ... MQTT and REST (HTTP POST requests). Assuming that a project will have sensors with direct TCP connectivity to IBM cloud over the Internet, what might we consider as the potential distinctions between the two technologies? What factors would case us to choose MQTT or REST as the technology to use? Are there any substantial performance differences at the final mile at the IBM end that would say that one technology is preferred over another?
MQTT is designed to be a fast and lightweight messaging protocol, and is as a result, faster and more efficient at this than HTTP when used to do the equivalent. More efficient not only means less traffic data and more speed, but sometimes it can mean less electrical power as well. MQTT is particularly good where bandwidth is a concern.
MQTT does, however, need a client implementation (like Paho) which is possibly a rarer thing than an HTTP client implementation, which would be more ubiquitous and therefore more likely/easily available on any given device.
There are also TCP/IP port considerations, where some network hardware may require HTTP ports 80 or 443 (although IoTF supports MQTT and MQTTWS on port 443).
There may also be an ideological or philosophical reason for choosing HTTP instead of MQTT (or COAP for that matter), but usually, I would say the reasons for choosing HTTP instead of MQTT would be network related or client support related.
There is no official paper on the performance differences yet, but safe to say MQTT will be more efficient and faster given just about any messaging scenario (long lived connections or adhoc etc.)
I would summarize the considerations as:
mqtt will support higher throughout and the API is much simpler compared to a REST api
REST API is likely much more readily available on iot devices, BUT this could be changing as mqtt is gaining in popularity and big players like Google Cloud Platform and IBM Bluemix support mqtt in their iot service.
I need to make TCP based decentralised chat app for local network. By decentralised I mean there is no central server. Each entity on a network should have server/client architecture. When app starts it should check which user is online ( already running the app ). My question is how can i check that? Can i do it by trying to connect via connect() function from socket library? I'm new to programming, especially socket programing, so if it's a dumb question sorry in advance.
You should definitely study how other decentralized applications do this. There are lots of techniques.
Each instance of the application should, as part of its server functionality, track the addresses of other instances of the application. Each instance should, as part of its client functionality, keep track of a few instances it can connect to. Prefer instances that have been around for a long time.
The software should include a list of servers that have been running for a long time and are expected to typically be available. You may wish to include a fallback method such DNS, maintained by anyone willing to keep a list of well-known servers offering access through a well-known port. The fallback method can also be IRC or HTTP.
If you want to stay decentralized, you might want to try multicasting or broadcasting a request packet to all hosts on the network to discover other instances of your chat application.
Something similar has been implemented in Pidgin, named Bonjour. It works quite nicely and provides chatting capabilities on a local network. More specifically, it is defined as a Serverless Messaging part of XMPP.
If you are looking for code examples, have a look at one of my projects where I use multicast to discover hosts on the local network that provide a specific service: Headers and implementation.
I am interested in making communications of commands between different MQTT clients and to perform the regarding actions on both end. Is it possible to have two way communication using MQTT? I am working on Raspberry PI.
Yes its possible by using different publisher and subscriber topic for same client. Also you need some handler on client side to act.
Not sure what you mean by two way communication. You have subscribers and publishers in Mqtt. You could have a subscriber sitting out there listening to a particular topic and have it react to certain messages.
The way you would interact with that subscriber is by a publisher. Have it send a message to that subscriber's topic it's listening on.
A client that you have subscribing and publishing is actually initiating the communication in both cases. Since the broker is ack and responding, the broker need not know the IP address of the client since it responds back through the client's TCP path to the socket. This, to an extent assists in security of the client that is behind a firewall since the client does not need port forwarding for the reason mentioned.
I have built a WebSockets server that acts as a chat message router (i.e. receiving messages from clients and pushing them to other clients according to a client ID).
It is a requirement that the service be able to scale to handle many millions of concurrent open socket connections, and I wish to be able to horizontally scale the server.
The architecture I have had in mind is to put the websocket server nodes behind a load balancer, which will create a problem because clients connected to different nodes won't know about each other. While both clients A and B enter via the LoadBalancer, client A might have an open connection with node 1 while client B is connected to node 2 - each node holds it's own dictionary of open socket connections.
To solve this problem, I was thinking of using some MQ system like ZeroMQ or RabbitMQ. All of the websocket server nodes will be subscribers of the MQ server, and when a node gets a request to route a message to a client which is not in the local connections dictionary, it will pub-lish a message to the MQ server, which will tell all the sub-scriber nodes to look for this client and issue the message if it's connected to that node.
Q1: Does this architecture make sense?
Q2: Is the pub-sub pattern described here really what I am looking for?
ZeroMQ would be my option - both architecture-wise & performance-wise
-- fast & low latency ( can measure your implementation performance & overheads, down to sub [usec] scale )
-- broker-less ( does not introduce another point-of-failure, while itself can have { N+1 | N+M } self-healing architecture )
-- smart Formal Communication Pattern primitives ready to be used ( PUB / SUB is the least cardinal one )
-- fair-queue & load balancing architectures built-in ( invisible for external observer )
-- many transport Classes for server-side internal multi-process / multi-threading distributed / parallel processing
-- ready to almost linear scaleability
Adaptive node re-discovery
This is a bit more complex subject. Your intention to create a feasible architecture will have to drill down into more details to solve.
Node authentication vs. peer-to-peer messaging
Node (re)-discovery vs. legal & privacy issues
Node based autonomous self-organising Agents vs. needs for central policy enforcement
To update this for 2021, we just solved this problem where we needed to design a system that could handle millions of simultaneous WS connections from IoT devices. The WS server just relays messages to our Serverless API backend that handles the actual logic. We chose to use docker and the node ws package using an auto-scaling AWS ECS Fargate cluster with an ALB in front of it.
This solved the main problem of routing messages, but then we had the same issue of how do we route response messages from the server. We initially thought of just keeping a central DB of connections, but routing messages to a specific Fargate instance behind an ALB didn't seem feasible.
Instead, we set up a simple sub/pub pattern using AWS SNS (https://aws.amazon.com/pub-sub-messaging/). Every WS server receives the response and then searches its own WS connections. Since each Fargate instance handles just routing (no logic), they can handle a lot of connections when we vertically scale them.
Update: To make this even more performant, you can use a persistent connection like Redis Pub/Sub to allow the response message to only go to one single server instead of every server.
At the bottom, it's all about socket communications. If there is some way to get the ip of the both users, why can't the connection be directly setup between the users instead of having to go thru a server in the middle?
My 2 cents:
No one out there forces us to have a server based real-time communication model. Infact XMPP have an extension called "Serverless Messaging" which defines how to communicate over local or wide-area networks using the principles of zero-configuration networking for endpoint discovery and the syntax of XML streams and XMPP messaging for real-time communication. This method uses DNS-based Service Discovery and Multicast DNS to discover entities that support the protocol, including their IP addresses and preferred ports.
P2P chat applications have been for over a decade now. Having a server in the middle is purely a decision dependent upon your application needs. If your application can live with chats getting lost while the user was transitioning between online/offline status, then you can very well have a direct P2P model going. Similarly, there are a loads and loads of advantages (contact list management, avatars, entity discovery, presence authorization, offline messages, ....) when it comes to choosing a server based messaging model. If you try to have all this right inside your P2P based clients, they might die or under-perform because of all the work they will need to perform by themselves.
"WebSockets" were not designed for P2P/Serverless communication, rather they were designed to provide a standardized PUSH semantic over stateless HTTP protocol. In short, "WebSockets" is a standardized way replacing hacky comet, long-polling, chunked-encoding, jsonp, iframe-based and various other technique developers have been using to simulate server push over HTTP.
Named WebSockets (if someday it is fully and widely supported) could be the solution.
http://namedwebsockets.github.io/spec/
Named WebSockets are useful in a variety of collaborative local device
and local network scenarios: Discover matching peer services on the
local device and/or the local network.
Direct communication between users is possible in Peer To Peer (P2P) networks. In P2P each participant can act as client as well as server. But for P2P networks you need to write a separate program to make the communication possible.
Web Sockets let you leverage existing common browsers as clients. All depends on what is the purpose of your application and how you want to deploy it.
If there is some way to get the ip of the both users
You nailed the answer right in your question.
Most machines I use have IP address of 192.168.0.10 (or similar from 192.168. private network) and are deep, deep behind several layers of NAT. With the end of free IPv4 address pool and IPv6 nowhere near sight, this is the reality most users live. Having a stable intermediary of known, routable address helps a ton working around this issue.
WebSockets don't allow the socket to listen for connections, only to connect as a client to a server (not reverse). Technically they could make it allow this, but as far as I understand the spec doesn't currently (nor is it expected to) allow listen functionality for WebSockets.
The new WebRTC (http://www.webrtc.org/) spec looks like it might support peer-to-peer connections. I have not played with WebRTC at all so I'm not in a position to comment on it. I think it would be a bit more involved than WebSocket stuff. Maybe someone who knows WebRTC better can chime in. (Also apart from the latest version of Chrome I'm not sure if any of the other browsers really support WebRTC yet).