I am looking for advice/guidance on how to achieve the following:
I have a circuit mounted and connected to an Arduino and I am able to easily retrieve data from it, using Python and the pySerial module. It allows me to determine the value of an analog input over time.
At the moment I am storing that data to a file, with a time stamp and the correspondent value and I would love to hear opinions and thoughts on how I could 'share' this data to a web server and 'play' it live.
Is it possible to 'stream' the values into the dump file and retrieve data from it at the same time through an AJAX request or should I look into event-driven web servers like 'Tornado', 'Twisted'...
I am a bit lost here. Just for the record, I am comfortable with PHP and JavaScript for the final output, I just don't have a clue on how to constantly 'stream' the data I need.
Thanks in advance.
If you don't plan to update the Ardunio device too much then it would make sense to have the Python component continue to collect the data over the serial port and publish it in a way that can easily be consumed by a service which can distribute the information in a more efficient, and probably flexible, manner.
e.g.
read the data from the serial port and publish messages onto a message queue. The message queue can then be read by any other component and the data can then be distributed to other applications/clients.
Make a web call to a server that can process each update and distribute to other applications/clients.
You could use something like Pusher (who I work for) and make a call to the REST API to deliver each message to any connected clients. Whilst this is a good way of distributing your data you will be publishing your data even if no clients are listening so I think you are best to get the data to a component like a web server first.
Assuming you go with 1 or 2, you can then use realtime web solution to distribute the data to any number of clients. You could use Pusher here or you could use a self hosted solution.
So, the data flow as I see it would be:
Ardunio -> small Python app -> Queue (or HTTP request to Web server) -> Realtime Web Technology -> Many clients
Related
I'm a PHP developer with a lack of experience on other sever side langages.
I’d like you to give me leads, advice, keywords or whatever that could help me refine my research better.
What I want to do is basically to create a one to one mobile app chat that will scale.
There will be channels of 3 users: User A, User B and the "computer" pushing some messages according to some channels informations like the last time a message has been sent, etc.
User A should know if User B is online, writing, etc.
Every conversation should be stored in a database and would be analyzed by some algorithms. Those algorithms would also analyzed stuffs on user Facebook open graph.
The application should be able to send notification on IOS and Android.
A web administration should allow admin to set some stuff that will define what kind of message would be sent by the "computer".
I'v read lot of posts about websocket, xmpp, node.js, socket.io, etc but I don't have enough knowledge in those areas to decide what architecture should I build to make everything works together. Or maybe there is some cloud base solutions that would fit my needs...
Thanks
As you've stated there are many ways to implement that kind of structure but I am going to write about node.js + socket.io part;
1) It is scalable. You can use cluster, nginx, haproxy. etc. to apply load balancing to your socket.io application (see here) Of course you've to use redis or mongo or some kind of store for socket.io that different servers and processes can communicate each other. (see here)
2) socket.io have rooms. That means clients and any computer bots can join that room to share events with each other. So, in your scenario User A, User B and a computer bot should join to same room and events sent to that room will be broadcasted to every room member. (events can vary as online, typing, new message, anything) (see here)
3) node.js can send push notifications both for iOS and Android.
4) You can write every message to database of your choice on new message event.
5) You can create a REST api with Express framework for your Administration page. And you can use passport for authentication and authorization purposes. You can consume the rest api with Jquery or some other frontend framework like React etc.
Meteor is very well suited for something like this and gives you everything you need. There are also open sourced chat systems built with meteor already to get an idea of where you need to go. The more custom route would be to do what #cdagli said.
I understand the main principles behind both. I have however a thought which I can't answer.
Benchmarks show that WebSockets can serve more messages as this website shows: http://blog.arungupta.me/rest-vs-websocket-comparison-benchmarks/
This makes sense as it states the connections do not have to be closed and reopened, also the http headers etc.
My question is, what if the connections are always from different clients all the time (and perhaps maybe some from the same client). The benchmark suggests it's the same clients connecting from what I understand, which would make sense keeping a constant connection.
If a user only does a request every minute or so, would it not be beneficial for the communication to run over REST instead of WebSockets as the server frees up sockets and can handle a larger crowd as to speak?
To fix the issue of REST you would go by vertical scaling, and WebSockets would be horizontal?
Doe this make sense or am I out of it?
This is my experience so far, I am happy to discuss my conclusions about using WebSockets in big applications approached with CQRS:
Real Time Apps
Are you creating a financial application, game, chat or whatever kind of application that needs low latency, frequent, bidirectional communication? Go with WebSockets:
Well supported.
Standard.
You can use either publisher/subscriber model or request/response model (by creating a correlationId with each request and subscribing once to it).
Small size apps
Do you need push communication and/or pub/sub in your client and your application is not too big? Go with WebSockets. Probably there is no point in complicating things further.
Regular Apps with some degree of high load expected
If you do not need to send commands very fast, and you expect to do far more reads than writes, you should expose a REST API to perform CRUD (create, read, update, delete), specially C_UD.
Not all devices prefer WebSockets. For example, mobile devices may prefer to use REST, since maintaining a WebSocket connection may prevent the device from saving battery.
You expect an outcome, even if it is a time out. Even when you can do request/response in WebSockets using a correlationId, still the response is not guaranteed. When you send a command to the system, you need to know if the system has accepted it. Yes you can implement your own logic and achieve the same effect, but what I mean, is that an HTTP request has the semantics you need to send a command.
Does your application send commands very often? You should strive for chunky communication rather than chatty, so you should probably batch those change request.
You should then expose a WebSocket endpoint to subscribe to specific topics, and to perform low latency query-response, like filling autocomplete boxes, checking for unique items (eg: usernames) or any kind of search in your read model. Also to get notification on when a change request (write) was actually processed and completed.
What I am doing in a pet project, is to place the WebSocket endpoint in the read model, then on connection the server gives a connectionID to the client via WebSocket. When the client performs an operation via REST, includes an optional parameter that indicates "when done, notify me through this connectionID". The REST server returns saying if the command was sent correctly to a service bus. A queue consumer processes the command, and when done (well or wrong), if the command had notification request, another message is placed in a "web notification queue" indicating the outcome of the command and the connectionID to be notified. The read model is subscribed to this queue, gets messessages and forward them to the appropriate WebSocket connection.
However, if your REST API is going to be consumed by non-browser clients, you may want to offer a way to check of the completion of a command using the async REST approach: https://www.adayinthelifeof.nl/2011/06/02/asynchronous-operations-in-rest/
I know, that is quite appealing to have an low latency UP channel available to send commands, but if you do, your overall architecture gets messed up. For example, if you are using a CQRS architecture, where is your WebSocket endpoint? in the read model or in the write model?
If you place it on the read model, then you can easy access to your read DB to answer fast search queries, but then you have to couple somehow the logic to process commands, being the read model the responsible of send the commands to the write model and notify if it is unable to do so.
If you place it on the write model, then you have it easy to place commands, but then you need access to your read model and read DB if you want to answer search queries through the WebSocket.
By considering WebSockets part of your read model and leaving command processing to the REST interface, you keep your loose coupling between your read model and your write model.
These applications stream video from client app to their own server. I am interested in knowing what type of protocol they use? I am planning on building a similar application but I dont know how to go about the video streaming. Once I get the stream to my server I will use OpenCV to do some processing and return the result to the client.
I recommend you to send only a minimum of data and do the processing as much as possible on the client. Since sending the whole video stream is a huge waste of traffic (and can not be done in realtime I think)
I would use a TCP connection to send an intermediate result to the server, that the server can process further. The desing of that communication depends on what you are sending and what you want to do with it.
You can wrap it in xml for instance, or serialize an object and so on.
How do they do this? I would like to have web pages with data fields that change in real time as a person views the web page. Here is an example.
How do they do this? JQuery? PHP?
I need to connect my field data to mySQL database.
There are two approaches:
Polling
Client requests data on a regular basis. Uses network and server resources even when there is no data. Data is not quite 'live'. Extremely easy to implement, but not scalable.
Push
Server sends data to the client, so client can simply wait for it to arrive instead of checking regularly.
This can be achieved with a socket connection (since you are talking about web pages, this doesn't really apply unless you are using Flash, since support for sockets in the browser in the browser is currently immature) - or by using the technique known as 'comet'.
Neither socket connections nor comet are particularly scalable if the server end is implemented naively.
- To do live data on a large scale (without buying a boat load of hardware) you will need server software that does not use a thread for each client.
I did it with JavaScript timer set execution in milliseconds, each time timer executed function that queried Server with Ajax and returned value(possibly JSON format), then you you update your field with the value. I did it each 5 sec and it works perfectly. In ASP.NET I think it called Ajax Timer Control.
There are two things needed to do this:
Code that runs on the browser to fetch the latest data. This could be Javascript or something running in a plugin such as Silverlight or Flash. This will need to periodically request updated content from the server.
Which leads to a need for...
Code that runs on the server to retrieve and return the latest data (from the database). This could be created with any server sided scripting language.
I unsuccessfully searched Google for a good definition and understanding of streaming data and its characteristics. My questions are:
What is streaming data?
How can it be detected?
Correction:
"How can it be detected" is not an appropriate question. Instead my question is:
How is it different from buffered data and other data transfer mechanisms?
It depends in what context you mean but basically streaming data is analagous to asynchronous data. Take the Web as an example. The Web (or HTTP specifically) is (basically) a request-response mechanism in that a client makes a request and receives a response (typically a Web page of some kind).
HTTP doesn't natively support the ability for servers to push content to clients. There are a number of ways this can be faked, including:
Polling: forcing the client to make repeated requests, typically inconspicuously (as far as the client is concerned);
Long-lived connections: this is where the client makes a normal HTTP request but instead of returning immediately the server hangs on to the request until there's something to send back. When the request times out or a response is sent th eclient sends another request. In this way you can fake server push;
Plug-ins: Java applets, Flash, Silverlight and others can be used to achieve this.
Anything where the server effectively sends data to the client (rather than the client asking for it)--regardless of the mechanism and whether or not the client is polling for that data--can be characterised as streaming data.
With non-HTTP transports (eg vanilla TCP) server push is typically easier (but can still run afoul of firewalls and th elike). An example of this might be a sharetrading application that receives market information from a provider. That's streaming data.
How do you detect it? Bit of a vague question. I'm not really sure what you're getting at.
When you say streaming data I think of the following, although I'm not sure if this is what you're getting at. To me it's playing a video/audio file while it's downloading. That's what happens when you go to YouTube and watch a video and it starts playing even though you haven't downloaded the whole video yet. But you can see the video downloading - I'm sure you're familiar with the seek bar filling up as the file downloads. It doesn't necessarily have to be a video or audio file but that's the most common.