Whats the best way to send large files from one node.js server to another? We tried to encode it with base64 and send it over an allready existent tls socket connection, but the base64 string is to long so the socket splits it in several parts. We also thought to send it via http methods but that seems not the best way for us. Any ideas?
Unless there are no special requirements, I'd use HTTP. HTTP cliens and servers are both available and rather mature in node.js, and HTTP gives you additional features (i.e. Caching, optimistic transactional behaviour, content negotiation, partial requests, etc.).
Don't roll your own protocol based on plain sockets, you are reinventing the wheel. But you might consider other protocols such as FTP as well.
Related
I'm new to network programming and have recently been playing around with using sockets in C++.
I have a pretty decent handle on it at this point, and I understand HTTP/TCP/IP packets pretty well.
However, upon doing some research online it seems like the bulk of network programmers suggest using external libraries such as libcurl (or curl++ for c++) for sending HTTP requests.
Considering that HTTP is a text-based protocol, why is this more beneficial/easier than simply sending HTTP requests as text messages using socket programming?
I found a few sites that show that you can do this without too much difficulty: HTTP Requests in C++ without external libraries?,
Simple C example of doing an HTTP POST and consuming the response
It seems like sending HTTP requests is simply a matter of getting the formatting correct and then sending it via a TCP socket. How is this made easier with external libraries?
Please bear with me as I'm new to network programming and eager to learn.
The links you've provided in your question are in a way a pretty good explanation on why you should not code HTTP yourself it: the first link only points to the socket API and does not say anything about HTTP while the second one provides some examples and code which are too much simplified for real world use and will not even work with with typical setup of multiple domains on the same host since the requests are missing the Host field. In other words: these are resources which might look useful to the inexperienced developer but they will actually lead you into trouble.
HTTP is not the simple as it might look. HTTP/0.9 was simple but is no longer supported by many clients. HTTP/1.0 is still kind of simple if restricted to the basic aspects. But there are already enough pitfalls, like using the wrong delimiter between lines and request header/body or not using a Host field when accessing multi-domain hosts.
And once you want to get efficient you want to have multiple requests per TCP connection and compression and then it gets more complex. With HTTP/1.1 it gets even more complex due to the use of chunked data encoding and with HTTP/2 it gets more efficient but way more complex with a binary protocol and interleaved requests and responses.
And this was only HTTP. With HTTPS you have the additional and not trivial TLS layer which has its own pitfalls.
Thus, if you just want to use HTTP and HTTPS it is much better to use established libraries. Of course if you want to learn the innards of HTTP it might be useful to read all the relevant standards, look at packet traces and try to implement your own.
Background
We are writing a Messenger-like app. We have setup Websockets to Inbox and Chat.
Question
My question is simple. What are the advantages and disadvantages when sending data from Client to Server using REST instead of Websockets? (I am not interested in updates now.)
We know that REST has higher overhead in terms of message sizes and that WS is duplex (thus open all time). What about the other things we didn't keep in mind?
Here's a summary of the tradeoffs I'm aware of.
Reasons to use webSocket:
You need/want server-push of data.
You are sending lots of small pieces of data from client to server and doing it very regularly. Using webSocket has significantly less overhead per transmission.
Reasons to use REST:
You want to use server-side frameworks or modules that are built for REST, not for webSocket (such as auth, rate limiting, security, streaming, etc...).
You aren't sending data very often from client to server and thus the server-side burden of keeping a webSocket connection open all the time may lessen your server scalability.
You want your client to run in places where a long-connected webSocket during inactive periods of time may not be practical (perhaps mobile).
You want your client to run in old browsers that don't support webSocket.
You want the browser to enforce same-origin restrictions (those are enforced for REST Ajax calls, but not for webSocket connections).
You don't want to have to write code that detects when the webSocket connection has died and then auto-reconnects and handles back-offs and handles mobile issues with battery usage issues, etc...
You need to run in situations where there are proxies or other network infrastructure that may not support long running webSocket connections.
If you want request/response built in. REST is request/response. WebSocket is not - it's message based. Responses from a webSocket are done by sending a messge back. That message back is not, by itself, a response to any specific request, it's just data being sent back. If you want request/response with webSocket, then you have to build some infrastructure yourself where you tag an id into a request and the response for that particular request contains that specific id. Otherwise, if there are every multiple requests in flight at the same time, then you don't know which response belongs with which request because all the data is being sent over the same connection and you would have no way of matching response with request.
If you want other clients to be able to carry out this operation via an Ajax call.
So, if you already have a webSocket implementation, don't have any problem with it that are lessened with REST and aren't interested in any of the reasons that REST might be better, then stick with your webSocket implementation.
Related references:
websocket vs rest API for real time data?
Ajax vs Socket.io
Adding comments per your request:
It sounds like you're expecting someone to tell you the "right" way to do it. There are reasons to pick one way over the other. If none of those reason compel you one way vs. the other, then it's just an architectural choice and you must take in the whole context of what you are doing and decide which architectural choice makes more sense to you. If you already have the reliably established webSocket connection and none of the advantages of REST apply to your situation then you can optimize for "efficiency" and send your data to the server over the webSocket connection.
On the other hand, if you wanted there to be a simple API on your server that could be reached with an Ajax call from other clients, then you'd want your server to support this operation via REST so it would simplest for these other clients to carry out this one operation. So, it all depends upon which direction your requirements drive you and, if there is no particular driving reason to go one way or the other, you just make an architectural choice yourself.
I was reading through HTTP2.0 as new HTTP protocol and advantages with binary header and multiplexing. But i would like to know for Rest calls does migrating from HTTP1.1 to HTTP2.0 provide any reasonable advantage. I am not able to find any specific gain for REST full calls with HTTP2.0
Thanks in advance.
It does, in several ways:
Better support for streaming transfers. The conventional alternative is a combination of HTTP/1.1's chunked transfer encoding, connection header Voodoo and the willingness or not of any of the parts to implement HTTP pipelining (which, e.g., curl enables by default). In my experience it takes a lot more of work to get the three to work well together than to just slap HTTP/2. There is no need for chunked transfer-encoding with HTTP/2, and the protocol does not support it.
With HTTP/2, you can have many requests in flight, REST or not, with zero time for establishing connections. This is a blessing both for the browser and for the server, which has to allocate less files descriptors per client.
Header compression also applies to HTTP/2 REST requests, together with the associated bandwidth reduction.
So, if in doubt, go always for HTTP/2. There are also excellent tools out there to develop HTTP/2 applications. Some, like ShimmerCat, even remove the drudgery of setting up certificates and DNS alias, so that starting with HTTP/2 from day one becomes a no-brainer.
Helo, I'm working on a mobile game which needs realtime communication from client to server.
Usually I'll implement a TCP socket server and use some private binary protocol to enable bidirectional communication, and now I also looking into XMPP server like Ejabberd which is based on standard. But XML in some way it's really redundant and inefficient, especially for mobile app it could means more traffic and memory consumption.
Is it a MUST that XMPP use XML?
Is there any XMPP implementation that uses binary as low level data format instead of using XML? (or I shouldn't choose XMPP and start with other standard or technology.)
Any strategy to reduce overhead of sending complex data object (not big file object) using XMPP?
XML is required by the XMPP specification, so there are no binary implementations. It does indeed contain much more overhead, but you have to keep in mind the problem XMPP is designed to solve - an active chat connection can be expected to transmit maybe one message per second.
As for the Google talk api: they use a non-xml protocol for client - Google server connections. When I send a message in the Gmail client, the request body just contains a bunch of post data:
count=1&ofs=16&req0_type=m&req0_to=my.friend%40gmail.com&req0_id=6A8466CBC59CBB0C_0&req0_text=test&req0_chatstate=active&req0_iconset=classic&req0__sc=c
That part is not XMPP. The server which accepts this request then does the job of creating and sending out the XMPP requests. The XMPP is still in XML, they just use a different protocol between the client and Google server.
We are building a proprietary system involving a client and a server linked over TCP/IP. Occasionally, the server will have some new expected or unexpected information that would be of interest to the client. My understanding is that SOAP and REST are query/response systems that require the client to request something and the server responds back. We need the client to register and the server to respond back if and when the new information is available. Are these protocols capable of such behavior? In addition, we need standard database-like queries requiring immediate response.
If you are in the Microsoft world, take a look at WCF, it is capable of this type of communication.
Whether you are in the Microsoft world or not, I would suggest looking into XMPP and AMQP.
First, REST isn't a protocol, it's more of an architecture style.
Second, neither of these are designed for push data delivery.
You could use either for the query-type messages, but you'll need to devise an out-of-band communication protocol to use for push notifications. If your data is fairly simple you could just use a basic TCP socket.