I was reading through HTTP2.0 as new HTTP protocol and advantages with binary header and multiplexing. But i would like to know for Rest calls does migrating from HTTP1.1 to HTTP2.0 provide any reasonable advantage. I am not able to find any specific gain for REST full calls with HTTP2.0
Thanks in advance.
It does, in several ways:
Better support for streaming transfers. The conventional alternative is a combination of HTTP/1.1's chunked transfer encoding, connection header Voodoo and the willingness or not of any of the parts to implement HTTP pipelining (which, e.g., curl enables by default). In my experience it takes a lot more of work to get the three to work well together than to just slap HTTP/2. There is no need for chunked transfer-encoding with HTTP/2, and the protocol does not support it.
With HTTP/2, you can have many requests in flight, REST or not, with zero time for establishing connections. This is a blessing both for the browser and for the server, which has to allocate less files descriptors per client.
Header compression also applies to HTTP/2 REST requests, together with the associated bandwidth reduction.
So, if in doubt, go always for HTTP/2. There are also excellent tools out there to develop HTTP/2 applications. Some, like ShimmerCat, even remove the drudgery of setting up certificates and DNS alias, so that starting with HTTP/2 from day one becomes a no-brainer.
Related
I'm new to network programming and have recently been playing around with using sockets in C++.
I have a pretty decent handle on it at this point, and I understand HTTP/TCP/IP packets pretty well.
However, upon doing some research online it seems like the bulk of network programmers suggest using external libraries such as libcurl (or curl++ for c++) for sending HTTP requests.
Considering that HTTP is a text-based protocol, why is this more beneficial/easier than simply sending HTTP requests as text messages using socket programming?
I found a few sites that show that you can do this without too much difficulty: HTTP Requests in C++ without external libraries?,
Simple C example of doing an HTTP POST and consuming the response
It seems like sending HTTP requests is simply a matter of getting the formatting correct and then sending it via a TCP socket. How is this made easier with external libraries?
Please bear with me as I'm new to network programming and eager to learn.
The links you've provided in your question are in a way a pretty good explanation on why you should not code HTTP yourself it: the first link only points to the socket API and does not say anything about HTTP while the second one provides some examples and code which are too much simplified for real world use and will not even work with with typical setup of multiple domains on the same host since the requests are missing the Host field. In other words: these are resources which might look useful to the inexperienced developer but they will actually lead you into trouble.
HTTP is not the simple as it might look. HTTP/0.9 was simple but is no longer supported by many clients. HTTP/1.0 is still kind of simple if restricted to the basic aspects. But there are already enough pitfalls, like using the wrong delimiter between lines and request header/body or not using a Host field when accessing multi-domain hosts.
And once you want to get efficient you want to have multiple requests per TCP connection and compression and then it gets more complex. With HTTP/1.1 it gets even more complex due to the use of chunked data encoding and with HTTP/2 it gets more efficient but way more complex with a binary protocol and interleaved requests and responses.
And this was only HTTP. With HTTPS you have the additional and not trivial TLS layer which has its own pitfalls.
Thus, if you just want to use HTTP and HTTPS it is much better to use established libraries. Of course if you want to learn the innards of HTTP it might be useful to read all the relevant standards, look at packet traces and try to implement your own.
Recently Chrome, Firefox, cURL etc announced their support for HTTP/3 (it was earlier termed as HTTP-over-QUIC).
How do you see its adaptation impact from perspective of changes in:
Applications (web-based, mobile, pure socket based etc)
Hosting infrastructures (web/app servers, firewalls, loadbalancers, CDNs, router, switches etc) & ISPs etc.
Security (new threats, vulnerabilities, landscape of VAPT tools etc.)
Congestion-control
A very subjective question so not sure a great fit for here but here’s my two cents:
Shouldn’t make any further difference behind what HTTP/2 did. It closes one edge-case of that (lost packets can make HTTP/2 slower on HTTP/2 than HTTP/1.1) and also potentially brings some performance improvements to the initial connection setup. If you’ve not move to, or optimised for, HTTP/2 then may wish to consider that in preparation. Priorities are also due to get a rethink in HTTP/3 but not decided how yet. At the end of the day it’s a transport layer change and the basic semantics of HTTP/2 doesn’t change so to higher level apps it should be fairly seamless - like HTTP/2 was mostly to HTTP/1.1 users.
It’s UDP based (with fallback to TCP-based HTTP/2 and/or HTTP/1.1) which will be fun to enable and setup! Also TLS libraries need to change to support it so could be a while before we see it in some servers. CDNs are already leading the way and will be easiest way of getting this. Like HTTP/2, it’s probably most important to have it on your edge server and then down grade to HTTP/2 or even HTTP/1.1 for internal traffic beyond that. It’s also more fully encrypted which will make it difficult to sniff and reroute traffic as less information is readable to middle boxes than was in TCP.
See answer 2 above which will make it more difficult to sniff traffic. Also it’s very new (not even fully finished and approved at this time of writing) so there may be implementation bugs like there were for HTTP/2 (for example), and some products will not support it initially. On plus side it’s only available over HTTPS which is good for security.
The goal is to introduce a transport and application layer protocol that is better in its latency and network throughput. Currently, the application uses REST with HTTP/1.1 and we experience a high latency. I need to resolve this latency problem and I am open to use either gRPC(HTTP/2) or REST/HTTP2.
HTTP/2:
Multiplexed
Single TCP Connection
Binary instead of textual
Header compression
Server Push
I am aware of all the above advantages. Question No. 1: If I use REST with HTTP/2, I am sure, I will get a significant performance improvement when compared to REST with HTTP/1.1, but how does this compare with gRPC(HTTP/2)?
I am also aware that gRPC uses proto buffer, which is the best binary serialization technique for transmission of structured data on the wire. Proto buffer also helps in developing an language agnostic approach. I agree with that and I can implement the same feature in REST using graphQL. But my concern is over serialization: Question No. 2: When HTTP/2 implements this binary feature, does using proto buffer give an added advantage on top of HTTP/2?
Question No. 3: In terms of streaming, bi-directional use-cases, how does gRPC(HTTP/2) compare with (REST and HTTP/2)?
There are so many blogs/videos out in the internet that compares gRPC(HTTP/2) with (REST and HTTP/1.1) like this. As stated earlier, I would like to know the differences, benefits on comparing GRPC(HTTP/2) and (REST with HTTP/2).
gRPC is not faster than REST over HTTP/2 by default, but it gives you the tools to make it faster. There are some things that would be difficult or impossible to do with REST.
Selective message compression. In gRPC a streaming RPC can decide to compress or not compress messages. For example, if you are streaming mixed text and images over a single stream (or really any mixed compressible content), you can turn off compression for the images. This saves you from compressing already compressed data which won't get any smaller, but will burn up your CPU.
First class load balancing. While not an improvement in point to point connections, gRPC can intelligently pick which backend to send traffic to. (this is a library feature, not a wire protocol feature). This means you can send your requests to the least loaded backend server without resorting to using a proxy. This is a latency win.
Heavily optimized. gRPC (the library) is under continuous benchmarks to ensure that there are no speed regressions. Those benchmarks are improving constantly. Again, this doesn't have anything to do with gRPC the protocol, but your program will be faster for having used gRPC.
As nfirvine said, you will see most of your performance improvement just from using Protobuf. While you could use proto with REST, it is very nicely integrated with gRPC. Technically, you could use JSON with gRPC, but most people don't want to pay the performance cost after getting used to protos.
I am not an expert on this by any means and I have no data to back any of this up.
The "binary feature" you're talking about is the binary representation of HTTP/2 frames. The content itself (a JSON payload) will still be UTF-8. You can compress that JSON and set Content-Encoding: gzip, just like HTTP/1.
But gRPC does gzip compression as well. So really, we're talking about the difference between gzip-compressed JSON vs gzip-compressed protobufs.
As you can imagine, compressed protobufs should beat compressed JSON in every way, or else protobufs have failed at their goal.
Besides the ubiquity of JSON vs protobufs, the only downside I can see to using protobufs is that you need the .proto to decode them, say in a tcpdump situation.
What I found out is,
If you want to send files such as txt, img or video files then REST/HTTP is much faster than gRPC.
If you want to send objects over the wire, then gRPC is effecient.
Sources:
For sending files,, sending large files,, another blog
I'm working on a project in which I want to display biosensor EEG/ECG data measured by a portable device (e.g., a micro controller with wireless data transmission via Wifi or Bluetooth). For this purpose, I need to interface with the portable device/microcontroller, for which the many or some of the device seem to use RESTful interfaces, but offer also probably sockets.
One example of microcontroller with wifi is the "spark.io", which is based on a cortex m3 and CC3000 wireless controller for WiFi access on-board. The data to be transferred are around 500 to 1000 float values per second, which should arrive at the REST client with as little delay as possible. Probably an non-REST approach like sockets would fit better, but I would still like to test an approach based on a RESTFul interface (a tiny argument for this would be that transferring data via RESFul interface seems very common and has good library support).
Q: The question is, what is the best approach for a performant (in the sense of near-realtime) implementation that interfaces with this via REST interface?
I am sure this problem has been solved before, but I could not quickly find a paper via google scholar or technical/scientific blog post that explains this. The only link I found is on "rest hooks", but I am not sure if this is a good approach. Searching on SE didn't reveal a past question on this.
Side note: My approach would be to implement the interface in haskell first to test the design and performance of the RESFull interface. Later the working approach should be ported or implemented with Java/Android/spark.io/some other microcontroller.
(Please note this question is entirely about the architecture and not at all about haskell libraries or anything. If using REST is the stupiest thing, I will accept that as an answer if it is argumented. Also then the question is then whether in general microcontroller web-interfaces and specically their APIs, like that of "spark.io", are in general a stupid idea, if they are implemented via REST. Is this the case? If not, what definition of "near real time" justifies that a REST interface is a bad idea and thus other means of communcation are better. Like: one sensor read per minute? Or, one per second, by 1/10 second, by 1/100 second, by 1/1000 second?)
Okay, let's go through this.
REST is not necessarily a bad idea but it has a lot of features which you may not need. For example, there are REST verbs not just for retrieval, but also updating, deleting, and creating resources. If those functions are important (e.g. you need to send certain control data to the EEG controller) then REST will be nice. If you just want fast access to the stream of data, consider raw TCP instead.
Similarly, REST will package messages into "requests" and their "responses" which come with a bunch of "headers" indicating things like whether the request could be fulfilled, whether it's compressed, etc. These can be great features but may be bloat. You'll probably want to emit enough data on each request so that the ~1kB of headers are a small fraction of it. But given 8-byte floats (doubles), that requires transmitting 500-1000 data points, which you've said will take about one second. Is that our fate -- to always have 1s of latency?
REST will allow you to avoid some of that bloat by declaring a Transfer-Encoding: chunked so that the client can operate on individual chunks as they become available. So that's an architectural decision that I think will need to be made.
I would definitely get Keep-Alive working as soon as possible, and it would be my chief feature when looking for what library to use on the server. Keep-Alive is a standard extension to HTTP which avoids tearing down and rebuilding the TCP stack for each HTTP request. If you don't do this then you have some heavy protocol negotiations each time you send a request.
A crucial decision you'll have to make involves whether you want to do HTTP pipelining or not. You can combine HTTP pipelining with longer-lived requests (ones where you don't expect an immediate response) to essentially "send the data when it becomes available" (i.e. send the headers first and let the server push out the data when it's good and ready). This is an alternative to chunked transfers.
If you can work those out, then HTTP is regularly used to send megabytes per second, so your use case fits well within what REST is capable of. In terms of REST/HTTP libraries for Haskell, if you have to somehow program the controller yourself, the big options are wai, yesod, snap, and rest. If you just need an HTTP client there are a few of those too.
Whats the best way to send large files from one node.js server to another? We tried to encode it with base64 and send it over an allready existent tls socket connection, but the base64 string is to long so the socket splits it in several parts. We also thought to send it via http methods but that seems not the best way for us. Any ideas?
Unless there are no special requirements, I'd use HTTP. HTTP cliens and servers are both available and rather mature in node.js, and HTTP gives you additional features (i.e. Caching, optimistic transactional behaviour, content negotiation, partial requests, etc.).
Don't roll your own protocol based on plain sockets, you are reinventing the wheel. But you might consider other protocols such as FTP as well.