I am exploring HTTP 1.1 persistent connection over single TCP socket for multiple HTTP request from client side. One thing I observed in wireshark is that, after each request-response my client sends an ACK to server. Is this ACK message call right according to protocol standard? Is there any way I can skip this ACK call. I compared the communication behaviour of my client with browser's communication pattern. I think browser does not send any tcp messages to server once tcp handshake is completed to establish connection.
ACK is part of TCP. You can't have a TCP connection without ACK, that's how it works. Data that is received is ACK'ed so the sender does not retransmit it.
HTTP is not dependent on TCP, you could implement HTTP on other protocols. The two protocols should be seen as separate layers, and should not influence each others.
Related
How is HTTP Keep Alive implemented? Does it internally use TCP Keep Alive? If not, how does the server detect if the client is dead or alive?
I know this is an old question, but still:
HTTP Keep-Alive is a feature that allows HTTP client (usually browser) and server (webserver) to send multiple request/response pairs over the same TCP connection. This decreases latency for 2nd, 3rd,... HTTP request, decreases network traffic and similar.
TCP keepalive is a totally different beast. It keeps TCP connection opened by sending small packets. Additionally, when the packet is sent this serves as a check so the sender is notified as soon as connection drops (note that this is NOT the case otherwise - until we try to communicate through TCP connection we have no idea if it is ok or not).
To answer your questions about HTTP Keep-Alive:
How is HTTP Keep Alive implemented?
To put it simply, the HTTP server doesn't close the TCP connection after each response but waits some time if some other HTTP request will come over it too. After some timeout it closes it anyway.
Does it internally use TCP Keep Alive?
No, at least I see no point in it.
If not, how does the server detect if the client is dead or alive?
It doesn't - it doesn't need to. If a client sends a request, it will get the response. If the client doesn't send anything over TCP connection (maybe because the connection is dead) then a timeout will close the connection; client will of course notice this and will send request through another TCP connection if needed.
HTTP Keep-Alive is a feature of HTTP protocol. The web-server, implementing Keep-Alive Feature, has to check the connection/socket periodically (for incoming HTTP request) for the time span since it sent the last HTTP response (in case there was corresponding HTTP Request). If no HTTP request is received by the time of the configured keep-alive time (seconds) the web server closes the connection. No further HTTP request will be possible after the 'close' done by Web Server. On the other hand, TCP Keep-Alive is managed by OS in the TCP layer. HTTP Keep-Alive and TCP Keep-Alive is totally unrelated things.
HTTP keep-alive, a.k.a., HTTP persistent connection, is an instruction that allows a single TCP connection to remain open for multiple HTTP requests/responses.
By default, HTTP connections close after each request. When someone visits your site, their browser needs to create new connections to request each of the files that make up your web pages (e.g. images, Javascript, and CSS stylesheets), a process that can lead to high page load times.
Enabling the keep-alive header allows you to serve all web page resources over a single connection. Keep-alive also reduces both CPU and memory usage on your server.
Source: https://www.imperva.com/learn/performance/http-keep-alive/
http keep-alive is just making tcp living longer in order to transfer multi http request.After keep-alive timeout, the tcp connection will be closed.
tcp keep-alive is just a mechanism keeping the tcp connection,or check the tcp connection is not closed
Here is my understanding of how a client-server HTTP server works.
The client creates a TCP socket connection to connect to the server and sends data.
The server creates a TCP socket connection to listen for incoming requests.
So it looks like both the client and the server need to agree on the use of the Transport protocol to use (in this case TCP). But if we want a website to work over UDP/QUIC protocol then we need both the client and the server to create a UDP socket connection. But some websites use TCP and others use UDP...
So does it mean it would need to look like this? To know beforehand which protocol a website uses?
if (URI == 'https://www.google.com') {
// Website that works over UDP
client.create.UDP.socket
client.sendData
server.create.UDP.socket
server.receive.data
} else {
// Website that works over TCP
client.create.TCP.socket
client.sendData
server.create.TCP.socket
server.receive.data
}
So the client needs to keep a record of which website uses TCP and which websites use UDP/QUIC and create that kind of socket to communicate with it?
If a protocol supports both TCP and UDP, the server listens both on the TCP port and the UDP port. The port number is generally the same, e.g. DNS uses TCP port 53 and UDP port 53.
Usually the client has a preference. Let's say it prefers TCP. The client will first try to connect with TCP. If the server does not respond, the client will retry with UDP. Alternatively, the server can respond over TCP but ask the client to switch to UDP. The client can then decide to continue with TCP or switch to UDP.
For QUIC 1 2, the browser will first connect using HTTP over TCP. The server will respond with a message stating that it also supports QUIC. If the browser also supports this protocol, the browser will reconnect to the server using QUIC.
I have the following queries:
1) Does TCP guarantee delivery of packets and thus is thus application level re-transmission ever required if transport protocol used is TCP. Lets say I have established a TCP connection between a client and server, and server sends a message to the client. However the client goes offline and comes back only after say 10 hours, so will TCP stack handle re-transmission and delivering message to the client or will the application running on the server need to handle it?
2) Related to the above question, is application level ACK needed if transport protocol is TCP. One reason for application ACK would be that without it, the application would not know when the remote end received the message. Is there any reason other than that? Meaning is the delivery of the message itself guaranteed?
Does TCP guarantee delivery of packets and thus is thus application level re-transmission ever required if transport protocol used is TCP
TCP guarantees delivery of message stream bytes to the TCP layer on the other end of the TCP connection. So an application shouldn't have to bother with the nuances of retransmission. However, read the rest of my answer before taking that as an absolute.
However the client goes offline and comes back only after say 10 hours, so will TCP stack handle re-transmission and delivering message to the client or will the application running on the server need to handle it?
No, not really. Even though TCP has some degree of retry logic for individual TCP packets, it can not perform reconnections if the remote endpoint is disconnected. In other words, it will eventually "time out" waiting to get a TCP ACK from the remote side and do a few retries. But will eventually give up and notify the application through the socket interface that the remote endpoint connection is in a dead or closed state. Typical pattern is that when a client application detects that it lost the socket connection to the server, it either reports an error to the user interface of the application or retries the connection. Either way, it's application level decision on how to handle a failed TCP connection.
is application level ACK needed if transport protocol is TCP
Yes, absolutely. Most client-server protocols has some notion of a request/response pair of messages. A TCP socket can only indicate to the application if data "sent" by the application is successfully queued to the kernel's network stack. It provides no guarantees that the application on top of the socket on the remote end actually "got it" or "processed it". Your protocol on top of TCP should provide some sort of response indication when ever a message is processed. Use HTTP as a good example here. Imagine if an application would send an HTTP POST message to the server, but there was not acknowledgement (e.g. 200 OK) from the server. How would the client know the server processed it?
In a world of Network Address Translators (NATs) and proxy servers, TCP connections that are idle (no data between each other) can fail as the NAT or proxy closes the connection on behalf of the actual endpoint because it perceives a lack of data being sent. The solution is to have some sort of periodic "ping" and "pong" protocol by which the applications can keep the TCP connection alive in the absences of having no data to send.
I am writing a UDP based client server and have got pretty much the code, but I am confused about how UDP server sends the response back to UDP client, this is my understanding till now:
Suppose a UDP client wants to communicate with a UDP server, so it will send a request to the UDP server (using the UDP socket opened at client's end), now this will reach the UDP module at the UDP server, where UDP module will identify the UDP service using the port number and will send that request to that UDP service/server.
Now, since UDP is a connection-less protocol so unlike TCP, UDP server will not send response over some connection, instead, UDP server will extract the source IP address and source port from the request and send the response back to client.
My confusion is that at server side, there is a socket which is bound to a UDP port and "continuously" listening for any UDP client request, but this is not true at client side, UDP client will open a socket to send the request to UDP server and then that's it, I think it cannot keep that port hanging for UDP server to respond, and if that port closes then how client will receive the response back.
I mean ofcourse, UDP server's response will reach back the UDP client because IP address is there, but once that response has reached UDP module of the client, even though there will be a port but how UDP module can send it to the client who originally sent the request because it would have closed the socket bound to that port? Or it will not?
I am looking for answer which clearly describes the UDP communication (I am not interested in contrasting it with TCP or explaining TCP since I have already fair understanding of TCP), especially how the response will reach back the UDP client.
My confusion is that at server side, there is a socket which is bound to a UDP port and "continuously" listening for any UDP client request, but this is not true at client side, UDP client will open a socket to send the request to UDP server and then that's it, I think it cannot keep that port hanging for UDP server to respond, and if that port closes then how client will receive the response back.
I agree. This is your confusion. Why do you think can't keep the socket open and do a receive on it? It can.
I mean ofcourse, UDP server's response will reach back the UDP client because IP address is there, but once that response has reached UDP module of the client, even though there will be a port but how UDP module can send it to the client who originally sent the request because it would have closed the socket bound to that port?
Why?
Or it will not?
Not.
The client:
creates a socket
sends a datagram
calls recvfrom() or friends to receive the response.
Of course if the client isn't interested in the response it can close the socket, but this is not the normal case.
I am looking for answer which clearly describes the UDP communication (I am not interested in contrasting it with TCP or explaining TCP since I have already fair understanding of TCP), especially how the response will reach back the UDP client.
So don't tag your question with the tcp tag.
Yes, UDP server can send back to client. Here is an example:
https://www.cs.rutgers.edu/~pxk/417/notes/sockets/udp.html and code demo https://www.cs.rutgers.edu/~pxk/417/notes/sockets/demo-udp-04.html
We now have a client sending a message to a server. What if the server wants to send a message back to that client? There is no connection so the server cannot just write the response back. Fortunately, the recvfrom call gave us the address of the server. It was placed in remaddr:
recvlen = recvfrom(s, buf, BUFSIZE, 0, (struct sockaddr *)&remaddr, &addrlen);
The server can use that address in sendto and send a message back to the recipient's address.
sendto(s, buf, strlen(buf), 0, (struct sockaddr *)&remaddr, addrlen)
The client uses some random but unique source port. The server sends the response back to this unique port. Server will never receive 2 request at a time from single port. Server uses this fact to map response to request. Only after receiving the response, client closes this source port/ socket. The client can send as much request as the count of ports available for its disposal at any given point of time. Since the port is closed as soon as the response is received, it becomes available again.
Reference: https://www.slashroot.in/how-does-udp-work
Usually we can set up a proxy server by some kind of tools such as CCProxy which provides proxy services for HTTP, SOCKS, FTP packets etc.
Also, Proxifier or Proxycap is used to forward the packets of specific application on the client PC.
However, when UDP packets are forwarded to the proxy server, these packets cannot be correctly forwarded to the destination originally the application wanted them to go.
When I use a network analyzer to observe the UDP traffic flow, the UDP packets are just passed to the proxy server from my PC without going to the correct destination finally.
Besides, someone suggested that the UDP relay is not enabled in the proxy server so the UDP packets cannot be routed successfully. How can I enable the UDP relay function on the proxy server?(Assume that I can control the proxy server completely)
Any kind of proxy, whether it is for TCP or UDP, needs to be told where to forward outgoing packets to. That also allows the proxy to know who is requesting the forwarding so it can route matching inbound packets back to that same requester.
Lets assume SOCKS, for example. SOCKS v4 does not support UDP (or IPv6), but SOCKS v5 does. However, it requires the requesting app to establish a TCP connection to the SOCKS proxy and ask it to forward UDP packets on the app's behalf until that TCP connection is closed.
Tools like CCProxy, Proxycap, Proxifier, etc work (for TCP, anyway) by intercepting outgoing TCP conections and redirecting them to the proxy server, transparently handling any proxy handshaking to set up forwarding, before then allowing any application data to flow through the TCP connection. Once the TCP connection has been established, the proxifier does not need to do anything more with the connection since the app is now talking directly to the proxy.
I do not know if such tools support UDP. It would be much harder to implement, since there is no outgoing connection to redirect. Every outbound UDP packet would have to be intercepted, then the proxifier would have to check if it already has its own SOCKS v5 TCP connection associated with the packet's local/remote tuple and if not then create a new one and send the necessary UDP forwarding handshake, then encapsulate every outbound UDP packet for that tuple and send it to the proxy's outbound IP/Port, and receive every matching inbound UDP packet for that tuple from the proxy so it can be de-encapsulated and forwarded to the app's local IP/Port that sent out the original outbound UDP packet. And because UDP is connection-less, the proxifier would have to also implement a timeout mechanism on its SOCKS v5 TCP connection to the proxy so it can eventually be closed after a period of UDP traffic being idle.
That is a LOT more work for a UDP proxifier to do compared to TCP.
And that is just for SOCKS. HTTP/FTP proxies do not support UDP at all (since HTTP/FTP are TCP-based protocols). And there are other tunnel/proxy protocols as well, which may or may not have their on UDP capabilities.
So you have to check the capabilities of your proxifier tool to see if it supports UDP or not.