I use a C++ library called restbed as a webserver to deliver static HTML files. On the same machine I have another webserver running, and I would like to redirect some of the incoming connections to restbed. Depending on the request I would make the decision to redirect certain requests to the other server.
Is it technically possible and advised to connect two sockets with each other, if I get access to the underlying socket of the incoming HTTP connection?
If not, what would be a common approach for this? I can only have one TCP port for both services.
Yes, you can respond to an HTTP request by opening a connection to another HTTP server, forwarding the request to that server, and then forwarding the response back to the original client. In fact it's common for Internet-facing systems to include some kind of "front end" or "reverse proxy" or "L7 load balancer" or "API gateway" that does exactly this, often applying some kind of authentication, input validation, or routing logic in the process.
If you're building this yourself, it's not quite as simple as just opening a socket to the second HTTP server and forwarding the request verbatim. You should use some HTTP client library to send the request to the second server. In other words, the HTTP server that receives the original request should then turn around and be an HTTP client for the second server. When preparing the request for the second server, you should copy some but not all of the data out of the original request.
You should copy the HTTP method and URL.
You should probably not copy the scheme (http: or https:) because how the client chose to connect to the original server doesn't have to influence how that server will connect to the second server; you might be using HTTPS for the original server but forward requests using HTTP.
You should not copy the Host header unless for some reason the second server has been configured to respond to the same host name as the original server.
You should not copy headers that will confuse the HTTP client library you're using to connect to the second server. For example, if the client sends you Accept-Encoding: gzip then it is claiming to be able to accept gzipped responses, but if you forward that header, the second server will think that the HTTP client library you're using in your server can accept gzipped responses, whether it actually can or not.
You should forward cache control headers if you want the second server to be able to send 304 Not Modified if the client already has the file.
If you're just serving static files from the second server, then you can probably get something to work just by sending the HTTP method and URL only and ignoring the other request headers.
It's a similar story on the response side. You should probably copy some headers like Content-Type, but others, like Content-Length, will be set by your server, so you should not copy those headers. Try starting out by copying no headers and see if it works, then copy individual headers to address issues you discover. You will probably at least need to copy Content-Type.
HTTP has a lot of features, and I can't hope to go through all the possible situations here. The point I want to get across is that you can't just copy all the headers from one request or response into the other, because some of them may not apply, but you can't just copy none of them either. You have to understand what the headers do and handle them appropriately.
Which headers you should preserve depends a lot on how much handling of the request and response you're doing in the first server. The more the first server handles or interprets the request and/or response, the more its interaction with the second server becomes independent of its interaction with the client, and the fewer headers you should copy.
Related
We have 2 Windows services (same machine) that communicate on top of HTTP Protocol.
On specific machine we see the HTTP POST being sent from the client (Windows service) and arrives to the server (Windows service listening to REST CALLs) - 2 times, meaning i get 2 exact HTTP Post request on the service, but we see on client it was executed only 1 time.
Before going to wireshark/analyze the HTTP protocol, I wish to understand what explain this behavior.
When going to https://www.rfc-editor.org/rfc/rfc7231#section-4.3.3
"the origin server SHOULD send a 201 (Created) response containing a Location header
field that provides an identifier for the primary resource created"
I guess we should look in wireshark for 201 response? And if no response? Does the HTTP or network framework for my C# application is retrying the POST on the server side? because we dont see 2 requests sent from client code.
POST reply behavior
While true, more often than not the server replies with a 200-ok status code and some extra information.
Whether this is by mistake or to avoid chatty apis or some other architecture/design consideration, only the developer can tell.
So in theory you get a 201 with an identifier and then make a GET request with said identifier to retrieve details.
In practice a lot of times this does not occur. So it is not safe to assume this behavior.
Your problem
I highly doubt that there is a built in mechanism that retries post. There are plenty of reasons for that:
Duplicating entries. Imagine creating a PayPal payment. If the network has an error and you just did not receive the answer, the built in mechanism will charge you twice.
There are libraries that do that only when you are sure that the request is idempotent, that is the post contained some sort of identifier and the second request will fail.
First, the calls are HTTP GET (not POST).
We define the URL with hostname/FQDN, the solution to avoid duplicated calls was to work with ip address instead of hostname when sending the Rest API.
This is the long explanation of the problem, no root cause yet.
Used both Wireshark/Process Monitor to diag, not sure for the root cause.
Process Monitor: Filtering to display network calls
Wireshark: Filter to show only HTTP
The Client send a single HTTP Get request to:
/DLEManagement/API/Engine/RunLearningPeriod
The call was executed at 11:08:16.931906
We can see 2nd call at 11:08:54.511909 - We did not trigger.
HTTP Get executed from *Server.exe (in red) and the Server is at *Management.Webservice.exe (in red).
We see that a *Client.exe (Antivirus process, in blue) is sending TCPCopy packets in the window between we sent and received.
Also, we can see that the first request was made with APIPA IPv6 and the 2nd call is IPv4, We checked the network interface and it was disabled.
Wireshark screenshot:
Process Monitor screenshot:
Network configuration:
Due to business limitations we need to serve both HTTP & HTTPS requests on the same port.
As I understand this is not possible at the HttpListener level.
So I'm trying to do this with TcpListener, by reading the first bytes of the request to understand whether it's an HTTP or HTTPS request and handle & respond to it accordingly. In case it's HTTPS I can use the SslStream, however this does not include HTTP layer data.
Since I already received data from the socket, I can no longer use HttpListener.
Also, I prefer not to implement the HTTP layer related code by myself.
Are there any HTTP classes in the .NET framework which can do the parsing of the request & help build the response to send back on the socket?
I came across a challenge that can't seem to find a way to handle. I have a software that runs a camera system. This software is only capable of sending a POST requests. It allows me to type in the host IP, user name, password and a body of a request. I wanted to send an http request to close/open an output on one of the cameras, but from the camera manufacturer API documentation the only way to close an output via http request is to send a request with PUT method. Is there a way to change a method of a request from the body of the request, or from the host IP field? Or maybe there is a standalone software that could act as a proxy to convert that request? Finally if there's no solution out of the box does anybody know if I could set up a lightweight server with which I could accept the POST request, analyze it and send it as a PUT request?
Really appreciate your input.
Thank you.
Well if you ask, there surely is burpsuite which can be used to manipulate your http request packet. Just add the port to proxy option and turn the interceptor on so before the request reaches to api you can manipulate the http packet.
Hope this is what you were looking for.
I'm trying to upload a file onto my personal server.
I've written a small php page that works flawlessy so far.
The little weird thing is the fact that I generate all the body of the HTTP message I'm going to send (let's say that amounts to ~4 mb) and then I send the request to my server.
The server, then, asks for an HTTP challenge and my delegate connection:didReceiveAuthenticationChallenge:challenge replies to the server with the proper credentials and the data.
But, what's happened? The data has been sent twice!
In fact I've noticed that when I added the progressbar.. the apps sends the data (4mb), the server asks for authentication, the apps re-sends the data with the authentication (another 4mb). So, at the end, I've sent 8mb. That's wrong.
I started googling and searching for a solution but I can't figure out how to fix this.
The case scenarios are two (my guess):
Share the realm for the whole session (a minimal HTTP request, then challenge, then data)
Use the synchronized way to perform an HTTP connection (things that I do not want to do since it seems an ugly way to handle this kind of stuff to me)
Thank you
You've run into a flaw into the http protocol: you have to send all the data before getting the response with the auth challenge (when you send a request with no credentials). You can try doing a small round trip as the first request in the same session (as you've mentioned), like a HEAD request, then future requests will share the same nonce.
Too late to answer the original requester, but in time if somebody else read this.
TL;DR: Section 8.2.3 of RFC 2616 describes the 100 Continue status which is all what you need (were needing) in such a situation.
Also have a look at sections 10.1.1 and 14.20.
The client sends a request with an "Expect: 100-continue" header, pausing the request before sending the body. The server uses the already received headers to make its decision whether this request may be accepted or not (if the entity –the body– to be received is not too large, if the user's credentials are correct...). If the request is acceptable for the server, it replies with a "100 Continue" status code, the client sends the body and the server replies with the final status code for that request. To the contrary, if the request is not acceptable, the server replies with a 4xx status code ("413 Request Entity Too Large" if the provided body size is... too large, or a "401 Unauthorized" + the WWW-Authenticate: header) and the client does not send the body. Being answered with a 401 status code and the corresponding WWW-Authenticate: information, the client can now perform the request again and provides its credentials.
I am writing a collection of web services, one of which needs to implement server push.
The client will be native Objective-C. I want this to be as simple, fast, and lightweight as possible. The data transmitted will be JSON. Is it possible to do this without using a message broker?
There's an HTTP technique called COMET in which the client spins up a thread that makes a potentially very long-lived request to the HTTP server. Whenever the server wants to send something to the client, it sends a response to this request. The client processes this response and immediately makes another long-lived request to the server. In this way the server can send information while other things happen in the client's main execution thread(s). The information sent by the serve can be in any format you like. (In fact, for clients in a web browser doing COMET with a Javascript library, JSON is perfect.)
#DevDevDev: It's true that COMET is most often associated with a Javascript-enabled browser, but I don't think it has to be. You might check out iStreamLight, which is an Objective-C client for the iPhone that connects to COMET servers. It's also discussed in this interview with the author.