I am trying to send Http Requests from Android phone using Apache HttpClient to a server routing my requests via Fiddler. For certain requests, the DefaultHttpClient.execute throws a NoHttpResponseException, but this particular request is not seen in the fiddler at all.
The same thing happens if i direct my traffic without Fiddler but directly over wifi. The code for execute works fine generally.
Fixed this by handling the NoHttpResponseException and re-sending the request. When the same requests gets sent again, it goes through fine.
I would be interested to know the root cause of this issue - but for the time being this works for me.
Related
Fiddler doesn't show outgoing API requests.
Using IFEO debugger shows localhost requests that are not usefull for me, I need actual domains.
Someone told me that client ignores windows proxy and i need application proxy but client has protection against it so there's even more to it.
Has anyone tracked lol/riot's requests before and know how to do that?
We have 2 Windows services (same machine) that communicate on top of HTTP Protocol.
On specific machine we see the HTTP POST being sent from the client (Windows service) and arrives to the server (Windows service listening to REST CALLs) - 2 times, meaning i get 2 exact HTTP Post request on the service, but we see on client it was executed only 1 time.
Before going to wireshark/analyze the HTTP protocol, I wish to understand what explain this behavior.
When going to https://www.rfc-editor.org/rfc/rfc7231#section-4.3.3
"the origin server SHOULD send a 201 (Created) response containing a Location header
field that provides an identifier for the primary resource created"
I guess we should look in wireshark for 201 response? And if no response? Does the HTTP or network framework for my C# application is retrying the POST on the server side? because we dont see 2 requests sent from client code.
POST reply behavior
While true, more often than not the server replies with a 200-ok status code and some extra information.
Whether this is by mistake or to avoid chatty apis or some other architecture/design consideration, only the developer can tell.
So in theory you get a 201 with an identifier and then make a GET request with said identifier to retrieve details.
In practice a lot of times this does not occur. So it is not safe to assume this behavior.
Your problem
I highly doubt that there is a built in mechanism that retries post. There are plenty of reasons for that:
Duplicating entries. Imagine creating a PayPal payment. If the network has an error and you just did not receive the answer, the built in mechanism will charge you twice.
There are libraries that do that only when you are sure that the request is idempotent, that is the post contained some sort of identifier and the second request will fail.
First, the calls are HTTP GET (not POST).
We define the URL with hostname/FQDN, the solution to avoid duplicated calls was to work with ip address instead of hostname when sending the Rest API.
This is the long explanation of the problem, no root cause yet.
Used both Wireshark/Process Monitor to diag, not sure for the root cause.
Process Monitor: Filtering to display network calls
Wireshark: Filter to show only HTTP
The Client send a single HTTP Get request to:
/DLEManagement/API/Engine/RunLearningPeriod
The call was executed at 11:08:16.931906
We can see 2nd call at 11:08:54.511909 - We did not trigger.
HTTP Get executed from *Server.exe (in red) and the Server is at *Management.Webservice.exe (in red).
We see that a *Client.exe (Antivirus process, in blue) is sending TCPCopy packets in the window between we sent and received.
Also, we can see that the first request was made with APIPA IPv6 and the 2nd call is IPv4, We checked the network interface and it was disabled.
Wireshark screenshot:
Process Monitor screenshot:
Network configuration:
We have a Client Toolkit provided by our partner that allows us to access their web services. It started giving errors yesterday on any call and initially their support wanted us to provide a Fiddler log. I tried to do so, however there was no activity shown in Fiddler when the call was made.
From this I would have assumed that the error would have to have occurred before an actual web request was sent out. However, the issue turned out to be an update they did that requires an SSL connection. They rolled back the change but advised us to update our calls to use https so they can re-implement their update.
So if the change was on their end, that means that communications obviously were going on with their server. Why wouldn't that have shown up in Fiddler? Are there scenarios where communications occur but a request isn't fully created or something like that? I just assumed that if there was any communication whatsoever that "something" would show up in Fiddler.
We're getting some HTTP responses with status 200 but with incomplete content (cut off at a different position in each request).
I wonder if there's something wrong with that network or if this kind of response errors must be expected and dealt by our application, i.e. if HTTP clients must check if the response content is complete. I haven't seen this behavior before.
More information:
The issue happens in a place where they have WiFi. If the client, the app, uses the mobile network, the response is always complete. We suspect that there's something wrong with that WiFi because we have not detected this issue in other places (other WiFis). The problem is not caused by the client since I could also reproduce it using Postman.
I made several tests at that place (using the app and also Postman) and found that on mobile data the response always comes OK, but when connecting to that WiFi the response sometimes gets truncated. Also, I saw that other API requests get the response correctly, even when the content length is 10 times bigger (I thought that maybe the response was too big, but we're talking about only 10Kb).
The failing request is a regular HTTP POST request using JSON and sent to our REST API made with Dropwizard and hosted on Heroku. The request gets processed correctly on the server, which returns status 200 and the content. The client gets the status 200 but the content is truncated, so the whole operation can't be finished successfully.
I'm trying to programmatically submit a form on a web site that I do not own. I'm trying to simulate what I would manually do with a web browser. I am issuing an HTTP POST request using an HTTP library.
For a reason that I don't know I am getting a different result (an error, a different response, ...) when I programmatically submit the form compared to a manual submission in a web browser.
How can that be and how can I find out what mistake I have made?
This question is intentionally language and library agnostic. I'm asking for the general procedure for debugging such issues.
All instances of this problem are equivalent. Here is how to resolve all of them:
The web site you are posting to cannot tell different clients apart. It cannot find out whether you are using a web browser or an HTTP library. Therefore, only what you send matters for the decision of the server on how to react.
If you observe different responses from the server this means that you are sending different requests.
A few important things that you probably have to send correctly:
URL
Verb (GET or POST)
Headers: Host, User-Agent, Content-Length
Cookies (the Cookie and Set-Cookie headers)
The request body
Use an HTTP sniffer like Fiddler to capture what you are programmatically sending and what your browser is sending. Compare the requests for differences. Eliminate the differences one by one to see which one caused the problem. You can drag an HTTP request into the Composer Window to be able to modify and reissue it.
It is impossible to still get a different result if you truly have eliminated all differences between the manual and the programmatic requests.