What is a good way to save / write / log (request, response) pairs on both ends (client-facing, server-facing) of mitmproxy? That is, all 4 of:
c->p: original request from the client
p->c: final response to the client, possibly rewritten by the proxy
p->s: request forwarded to the server and possibly rewritten by the proxy
s->p: original response from the server
c->p p->s
original forwarded rewritten
request request
---------------------> -------------->
client proxy server
<--------------------- <--------------
p->c s->p
forwarded rewritten original
response response
As far as I can see, the -w flag (of mitmdump) only saves single (request, response) pairs, so either the original or the forwarded request / response are not saved (and possibly neither of those, only some intermediate stage, before / after other addons that modify request / response).
Thanks for help.
Related
I launched my app on heroku.
And ever since all of my axios get request return a blank html.
axios.post(process.env.REACT_APP_baseServerurl + '/create/get-users')
axios.get(process.env.REACT_APP_baseServerurl + '/create/get-users')
the response i get is always blank.
request
response
but if i change the same request to a post it works fine.
The get request works fine when i try it on my localhost. But when i deploy it on heroku it returns blank.
router.post('/get-users', (req,res) => {
res.json("asdsadas")
})
router.route('/get-users').get((req,res) => {
res.json("yes")
})
the POST request works but GET request returns a 304.
The 304 HTTP Status code means that the resource has not been modified since the last get request. This indicates that the server thinks you already have a copy of up-to-date data. The Server makes this assumption by looking at the If-None-Match and If-Modified-Since in your request header. (https://en.wikipedia.org/wiki/List_of_HTTP_status_codes#3xx_redirection)
These are the actions you could take:
Cache the data manually on your first successful request (if there is one, otherwise there must be an error in the usage of the If-None-Match and If-Modified-Since headers) and re-use the cached data if the server returns a 304 status code.
Disable the mechanism and live with the fact that you might request the same data the server already gave you.
Usually when I send a response to the browser I have to enter content-length in the http headers, otherwise the browser never stops loading (wait for more data)
But recently, I tested rust code:
let response = format!("HTTP/1.1 200 OK\r\n\r\n{}", contents);
stream.write(response.as_bytes()).unwrap();
The browser receives this without any problems, stops loading after receiving the response.(even though content-length is not specified in the response)
Can someone pls explain this?... What makes the browser satisfied with the response in this scenario (even though it does not contain: Content-length)
Content-length is optional as long as the connection is closed after the response is done. From RFC 7230 section 3.3.3 Message Body Length:
Otherwise, this is a response message without a declared message
body length, so the message body length is determined by the
number of octets received prior to the server closing the
connection.
I have an angular client which is making a POST call to my server. This server needs to get a response by calling another server(server2) with a POST call and pass the response from the server2 to the client. I tried the following approaches.
public Response call(){
String server2Url = "http://server2/path"
RestClient restClient = new RestClient();
return Response.fromResponse(restClient.post(server2Url)).build();
}
But in the above case the HTTP status code gets transferred but not the response body. The response body is empty
Then I tried:
public Response call() throws URISyntaxException{
String server2Url = "http://server2/path"
RestClient restClient = new RestClient();
return Response.temporaryRedirect(new URI(server2Url)).build();
}
but the browser client ends up making an OPTIONS call to the server2Url instead of a POST
and I tried.
public Response call() throws URISyntaxException{
String server2Url = "http://server2/path"
RestClient restClient = new RestClient();
return Response.seeOther(new URI(server2Url)).build();
}
but this ends up making a GET call instead of a POST.
How do I make the browser client make a POST call to server2
You can use Html Client from JAX-RS to make your own requests (from server1 to server2) and then return the response from server2 to the angular client.
public Response call() {
String url = "server2 url";
Response response;
try {
response = ClientBuilder
.newClient()
.target(url)
.request()
.post(Entity.json(null), Response.class);
}
catch (Exception e) {
// Whatever you want
return null; // or error
}
// Return the status returned by server 2
return Response.status(response.getStatus()).build();
}
What you are trying to accomplish is covered in the RFC 2616 I just found here.
If the 302 status code is received in response to a request other than GET or HEAD, the user agent MUST NOT automatically redirect the request unless it can be confirmed by the user, since this might change the conditions under which the request was issued.
So it looks like this is out of your hands if you´re not implementing the client.
Edit because I was told that RFC 2616 must not be used any longer.
RFC 7231 states that:
302 Found
The 302 (Found) status code indicates that the target resource
resides temporarily under a different URI. Since the redirection
might be altered on occasion, the client ought to continue to use the
effective request URI for future requests.
The server SHOULD generate a Location header field in the response
containing a URI reference for the different URI. The user agent MAY
use the Location field value for automatic redirection. The server's
response payload usually contains a short hypertext note with a
hyperlink to the different URI(s).
Note: For historical reasons, a user agent MAY change the request
method from POST to GET for the subsequent request. If this
behavior is undesired, the 307 (Temporary Redirect) status code
can be used instead.
What is:
307 Temporary Redirect
The 307 (Temporary Redirect) status code indicates that the target
resource resides temporarily under a different URI and the user agent
MUST NOT change the request method if it performs an automatic
redirection to that URI. Since the redirection can change over time,
the client ought to continue using the original effective request URI
for future requests.
The server SHOULD generate a Location header field in the response
containing a URI reference for the different URI. The user agent MAY
use the Location field value for automatic redirection. The server's
response payload usually contains a short hypertext note with a
hyperlink to the different URI(s).
Note: This status code is similar to 302 (Found), except that it
does not allow changing the request method from POST to GET. This
specification defines no equivalent counterpart for 301 (Moved
Permanently) ([RFC7238], however, defines the status code 308
(Permanent Redirect) for this purpose).
We are implementing Camel routes to handle service requests.
We have the requirement to deliver a huge response to a certain request. This response can be that large that it may cause memory problems.
For now we write the response in a file, move that file to an external ftp server and return the name and location of the file as the http response.
What I really like to do is to stream the response via the http response in Camel. Unfortunately I could not find any doc/samples on how to do that.
This is our current implementation:
.rest(BASE_URL_VERSION)
.get("/generic/xxxxxx").produces(MIME_MEDIA_TYPE_JSON)
.route()
.process(new ResponseBuilderProcessor()) // this creates the temp file
.to(uriOut) // this moves the file to the external ftp
.process(new AnswerProcessor()); // provides the response where the file location is mentioned
Is this doable? How?
Consider making your service async
If the request is proper make the request as accepted(http 202) & give the client the details of the reponse availability (may be in a JMS queue or the file location as you mentioned) . The queue ID or file location can be send as response .
The client will hit the Queue or file location after a time interval & get the response if it is available .
When I send GET http requests to an EJB served by jetty, I often get a 401 response even though the auth parameters are correct.
When I look into jetty logs I see this :
2013-06-27 11:54:11.004:DBUG:oejs.Server:REQUEST /app/general/launch on AsyncHttpConnection#3adf0ddc,g=HttpGenerator{s=0,h=-1,b=-1,c=-1},p=HttpParser{s=-5,l=34,c=0},r=1
2013-06-27 11:54:11.021:DBUG:oejs.Server:RESPONSE /app/general/launch 401
2013-06-27 11:54:11.066:DBUG:oejs.Server:REQUEST /app/general/launch on AsyncHttpConnection#3adf0ddc,g=HttpGenerator{s=0,h=-1,b=-1,c=-1},p=HttpParser{s=-5,l=102,c=0},r=2
I suspect that the request is not fully read (too large request entity or too large headers?)
as it is parsed twice for a single request. Is there a way to fix this ?
what does HttpParser{s=-5,l=34,c=0} and HttpParser{s=-5,l=102,c=0} mean ?
when I desactivate authentication (security constraints using simple jetty realm). the request is only parsed once.
401 means that the server requires authentication credentials that the client either has not sent or the ones sent by the client have not been authorized.
Some client implementations will resend the request if they receive a 401 including the credentials. If your client is doing that, that would explain why you get the request twice on the server.
The HttpParser toString() method returns the current status of the HttpParser. Here's the code:
return String.format("%s{s=%d,l=%d,c=%d}",
getClass().getSimpleName(),
_state,
_length,
_contentLength);
So s is the state. -5 is STATE_HEADER. And l and c represent the length and the contentLength.