I recently use mitmproxy and I would like to intercept request having a body, like the node-http-mitm-proxy does (using the two events proxy.onRequestData(fn) and proxy.onRequestEnd(fn)).
Would it be possible with the mitmproxy to intercept the requests data chunk (the body) and eventually to have an event fired when request data (the body) is sent ?
I see that there is the possibility of intercepting the requests, but I do not see how we can intercept each request data chunk forming the body.
Any suggestions?
Related
What is a good way to save / write / log (request, response) pairs on both ends (client-facing, server-facing) of mitmproxy? That is, all 4 of:
c->p: original request from the client
p->c: final response to the client, possibly rewritten by the proxy
p->s: request forwarded to the server and possibly rewritten by the proxy
s->p: original response from the server
c->p p->s
original forwarded rewritten
request request
---------------------> -------------->
client proxy server
<--------------------- <--------------
p->c s->p
forwarded rewritten original
response response
As far as I can see, the -w flag (of mitmdump) only saves single (request, response) pairs, so either the original or the forwarded request / response are not saved (and possibly neither of those, only some intermediate stage, before / after other addons that modify request / response).
Thanks for help.
I am looking into the Swift Vapor framework.
I am trying to create a controller class that maps data obtained on an SSL link to a third party system (an Asterisk PBX server..) into a response body that is sent over some time down to the client.
So I need to send received text lines (obtained separately on the SSL connection) as they get in, without waiting for a 'complete response' to be constructed.
Seeing this example:
return Response(status: .ok) { chunker in
for name in ["joe\n", "pam\n", "cheryl\n"] {
sleep(1)
try chunker.send(name)
}
try chunker.close()
}
I thought it might be the way to go.
But what I see connecting to the Vapor server is that the REST call waits for the loop to complete, before the three lines are received as result.
How can I obtain to have try chunker.send(name) send it's characters back the client without first waiting for the loop to complete?
In the real code the controller method can potentially keep an HTTP connection to the client open for a long time, sending Asterisk activity data to the client as soon as it is obtained. So each .send(name) should actually pass immediately data to the client, not waiting for the final .close() call.
Adding a try chunker.flush() did not produce any better result..
HTTP requests aren't really designed to work like that. Different browsers and clients will function differently depending on their implementations.
For instance, if you connect with telnet to the chunker example you pasted, you will see the data is sent every second. But Safari on the other hand will wait for the entire response before displaying.
If you want to send chunked data like this reliably, you should use a protocol like WebSockets that is designed for it.
I am implementing REST services using Apache-CXF running on servicemix and for that I have a camel route that does some processing, sends the message over queue, process some more and send back the reply. Something like this:
from("direct:start")
.process(A)
.process("activemq:abc")
.process(B);
On this route I have applied some basic validation and exception handler and when I have to stop the route in both cases, I use something like this:
exchange.getOut().setBody(response);
exchange.setProperty(Exchange.ROUTE_STOP, Boolean.TRUE);
I use soap UI, restclient-UI and putty to make http requests and I get proper response body displayed in all of them. Now I wanted to preserve request headers so I made a little change everywhere in the code so that response bodies are set in exchange.getIn() only. For example: in case of validation failure I do:
exchange.getIn().setBody(response);
exchange.setProperty(Exchange.ROUTE_STOP, Boolean.TRUE);
Just with this little change, the rest clients I am using to make request stopped displaying the response body. As per the server logs, response is being generated and also as per the logs in rest client, I am getting the proper response but they are unable to display the response body only in case when I stop the route in between. Normal response is displaying just fine. Only the restclient-UI was considerate enough to show the error as to why they are not displaying body and the error is:
Byte array conversion from response body stream failed.
Diggin deeper, I found the only response header which was there in success response but missing in error response:
Transfer-Encoding chunked
Error response is around 1000 characters long and contains a header called content-length. I am not sure but I think the problem has something to do with this itself. I would really like to play with exchange.getIn but these different kind of responses prepared by camel are confusing me. How can I make sure my camel responses are always displayed properly?
The Content-Length header will be preserved from the original request so you need to remove it so that camel cxf can work out the new body length on the response and set Content-Length with that.
My problem is as follows: I have a TCP socket on which I asynchronously send messages based on a proprietary protocol. Upon receiving a response for a message I need to make sure that data gets sent back to the function that initially sent the message. The problem is I can only listen for responses on a "data" event on the socket (in this case in Node.js) and I have no way of easily determining which responses correspond to which messages.
Below is an example that I hope will give you an idea of what I am trying to accomplish.
function getTemperature (callback) {
socket.write(message);
// Good?
socket.on('data', callback);
}
function getVoltage (callback) {
socket.write(message);
// Other way to invoke callback?
}
getTemperature(function (err, temperature) {});
getVoltage(function (err, voltage) {});
socket.on('data', handleData function (data) {
// Somehow send data to correct function
}
One idea I had was registering multiple event handlers for the data event but this seems kind of wacky. What is the correct way to handle this?
When dealing with an asynchronous protocol, you would typically need to design the protocol so that each request has a unique ID that is included in the request data and gets sent back in the response. That way, you can receive responses out of order, and even receive unsolicited messages.
When you send a request, put the request ID in a list somewhere, along with a callback that will be called when the response arrives.
When you get a data event, append the raw data to a buffer, and then parse out only complete messages from the buffer, leaving incomplete messages for later completion. For each complete message parsed out, check if it has a request ID. If no, the message is unsolicited, so process it accordingly. Otherwise, look for the request ID in the list, and if found then call the callback associated with it.
I want to pass some data within request body, but I'm using GET request, because I just want to modify this data and send it back.
I know that it is bad practice to use body with GET requests.
But what should I do with this situation if I want to build correct RESTful service?
P.S. I'm not changin any object on server.
I'm not putting any new object on server.
You want a POST. Something like
POST /hashes
{
"myInput": ...
}
The response would be the hashed value. There's no rule that the created resource must be retained by the server.
From the RFC:
The action performed by the POST method might not result in a
resource that can be identified by a URI. In this case, either 200
(OK) or 204 (No Content) is the appropriate response status,
depending on whether or not the response includes an entity that
describes the result.