What is a good way to save / write / log (request, response) pairs on both ends (client-facing, server-facing) of mitmproxy? That is, all 4 of:
c->p: original request from the client
p->c: final response to the client, possibly rewritten by the proxy
p->s: request forwarded to the server and possibly rewritten by the proxy
s->p: original response from the server
c->p p->s
original forwarded rewritten
request request
---------------------> -------------->
client proxy server
<--------------------- <--------------
p->c s->p
forwarded rewritten original
response response
As far as I can see, the -w flag (of mitmdump) only saves single (request, response) pairs, so either the original or the forwarded request / response are not saved (and possibly neither of those, only some intermediate stage, before / after other addons that modify request / response).
Thanks for help.
I am using feign to call an endpoint that returns a large binary file, and then I'm using this to get the InputStream:
Response.Body.asInputStream();
Will feign return the Response before the entire file has downloaded, so that I can stream it as it comes in? Or does it wait to return until the entire file is downloaded from the Rest endpoint?
The fact that it provides an InputStream implies to me that it will return before the whole file has downloaded, but I can't find documentation on that.
I'm writing a C# Web API server application, and will send JSON to it via a Mirth HTTP Sender destination. This post is about how to handle error conditions. Specifically, there are three scenarios I want to handle:
Sometimes we take the C# application server offline for a short period for system upgrade or maintenance, and Mirth is unable to connect at all. I want Mirth to queue all messages in order, and when the server is available, process them in the order they were received.
The server receives the request, but rejects it due to a problem with the content of the request, e.g., missing a required field. In accordance with REST conventions, the server will return a 400-level HTTP response. This message would be rejected every time it's submitted, so it should not be re-sent; just log the failure and move on to the next message.
The server receives the request, but something goes wrong on the server, and the server returns an HTTP 500 Server Error response. This would be the appropriate response, for example, when something in the server environment has gone wrong. One real-world example was the time the Web API server was running, but somebody rebooted the database server. REST conventions would suggest we continue to resend the message until the transient problem has been resolved.
For #1, initially I had it queue on failure/always, but it appears the response transformer never runs for messages that were queued (at least, the debug statements never showed in the log). I have turned queueing off, and set it to retry every ten seconds for an hour, and that seems to give the desired behavior. Am I on the right track here, or missing something?
For #2 and #3, returning any HTTP 400 or 500 error invokes the 1-hour retries. What I want is to apply the 1-hour retries for the 500 errors, but not the 400 errors. I’ve tried responseStatus = SENT in the response transformer, but the response transformer only runs once, after the hour has expired, and not for each retry.
This seems like a common problem, yet I’m not finding a solution. How are the rest of you handling this?
You're close!
So by default, the response transformer will only run if there's a response payload to transform. For connection problems, or possibly for 4xx/5xx responses that contain no payload, the response transformer won't execute.
However, if you set your response data types (From the Summary -> Set Data Types dialog, or from the Destinations -> Edit Response, Message Templates tab) to Raw, then the response transformer will execute all the time. The reason being that the Raw data type considers even an empty payload to be "transformable".
So turn queuing back on, and set your response data types to Raw. Then in the response transformer, if you look at the Reference tab there's a category for HTTP Sender:
You'll want the "response status line", that's the "HTTP/1.1 200 OK" line of the response that contains the response code. Here's a response transformer script that forces 4xx responses to error:
if (responseStatus == QUEUED) {
var statusLine = $('responseStatusLine');
if (statusLine) {
var parts = statusLine.split(' ');
if (parts.length >= 2) {
var responseCode = parseInt(parts[1], 10);
// Force 4xx responses to error
if (responseCode >= 400 && responseCode < 500) {
responseStatus = ERROR;
responseStatusMessage = statusLine;
}
}
}
}
I am implementing REST services using Apache-CXF running on servicemix and for that I have a camel route that does some processing, sends the message over queue, process some more and send back the reply. Something like this:
from("direct:start")
.process(A)
.process("activemq:abc")
.process(B);
On this route I have applied some basic validation and exception handler and when I have to stop the route in both cases, I use something like this:
exchange.getOut().setBody(response);
exchange.setProperty(Exchange.ROUTE_STOP, Boolean.TRUE);
I use soap UI, restclient-UI and putty to make http requests and I get proper response body displayed in all of them. Now I wanted to preserve request headers so I made a little change everywhere in the code so that response bodies are set in exchange.getIn() only. For example: in case of validation failure I do:
exchange.getIn().setBody(response);
exchange.setProperty(Exchange.ROUTE_STOP, Boolean.TRUE);
Just with this little change, the rest clients I am using to make request stopped displaying the response body. As per the server logs, response is being generated and also as per the logs in rest client, I am getting the proper response but they are unable to display the response body only in case when I stop the route in between. Normal response is displaying just fine. Only the restclient-UI was considerate enough to show the error as to why they are not displaying body and the error is:
Byte array conversion from response body stream failed.
Diggin deeper, I found the only response header which was there in success response but missing in error response:
Transfer-Encoding chunked
Error response is around 1000 characters long and contains a header called content-length. I am not sure but I think the problem has something to do with this itself. I would really like to play with exchange.getIn but these different kind of responses prepared by camel are confusing me. How can I make sure my camel responses are always displayed properly?
The Content-Length header will be preserved from the original request so you need to remove it so that camel cxf can work out the new body length on the response and set Content-Length with that.
Is there any code samples to show how to handle trunked response messages? (http://spray.io/documentation/1.2.1/spray-can/http-client/#chunked-responses)
For example, I may use spray-can's http-client to access a service will response a big file in trucked way.
And I'd like to handle the trucked messages accordingly to save the big file in my local disk.
Thanks
Leon