find out how an LWP request was sent "over the wire" - perl

If I send a request with LWP, there is a convenience function as_string which tells me what request I just sent. Very handy, and in truth I have no problems with it. Except that I just noticed that it is surely lying to me. For instance, this code:
use v5.14.2;
use LWP;
my $response = LWP::UserAgent->new->get('http://user:pswd#example.com/');
say $response->request->as_string;
Gives this output
GET http://user:pswd#example.com/
User-Agent: libwww-perl/6.13
But surely the URL was not sent like that! The library must have parsed out the username and password, and added the appropriate headers, and added a host header, and so on. Is there an easy way to find out what was actually sent?

There's LWP::ConsoleLogger::Everywhere1, which you can just load to get all the details of both the request as well as the response. The request will be taken right before it's sent over the wire, and the response from when it comes back.
All you need to do is use LWP::ConsoleLogger::Everywhere anywhere in your code. If you want more control, the main module LWP::ConsoleLogger in that distribution will let you tweak settings easily.
However, this is not the real data that goes over the wire. If you want to know what it receives, you need to either monitor the connection with something like tcpdump and then take a look at it (which is quite advanced networking stuff), or maybe change the endpoint to your own IP address, or simply 127.0.0.1, and then use netcat to listen on a specific port.
$ nc -l 8080
If you send your request to that port, you'll see it in netcat.
1) Disclaimer: I'm a contributor for that module

Related

API Client testing tool for capturing requests

I need a tool which gives me a URL to make a HTTP request that will be recorded and it shows me what body was sent, which headers, which parameters, the method...
It's like something kind of opposite of Postman.
I tried to find such type of tool/service but didn't find any. If someone knows something similar, please let me know, thank you.
Actually, Postman has a feature which captures the request:
Enable it and just make the call to the set port (localhost:5555 by default) and it will be saved under the request history.

Kamailio - Dispatcher determine availability via http?

We are currently using the dispatcher module in kamailio to get the availability of gateways via the dispatch list.
It uses a health check based on if it can talk to the gateway via SIP by default. However, I would like to know if we can make the check better by also checking via a http health check?
The reason for this is because when the gateway on the other end is in courtesy shutdown the dispatcher still sends calls to it even though we would like the box to shutdown. This leads to the gateway always staying up.
Alternatively there might be a better way of handling this by sending a message back in the sip packet to kamailio.
I have read through the documentation but I can't seem to find anything like what I am looking for.
The Dispatcher module has Event Routes that can be called when a SIP destination goes down / up. There are no Event Routes for HTTP as it's not constantly queried in it's own thread by Dispatcher.
Alternatively there might be a better way of handling this by sending a message back in the sip packet to kamailio.
You can however set the dispatcher state using the ds_mark_dst([state]) function. Through this you could add a custom header in any SIP message from your box that's shutting down to tell Kamailio's Dispatcher to not use it as a destination in the future.
If we added an imaginary header called "X-SetState" with the value "Shutdown" and send it from our box that's shutting down to Kamailio in a SIP message we could pick it up with something like this:
is_present_hf("X-SetState"){ //If custom header is present
xlog("Received state change request ($ru) with value $hdr(X-SetState)")
if($hdr(X-SetState) == "Shutdown"){ //If value of header is Shutdown
ds_mark_dst("dp"); //Mark destatination as disabled & probing
}
}
Obviously you'd need to add your own code to select the right dispatcher to mark inactive and ensure that the X-SetState header was only parsed if it came from your internal boxes you want to mark as down but you get the idea.
However, I would like to know if we can make the check better by also checking via a http health check?
Dispatcher at the moment has no support for monitoring HTTP state, but adding it wouldn't be that difficult to implement, if you're handy at C you could add support or add a feature request.
Alternatively you could write an script to monitor HTTP status of each device and the using Kamcmd / Kamctl set the dispatcher group to down if it doesn't get a response.

subdomain redirect to a specific port using SRV?

Lets say I have the following:
subdomain: xyz.mydomain.com
my server's public DNS: xyz.fastserver.com
when someone goes to xyz.mydomain.com I want them to be redirected to
xyz.mydomain.com:8080
I have full access to all the typical A(host), C(NAME) as well as SRV records etc, tried different configurations but cant get it to work.
Any ideas?
You did not explicitly specify it, but I assume you mean HTTP (i.e. web browsing) and not FTP, SIP, SMTP... and lots of the other protocols on the internet.
In this case what you are trying to do is not possible. DNS A/AAAA/CNAME records are only used to get an IP address, so you can not get a port with these settings. And SRV records are not used within HTTP, so you can not use it to specify the port too.
Link to previous post that goes over the difference between redirect, rewrite, and vhosts.
DNS, however, has no concept of "port" unless you make a special record (SRV) and then a special request to get that record. It is much more transparent to use one of the HTTP methods described above.

POST from WinForms app using HttpWebRequest to webservice doesn't work when sent through Fiddler

I'm using HttpWebRequest in a VB.Net WinForms app to get data from an inhouse webservice. The code I'm using works for both GET and POST when run while Fiddler is not running. If I have Fiddler running the GETs work and are captured but a POST doesn't complete. In this case Fiddler captures the initial request but never gets the response and the application doesn't get a response.
The code builds a HttpWebRequest for the POST setting the appropriate properties, encodes the data to be sent into JSON and then does this.
Using postStream As Stream = webrequestobj.GetRequestStream()
postStream.Write(WebServiceByteData, 0, WebServiceByteData.Length)
End Using
I used WireShark to capture the generated network packets and noticed that when a POST is sent without going through Fiddler the following happens.
When "postStream As Stream = webrequestobj.GetRequestStream()" is executed a packet with all of the header info is sent that includes a "Expect: 100-continue" header but doesn't have the request data.
When the postStrean.Write call is executed an additional packet is sent that has the request data.
With Fiddler running nothing is put on the wire until after the postStream.Write is executed. At that point both the header packet with the "Expect: 100-continue" header and the request data packet are sent back to back before the service has responded with the "100 Continue". I'm guessing that this confuses the webservice as it doesn't expect to get the request data packet yet. It doesn't respond with the requested data.
I used Composer to manually create the request without the "Expect: 100-continue" header. When this is executed the same two packets are generated and the service responds with the expected data.
So, in order to be able to use Fiddler to capture the POST traffic it looks like I need to either be able to tell HttpWebRequest to not issue the "Expect: 100-continue" header (I've looked but haven't found a way to do this) or for Fiddler to handle the packets differently, maybe not sending the second packet until it sees the "100 Continue" response or by stripping out the "Expect: 100-continue" header.
It's possible that I've missed a setup option in Fiddler but nothing I've tried so far makes any difference.
Thanks,
Dave
Old question, but the short answer is that the lack of a 100/Continue response shouldn't have mattered at all.
To learn more about Expect: Continue, including how to remove this header if you like, see http://blogs.msdn.com/b/fiddler/archive/2011/11/05/http-expect-continue-delays-transmitting-post-bodies-by-up-to-350-milliseconds.aspx

REST performing an action

I'm designing a REST API where I need to give the clients the ability to copy resources.
Say for example you have the following resource: customer/{customerno}/address
And you will provide the client with functionallity for copying the address to shippingaddress.
I could do like this: customer/{customerno}/address/copytoshipping
But then it would be RPC and not REST. What is the correct way to do something like this?
Does that have to be in the REST API? The client could just GET the representation of the (billing?) address and PUT it to the shipping address. Yes, you might well hide the details of that in your client-side code (that seems very reasonable to me) but there's no particular reason to clutter the REST interface with special operations to do the copying.
GET the address using Rel="CustomerAddress"
POST using Rel="CustomerShippingAddress" example URL customer/{customerno}/address/shipping
A different approach could be:
Expose a resource for "customer address" (as you stated above)
GET /customer/{number}/address
Expose a resource for "customer shipping address"
fetch:
GET /customer/{number}/shipping-address
update with actual address:
PUT /customer/{number}/shipping-address
Content-Type: application/xml
(xml/json of a single address)
update using a URI pointer to another address
PUT /customer/number/shipping-address HTTP/1.1
Content-Type: application/atom+xml
(xml/json of an atom:link with href pointer to main address ('/customer/123/address')
Perhaps more complex than you want but puts the copy burden on the server rather than the client.