I am a newbie in Perl and in ActiveMQ.
I have downloaded this Perl nagios program to check the ActiveMQ queues. The problem is the program exist in the main Perl line:
my $page = get "http://admin:admin\#$address:$port/admin/xml/queues.jsp" or die "Cannot get XML file: $!\n";;
I substituted that line with this other lines to check the return code:
my $ua = LWP::UserAgent->new;
$ua->timeout(10);
$ua->env_proxy;
my $page = $ua->get("http://admin:admin\#$address:$port/admin/xml/queues.jsp");
if ($page->is_success) {
print $page->decoded_content; # or whatever
}
else {
die $page->status_line;
}
Now, it reports:
401 Unauthorized
But wget is still able to download the page:
Connecting to 127.0.0.1:8161... connected.
HTTP request sent, awaiting response... 401 Unauthorized
Reusing existing connection to 127.0.0.1:8161.
HTTP request sent, awaiting response... 200 OK
Length: 2430 (2.4K) [text/xml]
Saving to: `queues.jsp'
How can I config the UserAgent to make the get call imitate the wget behaviour?
Do you know another script/program to monitor de ActiveMQ queues?
Is there any way to get the queues values in plain text? Then I would write down my own bash script.
update 1
As #mob requested, here it is the output of wget --debug
DEBUG output created by Wget 1.12 on linux-gnu.
--2017-09-06 19:27:15-- http://admin:*password*#127.0.0.1:8161/admin/xml/queues.jsp
Connecting to 127.0.0.1:8161... connected.
Created socket 3.
Releasing 0x0000000002586c10 (new refcount 0).
Deleting unused 0x0000000002586c10.
---request begin---
GET /admin/xml/queues.jsp HTTP/1.0
User-Agent: Wget/1.12 (linux-gnu)
Accept: */*
Host: 127.0.0.1:8161
Connection: Keep-Alive
---request end---
HTTP request sent, awaiting response...
---response begin---
HTTP/1.1 401 Unauthorized
WWW-Authenticate: basic realm="ActiveMQRealm"
Cache-Control: must-revalidate,no-cache,no-store
Content-Type: text/html;charset=ISO-8859-1
Content-Length: 1293
Connection: keep-alive
Server: Jetty(7.6.9.v20130131)
---response end---
401 Unauthorized
Registered socket 3 for persistent reuse.
Skipping 1293 bytes of body: [<html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
<title>Error 401 Unauthorized</title>
</head>
<body>
<h2>HTTP ERROR: 401</h2>
<p>Problem accessing /admin/xml/queues.jsp. Reason:
<pre> Unauthorized</pre></p>
<hr /><i><small>Powered by Jetty://</small></i>
</body>
</html>
] done.
Reusing existing connection to 127.0.0.1:8161.
Reusing fd 3.
---request begin---
GET /admin/xml/queues.jsp HTTP/1.0
User-Agent: Wget/1.12 (linux-gnu)
Accept: */*
Host: 127.0.0.1:8161
Connection: Keep-Alive
Authorization: Basic xxxxxxxxxxxxxxxxxxxx
---request end---
HTTP request sent, awaiting response...
---response begin---
HTTP/1.1 200 OK
Set-Cookie: JSESSIONID=o7kaw1kbzcy91dozx82c8dq2j;Path=/admin
Expires: Thu, 01 Jan 1970 00:00:00 GMT
Content-Type: text/xml;charset=ISO-8859-1
Content-Length: 2430
Connection: keep-alive
Server: Jetty(7.6.9.v20130131)
---response end---
200 OK
Stored cookie 127.0.0.1 8161 /admin <session> <insecure> [expiry none] JSESSIONID o7kaw1kbzcy91dozx82c8dq2j
Length: 2430 (2.4K) [text/xml]
Saving to: `queues.jsp'
100%[================================================================================>] 2,430 --.-K/s in 0s
2017-09-06 19:27:15 (395 MB/s) - `queues.jsp' saved [2430/2430]
The only difference in the ---request begin--- sections of both attempts is
Authorization: Basic xxxxxxxxxxxxxxxxxxxx
found only in the second try.
LWP::UserAgent does not parse the username:password part of the URL. I suspect this is to discourage the insecure practice of putting the username:password in the URL where it will then be easily stolen from programs and in server logs.
You can override get_basic_credentials to pull the username and password out of the URL. This doesn't solve the security problem.
Or you can call authorization_basic on an HTTP::Request object to set the username and password for this particular request.
my $ua = LWP::UserAgent->new;
my $req = HTTP::Request->new(GET => $url);
$req->authorization_basic($user, $password);
my $res = $ua->request($req);
Or you can call credentials on the UserAgent to set up passwords for various host/port combinations rather than for just one request. This is like storing your passwords in the browser, where $ua is the browser.
my $ua = LWP::UserAgent->new;
$ua->credentials("$host:$port", $realm, $user, $password);
my $req = $ua->get($url);
Or you can switch to the less featureful but better designed HTTP::Tiny or the slightly more featureful HTTP::Tiny::UA. It will parse and use the username:password part of the URL.
use HTTP::Tiny;
my $ua = HTTP::Tiny->new;
my $res = $ua->get($url);
Related
I'm sending GET requests via proxy that only supports HTTP (no HTTPS support). When I use that proxy (or any other http-only proxy) to request HTTPS it returns 403 when I use curl (and it seems to be the correct status). But if I just use CONNECT and GET I get 200.
Curl - 403 Forbidden:
curl -x proxyhost:proxyport -I https://example.com -vvv
* Trying PROXYHOST:8080...
* TCP_NODELAY set
* Connected to PROXYHOST (PROXYHOST) port 8080 (#0)
* allocate connect buffer!
* Establish HTTP proxy tunnel to www.example.com:443
> CONNECT www.example.com:443 HTTP/1.1
> Host: www.example.com:443
> User-Agent: curl/7.68.0
> Proxy-Connection: Keep-Alive
>
< HTTP/1.1 403 Forbidden
HTTP/1.1 403 Forbidden
< Date: Fri, 15 Oct 2021 15:37:31 GMT
Date: Fri, 15 Oct 2021 15:37:31 GMT
< Server: Apache
Server: Apache
< Content-Length: 202
Content-Length: 202
< Content-Type: text/html; charset=iso-8859-1
Content-Type: text/html; charset=iso-8859-1
<
* Received HTTP code 403 from proxy after CONNECT
* CONNECT phase completed!
* Closing connection 0
curl: (56) Received HTTP code 403 from proxy after CONNECT
Pure HTTP - 200 OK:
CONNECT PROXYHOST:PROXYPORT HTTP/1.0
GET https://www.example.com:443 HTTP/1.0
HTTP/1.0 200 OK
Why do I get 200 with CONNECT + GET?
Additional info:
PROXYHOST is just a random host without any proxy setting or software. It just happens so that when you use it as proxy for http GET requests it returns correct status (200 if requested page exists, 404 if it does not, etc) and its own html instead of requested body. At the same time it always returns 403 if you try to use it as proxy for requesting https via anything but CONNECT + GET.
I also tried python requests and got this result (with verbose logs):
proxy = {'https': 'http://PROXYHOST:8080', 'http': 'http://PROXYHOST:8080'}
requests.get('https://example.com', proxies=proxy)
# logs
send: b'CONNECT example.com:443 HTTP/1.0\r\n'
send: b'\r\n'
# exception
File "/usr/lib/python3.8/http/client.py", line 276, in _read_status
raise RemoteDisconnected("Remote end closed connection without"
http.client.RemoteDisconnected: Remote end closed connection without response / (Caused by ProxyError('Cannot connect to proxy.', RemoteDisconnected('Remote end closed connection without response')))
requests.get('http://example.com', proxies=proxy)
# logs
send: b'GET http://example.com/ HTTP/1.1\r\nHost: example.com\r\nUser-Agent: python-requests/2.25.0\r\nAccept-Encoding: gzip, deflate\r\nAccept: */*\r\nConnection: keep-alive\r\n\r\n'
reply: 'HTTP/1.1 200 OK\r\n'
header: Date: Fri, 15 Oct 2021 17:10:00 GMT
header: ...
<Response [200]>
I've got a simple class making a HTTP POST request (to a Neo4j server, but that shouldn't matter) using URLSession, and an integration test that tests that it succeeds.
Through Xcode, via the iOS simulator, I can run this test a million times - it succeeds every time.
However, when I run swift test on Ubuntu Linux (I use the docker image provided by IBM) then I very often will get a test failure saying I got a 401 response.
Replicating that via cURL in the container does not fail at all, so I do not believe this to be a problem with my container.
I have added a packet dump (that I inspect via Charles Proxy) where our of 15 test runs, 7 test runs (and thus 7 requests) fail. All of the failed requests complain that no authentication header was supplied. And from the dump, that is correct, the dump does not contain the authentication header for those requests that fail. But why not? In fact, all of the header flags are different: a successful run has these headers:
POST /db/data/cypher HTTP/1.1
Host: 192.168.0.18:7474
Accept-Encoding: deflate, gzip
Authorization: Basic bmVvNGo6c3RhY2swdmVyRmxvdw==
Content-Type: application/json; charset=utf-8
Accept: application/json; charset=utf-8
Connection: keep-alive
User-Agent: urlsessionTestPackageTests.xctest (unknown version) curl/7.35.0
Content-Length: 135
while an unsuccessful one has these:
POST /db/data/cypher HTTP/1.1
Host: 192.168.0.18:7474
Accept: */*
Accept-Encoding: deflate, gzip
Connection: keep-alive
User-Agent: urlsessionTestPackageTests.xctest (unknown version) curl/7.35.0
Content-Length: 135
All the 200 results have the same headers, and all the 401 results have the same headers. Can you see anything in my code that should warrant such a random request?
I'm creating a REST interface (aren't we all), and I want to UPDATE a resource.
So, I think to use a PUT.
So, i read this.
My take away is that i PUT to a URL like this
/hc/api/v1/organizer/event/762d36c2-afc5-4c51-84eb-9b5b0ef2990c
with a payload, then a permanent redirect to the URL that it can GET an updated version of the resource.
In this case it happens to be the same URL, different action.
So my questions are:
Is my understanding of updating a resource correct in using a PUT, and is my understanding of the use of the PUT correct.
When a client gets a redirect does it do the same action on the redirected URL as it did on the original URL? If its "depends" is there a standard most clients follow?
I ask the 2nd question, because POSTMAN and my JQuery AJAX calls are choking. JQuery because of net::ERR_TOO_MANY_REDIRECTS. So is it redirecting and trying the PUT again, which it will get another REDIRECT?
curl blows up too but even though it says if it gets a 301 it will switch to a GET, it doesn't really seem to do that when i look at the output (below).
When curl follows a redirect and the request is not a plain GET (for example POST or PUT), it will do the following request with a GET if the HTTP response was 301, 302, or 303. If the response code was any other 3xx code, curl will re-send the following request using the same unmodified method.
CURL OUTPUT (edited for brevity) (also note how it says its going to switch to a GET [incorrectly from a POST], but then it seems to do a PUT anyway):
curl -X PUT -H "Authorization: Basic AUTHZ==" -H "Content-Type: application/json" -H "Cache-Control: no-cache" -H "Postman-Token: e80657f0-a8f5-af77-1d9d-d7bc22ed0b30" -d '{ JSONDATA"}' http://localhost:8080/hc/api/v1/organizer/event/762d36c2-afc5-4c51-84eb-9b5b0ef2990c -v -L
* Hostname was NOT found in DNS cache
* Trying 127.0.0.1...
* Connected to localhost (127.0.0.1) port 8080 (#0)
> PUT /hc/api/v1/organizer/event/762d36c2-afc5-4c51-84eb-9b5b0ef2990c HTTP/1.1
> User-Agent: curl/7.37.1
> Host: localhost:8080
> Accept: */*
> Authorization: Basic AUTHZ==
> Content-Type: application/json
> Cache-Control: no-cache
> Postman-Token: e80657f0-a8f5-af77-1d9d-d7bc22ed0b30
> Content-Length: 203
>
* upload completely sent off: 203 out of 203 bytes
< HTTP/1.1 301 Moved Permanently
< Connection: keep-alive
< X-Powered-By: Undertow/1
< Set-Cookie: rememberMe=deleteMe; Path=/hc; Max-Age=0; Expires=Fri, 20-Feb-2015 03:53:28 GMT
< Set-Cookie: JSESSIONID=uwI3_41LAa7vlvapTsrZdw10.macbook-air; path=/hc
* Server WildFly/8 is not blacklisted
< Server: WildFly/8
< Location: /hc/api/v1/organizer/event/762d36c2-afc5-4c51-84eb-9b5b0ef2990c
< Content-Length: 0
< Date: Sat, 21 Feb 2015 03:53:28 GMT
<
* Connection #0 to host localhost left intact
* Issue another request to this URL: 'http://localhost:8080/hc/api/v1/organizer/event/762d36c2-afc5-4c51-84eb-9b5b0ef2990c'
* Switch from POST to GET
* Found bundle for host localhost: 0x7f9e4b415430
* Re-using existing connection! (#0) with host localhost
* Connected to localhost (127.0.0.1) port 8080 (#0)
> PUT /hc/api/v1/organizer/event/762d36c2-afc5-4c51-84eb-9b5b0ef2990c HTTP/1.1
> User-Agent: curl/7.37.1
> Host: localhost:8080
> Accept: */*
> Authorization: Basic dGVzdHVzZXIxOlBhc3N3b3JkMQ==
> Content-Type: application/json
> Cache-Control: no-cache
> Postman-Token: e80657f0-a8f5-af77-1d9d-d7bc22ed0b30
>
< HTTP/1.1 500 Internal Server Error
< Connection: keep-alive
< Set-Cookie: JSESSIONID=fDXxlH2xI-0-DEaC6Dj5EhD9.macbook-air; path=/hc
< Content-Type: text/html; charset=UTF-8
< Content-Length: 8593
< Date: Sat, 21 Feb 2015 03:53:28 GMT
<
...failure ensues... It actually does a PUT
thanks in advance.
I think you're reading too much into the 301 redirect section.
If you want to update a resource using PUT, return:
201: if the resource was created
200: with the updated resource
The 301 in question only applies if there actually is a redirect in question - like, if something can be identified by name, and you need to redirect it to a url that has the id or something. (Maybe you refactor and people are still consuming the old endpoint).
So, do you really need to redirect your PUT requests? Because you should be sending back the updated resource within the same loop using 200, like stated above, instead of "redirecting to GET".
EDIT: Fix some spelling.
The server is responding with a less than helpful message.
Unable to get a request token: Request for https://etwssandbox.etrade.com/oauth/sandbox/request_token?oauth_callback=oob&oauth_consumer_key=aaf0812a4bcc6e4c21783af47cf88237&oauth_nonce=3495463522&oauth_signature=ykqRaZc18GwIoqHtYqtxzsMq4xs%3D&oauth_signature_method=HMAC-SHA1&oauth_timestamp=1371092839&oauth_version=1.0 failed,HTTP/1.1 400 Bad Request
Connection: close
Content-Length: 62
Client-Date: Thu, 13 Jun 2013 03:07:19 GMT
Client-Peer: 12.153.224.230:443
Client-Response-Num: 1
Client-SSL-Cert-Issuer: /C=US/O=VeriSign, Inc./OU=VeriSign Trust Network/OU=Terms of use at https://www.verisign.com/rpa (c)10/CN=VeriSign Class 3 Secure Server CA - G3
Client-SSL-Cert-Subject: /C=US/ST=New York/L=New York/O=ETRADE FINANCIAL CORPORATION/OU=Global Information Security/CN=etwssandbox.etrade.com
Client-SSL-Cipher: RC4-MD5
<html><body><b>Http/1.1 400 Bad Request</b></body> </html>
OK I will try with headers. All required parameters are present.
$ wget -d -O- --header='Authorization: OAuth realm="",oauth_callback="oob",oauth_consumer_key="aaf0812a4bcc6e4c21783af47cf88237",oauth_nonce="3495463522",oauth_signature="ykqRaZc18GwIoqHtYqtxzsMq4xs%3D",oauth_signature_method="HMAC-SHA1",oauth_timestamp="1371092839",oauth_version="1.0"' 'https://etwssandbox.etrade.com/oauth/sandbox/request_token'
Setting --output-document (outputdocument) to -
Setting --header (header) to Authorization: OAuth realm="",oauth_callback="oob",oauth_consumer_key="aaf0812a4bcc6e4c21783af47cf88237",oauth_nonce="3495463522",oauth_signature="ykqRaZc18GwIoqHtYqtxzsMq4xs%3D",oauth_signature_method="HMAC-SHA1",oauth_timestamp="1371092839"
DEBUG output created by Wget 1.13.4 on cygwin.
URI encoding = `UTF-8'
--2013-06-12 23:08:33-- https://etwssandbox.etrade.com/oauth/sandbox/request_token
Resolving etwssandbox.etrade.com (etwssandbox.etrade.com)... 12.153.224.230, 198.93.34.230
Caching etwssandbox.etrade.com => 12.153.224.230 198.93.34.230
Connecting to etwssandbox.etrade.com (etwssandbox.etrade.com)|12.153.224.230|:443... connected.
Created socket 3.
Releasing 0x80733128 (new refcount 1).
---request begin---
GET /oauth/sandbox/request_token HTTP/1.1
User-Agent: Wget/1.13.4 (cygwin)
Accept: */*
Host: etwssandbox.etrade.com
Connection: Keep-Alive
Authorization: OAuth realm="",oauth_callback="oob",oauth_consumer_key="aaf0812a4bcc6e4c21783af47cf88237",oauth_nonce="3495463522",oauth_signature="ykqRaZc18GwIoqHtYqtxzsMq4xs%3D",oauth_signature_method="HMAC-SHA1",oauth_timestamp="1371092839"
---request end---
HTTP request sent, awaiting response...
---response begin---
HTTP/1.1 400 Bad Request
Content-Length:62
Connection: close
---response end---
400 Bad Request
2013-06-12 23:08:34 ERROR 400: Bad Request.
That still did not work. Let me verify the signature. Notice my key and secret are correct.
First URL encode all the parameters to form a base string for signing.
$ perl -MURI::Escape -e "print uri_escape('oauth_callback=oob&oauth_consumer_key=aaf0812a4bcc6e4c21783af47cf88237&oauth_nonce=3495463522&oauth_signature_method=HMAC-SHA1&oauth_timestamp=1371092839&oauth_version=1.0')"
oauth_callback%3Doob%26oauth_consumer_key%3Daaf0812a4bcc6e4c21783af47cf88237%26oauth_nonce%3D3495463522%26oauth_signature_method%3DHMAC-SHA1%26oauth_timestamp%3D1371092839%26oauth_version%3D1.0
Now hash with HMAC-SHA1, encode with Base64 (no newline at end), and URL encode the resulting signature.
There is an ampersand at the end of the consumer secret because we don't have a token secret yet (it is empty).
$ perl -MDigest::HMAC_SHA1=hmac_sha1 -MMIME::Base64 -MURI::Escape -e "print uri_escape(encode_base64(hmac_sha1('GET&https%3A%2F%2Fetwssandbox.etrade.com%2Foauth%2Fsandbox%2Frequest_token&oauth_callback%3Doob%26oauth_consumer_key%3Daaf0812a4bcc6e4c21783af47cf88237%26oauth_nonce%3D3495463522%26oauth_signature_method%3DHMAC-SHA1%26oauth_timestamp%3D1371092839%26oauth_version%3D1.0', 'xxxxxxxxxxxxxxxxxxxx&'), ''))"
ykqRaZc18GwIoqHtYqtxzsMq4xs%3D
This signature matches the above.
The specs are here: http://oauth.net/core/1.0a/#signing_process
ETrade specs are here: https://us.etrade.com/ctnt/dev-portal/getDetail?contentUri=V0_Documentation-AuthorizationAPI-GetRequestToken
ETrade's documentation is broken. They specify in the Sandbox environment uses different hosts and URLs
https://us.etrade.com/ctnt/dev-portal/getContent?contentUri=V0_Documentation-DeveloperGuides-Sandbox
but for OAuth they do not. That part is never mentioned and I had to look in the source code for one of their SDKs to find out.
|Environment| URL |
|Production |https://etws.etrade.com/{module}/rest/{API} |
|Sandbox |https://etwssandbox.etrade.com/{module}/sandbox/rest/{API} |
This works fine (redirecting to a different domain):
#! /usr/bin/perl
print "Location:http://AnyDomainBesidesMyOwn.com/\n\n";
close(STDOUT); close(STDIN); close(STDERR);
[some long process]
But the following way stalls the browser, refusing to redirect until the long process is finished. The only change from above is that I'm redirecting to another page on the same domain the script is running from.
#! /usr/bin/perl
print "Location:http://MyOwnDomain.com/\n\n";
close(STDOUT); close(STDIN); close(STDERR);
[some long process]
I know I can get this to work by forking a new process, but there's got to be a simpler way...right?
UPDATE: Here is the output I get from Live Headers in Firefox:
http://example.com/test3.cgi
GET /test3.cgi HTTP/1.1
Host: example.com
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:18.0) Gecko/20100101 Firefox/18.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate
Connection: keep-alive
HTTP/1.1 302 Found
Date: Sun, 27 Jan 2013 23:31:49 GMT
Server: Apache
Location: http://example.com/
Vary: Accept-Encoding
Content-Encoding: gzip
Content-Length: 187
Keep-Alive: timeout=2, max=100
Connection: Keep-Alive
Content-Type: text/html; charset=iso-8859-1
And then after it deigns to redirect:
http: //example.com/ [Have to include the space b/c stackoverflow limits the number of links I can include in a post]
GET / HTTP/1.1
Host: example.com
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:18.0) Gecko/20100101 Firefox/18.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate
Connection: keep-alive
HTTP/1.1 200 OK
Date: Sun, 27 Jan 2013 23:31:54 GMT
Server: Apache
Accept-Ranges: bytes
X-Mod-Pagespeed: 0.10.21.2-1381 [Same problem even on a domain w/o pagespeed installed]
Vary: Accept-Encoding
Content-Encoding: gzip
MS-Author-Via: DAV
Cache-Control: max-age=0, no-cache
Content-Length: 12189
Keep-Alive: timeout=2, max=99
Connection: Keep-Alive
Content-Type: text/html
I believe I have found a solution after some trial and error. The key is to use fork() and exit the parent process. Then close all the file handles in the child process:
#!/usr/bin/perl
print "Location: ../myHomePage.htm\n\n";
exit 0 if fork();
close(STDIN);
close(STDOUT);
close(STDERR);
# do some long process
open(SM, "| /usr/sbin/sendmail -ti -odq);
⋮
In my case, the mail program was taking several seconds to accept the message and queue it. The webpage seemed very sluggish. After implementing the above, the redirect happenes instantly and several seconds later the message appeares in the mail log.
You should exit after a redirect header sent out. If I am correct a space is needed after 'Location:'. You could use CGI redirect method to achive this.
#!/usr/bin/perl
print "Location: http://MyOwnDomain.com/\n\n";
close(STDOUT); close(STDIN); close(STDERR);
exit 0;
Redirect with CGI: http://perldoc.perl.org/CGI.html#GENERATING-A-REDIRECTION-HEADER
use CGI; # load CGI routines
$q = CGI->new;
print $q->redirect('http://somewhere.else/in/movie/land');