Trying to mirror a local intranet site and have found previous questions using 'wget'. It works great with sites that are anonymous, but I have not been able to use it against a site that is expecting username\password (IIS with Integrated Windows Authentication).
Here is what I pass in:
wget -c --http-user='domain\user' --http-password=pwd http://local/site -dv
Here is the debug output (note I replaced some with dummy values obviously):
Setting --verbose (verbose) to 1
DEBUG output created by Wget 1.11.4 on Windows-MSVC.
--2009-07-14 09:39:04-- http://local/site
Host `local' has not issued a general basic challenge.
Resolving local... seconds 0.00, x.x.x.x
Caching local => x.x.x.x
Connecting to local|x.x.x.x|:80... seconds 0.00, connected.
Created socket 1896.
Releasing 0x003e32b0 (new refcount 1).
---request begin---
GET /site/ HTTP/1.0
User-Agent: Wget/1.11.4
Accept: */*
Host: local
Connection: Keep-Alive
---request end---
HTTP request sent, awaiting response...
---response begin---
HTTP/1.1 401 Access Denied
Server: Microsoft-IIS/5.1
Date: Tue, 14 Jul 2009 13:39:04 GMT
WWW-Authenticate: Negotiate
WWW-Authenticate: NTLM
Content-Length: 4431
Content-Type: text/html
---response end---
401 Access Denied
Closed fd 1896
Unknown authentication scheme.
Authorization failed.
NTLM authentication is broken in wget 1.11, use 1.10 instead.
Curl is actually probably a better tool for fetching content from NTLM-authenticated web servers. You can get an equivalent function to your proposed wget command line by using:
curl --anyauth --user username:password http://someserver/site
I've seen references to being able to use the NTLM Authorization Proxy Server to get around these types of problems.
use --auth-no-challenge option (wget 1.11+) (it's now considered unsafe)
I found solution.
It is work-around for Basic auth IIS7.
When auth is successeful it send next http header:
'Authorization: < type > < credentials >'.
So we able to do authorization in browser and
copy this header params from browser (firebug addon) or generate:
$ echo -en 'username:password' | base64
dXNlcm5hbWU6cGFzc3dvcmQK
$ echo 'dXNlcm5hbWU6cGFzc3dvcmQK' | base64 -d
username:password
example:
$ wget --header="Authorization: Basic dXNlcm5hbWU6cGFzc3dvcmQK" http://example.com/
Related
I am using cURL to execute commands using the REST interface of BaseX like this:
curl http://localhost:8984/rest/?command=repo+list
There are also commands to manage the XQuery module repository. I am especially interested in REPO INSTALL to install a package. Is it somehow possible to execute this command using cURL and the REST interface but without having the package already on the target server? I want to provide the file in the body of the cURL request, similar to adding a resource to a database (source) which goes like this:
curl -i -X PUT -T "etc/xml/factbook.xml" "http://localhost:8984/rest/factbook"
Trying
curl -i -X PUT -T "/tmp/foo.xar" http://localhost:8984/rest/?command=repo+install
Gives me
HTTP/1.1 404 Not Found
Content-Type: text/plain;charset=UTF-8
Content-Length: 18
Connection: close
Server: Jetty(9.4.18.v20190429)
No path specified.
Adding -H "Content-Type: application/x-xar" does not help either.
And replacing PUT with POST gives me
HTTP/1.1 100 Continue
HTTP/1.1 400 Bad Request
Date: Tue, 03 Mar 2020 09:53:21 GMT
Content-Type: text/plain;charset=utf-8
Content-Length: 46
Server: Jetty(9.4.18.v20190429)
"" (Line 1): Content is not allowed in prolog.
The following works in case of standard modules (replace user/pass/server if needed):
$ curl http://admin:admin#localhost:8984/rest/?command=repo+install+http://www.xqueryfunctions.com/xq/functx-1.0.1-doc.xq
I'm trying to get an AWS APIGateway implementation going, and am trying to send a request from the HTTPie module rather than from Postman. It works perfectly from Postman, but HTTPie doesn't seem to work for me, and only throws a 307 Temporary Redirect.
Using the following command:
http POST {userid}.execute-api.ap-southeast-2.amazonaws.com/sqstest/message name=john
Outputs:
HTTP/1.1 307 Temporary Redirect
Connection: keep-alive
Content-Length: 185
Content-Type: text/html
Date: Mon, 16 Apr 2018 06:28:24 GMT
Location: https://{userid}.execute-api.ap-southeast-2.amazonaws.com/sqstest/message
Server: CloudFront
Via: ################(CloudFront)
X-Amz-Cf-Id: ######################
X-Cache: Redirect from cloudfront
I did notice that Content-Type was text/html, which was odd considering I needed to send a json - but no matter what variant of the command I tried, it would still return the same results.
From my understanding it should work the same as Postman as long as the headers are the same (they are minus the content-type, which doesn't change even if I define it using -j/--json).
Any help?
Cheers.
After a few hours of trial-and-error, determined that the error was in syntax.
Required a https:// on the command and to state it as json it needed a semicolon (:).
For example:
http POST https://{userid}.execute-api.ap-southeast-2.amazonaws.com/sqstest/message name:=john
As opposed to the statement in the question.
I make a POST request to REST API to upload a file. In Postman everything works fine. I add Basic authorization and custom CSRF (XSRF) token which I get from the server.
I want to make the same using cURL. I copied the code from Postman, and it does not seem to work.
I believe that the error is related to CSRF because if I turn off CSRF on server and make the same cURL call without CSRF token, everything works fine.
Now some more details:
That's what the command for cURL which Postman gives:
curl -X POST -H "XSRF: 79f51981-8e85-4e26-be1b-bf63aed92a42" -H "Authorization: Basic bbhjbjb=" -H "Cache-Control: no-cache" -H "Postman-Token: 76a7a43b-f407-15a2-aaff-5242b44d0f47" -H "Content-Type: multipart/form-data; boundary=----WebKitFormBoundary7MA4YWxkTrZu0gW" -F "package=#C:\Downloads\hello-world.zip" "http://host:port/api/import"
And that's the reply I get with --verbose
timeout on name lookup is not supported
Trying ::1...
Connected to localhost (::1) port 7777 (#0)
POST /api/import HTTP/1.1
Host: localhost:7777
User-Agent: curl/7.47.1
Accept: /
XSRF: 79f51981-8e85-4e26-be1b-bf63aed92a42
Authorization: Basic bbhjbjb=
Cache-Control: no-cache
Postman-Token: 76a7a43b-f407-15a2-aaff-5242b44d0f47
Content-Length: 31281
Expect: 100-continue
Content-Type: multipart/form-data; boundary=----WebKitFormBoundary7MA4YWxkTrZu0gW;
boundary=------------------------742d3475ac5f6aba
< HTTP/1.1 302 Found
< Set-Cookie: JSESSIONID=1qfjmbntrthxll;Path=/api < Expires: Thu, 01 Jan 1970 00:00:00 GMT
< Set-Cookie: XSRF=b29bd143-cc80-49ad-b495-711125678o;Path=/;Expires=Thu, 15-Dec-2016 10:28:46 GMT
< XSRF: b29bd143-cc80-49ad-b495-711125678o < Location:
http://localhost:7777/api/login/error.jsp?errorMessage=Access Denied
< Content-Length: 0
< Server: Jetty(9.2.17.v20160517)
HTTP error before end of send, stop sending
Closing connection 0
I am probably missing something very obvious here, but don't know what yet.
Looks like I am redirected to login page, not being authenticated correctly, but do not know why (I do send XSRF in cURL). I tried also adding sessionid in cURL - also didn't work.
Any ideas and directions about where to search would be very appreciated!!!
As mentioned in this post, add following option
--cookie "csrftoken=XXXXXX;sessionid=YYYYYYY"
along with
-H "X-CSRFToken: XXXXX"
It is unclear how your server side code is implemented. One visible difference can be seen here is the UserAgent string in request header User-Agent: curl/7.47.1. You may try with adding -A "Mozilla/5.0" with your curl request.
About the comment above regarding XSRF 1-time token; Your server is returning Set-Cookie header in response. It can happen that the postman is using that as cookie for second time request, and that's why it works for it over and over. You can try adding -H "Cookie: XSRF=b29bd143-cc80-49ad-b495-711125678o" at the end of your curl and see if that makes any difference.
Those are all wild guess. Better you add some code at your server side that can print the request-headers. Then make two requests, one from curl and other one from postman. After that check the difference between the request headers. That will give you some clue.
In the end it turned out that the session id was required (adding JSESSIONID in cURL solved the problem).
Without more info on the server side code, I'm not sure either. If you're making your call from cURL, and not Postman, do you really need the Postman-Token header? Maybe it will work if you remove -H "Postman-Token: 76a7a43b-f407-15a2-aaff-5242b44d0f47" from the code.
curl -X POST \
-H "XSRF: 79f51981-8e85-4e26-be1b-bf63aed92a42" \
-H "Authorization: Basic bbhjbjb=" \
-H "Cache-Control: no-cache" \
-H "Content-Type: multipart/form-data; boundary=----WebKitFormBoundary7MA4YWxkTrZu0gW" \
-F "package=#C:\Downloads\hello-world.zip" \
"http://host:port/api/import"
consider a web service written in play, which excepts POST request (for uploads). now, when testing this with a medium size image (~75K) I've found out a strange behaviour. well, code speaks more clearly than long explanations, so:
$ curl -vX POST localhost:9000/path/to/upload/API -H "Content-Type: image/jpeg" -d #/path/to/mascot.jpg
* Hostname was NOT found in DNS cache
* Trying 127.0.0.1...
* Connected to localhost (127.0.0.1) port 9000 (#0)
> POST /path/to/upload/API HTTP/1.1
> User-Agent: curl/7.35.0
> Host: localhost:9000
> Accept: */*
> Content-Type: image/jpeg
> Content-Length: 27442
> Expect: 100-continue
>
< HTTP/1.1 100 Continue
< HTTP/1.1 200 OK
< Content-Type: application/json; charset=utf-8
< Content-Length: 16
<
* Connection #0 to host localhost left intact
{"success":true}
as you can see, curl decides to add the header Content-Length: 27442, but it's not true, the real size is 75211, and in play, I indeed got a body in size only 27442. of coarse, this is not the intended behaviour. so I tried a different tool, instead of curl I used the POST tool from libwww-perl:
cat /path/to/mascot.jpg | POST -uUsSeE -c image/jpeg http://localhost:9000/path/to/upload/API
POST http://localhost:9000/path/to/upload/API
User-Agent: lwp-request/6.03 libwww-perl/6.05
Content-Length: 75211
Content-Type: image/jpeg
200 OK
Content-Length: 16
Content-Type: application/json; charset=utf-8
Client-Date: Mon, 16 Jun 2014 09:21:00 GMT
Client-Peer: 127.0.0.1:9000
Client-Response-Num: 1
{"success":true}
this request succeeded. so I started to pay more attention to the differences between the tools. for starter: the Content-Length header was correct, but also, the Expect header was missing from the second try. I want the request to succeed either way. so the full list of headers as seen in play (via request.headers) is:
for curl:
ArrayBuffer((Content-Length,ArrayBuffer(27442)),
(Accept,ArrayBuffer(*/*)),
(Content-Type,ArrayBuffer(image/jpeg)),
(Expect,ArrayBuffer(100-continue)),
(User-Agent,ArrayBuffer(curl/7.35.0)),
(Host,ArrayBuffer(localhost:9000)))
for the libwww-perl POST:
ArrayBuffer((TE,ArrayBuffer(deflate,gzip;q=0.3)),
(Connection,ArrayBuffer(TE, close)),
(Content-Length,ArrayBuffer(75211)),
(Content-Type,ArrayBuffer(image/jpeg)),
(User-Agent,ArrayBuffer(lwp-request/6.03 libwww-perl/6.05)),
(Host,ArrayBuffer(localhost:9000)))
So my current thoughts are: the simpler perl tool used a single request, which is bad practice. the better way would be to wait for a 100 continue confirmation (especially if you gonna' upload a several GB of data...). curl would continue to send data until it receives a 200 OK or some bad request error code. So why play sends the 200 OK response without waiting for the next chunk? is it because curl specifies the wrong Content-Length? if it's wrong at all... (perhaps this refers to the size of the current chunk?).
so where's the problem lies? in curl or in the play webapp? and how do I fix it?
the problem was in my curl command. I used the -d argument, which is a short for --data or --data-ascii, when I should have used --data-binary argument.
The server is responding with a less than helpful message.
Unable to get a request token: Request for https://etwssandbox.etrade.com/oauth/sandbox/request_token?oauth_callback=oob&oauth_consumer_key=aaf0812a4bcc6e4c21783af47cf88237&oauth_nonce=3495463522&oauth_signature=ykqRaZc18GwIoqHtYqtxzsMq4xs%3D&oauth_signature_method=HMAC-SHA1&oauth_timestamp=1371092839&oauth_version=1.0 failed,HTTP/1.1 400 Bad Request
Connection: close
Content-Length: 62
Client-Date: Thu, 13 Jun 2013 03:07:19 GMT
Client-Peer: 12.153.224.230:443
Client-Response-Num: 1
Client-SSL-Cert-Issuer: /C=US/O=VeriSign, Inc./OU=VeriSign Trust Network/OU=Terms of use at https://www.verisign.com/rpa (c)10/CN=VeriSign Class 3 Secure Server CA - G3
Client-SSL-Cert-Subject: /C=US/ST=New York/L=New York/O=ETRADE FINANCIAL CORPORATION/OU=Global Information Security/CN=etwssandbox.etrade.com
Client-SSL-Cipher: RC4-MD5
<html><body><b>Http/1.1 400 Bad Request</b></body> </html>
OK I will try with headers. All required parameters are present.
$ wget -d -O- --header='Authorization: OAuth realm="",oauth_callback="oob",oauth_consumer_key="aaf0812a4bcc6e4c21783af47cf88237",oauth_nonce="3495463522",oauth_signature="ykqRaZc18GwIoqHtYqtxzsMq4xs%3D",oauth_signature_method="HMAC-SHA1",oauth_timestamp="1371092839",oauth_version="1.0"' 'https://etwssandbox.etrade.com/oauth/sandbox/request_token'
Setting --output-document (outputdocument) to -
Setting --header (header) to Authorization: OAuth realm="",oauth_callback="oob",oauth_consumer_key="aaf0812a4bcc6e4c21783af47cf88237",oauth_nonce="3495463522",oauth_signature="ykqRaZc18GwIoqHtYqtxzsMq4xs%3D",oauth_signature_method="HMAC-SHA1",oauth_timestamp="1371092839"
DEBUG output created by Wget 1.13.4 on cygwin.
URI encoding = `UTF-8'
--2013-06-12 23:08:33-- https://etwssandbox.etrade.com/oauth/sandbox/request_token
Resolving etwssandbox.etrade.com (etwssandbox.etrade.com)... 12.153.224.230, 198.93.34.230
Caching etwssandbox.etrade.com => 12.153.224.230 198.93.34.230
Connecting to etwssandbox.etrade.com (etwssandbox.etrade.com)|12.153.224.230|:443... connected.
Created socket 3.
Releasing 0x80733128 (new refcount 1).
---request begin---
GET /oauth/sandbox/request_token HTTP/1.1
User-Agent: Wget/1.13.4 (cygwin)
Accept: */*
Host: etwssandbox.etrade.com
Connection: Keep-Alive
Authorization: OAuth realm="",oauth_callback="oob",oauth_consumer_key="aaf0812a4bcc6e4c21783af47cf88237",oauth_nonce="3495463522",oauth_signature="ykqRaZc18GwIoqHtYqtxzsMq4xs%3D",oauth_signature_method="HMAC-SHA1",oauth_timestamp="1371092839"
---request end---
HTTP request sent, awaiting response...
---response begin---
HTTP/1.1 400 Bad Request
Content-Length:62
Connection: close
---response end---
400 Bad Request
2013-06-12 23:08:34 ERROR 400: Bad Request.
That still did not work. Let me verify the signature. Notice my key and secret are correct.
First URL encode all the parameters to form a base string for signing.
$ perl -MURI::Escape -e "print uri_escape('oauth_callback=oob&oauth_consumer_key=aaf0812a4bcc6e4c21783af47cf88237&oauth_nonce=3495463522&oauth_signature_method=HMAC-SHA1&oauth_timestamp=1371092839&oauth_version=1.0')"
oauth_callback%3Doob%26oauth_consumer_key%3Daaf0812a4bcc6e4c21783af47cf88237%26oauth_nonce%3D3495463522%26oauth_signature_method%3DHMAC-SHA1%26oauth_timestamp%3D1371092839%26oauth_version%3D1.0
Now hash with HMAC-SHA1, encode with Base64 (no newline at end), and URL encode the resulting signature.
There is an ampersand at the end of the consumer secret because we don't have a token secret yet (it is empty).
$ perl -MDigest::HMAC_SHA1=hmac_sha1 -MMIME::Base64 -MURI::Escape -e "print uri_escape(encode_base64(hmac_sha1('GET&https%3A%2F%2Fetwssandbox.etrade.com%2Foauth%2Fsandbox%2Frequest_token&oauth_callback%3Doob%26oauth_consumer_key%3Daaf0812a4bcc6e4c21783af47cf88237%26oauth_nonce%3D3495463522%26oauth_signature_method%3DHMAC-SHA1%26oauth_timestamp%3D1371092839%26oauth_version%3D1.0', 'xxxxxxxxxxxxxxxxxxxx&'), ''))"
ykqRaZc18GwIoqHtYqtxzsMq4xs%3D
This signature matches the above.
The specs are here: http://oauth.net/core/1.0a/#signing_process
ETrade specs are here: https://us.etrade.com/ctnt/dev-portal/getDetail?contentUri=V0_Documentation-AuthorizationAPI-GetRequestToken
ETrade's documentation is broken. They specify in the Sandbox environment uses different hosts and URLs
https://us.etrade.com/ctnt/dev-portal/getContent?contentUri=V0_Documentation-DeveloperGuides-Sandbox
but for OAuth they do not. That part is never mentioned and I had to look in the source code for one of their SDKs to find out.
|Environment| URL |
|Production |https://etws.etrade.com/{module}/rest/{API} |
|Sandbox |https://etwssandbox.etrade.com/{module}/sandbox/rest/{API} |