I would like to download PDF file from online form. I used the fiddler to find out the header and post fields. However, I cannot get the header from Firefox to match with Powershell using Invoke-webrequest.
Original request
POST http://somesight/SPLeam/EAMDisplayReportWithParamValues.asp HTTP/1.1
Host: someserver
User-Agent: Mozilla/5.0 (Windows NT 6.1; rv:27.0) Gecko/20100101 Firefox/27.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate
Referer: http://someserver/SPLeam/EAMPickParamValues.asp?plant=01&reportuser=reports&reportuserpw=pw&ReportName=SPLEAM%5CMNT+%2D+MM+Tracking+Rpt%2Erpt
Cookie: ASPSESDFSDIDSSQAQBQC=APHPHMNCHIECSFSSPOIKBDCAM`
Connection: keep-alive
Content-Type: application/x-www-form-urlencoded
Content-Length: 132
Text0=12%2F1%2F2013&Text1=12%2F31%2F2013&Text2=MESGI&Drop2=MESCI&Text3=01&Text4=&Drop4=&Text5=R&Text6=N&cmdSaveParameters=Run+Report
POWERSHELL
POST http://someserver/SPLeam/EAMPickParamValues.asp?plant=01&reportuser=reports&reportuserpw=pw&ReportName=SPLEAM%5CMNT+%2D+MM+Tracking+Rpt%2Erpt HTTP/1.1
User-Agent: Mozilla/5.0 (Windows NT; Windows NT 6.1; en-US) WindowsPowerShell/3.0
Content-Type: application/x-www-form-urlencoded
Host: someserver
Content-Length: 129
Connection: Keep-Alive
cmdSaveParameters=Run%2bReport&Text3=01&Text5=R&Text1=12%252F31%252F2013&Drop2=MI&Text2=MI&Text4=&Text6=N&Text0=12%252F1%252F2013
I run it as
$f = invoke-webrequest -uri $url -UserAgent $useragent -method Post -Body $param
How can I make Powershell's request match what Firefox sent?
.NET's HTTPWebRequest object has separate properties for the Accept and Referer headers; you must manually set these properties instead of trying to add the headers directly.
(These header restrictions exist partly for security purposes; partial trust code cannot set the properties).
Related
Closed. This question needs details or clarity. It is not currently accepting answers.
Want to improve this question? Add details and clarify the problem by editing this post.
Closed 4 years ago.
Improve this question
Dears,
I have the file below in the following format
Success|Filter passed|[invalid field]|[invalid field]|Id-350a875b087965e58cbe1f4a
Accept: text/plain, text/plain, application/json, application/*+json, */*, */*
Host: api2.tim.com.br
User-Agent: Mozilla/5.0 (Windows NT 6.3; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36
Via: 1.1
X-Forwarded-For: 144.22.98.123
X-Forwarded-Host:
X-Forwarded-Server:
Success|Success in calling policy shortcut|[invalid field]|[invalid field]|[invalid field]|Id-350a875b087965e58cbe1f4a|Call 'Set Request Message'|GET
Accept: text/plain, text/plain, application/json, application/*+json, */*, */*
Host: api2.tim.com.br
User-Agent: Mozilla/5.0 (Windows NT 6.3; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36
Via: 1.1 api2.tim.com.br
X-Forwarded-For: 144.22.98.123
X-Forwarded-Host:
X-Forwarded-Server:
Content-Type: text/xml; charset="UTF-8"
I need to perform a search for the line that begins with the string "^ Success" and display all the items until there is a "^ Sucess" string again.
Here is an example of what I need to display:
Success|Filter passed|[invalid field]|[invalid field]|Id-350a875b087965e58cbe1f4a
Accept: text/plain, text/plain, application/json, application/*+json, */*, */*
Host: api2.tim.com.br
User-Agent: Mozilla/5.0 (Windows NT 6.3; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36
Via: 1.1
X-Forwarded-For: 144.22.98.123
X-Forwarded-Host:
X-Forwarded-Server:
What happens is that the amount of lines below after the match is very dynamic.
What happens is that the amount of lines below the match is very dynamic and in the same file there may be several lines
of the same match and I would need to display them also when the file is run.
Could you guys help me?
Perl has a "paragraph mode". You change the input record separator, $/ to read chunks of "multiline" text. This splits up your data on the double newline:
use v5.10;
$/ = "\n\n";
while( <INPUT> ) {
chomp;
say "==========\n$_\n----------\n";
}
Start your program with that and try to do whatever else you're trying to do. In your next question you'll have the small demonstration program you need to get better help.
Using Powershell 5.0.10586.117 I want to request a specific web-site (on IIS 8.5) containing a form to log-in. Here is the specific code:
$Url = Invoke-Webrequest -Uri "https://not-work/hello.aspx" -SessionVariable SesVar
$UrlForm = $Url.Forms[0]
When doing so Powershell hangs and never returns anything as long as I do not interrupt manually.
Using
$Url = Invoke-Webrequest -Verbose -Uri "https://not-work/hello.aspx" -SessionVariable SesVar
I can see the following error:
VERBOSE: Get https://not-work/hello.aspx with 0-byte payload.
Running the same website in Google Chrome 55.0.2883.87 it works. Using an older version of the application code on a different (likely older IIS) it works too in Powershell.
So I am wondering, if this issue has to do with this bug in Powershell?
Note: Even when using
$Url = Invoke-Webrequest -UseBasicParsing -Verbose -Uri "https://not-work/hello.aspx" -SessionVariable SesVar
it fails.
For the sake of completeness, curl also gives an HTTP status code 200 (like in Chrome), but there are some differences in comparison to the working version:
Non-working website
C:\Tools\curl>curl -k https://not-work/hello.aspx -I HTTP/1.1 200 OK
Cache-Control: private
Content-Length: 12671
Content-Type: text/html; charset=utf-8
Server: Microsoft-IIS/8.5 X-AspNet-Version: 4.0.30319
Set-Cookie: __AntiXsrfToken=d438e79fe7004ad09d950a3b8b4e507d; path=/; HttpOnly
Set-Cookie: .ASPXAUTH=; expires=Mon, 11-Oct-1999 22:00:00 GMT; path=/; HttpOnly
X-Powered-By: ASP.NET
Date: Thu, 12 Jan 2017 13:57:33 GMT
Working website
C:\Tools\curl>curl -k https://works-well/hello.aspx -I HTTP/1.1 200 OK
Cache-Control: private Content-Length: 12306 Content-Type: text/html; charset=utf-8
X-Frame-Options: SAMEORIGIN
X-Content-Type-Options: nosniff
Server: WWW Server/1.1 X-XSS-Protection: 1; mode=block
Set-Cookie: __AntiXsrfToken=6142de1c5d474da780530da39fea033f; path=/;HttpOnly; Secure
Date: Thu, 12 Jan 2017 15:56:47 GMT
By the way, in Internet Explorer (IE) 11.0.9600.18537 objects like images are not loaded (as I believe Powershell works "under the hood" using IE DLLs) and the "hourglass" is running permanently too next to the tab of the requested URI.
So again my question: Has this issue to do with this bug in Powershell or could it be something else?
I'm troubleshooting my own situation of the same nature. My website seems to have changed a few things around. One of which is the FORMS.
Can you confirm that your website has a FORMS option?
EX.
#GET WEBSESSION
$url = "https://myaccount.nytimes.com/"
$r = Invoke-WebRequest -Uri $url -SessionVariable ws -UseBasicParsing
return $r
Notice that I use -UseBasicParsing. For my nytimes example I have to. This is because nytimes is using JS and that doesn't seem to return well for PS. Removing -UseBasicParsing will result in my WebRequest hanging.
You will notice that with the above return the 'FORMS : ' is null.
Right now I do not have a solution. Other than, my website clearly requires credentials and I clearly need to provide. So I'm now writing a muttery of different attempts using -Body $bodyParam #{} and -Headers $headerParams #{}. So far it's to no avail.
Simply put, PS doesn't seem to behave well with JS type websites and FORMS seems to be used for only a specific set of website, not for all type sites.
Hopefully someone more skilled can assist cause this is racking my brain.
I currently expand my test suite to increase the test coverage. I want to test my controller and the html output that it renders, but I found a problem in using delete methods. Let me explain it in an example.
I have a route:
$r->delete('/backups/:id')
->to('backup#delete_backup')
->name('backup_delete');
that points to the following function in the backup controller:
sub delete_backup {
my $self = shift;
my $id = $self->param('id');
if ( something ) {
$self->flash( msg => "Backup id $id deleted!" );
}
else{
$self->flash( msg => "Cannot delete, backup id $id not found!" );
}
$self->redirect_to($self->url_for('backup_index'));
}
where the method that handles the route backup_index just displays the $msg and shows few other irrelevant data.
Next, I want to test this method, so I write a test:
$t_logged_in->ua->max_redirects(3);
my $page = $t_logged_in->app->url_for( 'backup_delete', id => $backup_id );
$t_logged_in->delete_ok($page)
->status_isnt( 404, "Checking: 404 $page" )
->status_isnt( 500, "Checking: 500 $page" );
The test is passed. But now, I want to check if the text is correct on the web page that is shown after redirecting. So I do the following:
$t_logged_in->ua->max_redirects(3);
my $page = $t_logged_in->app->url_for( 'backup_delete', id => $backup_id );
$t_logged_in->delete_ok($page)
->status_isnt( 404, "Checking: 404 $page" )
->status_isnt( 500, "Checking: 500 $page" )
->content_unlike(qr/Cannot delete,/i)
->content_like(qr/deleted/i);
The test fails. It fails because the content is empty, so the matching is done as there were:
'' =~ /deleted/i;
'' !~ /Cannot delete,/i;
and this is of course false in both cases. Of course, in the browser, the redirects work perfectly and I see everything as designed in the test. I can change the method to POST or GET but I wanted to make the routing properly in the way an API would be designed.
Question: how to design the test such that the content can be matched after the redirect?
For those who want to dig deeper, I give links to Github.
Routing definition line 303
Controller Backup function delete_backup line 135
Functional Test of Backup Controller line 53
Sorry no one answered this yet. I used to try to keep a closer eye on stack overflow but I got lazy :-P. This is a very interesting question.
Standard redirects (301/302) use the same verb as the original request UNLESS the original request was a POST Interestingly, this behavior was not really intended. A redirect is supposed to use the same request method as the original, but the POST->GET redirect is so commonly intended that most browsers added this behavior even though it was against the original definition. However some people really wanted a POST to remain a POST and now it depended on the implementation. In fact this got so bad that HTTP added response statuses 307/308 which always redirects as the same http verb (ie POST -> POST), even thought that was already the official behavior of the old ones but that ship had sailed. This means that POST->GET, but DELETE->DELETE is the defacto behavior of the 301/302.
This can be demonstrated in the following one liner ($_ in these one-liners is the controller):
perl -Mojo -E 'del "/delete" => sub { $_->redirect_to("/") }; any "/" => { inline => q[I got a <%= $c->req->method %>] }; app->start' get -r -v -M DELETE /delete
DELETE /delete HTTP/1.1
Host: 127.0.0.1:62690
Accept-Encoding: gzip
User-Agent: Mojolicious (Perl)
Content-Length: 0
HTTP/1.1 302 Found
Date: Sun, 18 Sep 2016 03:25:58 GMT
Server: Mojolicious (Perl)
Location: /
Content-Length: 0
DELETE / HTTP/1.1
Content-Length: 0
User-Agent: Mojolicious (Perl)
Host: 127.0.0.1:62690
Accept-Encoding: gzip
HTTP/1.1 200 OK
Content-Type: text/html;charset=UTF-8
Content-Length: 5
Date: Sun, 18 Sep 2016 03:25:58 GMT
Server: Mojolicious (Perl)
I got a DELETE
You can see that since my / route handles all methods we get a response, however the request was a DELETE. I see that in your app, you redirect to another named route whose url also handles DELETE. This is a source of your confusion as you are actually ending up there in your tests. Your redirect is effectively DELETE /backups/<<id>> -> DELETE /backups, if I'm reading your code correctly.
Now as opposed to the people who wanted to keep POST as POST, what you want here is the opposite, you want to redirect from DELETE to GET because you are trying to show a response that is not the original resource. This is the defined behavior of the little-known 303 response in which any request method is redirected to a GET. Indeed this is what early browsers should have insisted on rather than breaking the 302.
In the following example I modify the response code to 303 explicitly.
$ perl -Mojo -E 'del "/delete" => sub { $_->res->code(303); $_->redirect_to("/") }; any "/" => { inline => q[I got a <%= $c->req->method %>] }; app->start' get -r -v -M DELETE /delete
DELETE /delete HTTP/1.1
User-Agent: Mojolicious (Perl)
Host: 127.0.0.1:62716
Accept-Encoding: gzip
Content-Length: 0
HTTP/1.1 303 See Other
Location: /
Content-Length: 0
Server: Mojolicious (Perl)
Date: Sun, 18 Sep 2016 03:27:19 GMT
DELETE / HTTP/1.1
User-Agent: Mojolicious (Perl)
Accept-Encoding: gzip
Content-Length: 0
Host: 127.0.0.1:62716
HTTP/1.1 200 OK
Content-Type: text/html;charset=UTF-8
Server: Mojolicious (Perl)
Content-Length: 5
Date: Sun, 18 Sep 2016 03:27:19 GMT
I got a DELETE
But whoops, that's still a DELETE! That's because there was a bug in Mojo::UserAgent so that it didn't handle 303s correctly. I'd suggest you still make the change to your code since browsers seem to handle 303s correctly. However since Mojo::UserAgent powers Test::Mojo, your tests can't test that behavior yet. Once fixed you will see it work correctly, as my example does in my local branch:
$ perl -Ilib -Mojo -E 'del "/delete" => sub { $_->res->code(303); $_->redirect_to("/") }; any "/" => { inline => q[I got a <%= $c->req->method %>] }; app->start' get -r -v -M DELETE /delete
DELETE /delete HTTP/1.1
Content-Length: 0
User-Agent: Mojolicious (Perl)
Host: 127.0.0.1:64924
Accept-Encoding: gzip
HTTP/1.1 303 See Other
Date: Sun, 18 Sep 2016 04:59:37 GMT
Location: /
Server: Mojolicious (Perl)
Content-Length: 0
GET / HTTP/1.1
Content-Length: 0
User-Agent: Mojolicious (Perl)
Host: 127.0.0.1:64924
Accept-Encoding: gzip
HTTP/1.1 200 OK
Date: Sun, 18 Sep 2016 04:59:37 GMT
Content-Type: text/html;charset=UTF-8
Server: Mojolicious (Perl)
Content-Length: 12
I got a GET
To see more about redirect response types see https://en.wikipedia.org/wiki/List_of_HTTP_status_codes#3xx_Redirection
I have the following script at �://192.168.1.3/homeworks/hw10/testcookie.cgi:
#!/usr/bin/perl -wT
use CGI qw(:standard);
use CGI::Carp qw(warningsToBrowser fatalsToBrowser);
use CGI::Cookie qw(cookie);
use strict;
use warnings;
sub makecookie {
return my $cookie = cookie(-name=>'hw10',
-value=>shift,
-expires=>shift,
-path=>'/hw10/testcookie.cgi',
-domain=>'192.168.1.3',
-secure=>1);
}
my $cgi = CGI->new();
my $cookie = makecookie("192.168.1.3",'+3d');
print STDOUT $cgi->header(-cookie=>$cookie);
print STDOUT $cgi->start_html("Test Cookie");
print STDOUT "<h1>TEST</h1>";
print STDOUT $cgi->end_html();
I checked with Live headers that something was sent:
GET /homeworks/hw10/testcookie.cgi HTTP/1.1
Host: 192.168.1.3
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:34.0) Gecko/20100101 Firefox/34.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate
Connection: keep-alive
HTTP/1.1 200 OK
Date: Wed, 24 Dec 2014 16:11:24 GMT
Server: Apache/2.2.22 (Debian)
Set-Cookie: hw10=192.168.1.3; domain=192.168.1.3; path=/hw10/testcookie.cgi; expires=Sat, 27-Dec-2014 16:11:24 GMT; secure
Vary: Accept-Encoding
Content-Encoding: gzip
Content-Length: 258
Keep-Alive: timeout=5, max=100
Connection: Keep-Alive
Content-Type: text/html; charset=ISO-8859-1
----------------------------------------------------------
but firefox says there is no cookie associated with the site. Why does firefox reject the cookie?
The path component of the URL is
/homeworks/hw10/testcookie.cgi
yet you're trying to set a cookie for
/hw10/testcookie.cgi
The latter is not contained by the former, so that's an error. You might want to use
-path => $cgi->url( -absolute => 1 )
instead of
-path => '/hw10/testcookie.cgi'
This may not be the only error. You didn't provide the URL you requested, so I don't know if you're using HTTP or HTTPS. You'll have problems if you're using HTTP because you specified the cookie should only be provided over secure connections.
I am working on a series of Powershell scripts for my company to use to transfer data to and from Box.com. The one thing I just can't figure out is uploading files. The Box API requires a multipart POST operation for uploads, and I've seen a few answers here on SO indicating that I should be able to do that in Powershell (such as this one). But I can't seem to get it working.
Here's the code I have right now:
Function Post-File {
Param(
[Parameter(Mandatory=$True,Position=1)]
[string]$SourcePath,
[Parameter(Mandatory=$False,Position=2)]
[string]$FolderId = ############
)
#Variables for building URIs
$baseUrl = "https://upload.box.com/api/2.0/files/content"
#Set Authorization for API requests
$headers = #{}
$AccessToken = Refresh-Tokens #A reference to another function that definitely works
$headers.Add("Authorization", "Bearer $AccessToken")
#Set POST content
$body = #{}
$body.Add("filename", [IO.File]::ReadAllBytes($SourcePath))
$body.Add("parent_id", $FolderId)
#Upload the file
Invoke-RestMethod -Uri $baseUrl -Method Post -Headers $headers -ContentType "multipart/form-data" -Body $body
}
Here's the response I get back:
{
"type":"error",
"status":400,
"code":"invalid_request_parameters",
"help_url":"http://developers.box.com/docs/#errors",
"message":"Invalid input parameters in request",
"request_id":"1764475572534bcddfe04b7"
}
I've also tried a couple of other permutations that aren't working. I've tried using the -InFile switch in Invoke-RestMethod instead of -Body. I've also tried using Get-Content -Raw in place of [IO.File]::ReadAllBytes. Both of those return a more generic error: The server encountered an internal error or misconfiguration and was unable to complete your request..
I'm pretty sure this has something to do with my filename parameter not being correct, but I'm not sure how to fix it. According to the Box API, here's what it should look like in curl. Can someone help me properly translate this for Powershell?
https://upload.box.com/api/2.0/files/content \
-H "Authorization: Bearer ACCESS_TOKEN" \
-F filename=#FILE_NAME \
-F parent_id=PARENT_FOLDER_ID
There are a couple things that make this a little tricky in PowerShell:
The filename parameter specifies the name of the file, not the contents.
A request body is required in order to specify the file name and destination.
PowerShell treats -InFile and -Body arguments as mutually exclusive.
PowerShell does not appear to natively support multipart/form-data POSTs, per the question that you referenced.
Since the request body is required (1,2) -- and thus -InFile can't be used (3) -- I think you might need to roll your own multipart/form-data body (4) that contains the necessary metadata and file content. This blog post describes a method for doing so. The content there is a short string (a tweet) but the principle is the same.
Below is a Fiddler trace of an upload I just performed using the Box Windows SDK. This shows how the request should look as it goes across the wire. The $BOUNDARY is an arbitrary, unique string -- a GUID works great.
POST https://upload.box.com/api/2.0/files/content HTTP/1.1
Authorization: Bearer $TOKEN
Content-Type: multipart/form-data; boundary="$BOUNDARY"
Host: upload.box.com
Content-Length: 2118
Accept-Encoding: gzip, deflate
--$BOUNDARY
Content-Disposition: form-data; name="file"; filename="$FILENAME"
<THE CONTENT OF $FILENAME>
--$BOUNDARY
Content-Type: text/plain; charset=utf-8
Content-Disposition: form-data; name="metadata"
{"parent":{"id":"$PARENT_FOLDER_ID"},"name":"$FILENAME"}
--$BOUNDARY--
Here is the response I received:
HTTP/1.1 201 Created
Date: Mon, 14 Apr 2014 12:52:33 GMT
Server: Apache
Cache-Control: no-cache, no-store
Content-Length: 364
Connection: close
Content-Type: application/json
{"total_count":1,"entries":[{"type":"file","id":"$ID","name":"$FILENAME", ... }]}
I may sound dumb, but what happens when you do this?
$body.Add("filename", $([IO.File]::ReadAllBytes($SourcePath)))