I have a Perl script trying to send a zip file like so with LWP UserAgent module
my $req = POST $url, Content_Type => 'form-data',
Content => [
submit => 1,
upfile => [ $fname ]
];
where $fname is the path of the file. On the server side though it seems my POST array only has "submit".
Should I base64 encode the file and assign it to a variable? What's the best way to do this?
Make sure the filename can be resolved. You should get an error if it cannot be, though. At least I do in my version of HTTP::Request::Common.
You don't have to encode the binary content as Base64. (Unless, of course, the server-side app happens to expect that format.)
Here's a complete sample script:
use strict;
use warnings;
use LWP::UserAgent;
use HTTP::Request::Common 'POST';
my $ua = LWP::UserAgent->new;
my $url = 'http://localhost:8888'; # Fiddler
my $req = POST $url,
Content_Type => 'form-data',
Content => [
submit => 1,
upfile => [ 'C:\temp\bla.zip' ],
];
my $line = '=' x 78 . "\n";
print $line, $req->as_string;
my $rsp = $ua->request( $req );
print $line, $rsp->as_string;
Related
I am writing a Perl script to POST an attachment to JIRA using
REST::Client to access the API
but I am getting an error.
use REST::Client;
use warnings;
use strict;
use File::Slurp;
use MIME::Base64;
my $user = 'user';
my $pass = 'pass';
my $url = "http://******/rest/api/2/issue/BugID/attachments";
my $client = REST::Client->new();
$client->addHeader( 'Authorization', 'Basic' . encode_base64( $user . ':' . $pass ) );
$client->addHeader( 'X-Atlassian-Token', 'no-check' );
$client->setHost( $url );
# my %header = ('Authorization' => 'Basic'. encode_base64($user . ':' . $pass),'X-Atlassian-Token' => 'no-check');
my $attachment = "C:\\Folder\\Test.txt";
$client->POST(
$url,
'Content_Type' => 'form-data',
'Content' => [ 'file' => [$attachment] ]
);
if ( $client->responseCode() eq '200' ) {
print "Updated\n";
}
# print the result
print $client->responseContent() . "\n";
The error I get is
REST::Client exception: headers must be presented as a hashref at C:\Users\a\filename.pl line 24.
As shown in the code, I have tried setting headers in different ways but I still get same error.
Please suggest if there is any other method.
I have tried using JIRA module but it gives error too.
According to the documentation, the POST method:
Takes an optional body content and hashref of custom request headers.
You need to put your headers in a hashref, e.g.:
$client->POST($url, $content, {
foo => 'bar',
baz => 'qux'
});
But...it looks like you're expecting REST::Client to use HTTP::Request::Common to construct a multipart/form-data request. Unfortunately, that's not the case, so you'll have to build the content by hand.
You could use HTTP::Request::Common directly like this:
use strict;
use warnings 'all';
use 5.010;
use HTTP::Request::Common;
use REST::Client;
my $client = REST::Client->new;
my $url = 'http://www.example.com';
my $req = POST($url,
Content_Type => 'form-data',
Content => [ file => [ 'foo.txt' ] ]
);
$client->POST($url, $req->content(), {
$req->headers->flatten()
});
But this is a bit convoluted; I would recommend dropping REST::Client and using LWP::UserAgent instead. REST::Client is just a thin wrapper for LWP::UserAgent with a few convenience features, like prepending a default host to all requests. In this case, it's just getting in the way and I don't think the conveniences are worth the trouble.
From the documentation:
POST ( $url, [$body_content, %$headers] )
And you're doing:
$client->POST(
$url,
'Content_Type' => 'form-data',
'Content' => [ 'file' => [$attachment] ]
);
So - passing a list of scalars, with an arrayref at the end.
Perhaps you want something like:
$client->POST(
$url,
$attachment,
{ 'Content-Type' => 'form-data' }
);
Note the {} to construct an anonymous hash for the headers.
Although you probably want to open and include the 'attachment', because there's nothing in REST::Client about opening files and sending them automagically.
I am trying to upload a file to a public API with a code similar to this:
my $ua = LWP::UserAgent->new;
sub uploadbox {
my $url = "http://host/token";
my $response = $ua->put($url,
'Content_Type' => 'form-data',
'Content' => [
Filedata => [ "$codename.box", "$codename.box", Content_type => 'application/octet-stream' ]
]
);
}
uploadbox();
This code runs, and exits without uploading anything ( the uploaded files are 300MB big, so it shoud take time).
Am I passing the right parameters to the put subroutine ?
How to further debug this ?
I like to debug LWP::UserAgent scripts using LWP::ConsoleLogger::Easy. (Disclaimer: this is one of my own modules).
use LWP::ConsoleLogger::Easy qw( debug_ua );
my $ua = LWP::UserAgent->new;
debug_ua( $ua );
# insert the rest of your code here
You'll now get a huge amount of debugging information from both the request and the response printed to your terminal. That should hopefully give you a good starting point to figure out what's going on.
In the end I just decided to use curl, and I get a dynamic status line for free
sub uploadbox {
my ($url) = #_;
my $curl = "curl -X PUT $url --upload-file $codename.box";
$OUTPUT_AUTOFLUSH = 1;
open(CURL, '-|', $curl,) or die "error: $ERRNO";
while (<CURL>) { say; }
}
not the code I am most proud of but ...
I believe I have a general Perl problem, rather than an LWP::UserAgent problem... however its somewhat complex.
The task is to write a test-script that does a SWORD deposit.
I create tests by first writing code to prove the thing works, then add in the Test::More wrappers to make it a test.
BACKGROUND
A SWORD deposit is simply an http post request with a bunch of defined headers, and the content of the body being the thing to be ingested. This all works fine, I can perform the actions through CURL, and I've written scripts to do this.... but within a a larger application environment (that'll be EPrints.)
CODE
My problem, I believe, comes when I try to attach the contents of the file on the disk.
#!/home/cpan/bin/perl
use strict;
use warnings;
use LWP::UserAgent;
##use WWW::Mechanize;
use File::Slurp;
use MIME::Base64;
my $auth = 'username:password';
my $domain = 'devel.example.com';
my $ua = LWP::UserAgent->new();
my $basedir = "./test_files";
my $package = 'http://opendepot.org/europePMC/2.0';
my $filename = "$basedir/PMC165035.zip";
my $mime = 'application/zip';
print "filename: $filename\n";
my $deposit_url = $domain . '/sword-app/deposit/archive';
my $file = read_file( $filename, { binmode => ':raw' } );
# Set up the SWORD deposit
my $autho = "Basic " . MIME::Base64::encode( $auth, '' );
my %headers = (
'X-Packaging' => $package,
'X-No-Op' => 'false',
'X-Verbose' => 'true',
'Content-Disposition' => "filename=$filename",
'Content-Type' => $mime,
'User-Agent' => 'Broker Test Harness',
'Authorization' => $autho,
);
my $r = $ua->post( $deposit_url, %headers, Content => $file );
# DEBUG TEST
write_file('foo.zip', $file);
my $ret = $r->decoded_content;
print "Content: $ret\n";
if ( $r->is_success ) { print "Deposited $package successfully" }
WHAT WORKS, WHAT DOESN'T
This code is lifted pretty much directly from working code I have - the only difference is that the working code gets the content for $file via an object-call within EPrints.
I know the file exists on the disk, if I do an ls -l on the filename printed, I can see the file, and its readable
In the code above, there is a line write_file('foo.zip', $file); - that writes a file which unzip -l foo.zip happily tells me has 3 files in it.
The line print "Content: $ret\n"; should print an atom response - for me, it prints nothing....
The Access log reports an error 500, but there's diddly-squat in the error-log.
The help
What I need to know is how I get the actual contents of the .zip file into the content part of the LWP::UserAgent post request...
(I'm going to spend much time not trying to dig into EPrints, to track where the error-500 is coming from, and why nothing appears in the log file.... but that's probably going to be down to an issue with what's been posted)
The solution lies in realizing what LWP POST is doing.
my $filename = "$basedir/PMC165035.zip";
my $file = read_file( $filename, { binmode => ':raw' } );
my %headers = (
'X-Packaging' => $package,
'X-No-Op' => 'false',
'X-Verbose' => 'true',
'Content-Disposition' => "filename=$filename",
'Content-Type' => $mime,
'User-Agent' => 'Broker Test Harness',
'Authorization' => $autho,
);
All work by setting $filename to be something like /home/services/foo/testing/test_files/PMC165035.zip, and passing this (full) filename to the server example.com.
The problem is that the server is looking for a filename, not a filename-with-path... so when the service does its thing with the file by dumping the content into its temporary upload location, and then it looks for ~~~temp_location/home/services/foo/testing/test_files/PMC165035.zip, it can't find it!
The solution is to read in the file, but ensure that the filename given in the headers is just the filename, not with-a-path
I'm sure this has been asked but I could not find a good question to my answer. I've got two scripts, one makes a post using LWP::UserAgent and the other basically receives the data, in this case I'm just looking to write a file. The file does get written but it can't be opened and the size is 1262 which leads me to believe only some of it is being read.
Here's what I'm doing (It's worth noting, I was uri/base64 encoding the file and passing everything via json, which was working, but my new task is the split the files out and pass everything as form params):
post script:
open (IMAGE, "./flower.jpg") or die "$!";
$raw_string1 = do{ local $/ = undef; <IMAGE>; };
my $req = HTTP::Request->new(POST => $url);
$req->content_type("application/x-www-form-urlencoded");
$req->content("json_string=$json&file_1=$raw_string1");
my $ua = LWP::UserAgent->new;
$res = $ua->request($req);
print $res->content;
Receiver script:
$cgi = CGI->new;
my $json_post = $cgi->param('json_string');
my $file_1 = $cgi->param('file_1');
open my $fh, '>', "$path/flower.jpg" or die $!;
binmode $fh;
print $fh $file_1;
close $fh;
Thanks for help in advance!
As previously mentioned, you have an encoding problem. The solution is simple:
my $req = HTTP::Request->new(POST => $url, [
json_string => $json,
file_1 => $raw_string1,
]);
which is short for
my $req = HTTP::Request->new(POST => $url,
Content_Type => 'application/x-www-form-urlencoded',
Content => [
json_string => $json,
file_1 => $raw_string1,
]
);
It's far more typical to use multipart/form-data to upload files, though. And if you do, you can even let HTTP::Request load the file for you!
my $req = HTTP::Request->new(POST => $url,
Content_Type => 'multipart/form-data',
Content => [
json_string => $json,
file_1 => [ 'flower.jpg' ],
]
);
CGI.pm will handle that no problem. See the section of the docs titled "PROCESSING A FILE UPLOAD FIELD".
You're not encoding the $raw_string1 data before sticking it into the HTTP POST body. That means that if the data in flower.jpg includes a hex 0x26 byte (the & char) -- at position 1263, let's say -- then the POST data will look like this:
json_string={ ... JSON data ... }&file_1=...1262 bytes of raw JPEG data...&...more JPEG data...
... which means that anything parsing the form body for form variables will truncate file_1 after 1262 bytes.
I'd recommend continuing to encode the $raw_string1 data with base64 or something similar, even if you pass it through as its own POST variable rather than adding it to the JSON data.
I have the following code and want to attach a file using an API. This code is delivering me the URL but the file is not getting attached.
#!/usr/bin/perl
use LWP::UserAgent;
$ua = LWP::UserAgent->new;
my $response = $ua->post(Content_Type => 'application/xml');
#$ua->agent("Mozilla 8.0 blah...");
use HTTP::Request::Common qw(POST);
use LWP::UserAgent(POST);
my $request=(POST "http://Server/Test.jsp",
Content =>[
external => "false",
Filedata => "C:/Location.jpg"
]);
#$request = $ua->request($request);
my $results=$ua->request($request);
$content = $request->content;
print $content;
exit;
Well, first you have to specify the correct content-type.
my $request=(POST "http://Server/watson/api/bug/addAttachmentAPI.jsp",
Content_Type => 'form-data',
Content =>[
appGUID => "Test GUID",
Second, the file specification must be an array reference of the form [ $file, $name, ... ] where ... are optional header field/value pairs (if you don't include headers, the content-type of the file will be guessed).
Filedata => ["C:Test Location/Upload/APIs.jpg", 'APIs.jpg'],
]);
See HTTP::Request::Common for more information.