Perl LWP::UserAgent only reading first line of posted jpg file - perl

I'm sure this has been asked but I could not find a good question to my answer. I've got two scripts, one makes a post using LWP::UserAgent and the other basically receives the data, in this case I'm just looking to write a file. The file does get written but it can't be opened and the size is 1262 which leads me to believe only some of it is being read.
Here's what I'm doing (It's worth noting, I was uri/base64 encoding the file and passing everything via json, which was working, but my new task is the split the files out and pass everything as form params):
post script:
open (IMAGE, "./flower.jpg") or die "$!";
$raw_string1 = do{ local $/ = undef; <IMAGE>; };
my $req = HTTP::Request->new(POST => $url);
$req->content_type("application/x-www-form-urlencoded");
$req->content("json_string=$json&file_1=$raw_string1");
my $ua = LWP::UserAgent->new;
$res = $ua->request($req);
print $res->content;
Receiver script:
$cgi = CGI->new;
my $json_post = $cgi->param('json_string');
my $file_1 = $cgi->param('file_1');
open my $fh, '>', "$path/flower.jpg" or die $!;
binmode $fh;
print $fh $file_1;
close $fh;
Thanks for help in advance!

As previously mentioned, you have an encoding problem. The solution is simple:
my $req = HTTP::Request->new(POST => $url, [
json_string => $json,
file_1 => $raw_string1,
]);
which is short for
my $req = HTTP::Request->new(POST => $url,
Content_Type => 'application/x-www-form-urlencoded',
Content => [
json_string => $json,
file_1 => $raw_string1,
]
);
It's far more typical to use multipart/form-data to upload files, though. And if you do, you can even let HTTP::Request load the file for you!
my $req = HTTP::Request->new(POST => $url,
Content_Type => 'multipart/form-data',
Content => [
json_string => $json,
file_1 => [ 'flower.jpg' ],
]
);
CGI.pm will handle that no problem. See the section of the docs titled "PROCESSING A FILE UPLOAD FIELD".

You're not encoding the $raw_string1 data before sticking it into the HTTP POST body. That means that if the data in flower.jpg includes a hex 0x26 byte (the & char) -- at position 1263, let's say -- then the POST data will look like this:
json_string={ ... JSON data ... }&file_1=...1262 bytes of raw JPEG data...&...more JPEG data...
... which means that anything parsing the form body for form variables will truncate file_1 after 1262 bytes.
I'd recommend continuing to encode the $raw_string1 data with base64 or something similar, even if you pass it through as its own POST variable rather than adding it to the JSON data.

Related

Uploading a file with perl LWP PUT method silency fails

I am trying to upload a file to a public API with a code similar to this:
my $ua = LWP::UserAgent->new;
sub uploadbox {
my $url = "http://host/token";
my $response = $ua->put($url,
'Content_Type' => 'form-data',
'Content' => [
Filedata => [ "$codename.box", "$codename.box", Content_type => 'application/octet-stream' ]
]
);
}
uploadbox();
This code runs, and exits without uploading anything ( the uploaded files are 300MB big, so it shoud take time).
Am I passing the right parameters to the put subroutine ?
How to further debug this ?
I like to debug LWP::UserAgent scripts using LWP::ConsoleLogger::Easy. (Disclaimer: this is one of my own modules).
use LWP::ConsoleLogger::Easy qw( debug_ua );
my $ua = LWP::UserAgent->new;
debug_ua( $ua );
# insert the rest of your code here
You'll now get a huge amount of debugging information from both the request and the response printed to your terminal. That should hopefully give you a good starting point to figure out what's going on.
In the end I just decided to use curl, and I get a dynamic status line for free
sub uploadbox {
my ($url) = #_;
my $curl = "curl -X PUT $url --upload-file $codename.box";
$OUTPUT_AUTOFLUSH = 1;
open(CURL, '-|', $curl,) or die "error: $ERRNO";
while (<CURL>) { say; }
}
not the code I am most proud of but ...

Adding a .zip file to the body of an LWP::UserAgent POST request

I believe I have a general Perl problem, rather than an LWP::UserAgent problem... however its somewhat complex.
The task is to write a test-script that does a SWORD deposit.
I create tests by first writing code to prove the thing works, then add in the Test::More wrappers to make it a test.
BACKGROUND
A SWORD deposit is simply an http post request with a bunch of defined headers, and the content of the body being the thing to be ingested. This all works fine, I can perform the actions through CURL, and I've written scripts to do this.... but within a a larger application environment (that'll be EPrints.)
CODE
My problem, I believe, comes when I try to attach the contents of the file on the disk.
#!/home/cpan/bin/perl
use strict;
use warnings;
use LWP::UserAgent;
##use WWW::Mechanize;
use File::Slurp;
use MIME::Base64;
my $auth = 'username:password';
my $domain = 'devel.example.com';
my $ua = LWP::UserAgent->new();
my $basedir = "./test_files";
my $package = 'http://opendepot.org/europePMC/2.0';
my $filename = "$basedir/PMC165035.zip";
my $mime = 'application/zip';
print "filename: $filename\n";
my $deposit_url = $domain . '/sword-app/deposit/archive';
my $file = read_file( $filename, { binmode => ':raw' } );
# Set up the SWORD deposit
my $autho = "Basic " . MIME::Base64::encode( $auth, '' );
my %headers = (
'X-Packaging' => $package,
'X-No-Op' => 'false',
'X-Verbose' => 'true',
'Content-Disposition' => "filename=$filename",
'Content-Type' => $mime,
'User-Agent' => 'Broker Test Harness',
'Authorization' => $autho,
);
my $r = $ua->post( $deposit_url, %headers, Content => $file );
# DEBUG TEST
write_file('foo.zip', $file);
my $ret = $r->decoded_content;
print "Content: $ret\n";
if ( $r->is_success ) { print "Deposited $package successfully" }
WHAT WORKS, WHAT DOESN'T
This code is lifted pretty much directly from working code I have - the only difference is that the working code gets the content for $file via an object-call within EPrints.
I know the file exists on the disk, if I do an ls -l on the filename printed, I can see the file, and its readable
In the code above, there is a line write_file('foo.zip', $file); - that writes a file which unzip -l foo.zip happily tells me has 3 files in it.
The line print "Content: $ret\n"; should print an atom response - for me, it prints nothing....
The Access log reports an error 500, but there's diddly-squat in the error-log.
The help
What I need to know is how I get the actual contents of the .zip file into the content part of the LWP::UserAgent post request...
(I'm going to spend much time not trying to dig into EPrints, to track where the error-500 is coming from, and why nothing appears in the log file.... but that's probably going to be down to an issue with what's been posted)
The solution lies in realizing what LWP POST is doing.
my $filename = "$basedir/PMC165035.zip";
my $file = read_file( $filename, { binmode => ':raw' } );
my %headers = (
'X-Packaging' => $package,
'X-No-Op' => 'false',
'X-Verbose' => 'true',
'Content-Disposition' => "filename=$filename",
'Content-Type' => $mime,
'User-Agent' => 'Broker Test Harness',
'Authorization' => $autho,
);
All work by setting $filename to be something like /home/services/foo/testing/test_files/PMC165035.zip, and passing this (full) filename to the server example.com.
The problem is that the server is looking for a filename, not a filename-with-path... so when the service does its thing with the file by dumping the content into its temporary upload location, and then it looks for ~~~temp_location/home/services/foo/testing/test_files/PMC165035.zip, it can't find it!
The solution is to read in the file, but ensure that the filename given in the headers is just the filename, not with-a-path

set http header in perl to force download the content

I am trying to force download a file by sending http headers via perl. the current code is as follow:
#!/usr/bin/perl
use strict;
use Session;
use CGI::Carp qw(fatalsToBrowser);
use HTTP::Headers;
HTTP::Headers->new(
Content_type => 'text/plain',
Content_disposition => 'attachment; filename=export.txt',
);
print 'just some text';
exit;
I have also included HTTP::Headers however when I run this, it prints out the text instead of downloading the content...
You're just constructing the HTTP::Headers, but it's never printed to stdout. So you have to call also the as_string method:
my $h = HTTP::Headers->new(
Content_type => 'text/plain',
Content_disposition => 'attachment; filename=export.txt',
);
print $h->as_string;
But this is just printing the HTTP header without the separator between header and body. If you want to let libwww-perl to do this for you, you can also use HTTP::Message:
use HTTP::Headers;
use HTTP::Message;
my $h = HTTP::Headers->new(
Content_type => 'text/plain',
Content_disposition => 'attachment; filename=export.txt',
);
my $content = 'just some text';
my $msg = HTTP::Message->new($h, $content);
print $msg->as_string;
To be more correct, you should probably use "\r\n" as line terminators:
print $msg->as_string("\015\012");
Another alternative is to use CGI.pm, and use the header method, which can be used to set response HTTP headers. Actually, using CGI.pm is more common than using the HTTP::* classes. The latter are more common in use when dealing with dealing with LWP::UserAgent to fetch web pages.
When you are using CGI, you can set it when printing the header via CGI that way:
my $q = CGI->new();
print $q->header(
-type => 'text/plain',
-charset => 'iso-8859-1',
-attachment => 'filename.txt',
);
ok I figured out a simpler way...
instead of using HTTP::Headers I simply printed out the following lines:
print"Content-type:text/plain\n";
print"Content-disposition:attachment; filename=export.txt\n\n";
which did the trick...
This is the code using the CGI module:
use CGI qw(:standard);
print header(-type => "text/plain", -content_disposition => "attachment; filename=filename=export.txt");
print "just some text";

Multipart Entity for perl for attaching a file through API not working

I have the following code and want to attach a file using an API. This code is delivering me the URL but the file is not getting attached.
#!/usr/bin/perl
use LWP::UserAgent;
$ua = LWP::UserAgent->new;
my $response = $ua->post(Content_Type => 'application/xml');
#$ua->agent("Mozilla 8.0 blah...");
use HTTP::Request::Common qw(POST);
use LWP::UserAgent(POST);
my $request=(POST "http://Server/Test.jsp",
Content =>[
external => "false",
Filedata => "C:/Location.jpg"
]);
#$request = $ua->request($request);
my $results=$ua->request($request);
$content = $request->content;
print $content;
exit;
Well, first you have to specify the correct content-type.
my $request=(POST "http://Server/watson/api/bug/addAttachmentAPI.jsp",
Content_Type => 'form-data',
Content =>[
appGUID => "Test GUID",
Second, the file specification must be an array reference of the form [ $file, $name, ... ] where ... are optional header field/value pairs (if you don't include headers, the content-type of the file will be guessed).
Filedata => ["C:Test Location/Upload/APIs.jpg", 'APIs.jpg'],
]);
See HTTP::Request::Common for more information.

Perl post request to send zip file as base64?

I have a Perl script trying to send a zip file like so with LWP UserAgent module
my $req = POST $url, Content_Type => 'form-data',
Content => [
submit => 1,
upfile => [ $fname ]
];
where $fname is the path of the file. On the server side though it seems my POST array only has "submit".
Should I base64 encode the file and assign it to a variable? What's the best way to do this?
Make sure the filename can be resolved. You should get an error if it cannot be, though. At least I do in my version of HTTP::Request::Common.
You don't have to encode the binary content as Base64. (Unless, of course, the server-side app happens to expect that format.)
Here's a complete sample script:
use strict;
use warnings;
use LWP::UserAgent;
use HTTP::Request::Common 'POST';
my $ua = LWP::UserAgent->new;
my $url = 'http://localhost:8888'; # Fiddler
my $req = POST $url,
Content_Type => 'form-data',
Content => [
submit => 1,
upfile => [ 'C:\temp\bla.zip' ],
];
my $line = '=' x 78 . "\n";
print $line, $req->as_string;
my $rsp = $ua->request( $req );
print $line, $rsp->as_string;