I have the following code:
...
sub setImage {
my $self=shift;
my $filename=shift;
unless(-r $filename) {
warn "File $filename not found";
return;
}
my $imgn=shift;
my $operation=&URI::Escape::uri_escape_utf8(
(shift) ? "Удалить! (Delete)" : "Сохранить! (Store)");
my $FH=&::File::open($filename, 0, 0);
my $image;
# &utf8::downgrade($image);
sysread($FH, $image, 102400, 0);
close $FH;
my $imginfo=eval{&Image::Info::image_info(\$image)};
if($# or $imginfo->{"error"}) {
warn "Invalid image: ".($# || $imginfo->{"error"});
return undef;
}
my $fields=[
DIR => $self->url("fl"),
OPERATION => $operation,
FILE_NAME => ".photo$imgn",
# FILE => [$filename],
FILE => [undef, "image.".$imginfo->{"file_ext"},
# Content_Type => $imginfo->{"file_media_type"},
# Content_Type => 'application/octet-stream',
Content => $image,
],
];
my $response=&ZLR::UA::post(
&ZLR::UA::absURL("/cgi-bin/file_manager")."",
$fields,
Content_Type => "form-data",
);
print $response->decoded_content;
}
...
When I try to use function setImage it fails with error HTTP::Message content must be bytes at /usr/lib64/perl5/vendor_perl/5.8.8/HTTP/Request/Common.pm line 91. Worse that I can't reproduce this error without using all of my code and upgrading libwww-perl does nothing. What can cause it?
Versions of libww-perl: dev-perl/libwww-perl-5.836. HTTP::Request and HTTP::Request::Common came from libwww-perl package, versions: 5.827 and 5.824.
Trace:
HTTP::Message content must be bytes at /usr/lib64/perl5/vendor_perl/5.8.8/HTTP/Request/Common.pm line 91
at Carp::croak(unknown source)
at HTTP::Message::__ANON__(/usr/lib64/perl5/vendor_perl/5.8.8/HTTP/Message.pm:16)
at HTTP::Message::_set_content(/usr/lib64/perl5/vendor_perl/5.8.8/HTTP/Message.pm:136)
at HTTP::Message::content(/usr/lib64/perl5/vendor_perl/5.8.8/HTTP/Message.pm:125)
at HTTP::Request::Common::POST(/usr/lib64/perl5/vendor_perl/5.8.8/HTTP/Request/Common.pm:91)
at LWP::UserAgent::post(/usr/lib64/perl5/vendor_perl/5.8.8/LWP/UserAgent.pm:397)
at ZLR::UA::post(./zlrchecker.pl:71)
at ZLR::Info::setImage(./zlrchecker.pl:1754)
at main::main(./zlrchecker.pl:3893)
at main::(./zlrchecker.pl:4148)
Use Devel::SimpleTrace and paste the trace. Install the module with cpan. Then run your program with -MDevel::SimpleTrace like perl -MDevel::SimpleTrace ./myapp_run.pl
And paste the version of HTTP::Request:Common, HTTP::Message, and LWP.
My guess is you'll see this in the stack trace:
This seems to be the code likely causing the error:
*_utf8_downgrade = defined(&utf8::downgrade) ?
sub {
utf8::downgrade($_[0], 1) or
Carp::croak("HTTP::Message content must be bytes")
}
:
sub {
};
The docs in utf8 say this:
Fails if the original UTF-X sequence
cannot be represented in the native 8
bit encoding. On failure dies or, if
the value of FAIL_OK is true, returns
false.
You should be able to make a test case by running utf8::downgrade($http_message_content)
One effective solution I found to this problem was to parse any text we put into HTTP::Message, during a HTTP request, through unidecode() from Text::Unidecode.
Related
Somebody can help me, please. I have app in mojolicious Lite
I wanna block images from all without session login
when i type http://m.y.i.p:3000/images/imageX.jpg
i wanna show images only with session login.
my html code is
shwo image 1
Same way as any other content. Set up a handler for requests, render (or don't render) the content.
get '/images/*img' => sub {
my $c = shift;
if (!$c->session("is_authenticated")) {
return $c->render( text => "Forbidden", status => 403 );
}
my $file = $c->param("img");
if (!open(my $fh, '<', $IMAGE_DIR/$file)) {
return $c->render( text => "Not found", status => 404 );
}
my $data = do { local $/; <$fh> };
close $fh;
$c->render( data => $data, format => 'jpg' );
};
Your request handler will take priority over the default handler that serves content from public folders, but once you have this handler in place, you don't need to store the files it serves in a public folder.
another sololution is
get '/pays/*img' => sub {
my $self = shift;
my $img = $self->param('img');
plugin 'RenderFile';
my #imgext = split(/\./, $img);
my $ext = $imgext[-1];
$self->render_file(
'filepath' => "/directiry/$img",
'format' => "$ext", # will change Content-Type "application/x-download" to "application/pdf"
'content_disposition' => 'inline', # will change Content-Disposition from "attachment" to "inline"
# delete file after completed
);
It is use plugin 'RenderFile';
I am sending email using Dancer2 via the Dancer2::Plugin::Email package. The main code that I have for this is:
sub sendEmail {
my ($params,$email_address,$template) = #_;
my $text = '';
my $tt = Template->new({
INCLUDE_PATH => config->{views},
INTERPOLATE => 1,
OUTPUT => \$text
}) || die "$Template::ERROR\n";
my $out = $tt->process($template,$params);
my $email = email {
from => XXXXX,
to => $email_address,
subject => XXXXX,
body => $text,
'Content-Type' => 'text/html'
};
}
where I have hidden a couple of the fields. I have gotten the following error:
Route exception: open body: Invalid argument at
/usr/local/share/perl/5.22.1/MIME/Entity.pm line 1878. in
/usr/local/share/perl/5.22.1/Dancer2/Core/App.pm l. 1454
It is not occurring all of the time and I haven't been able to find a consistent piece of code that always fails.
I have set the host parameter of the mail server that I am using in the configuration as explained here: https://metacpan.org/pod/Dancer2::Plugin::Email Simple tests show it works, but I get sporadic errors that I can't track down.
I'm attempting to use a particular web service, and I can successfully perform the upload with the following command:
curl -X POST --header "Transfer-Encoding: chunked" -d #Downloads/file.pdf https://some.webservice/upload
I get back a json response indicate success.
However, I'm unable to figure out how to do the same with WWW::Mechanize.
$mech->post("https://" . $server . "/upload", Content_Type => 'multipart/form-data', Content => [upID => $upid, name => $dlfile, userID => 0, userK => 0, file_0 => [$dlfile]]);
This receives a similar json response with a big fat error message in it.
Do I need to explicitly set the Transfer-Encoding header first? Is there some other trick to it? Google's not shedding much light on this, Perlmonks neither, and the documentation's a little obtuse.
You can do it using HTTP::Request::StreamingUpload
my $starttime = time();
my $req = HTTP::Request::StreamingUpload->new(
POST => $url,
path => $file,
headers => HTTP::Headers->new(
'Transfer-Encoding' => 'chunked'
),
);
my $gen = $req->content;
die unless ref($gen) eq "CODE";
my $total = 0;
$req->content(sub {
my $chunk = &$gen();
$total += length($chunk);
print "\r$total / $size bytes ("
. int($total/$size*100)
. "%) sent, "
. int($total/1000/(time()-$starttime+1))
. " k / sec ";
return $chunk;
});
my $resp = $ua->request($req);
print "\n";
unless ($resp->is_success) {
die "Failed uploading the file: ", $resp->status_line;
}
my $con = $resp->content;
return $con;
Do you really need WWW::Mechanize? It is a subclass of LWP::UserAgent with additional functionality that gives browser-like functionality like filling in and submitting forms, clicking links, a page history with a "back" operation etc. If you don't need all of that then you may as well use LWP::UserAgent directly
Either way, the post method is inherited unchanged from LWP::UserAgent, and it's fine to use it directly as you have done
The way to send a chunked POST is to set the Content to a reference to a subroutine. The subroutine must return the next chunk of data each time it is called, and finally ann empty string or undef when there is no more to send
Is the data supposed to be a JSON string?
It's easiest to write a factory subroutine that returns a closure, like this
sub make_callback {
my ($data) = shift;
sub { substr($data, 0, 512, "") }
}
Then you can call post like this
my $payload = to_json(...);
$mech->post(
"https://$server/upload",
Content_Type => 'multipart/form-data',
Content => make_callback($payload)
);
Please be aware that all of this is untested
Environment settings
OS : RHEL 6.6 (kernel 2.6.32) - x86_64
httpd : httpd-2.2.15-39
perl : 5.10.1-136
CGI API :
perl-CGI-3.15-136
perl-CGI-Session-4.35-6
I am using a static html page with Perl-CGI defined variables in the static html. This html is read in through perl, and then passed to a perl CGI script for eval.
Note:
While reading the static html, I am using UTF-8 encoding like
open( IN ,"<:encoding(UTF-8)", $file_path )
After reading the status HTML page, the output is passed back to the CGI script through a variable and then pressed in to eval to evaluate the variables.
Finally, the eval(uated) output from CGI is print which can be read through http daemon.
In the CGI script I am using
binmode(STDIN, ':encoding(UTF-8)');
binmode(STDOUT, ':encoding(UTF-8)');
The static HTML looks something like this
When I check the output of print in the CGI script, I see the complete output as desired, like this
However on the Browser, the hidden input fields are getting truncated in an unwanted manner. Like this
When I checked the wireshark output for the text/html, which is being printed back from the server to the browser, this is also getting truncated.
Like this
The HTML header has proper Content-Type and charset declaration.
The same code is working fine with EN language
The same code is working fine with zh(chinese) language as well.
When the language is set to japanese in the browser, and we read from HTTP_ACCEPT_LANGUAGE that it is 'ja', than we print the japanese specific data.
We are not using cgid module of apache.
Are we supposed to use some special encoding for japanese language??
Or it is a double encoding issue. I have tried removing the encoding when I am reading the static html file, however, that also did not help.
The same code is working fine with RHEL 5.x (2.6.18-308), and perl-CGI-Session (4.42-2), perl (5.8.8-38) httpd (2.2.3-63), there was no perl-CGI in RHEL-5.x.
I kind of found the solution to the problem. In our code we used to add a data in the CGI::Session object, which had the language. In the following form
$session->param( KEY_SESSION_LANG, $code ).
Where $session is a CGI::Session object and KEY_SESSION_LANG is 'language' and $code is something that we get from HTTP_ACCEPT_LANGUAGE. When we see 'en' we used to set it as perl constant 'en', when we got 'ja' we used to set it as perl constant 'ja'.
When we used to form the session object we used to get a session file as (perl version RHEL 5.x (2.6.18-308) & perl-CGI-Session-4.42-2 ) for JA language we used to get the following in the cgi file
cat cgisess_e31c8d21af82b59fd064babc7ca25c01
$D = {'_SESSION_ID' => 'e31c8d21af82b59fd064babc7ca25c01','_SESSION_ETIME' => 6000,'language' => 'ja','permit' => 'yes','_SESSION_REMOTE_ADDR' => '192.168.101.1','_SESSION_CTIME' => 1441090386,'execute' => 'yes','_SESSION_ATIME' => 1441090387,'_SESSION_EXPIRE_LIST' => {}};*a = \undef;;$D
For perl CGI Session in RHEL 6.6 this is coming out to be
cat cgisess_e31c8d21af82b59fd064babc7ca25c01
$D = {'_SESSION_ID' => 'e31c8d21af82b59fd064babc7ca25c01','_SESSION_ETIME' => 6000,'language' => *a,'permit' => 'yes','_SESSION_REMOTE_ADDR' => '192.168.101.1','_SESSION_CTIME' => 1441090386,'execute' => 'yes','_SESSION_ATIME' => 1441090387,'_SESSION_EXPIRE_LIST' => {}};*a = \undef;;$D
language data for ja is becoming *a. The same is also reflected when we use perl dumper for getting in memory data.
I checked the /usr/share/perl5/vendor_perl/CGI/Session.pm and it had following information in it
=head1 A Warning about UTF8
Trying to use UTF8 in a program which uses CGI::Session has lead to problems. See RT#21981 and RT#28516.
In the first case the user tried "use encoding 'utf8';" in the program, and in the second case the user tried "$dbh->do(qq|set names 'utf8'|);".
Until this problem is understood and corrected, users are advised to avoid UTF8 in conjunction with CGI::Session.
For details, see: http://rt.cpan.org/Public/Bug/Display.html?id=28516 (and ...id=21981).
=head1 TRANSLATIONS
This document is also available in Japanese.
Now when I used the perl dumper following things happened. I quote below from my offical analysis presented on our local development portal
I think the problem is because of perl-CGI-Session OSS package, please see the analysis below.
Some inputs from the CGI session source code.
## Inputs ##
From the file /usr/share/perl5/vendor_perl/CGI/Session.pm
## Following are the status of CGI session, set internally after modification to any of the parameters ##
sub STATUS_NEW () { 1 } # denotes session that's just created
sub STATUS_MODIFIED () { 2 } # denotes session that needs synchronization
sub STATUS_DELETED () { 4 } # denotes session that needs deletion
sub STATUS_EXPIRED () { 8 } # denotes session that was expired.
--snip --
I::Session - persistent session data in CGI applications
=head1 SYNOPSIS
# Object initialization:
use CGI::Session;
$session = new CGI::Session();
$CGISESSID = $session->id();
We are setting the "language" parameter in the session object. (We create a CGI object, set cookie to it, to get sid, and through sid get the session object). For setting up the language parameter we do
$session->param( 'language', ); ---> language_value = en(english) or ja(japanese). When we have completed the printing of the HTML page in /opt/packageManager/pm_gui/cgi/status.cgi file, I checked the cgi session object it is as follows
For EN Language
Before executing session flush
[09/10/2015 16:13:41] [23722] <ERROR> status.cgi : 267 :
$VAR1 = bless( {
'_STATUS' => 2,
'_DATA' => {
'_SESSION_ETIME' => 6000,
'_SESSION_ID' => '995d11334f2c39b95b3fdb86cecd9655',
'permit' => 'yes',
'language' => 'en',
Then after this when I flush the session as $session->flush() and check the session object it is
[09/10/2015 16:13:41] [23722] <ERROR> status.cgi : 270 :
$VAR1 = bless( {
'_STATUS' => 0,
'_DATA' => {
'_SESSION_ETIME' => 6000,
'_SESSION_ID' => '995d11334f2c39b95b3fdb86cecd9655',
'permit' => 'yes',
'language' => 'en',
Inference 1: session status changed after doing flush. This is good, and should be done.
For JP Language
Before executing session flush
[09/10/2015 16:14:54] [31910] status.cgi : 267 :
$VAR1 = bless( {
'_STATUS' => 2,
'_DATA' => {
'_SESSION_ID' => '1cd1b7860af4c71264f3969fe74e7a44',
'_SESSION_ETIME' => 6000,
'language' => *a
Then after this, when I flush the session as $session->flush() and check the session object it is NOT THERE. SCRIPT CRASHES HERE IT SELF
Inference 2 : Doing flush with language JP, is terminating the session, and that is why the session gets destroyed. And that is why, ending data in response is truncated
Due to the wrong value being set in memory, in session object, and then the implicit flush by CGI session is failing on the disk. Which results in termination of the session object, and in between termination of session, and and data loss of HTML.
I checked the actual code in sessions.pm file and it seems to be coming in from here
sub param {
my ($self, #args) = #_;
--snip--
# USAGE: $s->param($name, $value);
# USAGE: $s->param($name1 => $value1, $name2 => $value2 [,...]);
# DESC: updates one or more **public** records using simple syntax
if ((#args % 2) == 0) {
my $modified_cnt = 0;
ARG_PAIR:
while (my ($name, $val) = each %args) {
if ( $name =~ m/^_SESSION_/) {
carp "param(): attempt to write to private parameter";
next ARG_PAIR;
}
$self->{_DATA}->{ $name } = $val; ----> HERE
++$modified_cnt;
}
$self->_set_status(STATUS_MODIFIED);
return $modified_cnt;
}
As a solution, we stopped putting 'ja' value as a perl constant, but now are putting it as a string "ja" and it seems to be working fine now.
I use WWW::Curl to upload files:
use WWW::Curl::Easy 4.14;
use WWW::Curl::Form;
my $url = 'http://example.com/backups/?sid=12313qwed323';
my $params = {
name => 'upload',
action => 'keep',
backup1 => [ '/tmp/backup1.zip' ], # 1st file for upload
};
my $form = WWW::Curl::Form->new();
foreach my $k (keys %{$params}) {
if (ref $params->{$k}) {
$form->formaddfile(#{$params->{$k}}[0], $k, 'multipart/form-data');
} else {
$form->formadd($k, $params->{$k});
}
}
my $curl = WWW::Curl::Easy->new() or die $!;
$curl->setopt(CURLOPT_HTTPPOST, $form);
$curl->setopt(CURLOPT_URL, $url);
my $body;
$curl->setopt(CURLOPT_WRITEDATA, \$body);
my $retcode = $curl->perform();
my $response_code = $curl->getinfo(CURLINFO_HTTP_CODE);
nothing special here and this code works well.
I want to upload large files and I don't want to preload everything in the memory. At least that is what I heard that libcurl is doing.
CURLOPT_READFUNCTION accepts callbacks which returns parts of the content. That means that I cannot use WWW::Curl::Form to set POST parameters but that I have to return the whole content through this callback. Is that right?
I think that the code could look like this:
use WWW::Curl::Easy 4.14;
my $url = 'http://example.com/backups/?sid=12313qwed323'
my $params = {
name => 'upload',
action => 'keep',
backup1 => [ '/tmp/backup1.zip' ], # 1st file for upload
};
my $fields;
foreach my $k (keys %{$params}) {
$fields .= "$k=".(ref $params->{$k} ? '#'.#{$params->{$k}}[0] : uri_escape_utf8($params->{$k}))."&";
}
chop($fields);
my $curl = WWW::Curl::Easy->new() or die $!;
$curl->setopt(CURLOPT_POST, 1);
$curl->setopt(CURLOPT_POSTFIELDS, $fields); # is it needed with READFUNCTION??
$curl->setopt(CURLOPT_URL, $url);
my #header = ('Content-type: multipart/form-data', 'Transfer-Encoding: chunked');
$curl->setopt(CURLOPT_HTTPHEADER, \#header);
#$curl->setopt(CURLOPT_INFILESIZE, $size);
$curl->setopt(CURLOPT_READFUNCTION, sub {
# which data to return here?
# $params (without file) + file content?
return 0;
});
Which data does CURLOPT_READFUNCTION callback have to return? $params + File(s) content? In which format?
Do I really have to create the data (returned by CURLOPT_READFUNCTION) by myself or is there a simple way to create it in the right format?
Thanks
Test 16formpost.t is relevant. As you can see, it's completely disabled. This fact and my fruitless experiments with various return values for the callback function lets me believe the CURLOPT_READFUNCTION feature is known broken in the Perl binding.
I have to return the whole content through this callback. Is that right?
No, you can feed it the request body piecewise, suitable for chunked encoding. The callback will be necessarily called several times, according to the limit set in CURLOPT_INFILESIZE.
Which data does CURLOPT_READFUNCTION callback have to return?
A HTTP request body. Since you do a file upload, this means Content-Type multipart/form-data. Following is an example using HTTP::Message. CURLOPT_HTTPPOST is another way to construct this format.
use HTTP::Request::Common qw(POST);
use WWW::Curl::Easy 4.14;
my $curl = WWW::Curl::Easy->new or die $!;
$curl->setopt(CURLOPT_POST, 1);
$curl->setopt(CURLOPT_URL, 'http://localhost:5000');
$curl->setopt(CURLOPT_HTTPHEADER, [
'Content-type: multipart/form-data', 'Transfer-Encoding: chunked'
]);
$curl->setopt(CURLOPT_READFUNCTION, sub {
return POST(undef, Content_Type => 'multipart/form-data', Content => [
name => 'upload',
action => 'keep',
backup1 => [ '/tmp/backup1.zip' ], # 1st file for upload
])->content;
});
my $r = $curl->perform;
The CURLOPT_READFUNCTION callback is only used for chunked tranfer encoding. It may work, but I haven't been able to get it to and found that doing so wasn't necessary anyway.
My use case was for upload of data to AWS, where it's not ok to upload the data as multi-part form data. Instead, it's a straight POST of the data. It does require that you know how much data you're sending the server, though. This seems to work for me:
my $infile = 'file-to-upload.json';
my $size = -s $infile;
open( IN, $infile ) or die("Cannot open file - $infile. $! \n");
my $curl = WWW::Curl::Easy->new;
$curl->setopt(CURLOPT_HEADER, 1);
$curl->setopt(CURLOPT_NOPROGRESS, 1);
$curl->setopt(CURLOPT_POST, 1);
$curl->setopt(CURLOPT_URL, $myPostUrl);
$curl->setopt(CURLOPT_HTTPHEADER,
['Content-Type: application/json']); #For my use case
$curl->setopt(CURLOPT_POSTFIELDSIZE_LARGE, $size);
$curl->setopt(CURLOPT_READDATA, \*IN);
my $retcode = $curl->perform;
if ($retcode == 0) {
print("File upload success\n");
}
else {
print("An error happened: $retcode ".$curl->strerror($retcode)."\n");
}
The key is providing an open filehandle reference to CURLOPT_READDATA. After that, the core curl library handles the reads from it without any need for callbacks.