I have a web server running nginx on http://192.168.x.x, and a perl module is installed using http://nginx.org/en/docs/http/ngx_http_perl_module.html
I need to do a GET request inside that module using LWP. This is what i have:
my $url = "http://192.168.x.x/app/test.php";
my $ua = LWP::UserAgent->new;
$ua->timeout(5); # muere a los 5 segundos sin respuesta.
$ua->default_header('Cookie' => "FOO=BAR;PATH=/");
my $resp = $ua->get($url);
if ($resp->is_success) {
my $message = $resp->decoded_content;
print STDERR "Received reply: $message\n";
} else {
print STDERR "HTTP GET error code: ", $resp->code, "\n";
print STDERR "HTTP GET error message: ", $resp->message, "\n";
};
Everytime i do that request i get error code 500 : read timeout. But if i use another server works just fine. The problem is when i use the same server where i run the script.
I think this is because nginx has only one worker thread. When you call your perl code inside nginx it blocks whole server. So one nginx can't process your blocking perl code and php handler at the same time. This is deadlock. You can try Nginx::HTTP which shouldn't block nginx while waiting http response. So, this will work, I think.
Related
I have this script to send email. Everything works except that after the email is sent I receive the error below. It has to be something simple I am missing. I checked and cgi file is 755 and since it gets to the sub and executes its got to be coder error. Any help greatly appreciated.
Internal Server Error The server encountered an internal error or misconfiguration and was unable to complete your request. More information about this error may be available in the server error log. Additionally, a 500 Internal Server Error error was encountered while trying to use an ErrorDocument to handle the request.
The error in the server log is:
[Tue Jan 17 16:00:23.475272 2023] [cgi:error] [pid 230679] [client 69.90.223.10:35014] End of script output before headers: test3.cgi
Here is the code I am using, Perl CGI on Linux
#!/usr/bin/perl -Tw
# use warnings;
use strict;
use Net::SMTP;
send_mail('mail.xxxxxxx.com', # Host
'order#xxxxxxxxx.com', #From
'yyyyyyy#gmail.com', #to
'Just a test, from mail.xxxxxx.com please ignore', #Message body
"Testing mail server email.\n" # Subject
);
exit;
sub send_mail {
my ($SMTP_HOST, $from, $to_addr, $body, $subject, $msg) = #_;
$msg = "MIME-Version: 1.0\n"
. "From: $from\n"
. "To: " . ( ref($to_addr) ? join(';', #$to_addr) : $to_addr ) . "\n"
. "Subject: $subject\n\n" # Double \n
. $body;
#
# Open a SMTP session
#
my $smtp = Net::SMTP->new( $SMTP_HOST,
Debug => 0, # Change to a 1 to turn on debug messages
Port => 587,
);
die("SMTP ERROR: Unable to open smtp session.\n")
if(!defined($smtp) || !($smtp));
die("Failed to set FROM address\n")
if (! ($smtp->mail( $from ) ) );
die("Failed to set receipient\n")
if (! ($smtp->recipient( ( ref($to_addr) ? #$to_addr : $to_addr ) ) ) );
$smtp->data( $msg );
$smtp->quit;
}
Checked File attributes they are 755
Since it ran the code an performed the send email the Char set should be correct
Being new to Perl not sure what else to check
This isn't a CGI program. You send no response for the request. Since there is no response (not even an invalid one), the script exits with nothing sent to standard output. The server realizes this is a problem, creates the 500 server error response, and adds to the error log that the script output ended before headers, which is the first thing the server expected to see.
You might want to start with a basic CGI tutorial to see how CGI works.
But, also realize that sending mail through a CGI program is a very 1990s thing to do. We don't typically do that anymore because we all figured out that letting anyone trigger mail through a public server was a bad idea.
I have migrated my scrilpts from CentOS 7 to 8 and there's a new Perl version. I have the folowing snippet that uses head to check if a URL exists:
#!/bin/perl
use strict;
use warnings;
use LWP::Simple;
my $sitemapurl = "https://www.prosinger.net";
if (head($sitemapurl)) {
...
}
else {
print "The $sitemapurl doesn't exist\n";
exit(1);
}
It now always returns that the URL doesn't exist. I'm quite sure that this has to do something with https (I have perl-LWP-Protocol-https installed), but I'm not sure how to get any feedback information from head method to check what the error code is.
Any ideas?
You can use LWP::UserAgent instead of LWP::Simple, which allows you to get an error message:
my $ua = LWP::UserAgent->new;
my $sitemapurl = "https://www.prosinger.net";
my $req = $ua->head($sitemapurl);
if ($req->is_success) {
...
} else {
die "Could not head($sitemapurl): " . $req->status_line;
}
Running this code prints:
Could not head(https://www.prosinger.net): 403 Forbidden at head.pl line 15.
You can fix this (for this specific website; this will not work for all website) by setting a User-Agent in your LWP::UserAgent object:
my $ua = LWP::UserAgent->new( agent => 'anything seems to work' );
Of interest is the decoded_content method of HTTP::Response that allows you to get the content of the request (you don't need it in that case, but you might later):
my $req = $ua->get(...);
if ($req->is_success) {
my $content = $req->decoded_content;
...
}
Your code that uses LWP::Simple and Dada's version that switches to LWP::UserAgent are basically doing the same thing, except that you can get details of the error when using LWP::UserAgent.
Running the LWP::UserAgent version gives this error:
Could not head(https://www.prosinger.net): 500 Can't connect to www.prosinger.net:443 (SSL connect attempt failed error:2707307E:OCSP routines:OCSP_check_validity:status not yet valid)
And Googling that error message gives this SO answer as the first result. Is it possible that the clocks on your your client machine and the server are out of sync?
I am trying to connect via HTTPS to a site using LWP::Simple. I have set the environment variables both inside of the script and in the shell. I continue to get a 500 Connection Timed Out. I can connect to a HTTP site just fine.
The proxy is letting communication through. I can connect to the HTTPS site through the proxy using curl without a problem.
Any suggestions?
#!/usr/bin/perl
use warnings;
use Net::SSL;
use LWP::UserAgent;
use LWP::Debug qw(+);
use Data::Dumper;
$ENV{PERL_NET_HTTPS_SSL_SOCKET_CLASS} = "Net::SSL";
$ENV{HTTPS_DEBUG} = 1;
$ENV{HTTPS_VERSION} = 2;
my $ua = LWP::UserAgent->new (verify_hostname => 0);
$ua->ssl_opts(verify_hostname => 0,
SSL_verify_mode => 0x00);
$ua->proxy('https' => 'http://x.x.x.x:3128');
print $ua->proxy('https');
print Dumper($ua);
my $response = $ua->get('https://qualys.com/');
print Dumper ($response);
if ($response->is_success) {
print $response->decoded_content; # or whatever
exit(0);
}
else {
print "\nFail:\n";
print $response->status_line ."\n";
exit(1);
}
HTTPS proxy support for LWP is (or at least was until some days ago) broken, at least if you use IO::Socket::SSL, which is the default for LWP versions >=6.0 because it provides better security. It should be much better now with LWP 6.0.6 and LWP::Protocol::https 6.0.6 which were released few days ago. If you cannot upgrade you might try using Net::SSLGlue::LWP which patches LWP for better https proxy support.
I have read a number of threads on a number of sites and am still unable to make this work.
I have a client machine (OSX) with OpenSSL 0.9.8r running perl 5.12.4, with LWP 6.0.4, updated Crypt::SSLeay, Net::SSL etc. I am trying to connect to an HTTPS site (https://github.com in the example) via a WinGate proxy that I have running on a Windows VM. Note that my actual application is attaching to an SSL webservice that I have no control over.
From Firefox, pointed to the proxy everything is copacetic. The page loads successfully and I see the connections in the Proxy software Activity monitor. I'll be darned if I can make it work in Perl though. I've started with the code from this Stack Overflow Question : How do I force LWP to use Crypt::SSLeay for HTTPS requests? And added some debugging and additional output. Here's were I stand now:
#!/usr/bin/perl
use strict;
use warnings;
use Net::SSL (); # From Crypt-SSLeay
BEGIN {
$Net::HTTPS::HTTPS_SSL_SOCKET_CLASS = "Net::SSL"; # Force use of Net::SSL
$ENV{HTTPS_PROXY} = 'https://192.168.1.11:80';
# $ENV{PERL_LWP_SSL_VERIFY_HOSTNAME} = 0;
$ENV{HTTPS_DEBUG} = 1; #Add debug output
}
use LWP::UserAgent;
my $ua = LWP::UserAgent->new();
my $req = HTTP::Request->new('GET','https://github.com/');
my $response = $ua->request($req);
print "--\n";
print "$_\n" for grep { $_ =~ /SSL/ } keys %INC;
print "--\n";
if ($response->is_success) {
print $response->decoded_content; # or whatever
exit(0);
}
else {
print "\nFail:\n";
print $response->status_line ."\n";
exit(1);
}
Here's the output from this code:
--
Crypt/SSLeay.pm
Crypt/SSLeay/X509.pm
Net/SSL.pm
--
Fail:
500 Can't connect to github.com:443 (Crypt-SSLeay can't verify hostnames)
If I then uncomment $ENV{PERL_LWP_SSL_VERIFY_HOSTNAME} = 0;, I do see a single connect to github.com:443 on the proxy and then nothing. (Note it works great from a web browser through the proxy). After much hanging I get the following output from the script:
SSL_connect:before/connect initialization
SSL_connect:SSLv2/v3 write client hello A
SSL_connect:before/connect initialization
SSL_connect:SSLv3 write client hello A
SSL_connect:failed in SSLv3 read server hello A
SSL_connect:before/connect initialization
SSL_connect:SSLv2 write client hello A
SSL_connect:failed in SSLv2 read server hello A
--
Crypt/SSLeay.pm
Crypt/SSLeay/X509.pm
Net/SSL.pm
Crypt/SSLeay/CTX.pm
Crypt/SSLeay/MainContext.pm
--
Fail:
500 SSL negotiation failed:
If anyone can provide some direction here I would greatly appreciate it!
I just uploaded the LWP::Protocol::connect module to CPAN.
This module adds the missing HTTP/CONNECT method support to LWP.
use LWP::UserAgent;
$ua = LWP::UserAgent->new();
$ua->proxy('https', 'connect://proxyhost.domain:3128/');
$ua->get('https://www.somesslsite.com');
With this module you can use the regular IO::Socket::SSL implementation for LWP >=6.00.
Why would you want "Force use of Net::SSL".
Try
#!/usr/bin/perl
use strict;
use warnings;
use LWP::UserAgent;
BEGIN {
$ENV{HTTPS_PROXY} = 'https://192.168.1.11:80';
# $ENV{PERL_LWP_SSL_VERIFY_HOSTNAME} = 0;
$ENV{HTTPS_DEBUG} = 1; #Add debug output
}
my $ua = LWP::UserAgent->new();
my $req = HTTP::Request->new('GET','https://github.com/');
my $response = $ua->request($req);
print $response->code ."\n";
An out put of 200 should mean that there were no errors.
A below sample code of mine works perfectly
#!/usr/bin/perl
use warnings;
use LWP::UserAgent;
BEGIN {
$ENV{HTTPS_PROXY} = 'https://176.9.209.113:8080'; #Valid HTTPS proxy taken from http://hidemyass.com/proxy-list/
$ENV{HTTPS_DEBUG} = 1;
}
my $ua = new LWP::UserAgent;
my $req = new HTTP::Request('GET', 'https://www.nodeworks.com');
my $res = $ua->request($req);
print $res->code, "\n";
Output-
200
SSL_connect:before/connect initialization
SSL_connect:SSLv2/v3 write client hello A
SSL_connect:SSLv3 read server hello A
SSL_connect:SSLv3 read server certificate A
SSL_connect:SSLv3 read server key exchange A
SSL_connect:SSLv3 read server done A
SSL_connect:SSLv3 write client key exchange A
SSL_connect:SSLv3 write change cipher spec A
SSL_connect:SSLv3 write finished A
SSL_connect:SSLv3 flush data
SSL_connect:SSLv3 read finished A
SSL_connect:before/connect initialization
SSL_connect:SSLv2/v3 write client hello A
SSL_connect:SSLv3 read server hello A
SSL_connect:SSLv3 read server certificate A
SSL_connect:SSLv3 read server key exchange A
SSL_connect:SSLv3 read server done A
SSL_connect:SSLv3 write client key exchange A
SSL_connect:SSLv3 write change cipher spec A
SSL_connect:SSLv3 write finished A
SSL_connect:SSLv3 flush data
SSL_connect:SSLv3 read finished A
Tool completed successfully
With https://github.com/ the output is-
200
SSL_connect:before/connect initialization
SSL_connect:SSLv2/v3 write client hello A
SSL_connect:SSLv3 read server hello A
SSL_connect:SSLv3 read server certificate A
SSL_connect:SSLv3 read server done A
SSL_connect:SSLv3 write client key exchange A
SSL_connect:SSLv3 write change cipher spec A
SSL_connect:SSLv3 write finished A
SSL_connect:SSLv3 flush data
SSL_connect:SSLv3 read finished A
Tool completed successfully
So having said all this. Your code version (below) should work fine-
use warnings;
use LWP::UserAgent;
BEGIN {
$ENV{HTTPS_PROXY} = 'https://176.9.209.113:8080';
$ENV{PERL_LWP_SSL_VERIFY_HOSTNAME} = 0; #works even with this
$ENV{HTTPS_DEBUG} = 1; #Add debug output
}
my $ua = new LWP::UserAgent;
my $req = new HTTP::Request('GET', 'https://github.com/');
my $res = $ua->request($req);
print $res->code, "\n";
if ($res->is_success) {
print $res->decoded_content; # or whatever
exit(0);
}
else {
print "\nFail:\n";
print $res->status_line ."\n";
exit(1);
}
I pretty much ran into the same problem. Here are the things that fixed it for me:
make sure you use Crypt::SSLeay 0.58 or lower. At least 0.64 blocked endlessly for me (as described at http://cpanforum.com/posts/13808 ), while 0.57/0.58 worked.
make sure Net::HTTP has been updated to at least 6.03 (to avoid https://rt.cpan.org/Public/Bug/Display.html?id=72790 )
Instead of using Net::SSL which does not provide much host verification (and no SNI) you can use Net::SSLGlue::LWP. This monkey-patches LWP so that https_proxy can be used with the default SSL backend IO::Socket::SSL:
use Net::SSLGlue::LWP; # do this first
use LWP::Simple;
... continue with normal LWP stuff..
I know this may be a dead question, but if anyone else hits it I've another angle... I can't promise any answers, but we've faced a long-standing problem at $work in this area, but with the Squid proxy, and maybe specific to use of X509 client certs.
The use of the Net::SSL override is part of the solution, but I would fear that WinGate could be the problem (and not something I can help with) although in our case we contact the proxy over http (not sure how LWP deals with proxy+https).
For the record, here's an example of the precise form of code we use:
use Net::SSL;
$ENV{PERL_NET_HTTPS_SSL_SOCKET_CLASS}="Net::SSL";
use LWP::UserAgent;
use LWP::Protocol::https;
my $ua = LWP::UserAgent->new;
$ENV{HTTPS_PROXY} = 'http://cache.local.employer.co.uk:80';
$ua->get("https://example.com/");
This is Perl 5.8.8 with recent CPAN installs (hence separation of L:P:https), so we have a fresh Net::HTTP.
I was going to mention some versions of Net::HTTP are crocked, but I just realised that was my CPAN bug in Martin's reply :)
Sorry if this doesn't add anything.
I've sent a pull-request on libwww-perl repository to fix (or may be workaround ...) the issue.
The comment of this PR shows a simple program that connects with https to github.com through a proxy. With this patch there's no need to mess around with %ENV in your program.
Another advantage is that you can re-use the usual https_proxy setup.
There has been a error in earl 5.8 and some further module where environment variable HTTP_PROXY does not appropriately sets the proxy connection.
Your case has a issue where the bug reported is as mentioned here https://bugzilla.redhat.com/show_bug.cgi?id=1094440
Better way to use this is without environment variable and use LWP UserAgent
`use LWP::UserAgent;
$ua = LWP::UserAgent->new();
$ua->proxy('https', 'connect://proxyhost.domain:3128/');`
#!/usr/bin/env perl
#
# mimvp.com
# 2017-03-28
use CGI;
use strict;
use LWP::UserAgent;
our %proxy_https = ("https", "connect://173.233.55.118:443");
our $mimvp_url = "https://proxy.mimvp.com/exist.php";
## https
## 1. download LWP-Protocol-connect (wget http://search.cpan.org/CPAN/authors/id/B/BE/BENNING/LWP-Protocol-connect-6.09.tar.gz)
## 2. tar zxvf LWP-Protocol-connect-6.09.tar.gz
## cd LWP-Protocol-connect-6.09
## perl Makefile.PL
## make
## sudo make install
sub test_connect {
my ($url, %proxy) = #_;
print "proxy : $proxy{'http'}\n";
print "https : $proxy{'https'}\n";
print "socks4 : $proxy{'socks4'}\n";
print "socks5 : $proxy{'socks5'}\n";
print "url : $url\n";
my $browser = LWP::UserAgent->new();
$browser->env_proxy();
# # 设置的代理格式
$browser->proxy(%proxy);
$browser->timeout(30);
$browser->agent('Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.115 Safari/537.36');
# my $req = new HTTP::Request('GET', $url);
# my $response = $browser->request($req);
my $response = $browser->get($url); # 爬取的网址
my $is_success = $response->is_success(); # 1
my $content_type = $response->content_type(); # text/html
my $content = $response->content(); # 网页正文
my $content_length = length($content); # 网页正文长度
print "$is_success\n";
print "$content_type\n";
print "$content_length\n";
print "$content\n";
}
test_connect($mimvp_url, %proxy_https); # https
## perl mimvp-proxy-perl.pl
I'm attempting to use an existing CAS server to authenticate login for a Perl CGI web script and am using the AuthCAS Perl module (v 1.3.1). I can connect to the CAS server to get the service ticket but when I try to connect to validate the ticket my script returns with the following error from the IO::Socket::SSL module:
500 Can't connect to [CAS Server]:443 (Bad hostname '[CAS Server]')
([CAS Server] substituted for real server name)
Symptoms/Tests:
If I type the generated URL for the authentication into the web browser's location bar it returns just fine with the expected XML snippet. So it is not a bad host name.
If I generate a script without using the AuthCAS module but using the IO::Socket::SSL module directly to query the CAS server for validation on the generated service ticket the Perl script will run fine from the command line but not in the browser.
If I add the AuthCAS module into the script in item 2, the script no longer works on the command line and still doesn't work in the browser.
Here is the bare-bones script that produces the error:
#!/usr/bin/perl
use strict;
use warnings;
use CGI;
use AuthCAS;
use CGI::Carp qw( fatalsToBrowser );
my $id = $ENV{QUERY_STRING};
my $q = new CGI;
my $target = "http://localhost/cgi-bin/testCAS.cgi";
my $cas = new AuthCAS(casUrl => 'https://cas_server/cas');
if ($id eq ""){
my $login_url = $cas->getServerLoginURL($target);
printf "Location: $login_url\n\n";
exit 0;
} else {
print $q->header();
print "CAS TEST<br>\n";
## When coming back from the CAS server a ticket is provided in the QUERY_STRING
print "QUERY_STRING = " . $id . "</br>\n";
## $ST should contain the received Service Ticket
my $ST = $q->param('ticket');
my $user = $cas->validateST($target, $ST); #### This is what fails
printf "Error: %s\n", &AuthCAS::get_errors() unless (defined $user);
}
Any ideas on where the conflict might be?
The error is coming from the line directly above the snippet Cebjyre quoted namely
$ssl_socket = new IO::Socket::SSL(%ssl_options);
namely the socket creation. All of the input parameters are correct. I had edited the module to put in debug statements and print out all the parameters just before that call and they are all fine. Looks like I'm going to have to dive deeper into the IO::Socket::SSL module.
As usually happens when I post questions like this, I found the problem. It turns out the Crypt::SSLeay module was not installed or at least not up to date. Of course the error messages didn't give me any clues. Updating it and all the problems go away and things are working fine now.
Well, from the module source it looks like that IO::Socket error is coming from get_https2
[...]
unless ($ssl_socket) {
$errors = sprintf "error %s unable to connect https://%s:%s/\n",&IO::Socket::SSL::errstr,$host,$port;
return undef;
}
[...]
which is called by callCAS, which is called by validateST.
One option is to temporarily edit the module file to put some debug statements in if you can, but if I had to guess, I'd say the casUrl you are supplying isn't matching up to the _parse_url regex properly - maybe you have three slashes after the https?