Unable to send pipelined http request over a perl socket - perl

I have the following perl script to send pipelined http request over a sinlge tcp connection. The first request goes through and i am able to get the response. The second reqeust also gets sent, but no response is received for this request. I am able to see both requests being received on the server in the trace.
use IO::Socket::INET;
$| = 1;
$socket = new IO::Socket::INET (
PeerHost => '10.102.218.56',
PeerPort => '80',
Proto => 'tcp',);
die "cannot connect to the server $!\n" unless $socket;
print "connected to the server\n";
# data to send to a server
my $req = "GET /test/file5.html HTTP/1.1\r\nHost:10.102.218.50\r\nAccept: */*\r\n\r\n\r\n";
my $size = $socket->send($req);
print "1. sent data of length $size\n";
$socket->recv($response, 1024);
print "received response: $response\n";
my $req = "GET /test/file5.html HTTP/1.1\r\nHost:10.102.218.50\r\nAccept: */*\r\n\r\n\r\n";
my $size = $socket->send($req);
print "1. sent data of length $size\n";
$socket->recv($response, 1024);
print "received response: $response\n";
sleep 1;
$socket->close();
On the Apache access_log i see only one instance of the HTTP request being received :
10.102.218.50 - - [02/Sep/2014:23:04:50 +0530] "GET /test/file5.html HTTP/1.1" 200 6 "-" "-"
I am guessing that I am missing some characters that indicate the end of an HTTP request of the server. But i am not able to find what they are.
What am i doing wrong here?
Edit: Could the Apache server be closing the connection after the first HTTP request is answered. Is there a setting for this?
Edit2: KeepAlive was set to Off on my apache server. Setting it to ON fixes the issue.

Found the problem. The KeepAlive was set to off on my Apache server. Setting the KeepAlive to On fixed the problem.

Related

Perl IO::Socket::SSL with web server hang

I've got a problem when using IO::Socket::SSL.
Everything works fine under normal operation but I had an issue where the web server (IIS) locked up and Perl got stuck, even after the web server became active again.
I'm running the Perl script as an exe under Windows and I can't actually see the program running as it is hidden - here is the code:
sub api_action
{
$api_action = $_[0];
use IO::Socket::SSL;
$EOL = "\015\012";
$BLANK = $EOL x 2;
$remote = IO::Socket::SSL->new( Proto => "tcp",
PeerAddr => "api.xxxxx",
PeerPort => "443",
SSL_verify_mode => SSL_VERIFY_NONE,
verify_hostname => 0,
Timeout => 120,
);
unless ($remote) { print "cannot connect to API\n"; return "ERROR"
}
$remote->autoflush(1);
print $remote "GET /API/?action=$api_action" . $BLANK;
return <$remote>;
close $remote;
}
My thought is that it has made the initial connection, but is still waiting for a return from the GET command. I would have thought setting the Timeout would just cause it to return the error but it doesn't look like it works.
Any ideas on where I'm going wrong or what I've missed?
You actually send this to the server inside the SSL connection:
GET /path\r\n
\r\n
This is no valid HTTP/1.0 or HTTP/1.1 request but a HTTP/0.9 request. Since HTTP/0.9 is obsolete since 20 years I would not expect to IIS still support this old protocol so it might just show unexpected behavior when confronted with such a request. A proper HTTP/1.0 request would look like this:
GET /path HTTP/1.0\r\n
Host: www.example.com\r\n
\r\n
For more information on how to send proper requests and how to properly deal with the response see the HTTP/1.0 and HTTP/1.1 standards.
Apart from that:
SSL_verify_mode => SSL_VERIFY_NONE,
verify_hostname => 0,
verify_hostname has no meaning in IO::Socket::SSL. You probably saw this with LWP::UserAgent and it is only relevant there.
Everything works fine under normal operation but I had an issue where the web server (IIS) locked up and Perl got stuck, even after the web server became active again.
Unfortunately this is not usable as an error description because it is unclear what you consider as "locked up" and "active again" but essentially if the server behaves erratically then the client might be affected by this.
Currently your are dealing with the request by reading until end of TCP connection. If the server behaves erratically and will not close this connection then you will wait forever.
Note, that the timeout you set might only be relevant for the initial connect and not for further reads. And it might not even work properly on Windows depending on the version of IO::Socket::SSL you use. You might instead add a alarm(60) or similar to make sure that the client does not wait forever on a broken server or you might try to work with non-blocking sockets to deal with the issue.

Answer Browser Request with IO::Socket::SSL

Question formulated in the end, first detailed description of the problem and what I have tested already
I'm writing some code to showcase some basic principles to others. The code will never be productive and simplification is intended.
My goal (appart from others) is to write a simple application which uses a Web-Certificat to encrypt its network traffic.
The starting point is an application which does not encrypt:
#!/usr/bin/env perl
use strict;
use warnings;
use IO::Socket::INET;
# auto-flush on socket
$| = 1;
# creating a listening socket
my $socket = new IO::Socket::INET (
LocalAddr => '0.0.0.0', # local server address
LocalPort => '7777', # local server port
Listen => 5, # queue size for connections
Proto => 'tcp', # protocol used
);
die "cannot create socket $!\n" unless $socket;
print "server waiting for client connection on port 7777\n";
while(1)
{
# waiting for a new client connection
my $client_socket = $socket->accept() or die "socket accept failed $!";
# get information about a newly connected client
my $client_address = $client_socket->peerhost();
my $client_port = $client_socket->peerport();
print "connection from $client_address:$client_port\n";
# read up to 1024 characters from the connected client
my $client_data = '';
sysread( $client_socket, $client_data, 1024);
print "received data: $client_data\n";
# write response data to the connected client
print $client_socket "Hey $client_data!";
# notify client that response has been sent
shutdown($client_socket, 1);
}
END {
$socket->close();
}
This Server-Application can be called with this Client:
#!/usr/bin/env perl
use IO::Socket::INET;
# auto-flush on socket
$| = 1;
# create a connecting socket
my $socket = new IO::Socket::INET (
PeerHost => '127.0.0.1',
PeerPort => '7777',
Proto => 'tcp',
);
die "cannot connect to the server $!\n" unless $socket;
print "connected to the server\n";
# data to send to a server
my $req = $ARGV[0] . '';
print $socket $req;
# notify server that request has been sent
shutdown($socket, 1);
# receive a response of up to 1024 characters from server
my $response = '';
sysread( $socket, $response,1024);
print "received response: $response\n";
$socket->close();
An interaction of Client and Server could look like this:
inet$ perl server.pl
server waiting for client connection on port 7777
connection from 127.0.0.1:40028
received data: Herbert
inet$ perl client.pl "Herbert"
connected to the server
received response: Hey Herbert!
The cool thing is: The server can also be called from a browser:
So the first conclusion: The code works and is good to showcase basic functionality of a simple Client-Server interaction.
Now the Programs should use SSL to communicate
The Server and Client are written so, that SSL-Ability can be achieved by just adding a few lines in the code:
$ diff inet/server.pl ssl/server.pl
7c7
< use IO::Socket::INET;
---
> use IO::Socket::SSL 'inet4';
13c13
< my $socket = new IO::Socket::INET (
---
> my $socket = IO::Socket::SSL->new (
17a18,19
> SSL_cert_file => 'cert.pem', # SSL certificate
> SSL_key_file => 'key.pem', # SSL certificate key
$ diff inet/client.pl ssl/client.pl
5c5
< use IO::Socket::INET;
---
> use IO::Socket::SSL 'inet4';
11c11
< my $socket = new IO::Socket::INET (
---
> my $socket = new IO::Socket::SSL (
14a15
> SSL_ca_file => 'cert.pem',
So the new SSL enabled code is:
#!/usr/bin/env perl
use strict;
use warnings;
use IO::Socket::SSL 'inet4';
# auto-flush on socket
$| = 1;
# creating a listening socket
my $socket = IO::Socket::SSL->new (
LocalAddr => '0.0.0.0', # local server address
LocalPort => '7777', # local server port
Listen => 5, # queue size for connections
Proto => 'tcp', # protocol used
SSL_cert_file => 'cert.pem', # SSL certificate
SSL_key_file => 'key.pem', # SSL certificate key
);
die "cannot create socket $!\n" unless $socket;
print "server waiting for client connection on port 7777\n";
while(1)
{
# waiting for a new client connection
my $client_socket = $socket->accept() or die "socket accept failed $!";
# get information about a newly connected client
my $client_address = $client_socket->peerhost();
my $client_port = $client_socket->peerport();
print "connection from $client_address:$client_port\n";
# read up to 1024 characters from the connected client
my $client_data = '';
sysread( $client_socket, $client_data, 1024);
print "received data: $client_data\n";
# write response data to the connected client
print $client_socket "Hey $client_data!";
# notify client that response has been sent
shutdown($client_socket, 1);
}
END {
$socket->close();
}
and
#!/usr/bin/env perl
use IO::Socket::SSL 'inet4';
# auto-flush on socket
$| = 1;
# create a connecting socket
my $socket = new IO::Socket::SSL (
PeerHost => '127.0.0.1',
PeerPort => '7777',
Proto => 'tcp',
SSL_ca_file => 'cert.pem',
);
die "cannot connect to the server $!\n" unless $socket;
print "connected to the server\n";
# data to send to a server
my $req = $ARGV[0] . '';
print $socket $req;
# notify server that request has been sent
shutdown($socket, 1);
# receive a response of up to 1024 characters from server
my $response = '';
sysread( $socket, $response,1024);
print "received response: $response\n";
$socket->close();
To run the code, a certificate has to be created first:
openssl req -x509 -newkey rsa:4096 -keyout key.pem -out cert.pem -days 365
Now Server and Client can be started and interact nicely:
ssl$ perl server.pl
Enter PEM pass phrase:
server waiting for client connection on port 7777
connection from 127.0.0.1:40041
received data: Sabine
ssl$ perl client.pl "Sabine"
connected to the server
received response: Hey Sabine!
But what does not work is a connection from a browser like Firefox or Chrome, even though I converted the certificate using:
openssl pkcs12 -export -in cert.pem -inkey key.pem -out webcert.p12
I also imported the newly created certificate into the browser over its import menu.
The connection just gets refused. The browser can't connect and the server fails at $socket->accept() without any helpful message.
UPDATE: There is a message in the exported variable $SSL_ERROR:
SSL accept attempt failed error:1407609C:SSL routines:SSL23_GET_CLIENT_HELLO:http request
I did some testing with the tool analyze ssl
p5-ssl-tools-master$ perl analyze-ssl.pl --show-chain --all-ciphers -v3 127.0.0.1:7777
+ checking host=127.0.0.1(127.0.0.1) port=7777
* version SSLv23 no verification, ciphers= -> TLSv1_2,ECDHE-RSA-AES128-GCM-SHA256
* version SSLv23 no verification, ciphers=HIGH:ALL -> TLSv1_2,ECDHE-RSA-AES128-GCM-SHA256
* version TLSv1_2 no verification, ciphers= -> TLSv1_2,ECDHE-RSA-AES128-GCM-SHA256
* version TLSv1_2 no verification, ciphers=HIGH:ALL -> TLSv1_2,ECDHE-RSA-AES128-GCM-SHA256
* version TLSv1_1 no verification, ciphers= -> TLSv1_1,ECDHE-RSA-AES256-SHA
* version TLSv1_1 no verification, ciphers=HIGH:ALL -> TLSv1_1,ECDHE-RSA-AES256-SHA
* version TLSv1 no verification, ciphers= -> TLSv1,ECDHE-RSA-AES256-SHA
* version TLSv1 no verification, ciphers=HIGH:ALL -> TLSv1,ECDHE-RSA-AES256-SHA
* version SSLv3, no verification, ciphers= -> FAIL! SSL connect attempt failed because of handshake problems error:14094410:SSL routines:SSL3_READ_BYTES:sslv3 alert handshake failure
+ 127.0.0.1 failed permanently 'tcp connect: Verbindungsaufbau abgelehnt', no more IP to try
tcp connect: Verbindungsaufbau abgelehnt
The server seems to accept certain requests and fails at the end, maybe in the same way as with the browser?
ssl$ perl server.pl
Enter PEM pass phrase:
server waiting for client connection on port 7777
connection from 127.0.0.1:40042
received data:
connection from 127.0.0.1:40043
received data:
connection from 127.0.0.1:40044
received data:
connection from 127.0.0.1:40045
received data:
connection from 127.0.0.1:40046
received data:
connection from 127.0.0.1:40047
received data:
connection from 127.0.0.1:40048
received data:
connection from 127.0.0.1:40049
received data:
socket accept failed at server.pl line 27.
My Question
Why is the request with the browser not working?
The code should support a browser request in general since it works without SSL.
It seems to be a browser specific SSL setting since the SSL-Perl-Client has no problem.
Or is the import of the certificate into the browser not working as intended?
Can anyone give me a hint or solution in this matter?
UPDATE: When I look at the error message "SSL23" in $SSL_ERROR and the error message in the client test "SSL3" there seems to be a compatibility problem with SSL versions? Do I need to specify explicitly which version I want? (On the other hand the Client-Test for "SSL23" seems to run through successfully...)
Thanks a lot.
UPDATE: Output with $IO::Socket::SSL::DEBUG = 3;
ssl$ perl server.pl
Enter PEM pass phrase:
DEBUG: .../IO/Socket/SSL.pm:2554: new ctx 42708208
server waiting for client connection on port 7777
DEBUG: .../IO/Socket/SSL.pm:799: no socket yet
DEBUG: .../IO/Socket/SSL.pm:801: accept created normal socket IO::Socket::SSL=GLOB(0x28ac158)
DEBUG: .../IO/Socket/SSL.pm:829: starting sslifying
DEBUG: .../IO/Socket/SSL.pm:873: Net::SSLeay::accept -> -1
DEBUG: .../IO/Socket/SSL.pm:1779: SSL accept attempt failed
DEBUG: .../IO/Socket/SSL.pm:1784: SSL accept attempt failed error:1407609C:SSL routines:SSL23_GET_CLIENT_HELLO:http request
socket accept failed: SSL accept attempt failed error:1407609C:SSL routines:SSL23_GET_CLIENT_HELLO:http request at server.pl line 28.
DEBUG: .../IO/Socket/SSL.pm:2587: free ctx 42708208 open=42708208
DEBUG: .../IO/Socket/SSL.pm:2599: OK free ctx 42708208
... SSL accept attempt failed error:1407609C:SSL routines:SSL23_GET_CLIENT_HELLO:http request
My guess is that you still use a http:// URL to access the server even though you would need to use a https:// URL. This means that the browser sends a HTTP request to the server instead of first doing the TLS handshake and only sending the HTTP request after the successful handshake.
Apart from that you count on the browser understanding the old and long deprecated HTTP 0.9 protocol which consists of only the response body without any HTTP header. Not all browser understand this.

Perl File::Fetch Failed HTTP response: 500 Internal Server Error

I'm using a simple perl script to try and download a file from the following URL:
http://www.londonstockexchange.com/products-and-services/trading-services/setsqx/ccp-securities.xls
use File::Fetch;
my $url = 'http://www.londonstockexchange.com/products-and-services/trading-services/setsqx/ccp-securities.xls';
my $ff = File::Fetch->new(uri => $url);
my $file = $ff->fetch() or die $ff->error;
I am getting:
Fetch failed! HTTP response: 500 Internal Server Error [500 Can't connect to www.londonstockexchange.com:80 (connect: Connection refused)]
Could not open socket to 'www.londonstockexchange.com', 'Connection refused
Any ideas how i can get past this?
Could not open socket to 'www.londonstockexchange.com', 'Connection refused
Any ideas how i can get past this?
Direct connection to this site is blocked somewhere.
If you are inside a company there is probably a firewall and some proxy you are required to use. Ask your system administrator.

Port hopping on IO::SOCKET::INET

i am try to get a better understanding on Perl and was stumble about sockets. i am try to understand a "simple" example i found on xmodulo. I was expecting that like in the example the port will be 7777. But it seems like the port used for this socket communication is somewhere above 35000 and on every call of the client script the port is incremented by +1. Why ist the port different to the 7777 and why it is incrementing on every call?
the server example looks like this:
use IO::Socket::INET;
# auto-flush on socket
$| = 1;
# creating a listening socket
my $socket = new IO::Socket::INET (
LocalHost => '0.0.0.0',
LocalPort => '7777',
Proto => 'tcp',
Listen => 5,
Reuse => 1
);
die "cannot create socket $!\n" unless $socket;
print "server waiting for client connection on port 7777\n";
while(1)
{
# waiting for a new client connection
my $client_socket = $socket->accept();
# get information about a newly connected client
my $client_address = $client_socket->peerhost();
my $client_port = $client_socket->peerport();
print "connection from $client_address:$client_port\n";
# read up to 1024 characters from the connected client
my $data = "";
$client_socket->recv($data, 1024);
print "received data: $data\n";
# write response data to the connected client
$data = "ok";
$client_socket->send($data);
# notify client that response has been sent
shutdown($client_socket, 1);
}
$socket->close();
the client example is:
use IO::Socket::INET;
# auto-flush on socket
$| = 1;
# create a connecting socket
my $socket = new IO::Socket::INET (
PeerHost => '192.168.1.10',
PeerPort => '7777',
Proto => 'tcp',
);
die "cannot connect to the server $!\n" unless $socket;
print "connected to the server\n";
# data to send to a server
my $req = 'hello world';
my $size = $socket->send($req);
print "sent data of length $size\n";
# notify server that request has been sent
shutdown($socket, 1);
# receive a response of up to 1024 characters from server
my $response = "";
$socket->recv($response, 1024);
print "received response: $response\n";
$socket->close();
A connection isn't defined by
local address
peer address
port
It's defined by
local address
local port
peer address
peer port
For example,
>netstat /a
Active Connections
Proto Local Address Foreign Address State
...
TCP 10.0.0.2:34208 stackoverflow:http ESTABLISHED
TCP 10.0.0.2:34212 stackoverflow:http ESTABLISHED
TCP 10.0.0.2:34213 stackoverflow:http ESTABLISHED
TCP 10.0.0.2:34224 stackoverflow:http ESTABLISHED
TCP 10.0.0.2:34226 stackoverflow:http ESTABLISHED
TCP 10.0.0.2:34227 stackoverflow:http ESTABLISHED
...
You didn't specify a local port for the client, so the system picked an available one. That's the right thing to do. There's no reason to limit the client to one port. It can even cause problems.
For example, let's say your web browser tried to bind its sockets to port 80 (the port on which web servers listen). Your web browser would only be able to have one request pending at a time. That would be bad. You want to be able to create multiple connections to the same service. This allows you to request two different images at the same time, this allows you to load two pages in two different tabs at the same time, and so on. In the example above, my machine had six connections to stackoverflow.com's port 80, but that wouldn't have been possible if the web browser has bound the socket to port 80.

500 SSL read timeout error in LWP perl

I have a perl script which will post HTTP request to specified server URL (Say: http://some-ip/here_action_url ). My problem is, Sometimes I am getting the below error.
Error:
500 SSL read timeout.
Sample Code:
my $ua = new LWP::UserAgent;
$ua->timeout(30);
my $res = $ua->post( $url, { 'data' => $my_data } );
if(! $res->is_success ) {
# Error Logging
print $res->status_line."\n";
}
else {
$response_content = $res->content;
}
I read about the error. Most of the documents are saying that it is because of the response delay in server side.
I just want to confirm, whether this error is coming because of server response delay? (or) Can be the problem with my perl script?
If you get a result some of the time, and the error at other times, then it looks like your code is working.
If you alway get the 500 error, it indicates a connection problem. Would need to know more about the service you are trying to connect to, does it require certificates or other authentication (which may be needed for secure socket layer connection)