If I use wget to retrieve something from the geonames.org server, it reports two IP addresses, and the first one fails but it gets it from the second:
Resolving ws.geonames.org (ws.geonames.org)... 5.9.41.208, 176.9.107.169
Connecting to ws.geonames.org (ws.geonames.org)|5.9.41.208|:80... failed: Connection refused.
Connecting to ws.geonames.org (ws.geonames.org)|176.9.107.169|:80... connected.
HTTP request sent, awaiting response... 200 OK
But unfortunately I have to access it through perl using LWP::UserAgent and HTTP::Request. How can I make them try the second IP if the first fails?
my $ua = LWP::UserAgent->new;
my $req = HTTP::Request->new(
GET =>
"http://ws.geonames.org/countrySubdivision?lat=$lat&lng=$long&radius=$radius&username=xyzzy");
my $res = $ua->request($req);
You can do it yourself: get all the IP addresses with the help of Net::DNS::Resolver, and then try all IP addresses until you get a successful response. Note that you have to supply the "Host" header yourself if working with an IP address, in case the server is doing name-based virtual hosts.
Something like the following lines could work. Maybe there's even a CPAN module for this, I did not check:
use Net::DNS;
use LWP::UserAgent;
my #addrs;
{
my $res = Net::DNS::Resolver->new;
my $query = $res->search("ws.geonames.org");
if ($query) {
for my $rr ($query->answer) {
if ($rr->type eq "A") {
push #addrs, $rr->address;
}
}
} else {
die "DNS query failed: ", $res->errorstring, "\n";
}
}
my $ua = LWP::UserAgent->new;
my $res;
for my $addr (#addrs) {
$res = $ua->get("http://$addr/countrySubdivision?lat=$lat&lng=$long&radius=$radius&username=xyzzy", Host => 'ws.geonames.org');
last if $res->is_success;
}
The solution from Slaven is OK except when the IP addresses are not directly accessible.
In that case, the following works for me:
local #LWP::Protocol::http::EXTRA_SOCK_OPTS = (
PeerAddr => 'my_hostname',
MultiHomed => 1,
);
my $response = $ua->post('https://my_hostname/...', ...);
Related
Using the concurrent asynchronous URL example for Net::Async::HTTP, the first encounter of bad URL (timeout, doesn't exist, etc) error causes the program to fail and exit completely, without continuing to the next URL in the array. Is the problem my code or the module?
I tried setting fail_on_error to 0, and even 1, but it had no obvious results.
#!/bin/perl
use IO::Async::Loop;
use Net::Async::HTTP;
use Future::Utils qw(fmap_void);
use strict;
use warnings;
use feature 'say';
my $ua_string = "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/43.0.2357.81 Safari/537.36";
my $timeout = 10;
my $max_redirects = 10;
my $max_in_flight = 10;
my $max_connections_per_host = 10;
my $stall_timeout = 10;
my $max_recurse = "10";
my $max_per_host = "10";
my #URLs = ( "http://cnn.com", "http://google.com", "http://sdfsdfsdf24.com", "http://msn.net" );
my $loop = IO::Async::Loop->new();
my $http = Net::Async::HTTP->new();
$loop->add($http);
my $future = fmap_void {
my ( $url ) = #_;
$http->configure(user_agent => $ua_string);
$http->configure(timeout => $timeout );
$http->configure(max_redirects => $max_redirects);
$http->configure(max_in_flight => $max_in_flight);
$http->configure(max_connections_per_host => $max_connections_per_host);
$http->configure(stall_timeout => $stall_timeout);
$http->configure(fail_on_error => '0' );
$http->GET($url)->on_done(
sub {
my $response = shift;
say "Response: $response->code";
}
)->on_fail(
sub {
my $fail = shift;
say "Failed: $fail";
}
);
}
foreach => \#URLs;
$loop->await($future);
Your example really works well without any proxy, I tested and did some changes:
Fetching URL: '. $url;
$http->GET($url)->on_done(
sub {
my $response = shift;
say "Response: ".$response->code();
}
)->on_fail(
sub {
my $fail = shift;
say "Failed: " . $fail;
}
);
Output:
Fetching URL: http://cnn.com
Response: 200
Fetching URL: http://google.com
Response: 302
Fetching URL: http://sdfsdfsdf24.com
Response: 403
Fetching URL: http://msn.net
Response: 200
As this example is not doing async call's the URL's are on a queue and being processed one by one.
Behind the scenes when you are doing a request to a target in your case to some URL's, at the low level the connection is made through a socket connection.
If you have a proxy which is not configured between your script and the intenet, there is no connection and it will raise an exception and your script will die like:
Fetching URL: http://cnn.com
Failed: Timed out
The variable $! is set and the error "Operation now in progress" appears, in fact your request didn't established any connection it just tried to establish one without success.
There are some points which you can check for example:
1 - Is the proxy working ?
2 - Do I have internet connection ?
3 - Is the URL I am testing working ?
If you are having problems with proxy, your script need a small adjust that you can get more info in the docs:
$http->configure( proxy_host => 'xx.xx.xx.xx');
$http->configure( proxy_port => 1234);
Supposing that your proxy is configured, you can check if you have fully access to the internet and aim some target like that URL's.
Trying to access the URLs it will provide you a response code and depending on the code you can do something.
As an alternative solution you could use LWP::UserAgent to make simple requests and check the response code.
use LWP::UserAgent;
my $ua = LWP::UserAgent->new;
$ua->timeout(10);
$ua->env_proxy;
my $response = $ua->get('http://search.cpan.org/');
if ($response->is_success) {
print $response->decoded_content; # or whatever
}
else {
die $response->status_line;
}
And even with some bad stats like 4XX for example Net::Async::HTTP won't be friendly to use this module for a simple purpose as it can't handle the exceptions like you want.
So my question is kind of hard but i give it a try:
My code is a script that use this website to extract domains from reverse ip address "http://www.yougetsignal.com/tools/web-sites-on-web-server/"
The problem is that whenever i scan until 12 try i cant get results because the website block My IP when i get to 12 search.
So i was trying to find a way to add proxy but i need to change proxy whenever i scan again.
How could I add proxy list and limit the search for 12 per proxy?
this is my code :http://pastebin.com/EY3cy5Vs
#!/usr/bin/perl
use HTTP::Request;
use LWP::UserAgent;
if($^O =~ /Win/){
system("cls");
system("color a");
system("title Get Sites by : ip");
}else{
system("clear");
}
print q{
*--------------------*
* Get *
* Sites *
* result in log.txt *
*--------------------*
};
print "\nPut Host or IP (host without http://) :";
my $host = <>;
chomp($host);
my $ua = LWP::UserAgent->new(agent => 'Mozilla/5.0 (Windows; U; Windows NT 5.1; fr; rv:1.9.1) Gecko/20090624 Firefox/3.5');
$ua->env_proxy;
#proxy
my $response = $ua->get("127.0.0.1");
#proxy
my $zz = $ua->post('http://domains.yougetsignal.com/domains.php',
{
remoteAddress => $host,
}
);
my $resulta = $zz->content;
while ($resulta =~ m/\[([^\]]*)\]/g)
{
$zeb = $1;
$zeb =~ m/\"(.*?)\", \"?\"/g;
open(a, ">>log.txt");
print a "http://$1/\n";
close(a);
}
print "\nresult in log.txt";
How could I add proxy list and limit the search for 12 per proxy?
You can't add a list using the environment (i.e env_proxy) but you can change the proxy whenever you want with
$ua->proxy(['http','https'],'http://proxy-ip:port');
This setting applies then to all requests done with $ua until you change the setting again.
I'm trying to lighten a part of my code where I was user an LWP::UA
To acces a Rest API using a SSL Layer.
I'm trying to achieve the same result with the bare IO::Socket::SSL module.
It work great on the first call with something like this :
my $SSLSocket = IO::Socket::SSL->new(
PeerHost => $server,
PeerPort => "https");
print $SSLSocket "GET /restapi/json/v1/resources?AUTHTOKEN=$token HTTP/1.0\r\n\r\n";
while(<$SSLSocket>){
$response = $_;
}
$resources = $json->decode($response);
But when I'm trying to reuse this socket I'm getting no awnser
my $SSLSocket = IO::Socket::SSL->new(
PeerHost => $server,
PeerPort => "https");
print $SSLSocket "GET /restapi/json/v1/resources?AUTHTOKEN=$token HTTP/1.0\r\n\r\n";
while(<$SSLSocket>){
$response = $_;
}
$resources = $json->decode($response);
print $SSLSocket ,"GET /restapi/json/v1/resources/3138/accounts?AUTHTOKEN=$token HTTP/1.0\r\n\r\n";
while(<$SSLSocket >){
$response = $_;
}
close($SSLSocket );
my $accounts = $json->decode($response);
The second read is empty.
Since I cannot use the recv or send method I'm a bit lost.
Is there any way to not open a new SSL Socket on each request ?
Thank you for you answers.
I want to send an email using perl ,but when i execute the command as follows:
#./sendmail.sh "par1" "par2" "par3"
i got the error msg "connect to localhost failed (Connection refused) no (more) retries"
sendmail.sh:
/usr/bin/perl /code/sendmail.pl "$1" "$2" "$3";
sendmail.pl:
#!/usr/bin/perl -w
use Mail::Sendmail;
my $event1 = shift(#ARGV);
my $event2 = shift(#ARGV);
my $time = shift(#ARGV);
#my $info = shift(#ARGV);
my $datetime = `/bin/date "+20%y-%m-%d %H:%M:%S"`;
chomp $datetime;
$msg = "This is Monitor System speak:\n
The system discovers the events at $datetime.
Something may be abnormal, please check it. The detail is below:\n";
$msg = $msg."$event1 and $event2 at $time\n";
$msg = $msg."\n";
$msg = $msg."Any problem, check it from http://map_test.php\n\n\n";
$mail_subject = "Abnormal";
sendmail(
From => 'localhost',
To => 'test#mail.com',
Subject => $mail_subject,
Message => $msg,
);
Any help appreciated.
smtp stands for simple mail transfer protocol.
When you need to send an email your mail client needs to talk to an smtp server which will accept the message. Normally your internet service provider will provide an smtp host. If you look at your mail client it will need to have an smtp server configured to be able to send mail.
Ok so when you install the Mail::Sendmail module, it doesn't know what your smtp server will be. It is up to you to tell it. It provides a default of localhost which would often be true if your server is running a sendmail daemon.
The configuration of Mail::Sendmail is stored in a variable called
%Mail::Sendmail::mailcfg
You can change the value of the sendmail server using this snippet of code:
unshift #{$Mail::Sendmail::mailcfg{'smtp'}} , 'my.smtp.server';
You need to add this line of code to your script to set the smtp server.
It adds this server to an array which also includes localhost.
So if neither of the hosts work it will still print an error message about localhost which is slightly confusing.
If you use Data::Dumper to print the contents of the mailcfg variable it will look something like this:
#!/usr/bin/perl
use Mail::Sendmail;
use Data::Dumper;
unshift #{$Mail::Sendmail::mailcfg{'smtp'}} , 'my.smtp.server';
print Dumper(\%Mail::Sendmail::mailcfg);
Should return:
$VAR1 = {
'retries' => 1,
'smtp' => [
'my.smtp.server',
'localhost'
],
'delay' => 1,
'port' => 25,
'from' => '',
'debug' => 0,
'tz' => '',
'mime' => 1
};
I tried the following to access a router via a central admin server as "ssh hop" server
#!/usr/bin/perl -X
use strict;
use Net::OpenSSH;
use Net::Telnet;
my $lhost = "linuxserver";
my $luser = "linuxuser";
my $lpass = "linuxpassword";
my $chost = "routername";
my $cpass = "Routerpassword";
my $prompt = '/(?:Password: |[>])/m';
my #commands = ("show users\r");
my $ssh = Net::OpenSSH->new($lhost,
'user' => $luser,
'password' => $lpass,
'master_opts' => [ '-t' ],
#'async' => 1 # if enabled then password cannot be set here
);
my ($pty, $err, $pid) = $ssh->open2pty("telnet $chost");
my $t = new Net::Telnet(
-telnetmode => 0,
-fhopen => $pty,
-prompt => $prompt,
-cmd_remove_mode => 1,
-output_record_separator => "\r",
#-dump_log => "debug.log",
);
my $end = 0;
while (!$end) {
my ($pre, $post) = $t->waitfor($prompt);
if ($post =~ /Password: /m) {
# send password
$t->print("$cpass");
}
elsif ($post =~ /[>#]/ && #commands) {
my $cmd = shift(#commands);
if ($cmd !~ /[\r\n]/) {
$t->print($cmd);
}
else {
print $t->cmd($cmd);
}
}
else {
$end = 1;
$t->cmd("exit");
}
}
#close $pty;
$t->close();
Unfortunately I always get the following error:
read error: Input/output error at test.pl line 71
Can somebody help me please or is there a better solution only to test if a telnet connection via the "hop" server is possible or not?
The connection looks like:
workstation --ssh-> server --telnet-> router
Thanks in advance.
I think best option is to make an SSH-tunnel to your admin server and use it for telnetting to the router.
Getting Net::Telnet to work over Net::OpenSSH sometimes is not as easy as it should be and it requires some experimentation to get to the right combination of flags and calls that make it work.
For instance, instead of telneting to the target host, use netcat to open a raw connection (or Net::OpenSSH support for TCP forwarding if tunnels are allowed on the proxy).
Expect + Net::OpenSSH may be a better option.