how to Add Proxy List to Perl? - perl

So my question is kind of hard but i give it a try:
My code is a script that use this website to extract domains from reverse ip address "http://www.yougetsignal.com/tools/web-sites-on-web-server/"
The problem is that whenever i scan until 12 try i cant get results because the website block My IP when i get to 12 search.
So i was trying to find a way to add proxy but i need to change proxy whenever i scan again.
How could I add proxy list and limit the search for 12 per proxy?
this is my code :http://pastebin.com/EY3cy5Vs
#!/usr/bin/perl
use HTTP::Request;
use LWP::UserAgent;
if($^O =~ /Win/){
system("cls");
system("color a");
system("title Get Sites by : ip");
}else{
system("clear");
}
print q{
*--------------------*
* Get *
* Sites *
* result in log.txt *
*--------------------*
};
print "\nPut Host or IP (host without http://) :";
my $host = <>;
chomp($host);
my $ua = LWP::UserAgent->new(agent => 'Mozilla/5.0 (Windows; U; Windows NT 5.1; fr; rv:1.9.1) Gecko/20090624 Firefox/3.5');
$ua->env_proxy;
#proxy
my $response = $ua->get("127.0.0.1");
#proxy
my $zz = $ua->post('http://domains.yougetsignal.com/domains.php',
{
remoteAddress => $host,
}
);
my $resulta = $zz->content;
while ($resulta =~ m/\[([^\]]*)\]/g)
{
$zeb = $1;
$zeb =~ m/\"(.*?)\", \"?\"/g;
open(a, ">>log.txt");
print a "http://$1/\n";
close(a);
}
print "\nresult in log.txt";

How could I add proxy list and limit the search for 12 per proxy?
You can't add a list using the environment (i.e env_proxy) but you can change the proxy whenever you want with
$ua->proxy(['http','https'],'http://proxy-ip:port');
This setting applies then to all requests done with $ua until you change the setting again.

Related

Can't use concurrent ascynrounous URLs with Net::Async::HTTP .. It quits and doesn't goto the next URL

Using the concurrent asynchronous URL example for Net::Async::HTTP, the first encounter of bad URL (timeout, doesn't exist, etc) error causes the program to fail and exit completely, without continuing to the next URL in the array. Is the problem my code or the module?
I tried setting fail_on_error to 0, and even 1, but it had no obvious results.
#!/bin/perl
use IO::Async::Loop;
use Net::Async::HTTP;
use Future::Utils qw(fmap_void);
use strict;
use warnings;
use feature 'say';
my $ua_string = "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/43.0.2357.81 Safari/537.36";
my $timeout = 10;
my $max_redirects = 10;
my $max_in_flight = 10;
my $max_connections_per_host = 10;
my $stall_timeout = 10;
my $max_recurse = "10";
my $max_per_host = "10";
my #URLs = ( "http://cnn.com", "http://google.com", "http://sdfsdfsdf24.com", "http://msn.net" );
my $loop = IO::Async::Loop->new();
my $http = Net::Async::HTTP->new();
$loop->add($http);
my $future = fmap_void {
my ( $url ) = #_;
$http->configure(user_agent => $ua_string);
$http->configure(timeout => $timeout );
$http->configure(max_redirects => $max_redirects);
$http->configure(max_in_flight => $max_in_flight);
$http->configure(max_connections_per_host => $max_connections_per_host);
$http->configure(stall_timeout => $stall_timeout);
$http->configure(fail_on_error => '0' );
$http->GET($url)->on_done(
sub {
my $response = shift;
say "Response: $response->code";
}
)->on_fail(
sub {
my $fail = shift;
say "Failed: $fail";
}
);
}
foreach => \#URLs;
$loop->await($future);
Your example really works well without any proxy, I tested and did some changes:
Fetching URL: '. $url;
$http->GET($url)->on_done(
sub {
my $response = shift;
say "Response: ".$response->code();
}
)->on_fail(
sub {
my $fail = shift;
say "Failed: " . $fail;
}
);
Output:
Fetching URL: http://cnn.com
Response: 200
Fetching URL: http://google.com
Response: 302
Fetching URL: http://sdfsdfsdf24.com
Response: 403
Fetching URL: http://msn.net
Response: 200
As this example is not doing async call's the URL's are on a queue and being processed one by one.
Behind the scenes when you are doing a request to a target in your case to some URL's, at the low level the connection is made through a socket connection.
If you have a proxy which is not configured between your script and the intenet, there is no connection and it will raise an exception and your script will die like:
Fetching URL: http://cnn.com
Failed: Timed out
The variable $! is set and the error "Operation now in progress" appears, in fact your request didn't established any connection it just tried to establish one without success.
There are some points which you can check for example:
1 - Is the proxy working ?
2 - Do I have internet connection ?
3 - Is the URL I am testing working ?
If you are having problems with proxy, your script need a small adjust that you can get more info in the docs:
$http->configure( proxy_host => 'xx.xx.xx.xx');
$http->configure( proxy_port => 1234);
Supposing that your proxy is configured, you can check if you have fully access to the internet and aim some target like that URL's.
Trying to access the URLs it will provide you a response code and depending on the code you can do something.
As an alternative solution you could use LWP::UserAgent to make simple requests and check the response code.
use LWP::UserAgent;
my $ua = LWP::UserAgent->new;
$ua->timeout(10);
$ua->env_proxy;
my $response = $ua->get('http://search.cpan.org/');
if ($response->is_success) {
print $response->decoded_content; # or whatever
}
else {
die $response->status_line;
}
And even with some bad stats like 4XX for example Net::Async::HTTP won't be friendly to use this module for a simple purpose as it can't handle the exceptions like you want.

How may I bypass LWP's URL encoding for a GET request?

I'm talking to what seems to be a broken HTTP daemon and I need to make a GET request that includes a pipe | character in the URL.
LWP::UserAgent escapes the pipe character before the request is sent.
For example, a URL passed in as:
https://hostname/url/doSomethingScript?ss=1234&activities=Lec1|01
is passed to the HTTP daemon as
https://hostname/url/doSomethingScript?ss=1234&activities=Lec1%7C01
This is correct, but doesn't work with this broken server.
How can I override or bypass the encoding that LWP and its friends are doing?
Note
I've seen and tried other answers here on StackOverflow addressing similar problems. The difference here seems to be that those answers are dealing with POST requests where the formfield parts of the URL can be passed as an array of key/value pairs or as a 'Content' => $content parameter. Those approaches aren't working for me with an LWP request.
I've also tried constructing an HTTP::Request object and passing that to LWP, and passing the full URL direct to LWP->get(). No dice with either approach.
In response to Borodin's request, this is a sanitised version of the code I'm using
#!/usr/local/bin/perl -w
use HTTP::Cookies;
use LWP;
my $debug = 1;
# make a 'browser' object
my $browser = LWP::UserAgent->new();
# cookie handling...
$browser->cookie_jar(HTTP::Cookies->new(
'file' => '.cookie_jar.txt',
'autosave' => 1,
'ignore_discard' => 1,
));
# proxy, so we can watch...
if ($debug == 1) {
$browser->proxy(['http', 'ftp', 'https'], 'http://localhost:8080/');
}
# user agent string (pretend to be Firefox)
$agent = 'Mozilla/5.0 (Windows; U; Windows NT 5.1; en-GB; rv:1.7.12) Gecko/20050919 Firefox/1.0.7';
# set the user agent
$browser->agent($agent);
# do some things here to log in to the web site, accept session cookies, etc.
# These are basic POSTs of filled forms. Works fine.
# [...]
my $baseURL = 'https://hostname/url/doSomethingScript?ss=1234&activities=VALUEA|VALUEB';
#values = ['Lec1', '01', 'Lec1', '02'];
while (1) {
if (scalar(#values) < 2) { last; }
my $vala = shift(#values);
my $valb = shift(#values);
my $url = $basEURL;
$url =~ s/VALUEA/$vala/g;
$url =~ s/VALUEB/$valb/g;
# simplified. Would usually check request for '200' response, etc...
$content = $browser->get($url)->content();
# do something here with the content
# [...]
# fails because the '|' character in the url is escaped after it's handed
# to LWP
}
# end
As #bchgys mentions in his comment, this is (almost) answered in the linked thread. Here are two solutions:
The first and arguably cleanest one is to locally override the escape map in URI::Escape to not modify the pipe character:
use URI;
use LWP::UserAgent;
my $ua = LWP::UserAgent->new();
my $res;
{
# Violate RFC 2396 by forcing broken query string
# local makes the override take effect only in the current code block
local $URI::Escape::escapes{'|'} = '|';
$res = $ua->get('http://server/script?q=a|b');
}
print $res->request->as_string, "\n";
Alternatively, you can simply undo the escaping by modifying the URI directly in the request after the request has been created:
use HTTP::Request;
use LWP::UserAgent;
my $ua = LWP::UserAgent->new();
my $req = HTTP::Request->new(GET => 'http://server/script?q=a|b');
# Violate RFC 2396 by forcing broken query string
${$req->uri} =~ s/%7C/|/;
my $res = $ua->request($req);
print $res->request->as_string, "\n";
The first solution is almost certainly preferable because it at least relies on the %URI::Escape::escapes package variable which is exported and documented, so that's probably as close as you're gonna get to doing this with a supported API.
Note that in either case you are in violation of RFC 2396 but as mentioned you may have no choice when talking to a broken server that you have no control over.

How to fail over

If I use wget to retrieve something from the geonames.org server, it reports two IP addresses, and the first one fails but it gets it from the second:
Resolving ws.geonames.org (ws.geonames.org)... 5.9.41.208, 176.9.107.169
Connecting to ws.geonames.org (ws.geonames.org)|5.9.41.208|:80... failed: Connection refused.
Connecting to ws.geonames.org (ws.geonames.org)|176.9.107.169|:80... connected.
HTTP request sent, awaiting response... 200 OK
But unfortunately I have to access it through perl using LWP::UserAgent and HTTP::Request. How can I make them try the second IP if the first fails?
my $ua = LWP::UserAgent->new;
my $req = HTTP::Request->new(
GET =>
"http://ws.geonames.org/countrySubdivision?lat=$lat&lng=$long&radius=$radius&username=xyzzy");
my $res = $ua->request($req);
You can do it yourself: get all the IP addresses with the help of Net::DNS::Resolver, and then try all IP addresses until you get a successful response. Note that you have to supply the "Host" header yourself if working with an IP address, in case the server is doing name-based virtual hosts.
Something like the following lines could work. Maybe there's even a CPAN module for this, I did not check:
use Net::DNS;
use LWP::UserAgent;
my #addrs;
{
my $res = Net::DNS::Resolver->new;
my $query = $res->search("ws.geonames.org");
if ($query) {
for my $rr ($query->answer) {
if ($rr->type eq "A") {
push #addrs, $rr->address;
}
}
} else {
die "DNS query failed: ", $res->errorstring, "\n";
}
}
my $ua = LWP::UserAgent->new;
my $res;
for my $addr (#addrs) {
$res = $ua->get("http://$addr/countrySubdivision?lat=$lat&lng=$long&radius=$radius&username=xyzzy", Host => 'ws.geonames.org');
last if $res->is_success;
}
The solution from Slaven is OK except when the IP addresses are not directly accessible.
In that case, the following works for me:
local #LWP::Protocol::http::EXTRA_SOCK_OPTS = (
PeerAddr => 'my_hostname',
MultiHomed => 1,
);
my $response = $ua->post('https://my_hostname/...', ...);

Perl: How can i test for a URL ( https ) accepting GET requests using "login" parameter

I have a CGI server side script that accepts GET and POST, with login parameters.
I want to test it to make sure it is not vulnerable. So the plan is to use Perl LWP, and send login parameters in GET and POST, and compare the results. the interface has been changed, so that only in POST we can send user-name and password in session cookies ( not sure if that is a great idea ) , so how do i test it ? Here is what i have so far:
#!/usr/bin/perl
use LWP;
print "This is libwww-perl-$LWP::VERSION\n";
# Create a user agent object
use LWP::UserAgent;
my $ua = LWP::UserAgent->new;
$ua->agent("MyApp/0.1 ");
# Create a request
#my $req = HTTP::Request->new(POST => 'http://search.cpan.org/search');
#my $req = HTTP::Request->new(GET => 'https://qa.co.net:443/cgi-bin/n-cu.cgi');
my $req = HTTP::Request->new(GET => 'https://qa.co.net:443/cgi-bin/n-cu.cgi?mode=frameset&JScript=1&remote_user&login=foo&password=foo HTTP/1.1');
$req->content_type('application/x-www-form-urlencoded');
$req->content('query=libwww-perl&mode=dist');
# Pass request to the user agent and get a response back
my $res = $ua->request($req);
# Check the outcome of the response
if ($res->is_success) {
print $res->content;
#print $res->code;
#print $res->message;
}
else {
print $res->status_line, "\n";
}
This is not going to do it, since it does not have the session cookie stuff. But might be a good start though. Is this the right way to test the GET and POST ?
Here is what was implemented in the cgi:
#cr_login for POST && login for GET -- leave GET param as it used to be.
if ($m eq 'GET' && defined($req->param('login'))) {
$msg = 'parameter "login" is invalid for this request type.';
+ my $seclog = $event_logging_directory . '/invalid_request.log';
+ open(S, ">>$seclog") or die $!;
+ my $logmsg = sprintf("%4d-%02d-%02d %02d:%02d:%02d",Today_and_Now())
+ . "|mode:" . $req->param('mode')
+ . "|login:" . $req->param('login')
+ . "|remote_addr:" . $ENV{REMOTE_ADDR}
+ . "|$msg\n";
+ print S $logmsg;
and :
POST request to n-cu.cgi should use parameter "cr_login". If the parameter "login" is passed in a post request, it should throw error and return to login screen.
GET request to n-cu.cgi should use the parameter "login". If the parameter "cr_login" is passed in a post request, it should throw error and return to login screen.
so here is how we do it:
Keep the session cookie and context alive :
my $browser = LWP::UserAgent->new(keep_alive => 10);
$browser->cookie_jar( {} );
$browser->agent('Mozilla/8.0');
#$browser->ssl_opts({ verify_hostname => 0 });
$browser->show_progress(1);
and later: print the response
print "Cookies:\n", Dumper($browser->cookie_jar()), "\n\n";
my $content = $response->as_string;
print "$content\n";
Sending password in a cookie? Nope.
Disallow GET for /login.
POST username and password to /login, over SSL.
In CGI, the GET/POST is indicated via the REQUEST_METHOD environment variable.
You cannot stop determined people from issuing a GET request to your server, but you can refuse to process it like so (untested code - you have to fill in details):
if ($ENV{REQUEST_METHOD} ne 'POST') {
# issue a redirect to a suitable error page, then return.
}
my $q = CGI->new();
my $user = $q->params('username');
my $password = $q->params('password');
my $encrypted_password = my_password_encryptor($password);
unless ( can_log_in($user, $encrypted_password) ) {
# issue an error message - redirect&return or fall-through...
}
else {
$session->set_user_logged_in();
}
Most people do not roll their own authentication or session handling. They mostly use one from CPAN, or one included with the larger app framework. If you're doing CGI, you can use CGI::Session.
You might give CGI::Application and/or its offspring a look. Those authors have already solved a bunch of the problems that you're encountering.

Special Characters in password causing Basic Auth Failure in Mojolicious UA

The following program fails when trying to go to an https web site that requires basic authentication.
use Mojo::UserAgent;
my $ua = Mojo::UserAgen->new;
my $user = "foobar";
my $pass = "Cant#change";
my $url = "https://$user:$pass\#site.foo.com";
my $tx = $ua->get($url);
if (my $res = $tx->success) {
say $res->body;
}
else {
my ($message, $code) = $tx->error;
say $code ? "$code response $message" : "Connection error: $message";
}
When I run with MOJO_USERAGENT_DEBUG=1 I get the following output:
-- Blocking request (https://foobar:cant#change#site.foo.com)
-- Connect (https:foobar:Cant:443)
Connection error: Couldn't connect
Using Mojolicious 3.35 updated from CPAN. Unfortunately, passwords will likely contain "special characters" (ascii #!#%^& and the like) and changing the password to something not containing a # is not an option. The web server handles the request correctly in web browsers, so I do not believe it is a web server configuration issue.
So is there another way to achieve this in Mojo?
The error is yours, not Mojo's. Specifically, the URL is incorrectly built. Fix:
use URI::Escape qw( uri_escape );
my $creds = uri_escape($user) . ':' . uri_escape($pass);
my $url = 'https://' . $creds . '#site.foo.com/';
use Mojo::Base -strict;
use Mojo::URL;
#1 Mojo way
my $url = Mojo::URL->new('http://google.com/')->userinfo('user:pa#ss');
say $url;
#2 or manually
use Mojo::Util qw/url_escape/;
my $auth = join ':', url_escape('user'), url_escape('pa#ss');
my $url2 = qq{http://$auth\#google.com/};
say $url2;