The following program fails when trying to go to an https web site that requires basic authentication.
use Mojo::UserAgent;
my $ua = Mojo::UserAgen->new;
my $user = "foobar";
my $pass = "Cant#change";
my $url = "https://$user:$pass\#site.foo.com";
my $tx = $ua->get($url);
if (my $res = $tx->success) {
say $res->body;
}
else {
my ($message, $code) = $tx->error;
say $code ? "$code response $message" : "Connection error: $message";
}
When I run with MOJO_USERAGENT_DEBUG=1 I get the following output:
-- Blocking request (https://foobar:cant#change#site.foo.com)
-- Connect (https:foobar:Cant:443)
Connection error: Couldn't connect
Using Mojolicious 3.35 updated from CPAN. Unfortunately, passwords will likely contain "special characters" (ascii #!#%^& and the like) and changing the password to something not containing a # is not an option. The web server handles the request correctly in web browsers, so I do not believe it is a web server configuration issue.
So is there another way to achieve this in Mojo?
The error is yours, not Mojo's. Specifically, the URL is incorrectly built. Fix:
use URI::Escape qw( uri_escape );
my $creds = uri_escape($user) . ':' . uri_escape($pass);
my $url = 'https://' . $creds . '#site.foo.com/';
use Mojo::Base -strict;
use Mojo::URL;
#1 Mojo way
my $url = Mojo::URL->new('http://google.com/')->userinfo('user:pa#ss');
say $url;
#2 or manually
use Mojo::Util qw/url_escape/;
my $auth = join ':', url_escape('user'), url_escape('pa#ss');
my $url2 = qq{http://$auth\#google.com/};
say $url2;
Related
I have a simple code like this:
use LWP::UserAgent;
use HTTP::Cookies;
use HTTP::Request;
my $cookies = HTTP::Cookies->new();
my $browser = LWP::UserAgent->new();
$browser->agent(' ... ');
$browser->cookie_jar($cookies);
my $request = HTTP::Request->new();
my $response;
my $url;
my $referer;
$referer = '';
$url = 'https:// ...'; # url #1
$request->url($url);
$request->method('GET');
$request->header('Referer' => $referer);
$response = $browser->request($request);
print $response->request()->uri() . "\n\n" .
$response->headers()->as_string . "\n\n" .
$response->content . "\n\n";
$referer = $response->request()->uri();
$url = 'https:// ... '; # url #2
$request->url($url);
$request->method('GET');
$request->header('Referer' => $referer);
$response = $browser->request($request);
print $response->request()->uri() . "\n\n" .
$response->headers()->as_string . "\n\n" .
$response->content . "\n\n";
Now, I want to see full HTTP request headers as well, not just response headers.
How can I do it? What has to be added to this code?
I think you almost have it in your existing code. You are accessing the request URI with $response->request()->uri(). The ->request() is your HTTP::Request object. I believe that you can use $response->request->headers->as_string to get what you want.
print $response->request->as_string
This will show you requests as well as responses.
use LWP::UserAgent;
use LWP::ConsoleLogger::Easy qw( debug_ua );
my $browser = LWP::UserAgent->new();
debug_ua( $browser );
$request->headers->as_string and $response->request->headers->as_string will you get you the headers of the first and last request passed to Net::HTTP by LWP[1], but these aren't quite what Net::HTTP sends. For example, Net::HTTP can add a Content-Length header, a TE header, and/or a number of others.
Net::HTTP doesn't keep a record of the headers it actually sends. You will need a wire sniffer (e.g. tcpdump) or a debugging proxy (e.g. Fiddler) for that. You could also use a debugger or trace statements to view the request prepared in Net::HTTP::Methods's format_request. The most convenient, however, might be to wrap Net::HTTP::Methods's format_request.
These are the same unless the initial request was redirected. To get all the requests (and responses), you can use:
while ($response) {
my $request = $response->request;
...
$response = $response->previous;
}
I'm trying to use Asana API with HTTP Basic Auth. The following program prints
{"errors":[{"message":"Not Authorized"}]}
It seems that LWP doesn't send the auth credentials to the server.
#!/usr/bin/perl
use v5.14.0;
use LWP;
my $ua = new LWP::UserAgent;
$ua->credentials('app.asana.com:443', 'realm', 'api_key_goes_here' => '');
my $res = $ua->get("https://app.asana.com/api/1.0/users/me");
say $res->decoded_content;
I've run into something similar (on a completely different service), and couldn't get it working. I think it's to do with a realm/hostname mismatch.
As you note - if you hit that URL directly, from a web browser, you get the same answer (without an auth prompt).
But what I ended up doing instead:
my $request = HTTP::Request -> new ( 'GET' => 'https://path/to/surl' );
$request -> authorization_basic ( 'username', 'password' );
my $results = $user_agent -> request ( $request );
I have a CGI server side script that accepts GET and POST, with login parameters.
I want to test it to make sure it is not vulnerable. So the plan is to use Perl LWP, and send login parameters in GET and POST, and compare the results. the interface has been changed, so that only in POST we can send user-name and password in session cookies ( not sure if that is a great idea ) , so how do i test it ? Here is what i have so far:
#!/usr/bin/perl
use LWP;
print "This is libwww-perl-$LWP::VERSION\n";
# Create a user agent object
use LWP::UserAgent;
my $ua = LWP::UserAgent->new;
$ua->agent("MyApp/0.1 ");
# Create a request
#my $req = HTTP::Request->new(POST => 'http://search.cpan.org/search');
#my $req = HTTP::Request->new(GET => 'https://qa.co.net:443/cgi-bin/n-cu.cgi');
my $req = HTTP::Request->new(GET => 'https://qa.co.net:443/cgi-bin/n-cu.cgi?mode=frameset&JScript=1&remote_user&login=foo&password=foo HTTP/1.1');
$req->content_type('application/x-www-form-urlencoded');
$req->content('query=libwww-perl&mode=dist');
# Pass request to the user agent and get a response back
my $res = $ua->request($req);
# Check the outcome of the response
if ($res->is_success) {
print $res->content;
#print $res->code;
#print $res->message;
}
else {
print $res->status_line, "\n";
}
This is not going to do it, since it does not have the session cookie stuff. But might be a good start though. Is this the right way to test the GET and POST ?
Here is what was implemented in the cgi:
#cr_login for POST && login for GET -- leave GET param as it used to be.
if ($m eq 'GET' && defined($req->param('login'))) {
$msg = 'parameter "login" is invalid for this request type.';
+ my $seclog = $event_logging_directory . '/invalid_request.log';
+ open(S, ">>$seclog") or die $!;
+ my $logmsg = sprintf("%4d-%02d-%02d %02d:%02d:%02d",Today_and_Now())
+ . "|mode:" . $req->param('mode')
+ . "|login:" . $req->param('login')
+ . "|remote_addr:" . $ENV{REMOTE_ADDR}
+ . "|$msg\n";
+ print S $logmsg;
and :
POST request to n-cu.cgi should use parameter "cr_login". If the parameter "login" is passed in a post request, it should throw error and return to login screen.
GET request to n-cu.cgi should use the parameter "login". If the parameter "cr_login" is passed in a post request, it should throw error and return to login screen.
so here is how we do it:
Keep the session cookie and context alive :
my $browser = LWP::UserAgent->new(keep_alive => 10);
$browser->cookie_jar( {} );
$browser->agent('Mozilla/8.0');
#$browser->ssl_opts({ verify_hostname => 0 });
$browser->show_progress(1);
and later: print the response
print "Cookies:\n", Dumper($browser->cookie_jar()), "\n\n";
my $content = $response->as_string;
print "$content\n";
Sending password in a cookie? Nope.
Disallow GET for /login.
POST username and password to /login, over SSL.
In CGI, the GET/POST is indicated via the REQUEST_METHOD environment variable.
You cannot stop determined people from issuing a GET request to your server, but you can refuse to process it like so (untested code - you have to fill in details):
if ($ENV{REQUEST_METHOD} ne 'POST') {
# issue a redirect to a suitable error page, then return.
}
my $q = CGI->new();
my $user = $q->params('username');
my $password = $q->params('password');
my $encrypted_password = my_password_encryptor($password);
unless ( can_log_in($user, $encrypted_password) ) {
# issue an error message - redirect&return or fall-through...
}
else {
$session->set_user_logged_in();
}
Most people do not roll their own authentication or session handling. They mostly use one from CPAN, or one included with the larger app framework. If you're doing CGI, you can use CGI::Session.
You might give CGI::Application and/or its offspring a look. Those authors have already solved a bunch of the problems that you're encountering.
This Perl code works with Anonymous access to an ASP.NET web service, but when integrated security is turned on, the service returns 401 errors. I think I need to use the NTLM module in conjunction with SOAP::Lite, but it's not clear how to do so. How can these components be integrated?
use SOAP::Lite;
use strict;
my $proxy = "http://localhost:28606/WebService.asmx";
my $method_name = "HelloWorld";
my $uri = "http://tempuri.org/";
my $methodAction = $uri . $method_name;
my $soap = SOAP::Lite
->uri( $uri )
->proxy( $proxy )
->on_action(sub{ $methodAction; });
my $method = SOAP::Data->name($method_name)->attr({xmlns=>$uri});
my $result = $soap->call($method);
print $result->result();
You can get SOAP::Lite to print some debugging output if you do:
use SOAP::Lite +trace;
instead of
use SOAP::Lite;
EDIT:
OK, I think I get it now. Turning on the integrated security feature makes IIS require NTLM authentication. There's a thread over at perlmonks.org that seems to reveal the answer.
I'm a bit late, but I just faced the same problem. Try this:
use LWP::UserAgent;
use LWP::Debug;
use SOAP::Lite on_action => sub { "$_[0]$_[1]"; };
import SOAP::Data 'name', 'value';
our $sp_endpoint = 'http://sp.example.com/sites/mysite/_vti_bin/lists.asmx';
our $sp_domain = 'sp.example.com:80';
our $sp_username = 'DOMAIN\username';
our $sp_password = 'xyz';
if ($debug) {
LWP::Debug::level('+');
SOAP::Lite->import(+trace => 'all');
}
my #ua_args = (keep_alive => 1);
my #credentials = ($sp_domain, "", $sp_usernam, $sp_password);
my $schema_ua = LWP::UserAgent->new(#ua_args);
$schema_ua->credentials(#credentials);
$soap = SOAP::Lite->proxy($sp_endpoint, #ua_args, credentials => \#credentials);
$soap->schema->useragent($schema_ua);
$soap->uri("http://schemas.microsoft.com/sharepoint/soap/");
I have some code I've written in PHP for consuming our simple webservice, which I'd also like to provide in Perl for users who may prefer that language. What's the simplest method of making a HTTP request to do that? In PHP I can do it in one line with file_get_contents().
Here's the entire code I want to port to Perl:
/**
* Makes a remote call to the our API, and returns the response
* #param cmd {string} - command string ID
* #param argsArray {array} - associative array of argument names and argument values
* #return {array} - array of responses
*/
function callAPI( $cmd, $argsArray=array() )
{
$apikey="MY_API_KEY";
$secret="MY_SECRET";
$apiurl="https://foobar.com/api";
// timestamp this API was submitted (for security reasons)
$epoch_time=time();
//--- assemble argument array into string
$query = "cmd=" .$cmd;
foreach ($argsArray as $argName => $argValue) {
$query .= "&" . $argName . "=" . urlencode($argValue);
}
$query .= "&key=". $apikey . "&time=" . $epoch_time;
//--- make md5 hash of the query + secret string
$md5 = md5($query . $secret);
$url = $apiurl . "?" . $query . "&md5=" . $md5;
//--- make simple HTTP GET request, put the server response into $response
$response = file_get_contents($url);
//--- convert "|" (pipe) delimited string to array
$responseArray = explode("|", $response);
return $responseArray;
}
LWP::Simple:
use LWP::Simple;
$contents = get("http://YOUR_URL_HERE");
LWP::Simple has the function you're looking for.
use LWP::Simple;
$content = get($url);
die "Can't GET $url" if (! defined $content);
Take a look at LWP::Simple.
For more involved queries, there's even a book about it.
I would use the LWP::Simple module.
Mojo::UserAgent is a great option too!
use Mojo::UserAgent;
my $ua = Mojo::UserAgent->new;
# Say hello to the Unicode snowman with "Do Not Track" header
say $ua->get('www.☃.net?hello=there' => {DNT => 1})->res->body;
# Form POST with exception handling
my $tx = $ua->post('https://metacpan.org/search' => form => {q => 'mojo'});
if (my $res = $tx->success) { say $res->body }
else {
my ($err, $code) = $tx->error;
say $code ? "$code response: $err" : "Connection error: $err";
}
# Quick JSON API request with Basic authentication
say $ua->get('https://sri:s3cret#example.com/search.json?q=perl')
->res->json('/results/0/title');
# Extract data from HTML and XML resources
say $ua->get('www.perl.org')->res->dom->html->head->title->text;`
Samples direct from CPAN page. I used this when I couldn 't get LWP::Simple to work on my machine.
Try the HTTP::Request module.
Instances of this class are usually passed to the request() method of an LWP::UserAgent object.
If it's in Unix and if LWP::Simple isn't installed, you can try:
my $content = `GET "http://trackMyPhones.com/"`;
I think what Srihari might be referencing is Wget, but I would actually recommend (again, on *nix without LWP::Simple) to use cURL:
$ my $content = `curl -s "http://google.com"`;
<HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8">
<TITLE>301 Moved</TITLE></HEAD><BODY>
<H1>301 Moved</H1>
The document has moved
here.
</BODY></HTML>
The -s flag tells curl to be silent. Otherwise, you get curl's progress bar output on standard error every time.