Why does LWP::UserAgent succeed and Mojo::UserAgent fail? - perl

If I make a request like this:
my $mojo_ua = Mojo::UserAgent->new->max_redirects(5);
$mojo_ua->inactivity_timeout(60)->connect_timeout(60)->request_timeout(60);;
$mojo_ua->transactor->name('Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Safari/537.36');
my $headers = {
'Accept' => 'application/json',
'Accept-Language' => 'en-US,en;q=0.5',
'Connection' => 'keep-alive',
'User-Agent' => 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Safari/537.36',
'x-csrf-token' => 'Fetch',
'Accept-Encoding' => 'gzip, deflate, br',
'DataServiceVersion' => '2.0',
'MaxDataServiceVersion' => '2.0',
'Referer' => 'https://blah.blas.com/someThing.someThing'
};
my $url = Mojo::URL->new('https://blah.blah.com/irj/go/sap/FOO_BAR_BAZ/');
my $tx = $mojo_ua->get($url, $headers);
$tx = $mojo_ua->start($tx);
my $res = $tx->result;
the request times out, but if I take the exact same request, built in the same way and do this:
my $lwp_ua = LWP::UserAgent->new;
my $req = HTTP::Request->parse( $tx->req->to_string );
$req->uri("$url");
my $res = $lwp_ua->request($req);
it succeeds.
It happens in a few cases that Mojo::UserAgent fails, and LWP::UserAgent succeeds with exactly the same transaction, and I'm starting to get curious.
Any idea as to why?

Your call to
$mojo_ua->get($url, $headers)
has already sent the HTTP request and received the response from the server, errored, or timed out. You don't need to call
$mojo_ua->start($tx)
as well, and that statement should be removed
If you really want to first build the transaction and then start it, you need
my $tx = $mojo_ua->build_tx(GET => $url, $headers);
$tx = $mojo_ua->start($tx);
but I don't see any reason why you should need to do it this way

Related

How to prevant request by guzzle get redirected by client?

How to prevant request by guzzle get redirected by client?
screnshoot
i already give the parameter allow_redirects set to false but still my request get redirected.
$client = new \GuzzleHttp\Client([
'headers' => [
'User-Agent' => 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.102 Safari/537.36',
], 'verify' => false
]);
$response = $client->post(
"{$value->url}/wp-json/apg/v1/webhooks",
[
'form_params' => [
'token_auth' => $this->setting->token_auth,
'webhooks_type'=> 'theme',
'theme' => $theme,
'url_download' => url('/storage/themes')
],
'allow_redirects' => FALSE
]
);
$jsonResponse = json_decode($response->getBody()->getContents());

Invalid character found in method name. HTTP method names must be tokens, persists even with http request

I am trying to warm up my controller so that the service gets hot during each deployment.
In order to do this i have written a perl script as below:
#!perl
use strict;
use warnings;
use WWW::Mechanize;
use HTTP::Request;
my $ua = WWW::Mechanize->new();
my $r = HTTP::Request->new(
'GET' =>
'http://gaurav_setia.microsoft.com:8080/b2h/homepage?_encoding=UTF8&opf_redir=1&portalDebug=1',
[
'Connection' => 'Keep-Alive',
'Via' => 'HTTP/1.1 ShoppingSchedule',
'Accept' =>
'text/x-html-parts,text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8',
'Accept-Charset' => 'UTF-8',
'Accept-Encoding' => 'identity',
'Accept-Language' => 'en-US',
'Host' => 'gaurav_setia.microsoft.com',
'User-Agent' =>
'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.80 Safari/537.36',
'Cookie' =>
'Original-X-Forwarded-For' => '10.45.103.166',
'X-MS-Internal-Ip-Class' => 'rfc1918',
'X-MS-Internal-Via' =>
'1.1 us-beta-opf-1a-1-67440dc2.us-east-1.ms.com (OPF)',
'X-MS-Urlspace' => 'NoPageType',
'X-MS-Portal-Customer-Id' => 'AMY4OD2PMM9T31',
'X-MS-Portal-Default-Merchant-Id' => 'BTLPDKIKX0DE41',
'X-MS-Portal-Device-Attr' => 'desktop',
'X-MS-Portal-Language' => 'en_US',
'X-MS-Portal-Marketplace-Id' => 'ATVPDKIKX0DER',
'X-MS-Portal-Page-Type' => 'AQGate',
'X-MS-Portal-Request-Attr' => 'internal, http, portal-debug',
'X-MS-Portal-Session-Id' => '1M0-493PO66-0596753',
'X-MS-Portal-Ubid' => '1P2-465OP632-8831161',
'X-MS-Portal-User-Attr' => 'business',
'X-MS-Rid' => 'G308MPK95BWTA69EY2MW',
'X-Forwarded-For' => '10.45.101.126',
'X-Forwarded-Host' => 'development.ms.com',
'X-Forwarded-Server' =>
'development.ms.com, b-hp-shpomnpng-na-2b-02af3555.us-west-2.amazon.com',
'X-Original-Args' => 'portalDebug=1',
'X-Original-Method' => 'GET',
'X-Original-Scheme' => 'http',
'X-Original-Uri' => '/',
],
);
my $res = $ua->request( $r, );
if ( $res->is_success() )
{
print $response->is_success();
}
print $response->status_line;
This script should run after each deployment.
But in the catalina.out logs i am getting the following error:
Dec 13, 2018 9:08:11 AM org.apache.coyote.http11.AbstractHttp11Processor process
INFO: Error parsing HTTP request header
Note: further occurrences of HTTP header parsing errors will be logged at DEBUG level.
java.lang.IllegalArgumentException: Invalid character found in method name. HTTP method names must be tokens
at org.apache.coyote.http11.AbstractNioInputBuffer.parseRequestLine(AbstractNioInputBuffer.java:235)
at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1055)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:684)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1539)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1495)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:748)
I am unable to find the fix!
Many answers say that this is due to https/http issue, but i am making a http call here itself!
In amongst your pile of headers, you have this:
'User-Agent' =>
'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.80 Safari/537.36',
'Cookie' =>
'Original-X-Forwarded-For' => '10.45.103.166',
Notice that there's no value for the Cookie header. That means all of the headers after that will be wrong (the names and values will be muddled up).
Either remove the Cookie line completely or set its value to undef.
'Cookie' => undef,
(Removing it is probably best)

Perl: Need an LWP & HTTP::Request POST code that actually works

I have been scratching my head trying to get LWP and HTTP::Request to actually pass a POST parameter to a web server. The web server can see the fact that the request was a POST transaction, but it is not picking up the passed parameters. I have been searching all day on this and have tried different things and I have yet to find something that works. (The web server is working, I am able to manually send post transactions and when running the whole script, I am getting '200' status but I am not seeing any posted elements. Any help would be appreciated. Tnx.
my $ua2 = LWP::UserAgent->new;
$ua2->agent("Mozilla/5.0 (compatible; MSIE 6.0; Windows 98)");
my $req2 = HTTP::Request->new(POST => "$url", [ frm-advSearch => 'frmadvSearch' ]);
$req2->content_type('text/html');
my $res2 = $ua2->request($req2);
$http_stat = substr($res2->status_line,0,3);
my $res = $ua->post($url,
Content => [
'frm-advSearch' => 'frmadvSearch',
],
);
which is short for
use HTTP::Request::Common qw( POST );
my $req = POST($url,
Content => [
'frm-advSearch' => 'frmadvSearch',
],
);
my $res = $ua->request($req);
Here's a Mojo::UserAgent example, which I find easier to debug:
use Mojo::UserAgent;
my $ua = Mojo::UserAgent->new;
$ua->transactor->name( 'Mozilla/5.0 (compatible; MSIE 6.0; Windows 98)' );
my $url = 'http://www.example.com/form/';
my $tx = $ua->post( $url, form => { 'frm-advSearch' => 'frmadvSearch' } );
say $tx->req->to_string;
The transaction in $tx knows about the request so I can look at that:
POST /form/ HTTP/1.1
Content-Type: application/x-www-form-urlencoded
User-Agent: Mozilla/5.0 (compatible; MSIE 6.0; Windows 98)
Accept-Encoding: gzip
Host: www.example.com
Content-Length: 26
frm-advSearch=frmadvSearch

Getting "500 Internal Server Error" retrieving page with LWP::UserAgent

I'm trying to retrieve a page using LWP::UserAgent but I keep getting a "500 Internal Server Error" as a response. Retrieving the exact same page in Firefox (using a fresh "Private Window" - so without any cookies set yet) succeeds without a problem.
I've duplicated the headers exactly as sent by Firefox, but that still does not make a difference. Here's my full code:
use strict;
use LWP::UserAgent;
my $browserObj = LWP::UserAgent->new();
$browserObj->cookie_jar( {} );
$browserObj->timeout(600);
my #header = (
'Host' => 'www.somedomain.com',
'User-Agent' => 'Mozilla/5.0 (Windows NT 6.1; WOW64; rv:53.0) Gecko/20100101 Firefox/53.0',
'Accept-Language' => 'en-US,en;q=0.5',
'Accept' => 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
'Accept-Encoding' => 'gzip, deflate, br',
'DNT' => '1',
'Connection' => 'keep-alive',
'Upgrade-Insecure-Requests' => '1'
);
my $URL = "https://www.somedomain.com";
my $response = $browserObj->get( $URL, #header );
if( $response->is_success ) {
print "Success!\n";
} else {
print "Error: " . $response->status_line . ".\n" );
}
The real web address is something other than "www.somedomain.com". In fact, it's a URL to an online casino, but I don't want my question be regarded as spam.
But anyone any idea what could be wrong?
On our corporate network which has a proxy (and an out of date perl version - there may be better options in newer versions) we tend to add the following for one-offs:
BEGIN {
$ENV{HTTPS_DEBUG} = 1; # optional but can help if you get a response
$ENV{HTTPS_PROXY} = 'https://proxy.server.here.net:8080';
}
If we don't do this the script simply fails to connect with no other information.
You may also want to add something like this if you want to inspect the messages:
$browserObj->add_handler("request_send", sub { shift->dump; return });
$browserObj->add_handler("response_done", sub { shift->dump; return });

Parsing Cyrillic sites with Zend Framework encoding issue

$url = "http://www.kinopoisk.ru/picture/791547/";
require_once 'Zend/Http/Client.php';
require_once 'Zend/Dom/Query.php';
$client = new Zend_Http_Client($url, array(
'maxredirects' => 0,
'timeout' => 30,
'useragent' => 'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/535.7 (KHTML, like Gecko) Chrome/16.0.912.36 Safari/535.7'));
$response = $client->request();
$html = $response->getBody();
$dom = new Zend_Dom_Query($html);
$title = $dom->query('title');
$titleText = $title->current()->textContent;
echo $titleText;
// Locally returns "Постеры: ВАЛЛ·И (WALL·E)"
// Remotely returns "Ïîñòåðû: ÂÀËË·È (WALL·E)"
.htaccess setting: AddDefaultCharset utf-8
Response headers in remote site:
Content-Encoding gzip
Vary Accept-Encoding
Though there are no such responses in local
So I think that it's the server fault? how to resolve this?