How to prevant request by guzzle get redirected by client?
screnshoot
i already give the parameter allow_redirects set to false but still my request get redirected.
$client = new \GuzzleHttp\Client([
'headers' => [
'User-Agent' => 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.102 Safari/537.36',
], 'verify' => false
]);
$response = $client->post(
"{$value->url}/wp-json/apg/v1/webhooks",
[
'form_params' => [
'token_auth' => $this->setting->token_auth,
'webhooks_type'=> 'theme',
'theme' => $theme,
'url_download' => url('/storage/themes')
],
'allow_redirects' => FALSE
]
);
$jsonResponse = json_decode($response->getBody()->getContents());
Related
After upgrading to Craft 3 from 2, I'm getting a 500 Internal Server Error on a select few pages. For example:
/reports (Doesn't work)
/reports/slug (Works)
/publications (Doesn't work)
/publications/slug (Works)
For some reason it's not the same for all paths. /media works just fine. This is what shows up in my logs but I have no idea what it means. Also, the error screen I get is a Craft-styled page.
2020-09-25 00:37:46 [-][1][80b3e94259710a336582ea71cac98d1a][info][application] $_GET = [
'p' => 'publications'
]
$_POST = []
$_FILES = []
$_COOKIE = [
'CraftSessionId' => '80b3e94259710a336582ea71cac98d1a'
'1031b8c41dfff97a311a7ac99863bdc5_identity' => 'f108309014ae583a6eb6ce545f6acdd1c25222a18bee9c382e0050742d3cd949a:2:{i:0;s:41:\"1031b8c41dfff97a311a7ac99863bdc5_identity\";i:1;s:246:\"[\"1\",\"[\\\"xS6JjF6hHAbX59SK5XGpi5Ktmw-FTf-phJhV7uix7aalULuGzl308wNhAVNTGY2dPm7xXOk6dLOgaR2NbBXEqhhZFF649LoNhLnu\\\",null,\\\"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.83 Safari/537.36\\\"]\",1209600]\";}'
'1031b8c41dfff97a311a7ac99863bdc5_username' => '9a33d470ce6b2e033bab618b31164d34798869ae07d8c7515cc445de0603e560a:2:{i:0;s:41:\"1031b8c41dfff97a311a7ac99863bdc5_username\";i:1;s:25:\"mollie#email.com\";}'
'CRAFT_CSRF_TOKEN' => '•••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••'
]
$_SESSION = [
'bd62416aa8538ede709019a5e113eea5__flash' => []
'1031b8c41dfff97a311a7ac99863bdc5__token' => '••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••'
'1031b8c41dfff97a311a7ac99863bdc5__id' => '1'
'1031b8c41dfff97a311a7ac99863bdc5__expire' => 1600997866
'bd62416aa8538ede709019a5e113eea5__auth_access' => [
0 => 'editStructure:5'
1 => 'editStructure:11'
2 => 'editStructure:9'
3 => 'editStructure:6'
4 => 'editStructure:8'
]
]
$_SERVER = [
'TMP' => '/srv/users/serverpilot/tmp/water-institute'
'TEMP' => '/srv/users/serverpilot/tmp/water-institute'
'TMPDIR' => '/srv/users/serverpilot/tmp/water-institute'
'PATH' => '/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin'
'USER' => 'serverpilot'
'HOME' => '/srv/users/serverpilot'
'SCRIPT_NAME' => '/index.php'
'REQUEST_URI' => '/publications'
'QUERY_STRING' => 'p=publications'
'REQUEST_METHOD' => 'GET'
'SERVER_PROTOCOL' => 'HTTP/1.0'
'GATEWAY_INTERFACE' => 'CGI/1.1'
'REDIRECT_QUERY_STRING' => 'p=publications'
'REDIRECT_URL' => '/publications'
'REMOTE_PORT' => '38390'
'SCRIPT_FILENAME' => '/srv/users/serverpilot/apps/water-institute/public/index.php'
'SERVER_ADMIN' => 'webmaster#'
'CONTEXT_DOCUMENT_ROOT' => '/srv/users/serverpilot/apps/water-institute/public'
'CONTEXT_PREFIX' => ''
'REQUEST_SCHEME' => 'https'
'DOCUMENT_ROOT' => '/srv/users/serverpilot/apps/water-institute/public'
'REMOTE_ADDR' => '209.555.242.105'
'SERVER_PORT' => '443'
'SERVER_ADDR' => '127.0.0.1'
'SERVER_NAME' => 'dev.thewaterinstitute.org'
'SERVER_SOFTWARE' => 'Apache/2.4.46 (Unix) OpenSSL/1.0.2g'
'SERVER_SIGNATURE' => ''
'LD_LIBRARY_PATH' => '/opt/sp/apache/lib:/opt/sp/apache/lib'
'HTTP_COOKIE' => 'CraftSessionId=80b3e94259710a336582ea71cac98d1a; 1031b8c41dfff97a311a7ac99863bdc5_identity=f108309014ae583a6eb6ce545f6acdd1c25222a18bee9c382e0050742d3cd949a%3A2%3A%7Bi%3A0%3Bs%3A41%3A%221031b8c41dfff97a311a7ac99863bdc5_identity%22%3Bi%3A1%3Bs%3A246%3A%22%5B%221%22%2C%22%5B%5C%22xS6JjF6hHAbX59SK5XGpi5Ktmw-FTf-phJhV7uix7aalULuGzl308wNhAVNTGY2dPm7xXOk6dLOgaR2NbBXEqhhZFF649LoNhLnu%5C%22%2Cnull%2C%5C%22Mozilla%2F5.0+%28Windows+NT+10.0%3B+Win64%3B+x64%29+AppleWebKit%2F537.36+%28KHTML%2C+like+Gecko%29+Chrome%2F85.0.4183.83+Safari%2F537.36%5C%22%5D%22%2C1209600%5D%22%3B%7D; 1031b8c41dfff97a311a7ac99863bdc5_username=9a33d470ce6b2e033bab618b31164d34798869ae07d8c7515cc445de0603e560a%3A2%3A%7Bi%3A0%3Bs%3A41%3A%221031b8c41dfff97a311a7ac99863bdc5_username%22%3Bi%3A1%3Bs%3A25%3A%22mollie%40email.com%22%3B%7D; CRAFT_CSRF_TOKEN=25773457aeb176b611944eb8242d07ca89d6db0513942ffa87741e60f6789b13a%3A2%3A%7Bi%3A0%3Bs%3A16%3A%22CRAFT_CSRF_TOKEN%22%3Bi%3A1%3Bs%3A208%3A%226WM7NPL9Vbakyko4DJ-9pHHorwTsXXoGGZTOowAj%7Cedd0c62c7d374da3e339e8d4cc1fa8143778f59331f0d207580bec8e76567a666WM7NPL9Vbakyko4DJ-9pHHorwTsXXoGGZTOowAj%7C1%7C%242y%2413%24SPzKwYvJwBZgGjgrnHip6u2WVgGOtTJhn4Rix9UqItYl1o%2FMzgRZK%22%3B%7D'
'HTTP_ACCEPT_LANGUAGE' => 'en-US,en;q=0.9,la;q=0.8'
'HTTP_ACCEPT_ENCODING' => 'gzip, deflate, br'
'HTTP_SEC_FETCH_DEST' => 'document'
'HTTP_SEC_FETCH_USER' => '?1'
'HTTP_SEC_FETCH_MODE' => 'navigate'
'HTTP_SEC_FETCH_SITE' => 'none'
'HTTP_ACCEPT' => 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9'
'HTTP_USER_AGENT' => 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.83 Safari/537.36'
'HTTP_UPGRADE_INSECURE_REQUESTS' => '1'
'HTTP_CONNECTION' => 'close'
'HTTP_X_FORWARDED_PROTO' => 'https'
'HTTP_X_FORWARDED_SSL' => 'on'
'HTTP_X_FORWARDED_FOR' => '209.555.242.105'
'HTTP_HOST' => 'dev.mydomain.org'
'SCRIPT_URI' => 'https://dev.mydomain.org/publications'
'SCRIPT_URL' => '/publications'
'HTTPS' => 'on'
'REDIRECT_STATUS' => '200'
'REDIRECT_SCRIPT_URI' => 'https://dev.mydomain.org/publications'
'REDIRECT_SCRIPT_URL' => '/publications'
'REDIRECT_HTTPS' => 'on'
'FCGI_ROLE' => 'RESPONDER'
'PHP_SELF' => '/index.php'
'REQUEST_TIME_FLOAT' => 1600994266.1376
'REQUEST_TIME' => 1600994266
'argv' => [
0 => 'p=publications'
]
'argc' => 1
'ENVIRONMENT' => 'dev'
'SECURITY_KEY' => '••••••••••••••••'
'DB_DRIVER' => 'mysql'
'DB_SERVER' => 'localhost'
'DB_USER' => 'e39b766ddf0a'
'DB_PASSWORD' => '••••••••••••••••'
'DB_DATABASE' => 'water-institute'
'DB_SCHEMA' => 'public'
'DB_TABLE_PREFIX' => 'craft'
'DB_PORT' => ''
'ROOT_URL' => 'https://dev.mydomain.org'
]
I am trying to warm up my controller so that the service gets hot during each deployment.
In order to do this i have written a perl script as below:
#!perl
use strict;
use warnings;
use WWW::Mechanize;
use HTTP::Request;
my $ua = WWW::Mechanize->new();
my $r = HTTP::Request->new(
'GET' =>
'http://gaurav_setia.microsoft.com:8080/b2h/homepage?_encoding=UTF8&opf_redir=1&portalDebug=1',
[
'Connection' => 'Keep-Alive',
'Via' => 'HTTP/1.1 ShoppingSchedule',
'Accept' =>
'text/x-html-parts,text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8',
'Accept-Charset' => 'UTF-8',
'Accept-Encoding' => 'identity',
'Accept-Language' => 'en-US',
'Host' => 'gaurav_setia.microsoft.com',
'User-Agent' =>
'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.80 Safari/537.36',
'Cookie' =>
'Original-X-Forwarded-For' => '10.45.103.166',
'X-MS-Internal-Ip-Class' => 'rfc1918',
'X-MS-Internal-Via' =>
'1.1 us-beta-opf-1a-1-67440dc2.us-east-1.ms.com (OPF)',
'X-MS-Urlspace' => 'NoPageType',
'X-MS-Portal-Customer-Id' => 'AMY4OD2PMM9T31',
'X-MS-Portal-Default-Merchant-Id' => 'BTLPDKIKX0DE41',
'X-MS-Portal-Device-Attr' => 'desktop',
'X-MS-Portal-Language' => 'en_US',
'X-MS-Portal-Marketplace-Id' => 'ATVPDKIKX0DER',
'X-MS-Portal-Page-Type' => 'AQGate',
'X-MS-Portal-Request-Attr' => 'internal, http, portal-debug',
'X-MS-Portal-Session-Id' => '1M0-493PO66-0596753',
'X-MS-Portal-Ubid' => '1P2-465OP632-8831161',
'X-MS-Portal-User-Attr' => 'business',
'X-MS-Rid' => 'G308MPK95BWTA69EY2MW',
'X-Forwarded-For' => '10.45.101.126',
'X-Forwarded-Host' => 'development.ms.com',
'X-Forwarded-Server' =>
'development.ms.com, b-hp-shpomnpng-na-2b-02af3555.us-west-2.amazon.com',
'X-Original-Args' => 'portalDebug=1',
'X-Original-Method' => 'GET',
'X-Original-Scheme' => 'http',
'X-Original-Uri' => '/',
],
);
my $res = $ua->request( $r, );
if ( $res->is_success() )
{
print $response->is_success();
}
print $response->status_line;
This script should run after each deployment.
But in the catalina.out logs i am getting the following error:
Dec 13, 2018 9:08:11 AM org.apache.coyote.http11.AbstractHttp11Processor process
INFO: Error parsing HTTP request header
Note: further occurrences of HTTP header parsing errors will be logged at DEBUG level.
java.lang.IllegalArgumentException: Invalid character found in method name. HTTP method names must be tokens
at org.apache.coyote.http11.AbstractNioInputBuffer.parseRequestLine(AbstractNioInputBuffer.java:235)
at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1055)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:684)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1539)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1495)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:748)
I am unable to find the fix!
Many answers say that this is due to https/http issue, but i am making a http call here itself!
In amongst your pile of headers, you have this:
'User-Agent' =>
'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.80 Safari/537.36',
'Cookie' =>
'Original-X-Forwarded-For' => '10.45.103.166',
Notice that there's no value for the Cookie header. That means all of the headers after that will be wrong (the names and values will be muddled up).
Either remove the Cookie line completely or set its value to undef.
'Cookie' => undef,
(Removing it is probably best)
If I make a request like this:
my $mojo_ua = Mojo::UserAgent->new->max_redirects(5);
$mojo_ua->inactivity_timeout(60)->connect_timeout(60)->request_timeout(60);;
$mojo_ua->transactor->name('Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Safari/537.36');
my $headers = {
'Accept' => 'application/json',
'Accept-Language' => 'en-US,en;q=0.5',
'Connection' => 'keep-alive',
'User-Agent' => 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Safari/537.36',
'x-csrf-token' => 'Fetch',
'Accept-Encoding' => 'gzip, deflate, br',
'DataServiceVersion' => '2.0',
'MaxDataServiceVersion' => '2.0',
'Referer' => 'https://blah.blas.com/someThing.someThing'
};
my $url = Mojo::URL->new('https://blah.blah.com/irj/go/sap/FOO_BAR_BAZ/');
my $tx = $mojo_ua->get($url, $headers);
$tx = $mojo_ua->start($tx);
my $res = $tx->result;
the request times out, but if I take the exact same request, built in the same way and do this:
my $lwp_ua = LWP::UserAgent->new;
my $req = HTTP::Request->parse( $tx->req->to_string );
$req->uri("$url");
my $res = $lwp_ua->request($req);
it succeeds.
It happens in a few cases that Mojo::UserAgent fails, and LWP::UserAgent succeeds with exactly the same transaction, and I'm starting to get curious.
Any idea as to why?
Your call to
$mojo_ua->get($url, $headers)
has already sent the HTTP request and received the response from the server, errored, or timed out. You don't need to call
$mojo_ua->start($tx)
as well, and that statement should be removed
If you really want to first build the transaction and then start it, you need
my $tx = $mojo_ua->build_tx(GET => $url, $headers);
$tx = $mojo_ua->start($tx);
but I don't see any reason why you should need to do it this way
$url = "http://www.kinopoisk.ru/picture/791547/";
require_once 'Zend/Http/Client.php';
require_once 'Zend/Dom/Query.php';
$client = new Zend_Http_Client($url, array(
'maxredirects' => 0,
'timeout' => 30,
'useragent' => 'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/535.7 (KHTML, like Gecko) Chrome/16.0.912.36 Safari/535.7'));
$response = $client->request();
$html = $response->getBody();
$dom = new Zend_Dom_Query($html);
$title = $dom->query('title');
$titleText = $title->current()->textContent;
echo $titleText;
// Locally returns "Постеры: ВАЛЛ·И (WALL·E)"
// Remotely returns "Ïîñòåðû: ÂÀËË·È (WALL·E)"
.htaccess setting: AddDefaultCharset utf-8
Response headers in remote site:
Content-Encoding gzip
Vary Accept-Encoding
Though there are no such responses in local
So I think that it's the server fault? how to resolve this?
I once wrote a simple 'crawler' to download http pages for me in JAVA.
Now I'm trying to rewrite to same thing to Perl, using LWP module.
This is my Java code (which works fine):
String referer = "http://example.com";
String url = "http://example.com/something/cgi-bin/something.cgi";
String params= "a=0&b=1";
HttpState initialState = new HttpState();
HttpClient httpclient = new HttpClient();
httpclient.setState(initialState);
httpclient.getParams().setCookiePolicy(CookiePolicy.NETSCAPE);
PostMethod postMethod = new PostMethod(url);
postMethod.addRequestHeader("Referer", referer);
postMethod.addRequestHeader("User-Agent", " Mozilla/5.0 (Windows; U; Windows NT 6.1; pl; rv:1.9.2.13) Gecko/20101203 Firefox/3.6.13");
postMethod.addRequestHeader("Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,/;q=0.8");
postMethod.addRequestHeader("Content-Type", "application/x-www-form-urlencoded");
String length = String.valueOf(params.length());
postMethod.addRequestHeader("Content-Length", length);
postMethod.setRequestBody(params);
httpclient.executeMethod(postMethod);
And this is the Perl version:
my $referer = "http://example.com/something/cgi-bin/something.cgi?module=A";
my $url = "http://example.com/something/cgi-bin/something.cgi";
my #headers = (
'User-Agent' => 'Mozilla/5.0 (Windows; U; Windows NT 6.1; pl; rv:1.9.2.13) Gecko/20101203 Firefox/3.6.13',
'Accept' => 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
'Referer' => $referer,
'Content-Type' => 'application/x-www-form-urlencoded',
);
my #params = (
'a' => '0',
'b' => '1',
);
my $browser = LWP::UserAgent->new( );
$browser->cookie_jar({});
$response = $browser->post($url, #params, #headers);
print $response->content;
The post request executes correctly, but I get another (main) webpage. As if cookies were not working properly...
Any guesses what is wrong?
Why I'm getting different result from JAVA and perl programs?
You can also use WWW::Mechanize, which is a wrapper around LWP::UserAgent. It gives you the cookie jar automatically.
You want to be creating hashes, not arrays - e.g. instead of:
my #params = (
'a' => '0',
'b' => '1',
);
You should use:
my %params = (
a => 0,
b => 1,
);
When passing the params to the LWP::UserAgent post method, you need to pass a reference to the hash, e.g.:
$response = $browser->post($url, \%params, %headers);
You could also look at the request you're sending to the server with:
print $response->request->as_string;
You can also use a handler to automatically dump requests and responses for debugging purposes:
$ua->add_handler("request_send", sub { shift->dump; return });
$ua->add_handler("response_done", sub { shift->dump; return });
I believe it has to do with $response = $browser->post($url, #params, #headers);
From the doc of LWP::UserAgent
$ua->post( $url, \%form )
$ua->post( $url, \#form )
$ua->post( $url, \%form, $field_name => $value, ... )
$ua->post( $url, $field_name => $value,... Content => \%form )
$ua->post( $url, $field_name => $value,... Content => \#form )
$ua->post( $url, $field_name => $value,... Content => $content )
Since your params and headers are as hashes, I would try this:
my $referer = "http://example.com/something/cgi-bin/something.cgi?module=A";
my $url = "http://example.com/something/cgi-bin/something.cgi";
my %headers = (
'User-Agent' => 'Mozilla/5.0 (Windows; U; Windows NT 6.1; pl; rv:1.9.2.13) Gecko/20101203 Firefox/3.6.13',
'Accept' => 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
'Referer' => $referer,
'Content-Type' => 'application/x-www-form-urlencoded',
);
my %params = (
'a' => '0',
'b' => '1',
);
my $browser = LWP::UserAgent->new( );
$browser->cookie_jar({});
$response = $browser->post($url, \%params, %headers);