why `wget` can not get redirection for certain website? - wget

wget hangs there while it accesses the following website. But when I use a browser to access it, it will be redirected to https://nyulangone.org. Does anybody know why wget can not get redirected in this case? Thanks.
$ wget http://nyumc.org
--2018-02-20 20:27:05-- http://nyumc.org/
Resolving nyumc.org (nyumc.org)... 216.165.125.106
Connecting to nyumc.org (nyumc.org)|216.165.125.106|:80...

When I used wget on the site you mentioned, this is what I get:
--2018-02-21 21:16:38-- http://www.nyumc.org/
Resolving www.nyumc.org (www.nyumc.org)... 216.165.125.112
Connecting to www.nyumc.org (www.nyumc.org)|216.165.125.112|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 179 [text/html]
Saving to: ‘index.html’
index.html 100%[==================================>] 179 --.-KB/s in 0s
2018-02-21 21:16:38 (8.16 MB/s) - ‘index.html’ saved [179/179]
In the index.html file, which bears the logo of NYU Langone Medical Center, it says: "The following URL has been rejected for security concerns. If you believe you have received this message in error, please summit an incident with our helpdesk at 212-263-6868..." So, it may not redirect because the website can detect that you are a bot and not a browser. You could attempt to change the user agent string and other HTTP headers to avoid detection, but I'm not sure why you wouldn't just turn wget on https://nyulangone.org. Judging from information on archive.org, nyumc.org has been redirecting to other sites for at least the last 5 years. It was redirecting to http://www.med.nyu.edu until 2016, at which point it started redirecting to https://www.nyulangone.org.
I hope that helps.

Related

Page on custom domain gives "too many redirects" error

I have a domain from Namecheap.com and hosted a page through github pages. Sometimes the my page fails to load and gives "too many redirects" error. I think the problem in somewhere in my advanced DNS. Take a look at it-enter image description here
Basically I have got 2 A records pointing to the IP address of Github and a CNAME record pointing to mygithubusername.github.io and an URL redirect record pointing to my domain. What am I doing wrong?
As far as I can see the problem comes from the URL redirect record. When trying to reach www.akshayud.me it redirects to akshayud.me and when it hit the URL redirect it redirects back to www.akshayud.me, which redirects to akshayud.me and is making an infinite loop.
To solve it remove the URL record from the DNS (which is my preferred solution) or fix the redirection from www to root on web server level.
Here is my wget output (I used only the URL record IP to guaranty 100% usage of the URL redirect record):
$ wget www.akshayud.me
--2017-12-19 08:03:18-- http://www.akshayud.me/
Resolving www.akshayud.me (www.akshayud.me)... 151.101.1.147, 151.101.65.147, 151.101.129.147, ...
Connecting to www.akshayud.me (www.akshayud.me)|151.101.1.147|:80... connected.
HTTP request sent, awaiting response... 301 Moved Permanently
Location: http://akshayud.me/ [following]
--2017-12-19 08:03:18-- http://akshayud.me/
Resolving akshayud.me (akshayud.me)... 162.255.119.148
Connecting to akshayud.me (akshayud.me)|162.255.119.148|:80... connected.
HTTP request sent, awaiting response... 302 Found
Location: http://www.akshayud.me/ [following]
--2017-12-19 08:03:18-- http://www.akshayud.me/
Connecting to www.akshayud.me (www.akshayud.me)|151.101.1.147|:80... connected.
HTTP request sent, awaiting response... 301 Moved Permanently
Location: http://akshayud.me/ [following]
--2017-12-19 08:03:18-- http://akshayud.me/
Connecting to akshayud.me (akshayud.me)|162.255.119.148|:80... connected.
HTTP request sent, awaiting response... 302 Found
Location: http://www.akshayud.me/ [following]
--2017-12-19 08:03:19-- http://www.akshayud.me/
Connecting to www.akshayud.me (www.akshayud.me)|151.101.1.147|:80... connected.
HTTP request sent, awaiting response... 301 Moved Permanently
Location: http://akshayud.me/ [following]
I had the same issue, and the solution was for me that in my github CNAME file, I had to change my webpage title. So after I changed it to "www.mywebpage.hu" (before it was without "www."), then no more redirect problems happened. I've checked your CNAME, and you don't have 'www.' too, maybe it will help if you complete your title with this.
I got into the same trouble and in my case,
How I Fixed :
the mistake was in my CNAME record. I pointed CNAME www to my github pages domain (username.github.io), so when I hit the domain it creates a loop of github domain -> custom domain -> github domain ..... as github domain auto redirects to the custom domain. Changing the CNAME value to my custom domain fixed the issue

Why can't I wget from this address?

I'm trying to wget from the "Download Now" link on this website but it returns the following error:
$wget https://www.spigotmc.org/resources/supervanish.1331/download?version=46330
--2015-09-27 19:13:53-- https://www.spigotmc.org/resources/supervanish.1331/download?version=46330
Resolving www.spigotmc.org (www.spigotmc.org)... 198.41.204.94, 198.41.205.94, 2400:cb00:2048:1::c629:cd5e, ...
Connecting to www.spigotmc.org (www.spigotmc.org)|198.41.204.94|:443... connected.
HTTP request sent, awaiting response... 503 Service Temporarily Unavailable
2015-09-27 19:13:53 ERROR 503: Service Temporarily Unavailable.
Here is the page:
https://www.spigotmc.org/resources/supervanish.1331
Here is the URL:
https://www.spigotmc.org/resources/supervanish.1331/download?version=46330
You cannot wget because the server returns an error.
If you want to retrieve an error page, perhaps this page will help you.

Wget gives up too quickly on a Express API

I want to download the result of a Express.js REST API which is very slow to process (~10 minutes). I tried few timeout options with wget but it gives up after few minutes while I ask it to wait around ~60 000 years.
wget "http://localhost:5000/slowstuff" --http-user=user --http-password=password --read-timeout=1808080878708 --tries=1
--2015-02-26 11:14:21-- http://localhost:5000/slowstuff
Resolving localhost (localhost)... ::1, 127.0.0.1
Connecting to localhost (localhost)|::1|:5000... connected.
HTTP request sent, awaiting response... 401 Unauthorized
Authentication selected: Basic realm="Authorization Required"
Reusing existing connection to [localhost]:5000.
HTTP request sent, awaiting response... No data received.
Giving up.
EDIT:
The problem doesn't come from the wget timeout value. With a timeout set to 4 seconds, the error is different: Read error (Connection timed out) in headers. And I have exactly the same problem with curl.
I think the problem comes from my API. It looks like a timeout of 2 minutes is set by default in NodeJS.
Now, I need to find how to change this value.
This
--http-password=password--read-timeout=1808080878708
is missing a blank. Use
--http-password=password --read-timeout=1808080878708

Booted Off Local Server - 302 error

I'll start with the log that I am receiving below:
Dec.15.11.56-Rf: Incoming Request URL: /
Dec.15.11.56-Rf: SECURE GET Path: / From: mlocal.cldeals.com Rewritten: www.cldeals.com
Dec.15.11.56-Rf: Received 302 Found [text/html; charset=UTF-8] response for /
Dec.15.11.56-Rf: Sending 302 text/html; charset=UTF-8 response for /
Dec.15.11.56-Rf: Stats. Total: 0.52088702, Upstream: 0.48212701, Processing: 0.00105600, ProcessingOther: 0.04037500
Basically, when I go to mlocal.cldeals.com, it loads fine. If I click on another page, say mlocal.cldeals.com/products, that loads fine as well. The issue seems to be when I go to the account page and try to switch back to the homepage, maybe some type of security issue? When I try to switch back to mlocal.cldeals.com, the home page, it boots me off and sends me to www.cldeals.com. Is there something I can add to force this from not happening? Additionally, is this just a local server issue that would go away when I launch it on Moovweb's server? Any help is greatly appreciated.
Thank you.
It looks like the backend response to https://www.cldeals.com is a 302 to http://www.cldeals.com:80/. Not sure why that is the case (see note below *)
curl -v -o /dev/null https://www.cldeals.com
This response contains a hardcoded Location header and your project is passing along the response as is, which is why you are being booted off your local server.
Because the Location header value has a port specified, you'll need to modify your config.json to include this line in the mapping:
{
"host_map": [
"$.cldeals.com => www.cldeals.com",
"$.cldeals.com => www.cldeals.com:80"
]
}
This way, the SDK knows to rewrite that specific host:port value... (By default all HTTP requests go through port 80, so that information isn't really necessary)
*This is might be bug in the backend implementation because once you log in, you should be in HTTPS mode until you log out. (I can see some pages with personal information being transmitted over plain HTTP)

Express Checkout Digital Goods : Proxy Error on sandbox.paypal.com/incontext

I have a Flash website. When I want to use Paypal Express Checkout with Digital Goods, I call this javascript code :
dg = new PAYPAL.apps.DGFlow();
dg.startFlow("http://mydomain.com/setup.php");
setup.php calls SetExpressCheckoutPayment function and redirect to https://www.sandbox.paypal.com/incontext?token=...&useraction=commit
With Firebug I can see this address returns a 302, and redirects to https://www.sandbox.paypal.com/webapps/checkout/webflow/sparta/expresscheckoutvalidatedataflow?exp_type=&cookiesBlocked=&token=...&useraction=commit
This adress returns also a 302 and redirects to https://www.sandbox.paypal.com/webapps/checkout/webflow/sparta/expresscheckoutvalidatedataflow?execution=e1s1
Here it hangs for several minutes and ends with this error message :
Proxy Error
The proxy server could not handle the request GET /webapps/checkout/webflow/sparta/expresscheckoutvalidatedataflow.
Reason: Error during SSL Handshake with remote server
I started to get this error sometimes last week, and I have it every time today.
It happens on my MAMP environment and on my website.
I don't have SSL certificate but I didn't last week and it was not a problem.
Do you know anything about this error message ?
Edit
I tried with Opera, proxyError comes at a different step : https://www.sandbox.paypal.com/webapps/checkout/webflow/sparta/expresscheckoutvalidatedataflow?execution=e1s4
And once this morning on Firefox I had another Proxy Error after the first redirection :
Proxy Error
The proxy server received an invalid response from an upstream server.
The proxy server could not handle the request GET /webapps/checkout/webflow/sparta/expresscheckoutvalidatedataflow.
Reason: Error reading from remote server
I don't have the Proxy Error anymore since yesterday. I didn't change anything so it seems PayPal servers are unstable...
I'm having the same issue since Sunday evening (sorry that I can't post this as a comment, don't have enough reputation yet).
I'm on LiquidWeb shared hosting, using the Merchant SDK ( https://github.com/paypal/merchant-sdk-php ). I was on merchant-sdk-php-2.1.96 when the errors began, and tried upgrading to merchant-sdk-php-2.2.98 but now it is worse (won't even do the first redirect, which is confusing). My code is server side, but getting the timeout and proxy error at the same urls:
$setECResponse = $PayPal_service->SetExpressCheckout($setECReq);
if($setECResponse->Ack == 'Success') {
$token = $setECResponse->Token;
$payPalURL = 'https://www.sandbox.paypal.com/incontext?token=' . $token;
$this->Redirect($payPalURL);
}