I'm getting error "Network unreachable: robots.txt unreachable" when trying to add my website on Google Webmaster tools -> http://www.hyponomist.com/
You can check my robots.txt at here and sitemap.xml at here
I have reading other posts here and there, but could not solve/understand. what is causing this issue. Also, I tried downloading a page with the Fetch as Googlebot tool but got same error.
Anyone knows?
Thanks in advance!
Your web server is returning a 503 error when the user-agent string says the request is from Googlebot, but 200 when it's from a browser. If you use an http diagnostic tool such as Fiddler (http://fiddler2.com/) you can see this.
If you use Fiddler to send the same request that a browser would send:
GET http://www.hyponomist.com/robots.txt HTTP/1.1
Host: www.hyponomist.com
Connection: keep-alive
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.72 Safari/537.36
Accept-Encoding: gzip,deflate,sdch
Accept-Language: en-US,en;q=0.8
The response is:
HTTP/1.1 200 OK
Server: nginx/1.4.4
Date: Fri, 10 Jan 2014 21:34:42 GMT
Content-Type: text/plain; charset=UTF-8
Transfer-Encoding: chunked
Connection: keep-alive
Retry-After: 18000
Last-Modified: Fri, 10 Jan 2014 20:43:28 GMT
Content-Encoding: gzip
If you change the user-agent to mimic Googlebot:
GET http://www.hyponomist.com/robots.txt HTTP/1.1
Host: www.hyponomist.com
Connection: keep-alive
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
User-Agent: Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Accept-Encoding: gzip,deflate,sdch
Accept-Language: en-US,en;q=0.8
Then the response is:
HTTP/1.1 503 Service Temporarily Unavailable
Server: nginx/1.4.4
Date: Fri, 10 Jan 2014 21:35:25 GMT
Content-Type: text/html; charset=iso-8859-1
Content-Length: 234
Connection: keep-alive
Retry-After: 18000
Exactly why it's doing this, I can't tell you. 503 is normally the error sent when a server is temporarily overloaded, but that's clearly not the case here. Maybe your firewall is poorly configured, and has blacklisted Googlebot based on request frequency? Take a look at your firewall settings and your server config.
Removing the trailing slash (use http://www.hyponomist.com instead of http://www.hyponomist.com/) may help
Related
Background:
I have set up an IPFS server behind an HAProxy server.
I have written a Flutter client to connect to this IPFS server through the proxy and add a file.
Problem:
Everything works as expected when I run the Flutter client as a desktop app (on Macos), but I get a 403 error when I run the Flutter client as a web app.
Details:
The headers for the successful call (captured via tcpdump) are as follows:
REQUEST:
POST /api/v0/add HTTP/1.1
Host: po.segito.net
User-Agent: Dart/2.11 (dart:io)
Accept-Encoding: gzip
Content-Type: multipart/form-data; boundary=dart-http-boundary-fmjPP-TIwnDcY7pJGniid4grt9mdDADtazmb7Pm8sP_PRJkV1oY
Content-Length: 39611
--dart-http-boundary-fmjPP-TIwnDcY7pJGniid4grt9mdDADtazmb7Pm8sP_PRJkV1oY
content-type: application/octet-stream
content-disposition: form-data; name="asset"
RESPONSE:
HTTP/1.1 200 OK
Access-Control-Allow-Headers: X-Stream-Output, X-Chunked-Output, X-Content-Length
Access-Control-Expose-Headers: X-Stream-Output, X-Chunked-Output, X-Content-Length
Content-Type: application/json
Server: go-ipfs/0.6.0
Trailer: X-Stream-Error
Vary: Origin
X-Chunked-Output: 1
Date: Sat, 07 Nov 2020 09:48:28 GMT
Transfer-Encoding: chunked
The headers for the unsuccessful call is as follows:
REQUEST:
POST /api/v0/add HTTP/1.1
Host: po.segito.net
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:82.0) Gecko/20100101 Firefox/82.0
Accept-Encoding: gzip, deflate, br
Content-Type: multipart/form-data; boundary=dart-http-boundary-HvcRoK0lGd4+oahuNaOejbIBd-cID5+0n3GFvwwOY4ZqfitRP1s
Content-Length: 39611
--dart-http-boundary-HvcRoK0lGd4+oahuNaOejbIBd-cID5+0n3GFvwwOY4ZqfitRP1s
content-type: application/octet-stream
content-disposition: form-data; name="asset"
RESPONSE:
HTTP/1.1 403 Forbidden
Content-Type: text/plain; charset=utf-8
Vary: Origin
X-Content-Type-Options: nosniff
Date: Sat, 07 Nov 2020 09:38:55 GMT
Content-Length: 16
403 - Forbidden
Notes:
There is no CORS issue because all files and REST calls (including
the Flutter HTML files) are all served via the same HAProxy server.
The HAProxy is an SSL terminator.
A CURL request works as expected.
All other REST calls through the HAProxy works as expected.
I'm learning Keycloak and I have the following problem.
I have a Java EE application protected through Keycloak (main-app). After logging into keycloak, it accesses another application (app-example, also protected by keycloak) that performs a check and depending on the result returns to main-app or shows an error.
The problem is that app-example redirects to keycloak and keycloak redirects back to app-example. At the end, I get the error: 400 bad request.
In front of keycloak I have an Apache server that acts as a proxy and all connections are via SSL.
First call:
Request URL: https://localhost/app-example
Request Method: GET
Status Code: 302 Found
Remote Address: xxx.xxx.xxx.xxx:443
Referrer Policy: no-referrer-when-downgrade
Connection: Keep-Alive
Content-Length: 0
Date: Mon, 17 Dec 2018 14:09:07 GMT
Keep-Alive: timeout=5, max=96
Location: https://localhost/auth/realms/my-realm/protocol/openid-connect/auth?response_type=code&client_id=appexample&redirect_uri=https%3A%2F%2Flocalhost%2Fapp-example%2F&state=e327185a-6306-4124-b384-215954f51bb7&login=true&scope=openid
Server: Apache/2.4.10 (Debian) mod_jk/1.2.37 OpenSSL/1.0.1t
Set-Cookie: OAuth_Token_Request_State=e327185a-6306-4124-b384-215954f51bb7; Version=1; Secure; HttpOnly
Set-Cookie: JSESSIONID=AxzD3hq5sfv-SJdbSa7HDLJe1lBWC1ExBoy86TdR; path=/app-example
Strict-Transport-Security: max-age=15768000
X-Powered-By: Undertow/1
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8
Accept-Encoding: gzip, deflate, br
Accept-Language: es-ES,es;q=0.9
Cache-Control: no-cache
Connection: keep-alive
Cookie: OAuth_Token_Request_State=6f634ef2-cc69-4cb8-bdaa-2d69929b26a8; JSESSIONID=BVzI9MiQB6FYFIUDLyK2dQU_o6OPSCyZB9NQDfXV
Host: localhost
Pragma: no-cache
Referer: https://localhost/main-app/
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.110 Safari/537.36
Second call:
Request URL: https://localhost/auth/realms/my-realm/protocol/openid-connect/auth?response_type=code&client_id=app-example&redirect_uri=https%3A%2F%2Flocalhost%2Fapp-example%2F&state=e327185a-6306-4124-b384-215954f51bb7&login=true&scope=openid
Request Method: GET
Status Code: 302 Found
Remote Address: xxx.xxx.xxx.xxx:443
Referrer Policy: no-referrer-when-downgrade
Cache-Control: no-store, must-revalidate, max-age=0
Connection: Keep-Alive
Content-Length: 0
Date: Mon, 17 Dec 2018 14:09:08 GMT
Keep-Alive: timeout=10
Location: https://localhost/app-example/?state=e327185a-6306-4124-b384-215954f51bb7&session_state=39fc0c37-e60d-42e7-876a-97501753bd6a&code=eyJhbGciOiJkaXIiLCJlbmMiOiJBMTI4Q0JDLUhTMjU2In0..pDbVJYc-fWIWXhsycldqHA.4nZs2gTZZyNhHRDDF3vjd99vCmM6INRMGk8T-Svcc6U1nxtFu7am_Ck2oTNZKYs0_zRHzgUkU4mzmdfRrTRoN64b4uosAVvZKSx_UDhaOERbaLk4p1wqjAUswc2N-48Vb92XBPr-ihET5WF3mzcGeb2TK6k1_GjaBLdEz4BlO8cYVisp35HNIq1APR7GPh9UbCrJaspLegxrvY5_lM9GPsF3NwWorWFE8G3BeUSQeqF7CwTY4YXJnTPkarbwaYX9.-uqvbrqkLz2JLLjeBl4n3g
P3P: CP="This is not a P3P policy!"
Referrer-Policy: no-referrer-when-downgrade
Server: Apache
Set-Cookie: AUTH_SESSION_ID=39fc0c37-e60d-42e7-876a-97501753bd6a.public:server-public-1; Version=1; Path=/auth/realms/my-realm/; Secure; HttpOnly
Set-Cookie: KC_RESTART=eyJhb....9Y3M; Version=1; Path=/auth/realms/my-realm/; Secure; HttpOnly
Set-Cookie: KEYCLOAK_SESSION=my-realm/f24b7ee7-b32e-4e93-821f-cfbf4708acf4/39fc0c37-e60d-42e7-876a-97501753bd6a; Version=1; Expires=Tue, 18-Dec-2018 00:09:08 GMT; Max-Age=36000; Path=/auth/realms/my-realm/; Secure
Set-Cookie: KEYCLOAK_REMEMBER_ME=; Version=1; Comment=Expiring cookie; Expires=Thu, 01-Jan-1970 00:00:10 GMT; Max-Age=0; Path=/auth/realms/my-realm/; Secure; HttpOnly
Strict-Transport-Security: max-age=15768000
X-Content-Type-Options: nosniff
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8
Accept-Encoding: gzip, deflate, br
Accept-Language: es-ES,es;q=0.9
Cache-Control: no-cache
Connection: keep-alive
Cookie: KEYCLOAK_LOCALE=es; AUTH_SESSION_ID=39fc0c37-e60d-42e7-876a-97501753bd6a.public:server-public-1; KEYCLOAK_SESSION=my-realm/f24b7ee7-b32e-4e93-821f-cfbf4708acf4/39fc0c37-e60d-42e7-876a-97501753bd6a; KEYCLOAK_IDENTITY=eyJhb.....Qgs
Host: localhost
Pragma: no-cache
Referer: https://localhost/main-app/
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.110 Safari/537.36
response_type: code
client_id: appexample
redirect_uri: https://localhost/app-example/
state: e327185a-6306-4124-b384-215954f51bb7
login: true
scope: openid
scope: openid
In the log I observe:
ERROR [io.undertow.request] (default task-73) UT005023: Exception handling request to /main-app/launcher.jsp: org.apache.jasper.JasperException: java.lang.RuntimeException: java.lang.IllegalStateException: UT000139: Exchange already complete
I checked that the json file I got from keycloak and integrated into the application is correct. What's the problem?
cheers
Have you tried clearing your cookies?
In case you're using chrome:
https://support.google.com/chrome/answer/6098869?visit_id=637347644493186330-3872337140&p=rl_error&hl=en-GB&rd=1#redirect
Also using private window will likely help. In my case this was caused by running multiple keycloak instances on localhost over time, which got their cookies mixed up.
A Laravel app is using the JSTree to display files.
If I get the tree under http://localhost:8000 I recive the correct tree.
We have a ngnix reverse Proxy setting to access the web site from behind a proxy.
But if I open the ngnix web site there are in some cases no data. The ajax response is correct, but JSTree doesn't render it.
Have anybody a idea?
First I tried out the jstree().last_error() function and it is a empty object.
Here are my header, I hope it helps:
Host: DOMAIN.de
User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:45.0) Gecko/20100101 Firefox/45.0
Accept: */*
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate
Content-Type: application/x-www-form-urlencoded; charset=UTF-8
X-Requested-With: XMLHttpRequest
Referer: http://DOMAIN.de/explorer/show/443
Content-Length: 6
Cookie: cartalyst_sentinel=eyJpdiI6I...iJ9; laravel_session=eyJp...J9
Connection: keep-alive
Pragma: no-cache
Cache-Control: no-cache
The response:
Cache-Control: private, must-revalidate
Connection: keep-alive
Content-Encoding: gzip
Content-Type: text/html; charset=UTF-8
Date: Tue, 12 Apr 2016 06:37:12 GMT
Expires: -1
Host: DOMAIN.de
Pragma: no-cache
Server: nginx/1.4.6 (Ubuntu)
Set-Cookie: laravel_session=eyJpdi......3D; expires=Tue, 12-Apr-2016 08:37:35 GMT; Max-Age=7200; path=/; httponly
Transfer-Encoding: chunked
X-Powered-By: PHP/5.6.19
The PHP header:
header('Content-Type: application/json; charset=utf-8');
The problem is, with ngnix the response has another Content-Type. ngnix put "application/json" to "text/html".
Are there any options to modify this?
I am trying to get the data of an application which is build on Oracle Application Development Framework.
Whenver i try to wget the application it returns nothing..no content only returns below content
<html>
<head>
</head>
<body>
<meta HTTP-EQUIV="REFRESH" content="0; url=...">
</body>
</html>
I have also tried to wget the application by passing header values to wget, but it does not make any difference.
wget command was as below :
wget --header="Host:example.com" --header="User-Agent:Mozilla/5.0 (X11; Ubuntu; Linux i686; rv:14.0) Gecko/20100101 Firefox/14.0." --header="Accept:text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8" --header="Accept-Language:en-us,en;q=0.5" --header="Proxy-Connection:keep-alive" --header="Referer:http://example.com/Internet/faces/sbig_home" --header="GET:/Internet/faces/sbig_home?_afrLoop=19302770119376498&_afrWindowMode=0&_afrWindowId=null HTTP/1.1" http://example.com/
Expected result is html content of website application.
Below is the output of wget with --debug option(I have changed the url with example.com)
DEBUG output created by Wget 1.13.4 on linux-gnu.
URI encoding = `UTF-8'
--2014-05-14 09:41:18-- http://exaple.com/
Resolving exaple.com (exaple.com)... 202.59.244.48
Caching exaple.com => 202.59.244.48
Connecting to exaple.com (exaple.com)|202.59.244.48|:80... connected.
Created socket 3.
Releasing 0x08823ed0 (new refcount 1).
---request begin---
GET / HTTP/1.1
User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux i686; rv:14.0) Gecko/20100101 Firefox/14.0.
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Host: exaple.com
Connection: Keep-Alive
Accept-Language: en-us,en;q=0.5
Proxy-Connection: keep-alive
Referer: http://exaple.com/Internet/faces/sbig_home
GET: /Internet/faces/sbig_home?_afrLoop=19302770119376498&_afrWindowMode=0&_afrWindowId=null HTTP/1.1
---request end---
HTTP request sent, awaiting response...
---response begin---
HTTP/1.1 200 OK
Date: Tue, 06 May 2014 19:33:42 GMT
Last-Modified: Sat, 03 Aug 2013 10:45:27 GMT
ETag: "27400a-73-4e308c5f11fc0"
Accept-Ranges: bytes
Vary: Accept-Encoding
Content-Type: text/html
Content-Language: en
Connection: Keep-Alive
Keep-Alive: timeout=20, max=999
Server: Oracle-Application-Server-11g Oracle-Web-Cache-11g/11.1.1.6.0 (G;max-age=0+0;age=0;ecid=99079302593879470,0:1)
Content-Length: 115
---response end---
200 OK
Registered socket 3 for persistent reuse.
Length: 115 [text/html]
Saving to: `index.html.2'
I am trying to upload a file to Azure Blob storage.
For some reason when I try to put a new block blob on in the storage it tells me the resource does not exist. I am sure it is something silly I am missing.
According to the documentation:
The Put Blob operation creates a new block blob or page blob, or updates the content of an existing block blob.
Updating an existing block blob overwrites any existing metadata on the blob. Partial updates are not supported with Put Blob; the content of the existing blob is overwritten with the content of the new blob. To perform a partial update of the content of a block blob, use the Put Block List (REST API) operation.
CORS is setup and that seems okay.
When I do a preflight and get this:
Request URL:https://<account>.blob.core.windows.net/test/image.png
Request Method:OPTIONS
Status Code:200 OK
Request Headers
-----------------
OPTIONS /test/image.png HTTP/1.1
Host: <account>.blob.core.windows.net
Connection: keep-alive
Cache-Control: no-cache
Pragma: no-cache
Access-Control-Request-Method: PUT
Origin: http://www.<site>.com
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/33.0.1750.117 Safari/537.36
Access-Control-Request-Headers: accept, content-type
Accept: */*
Referer: http://www.<site>.com/azure/
Accept-Encoding: gzip,deflate,sdch
Accept-Language: en-US,en;q=0.8
Response Headers
----------------
HTTP/1.1 200 OK
Transfer-Encoding: chunked
Server: Blob Service Version 1.0 Microsoft-HTTPAPI/2.0
x-ms-request-id: 0d372e95-1524-460a-ab9c-7973d42a7070
Access-Control-Allow-Origin: http://www.<site>.com
Access-Control-Allow-Methods: PUT
Access-Control-Allow-Headers: accept, content-type
Access-Control-Max-Age: 36000
Access-Control-Allow-Credentials: true
Date: Thu, 27 Feb 2014 22:43:52 GMT
But when I make the PUT request these are the results.
Request URL:https://<account>.blob.core.windows.net/test/image.png
Request Method:PUT
Status Code:404 The specified resource does not exist.
Request Headers
-----------------
PUT /test/image.png HTTP/1.1
Host: <account>.blob.core.windows.net
Connection: keep-alive
Content-Length: 22787
Cache-Control: no-cache
Pragma: no-cache
x-ms-blob-content-disposition: attachment; filename = "image.png"
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/33.0.1750.117 Safari/537.36
Content-Type: image/png
x-ms-blob-type: BlockBlob
Accept: application/json, text/plain, */*
x-ms-version: 2013-08-15
Origin: http://www.<site>.com
x-ms-date: Thu, 27 Feb 2014 23:19:19 GMT
Referer: http://www.<site>.com/azure/
Accept-Encoding: gzip,deflate,sdch
Accept-Language: en-US,en;q=0.8
Response Headers
----------------
HTTP/1.1 404 The specified resource does not exist.
Content-Length: 223
Content-Type: application/xml
Server: Blob Service Version 1.0 Microsoft-HTTPAPI/2.0
x-ms-request-id: d5a60c8b-356a-44ff-93af-0ea720b5591f
x-ms-version: 2013-08-15
Access-Control-Expose-Headers: x-ms-request-id,Server
Access-Control-Allow-Origin: http://www.<site>.com
Access-Control-Allow-Credentials: true
Date: Thu, 27 Feb 2014 23:22:42 GMT