Network unreachable: robots.txt unreachable - robots.txt

Question

I'm getting error "Network unreachable: robots.txt unreachable" when trying to add my website on Google Webmaster tools -> http://www.hyponomist.com/
You can check my robots.txt at here and sitemap.xml at here
I have reading other posts here and there, but could not solve/understand. what is causing this issue. Also, I tried downloading a page with the Fetch as Googlebot tool but got same error.
Anyone knows?
Thanks in advance!

Answer 1

Your web server is returning a 503 error when the user-agent string says the request is from Googlebot, but 200 when it's from a browser. If you use an http diagnostic tool such as Fiddler (http://fiddler2.com/) you can see this.
If you use Fiddler to send the same request that a browser would send:
GET http://www.hyponomist.com/robots.txt HTTP/1.1
Host: www.hyponomist.com
Connection: keep-alive
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.72 Safari/537.36
Accept-Encoding: gzip,deflate,sdch
Accept-Language: en-US,en;q=0.8
The response is:
HTTP/1.1 200 OK
Server: nginx/1.4.4
Date: Fri, 10 Jan 2014 21:34:42 GMT
Content-Type: text/plain; charset=UTF-8
Transfer-Encoding: chunked
Connection: keep-alive
Retry-After: 18000
Last-Modified: Fri, 10 Jan 2014 20:43:28 GMT
Content-Encoding: gzip
If you change the user-agent to mimic Googlebot:
GET http://www.hyponomist.com/robots.txt HTTP/1.1
Host: www.hyponomist.com
Connection: keep-alive
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
User-Agent: Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Accept-Encoding: gzip,deflate,sdch
Accept-Language: en-US,en;q=0.8
Then the response is:
HTTP/1.1 503 Service Temporarily Unavailable
Server: nginx/1.4.4
Date: Fri, 10 Jan 2014 21:35:25 GMT
Content-Type: text/html; charset=iso-8859-1
Content-Length: 234
Connection: keep-alive
Retry-After: 18000
Exactly why it's doing this, I can't tell you. 503 is normally the error sent when a server is temporarily overloaded, but that's clearly not the case here. Maybe your firewall is poorly configured, and has blacklisted Googlebot based on request frequency? Take a look at your firewall settings and your server config.

Answer 2

Removing the trailing slash (use http://www.hyponomist.com instead of http://www.hyponomist.com/) may help