Why is this particular website auto-formatting dates when parsed programatically? - date

Question

I am trying to get a list of certain links from a website using Jsoup 1.10.1. The following is a snippet which I've isolated from the rest of my code in an attempt to diagnose the problem:
public static void main(String[] args) throws IOException {
URL link = new URL("https://www.ncdc.noaa.gov/gibbs/availability/1979-01-01");
Document doc = Jsoup.parse(link, 600);
Elements links = doc.select(".availableChannels > a");
System.out.println(links.get(0));
}
In theory this should print out the contents of the first link under the .availableChannelsclass on the provided URL, which should be IR .
However, Jsoup instead auto-formats the yyyy-mm-dd date that appears within the a href, and as a result the code snippet prints out IR , which is undesired.
How do I stop Jsoup from automatically formatting dates?
UPDATE
I decided to write a similar program in Python 2.7 to see what would happen if I read from that particular page (https://www.ncdc.noaa.gov/gibbs/availability/1979-01-01). Turns out that the yyyy-mm-dd that appears in IR is still getting formatted into IR when I open and print the page's source using python.
import urllib
link = "https://www.ncdc.noaa.gov/gibbs/availability/1979-01-01";
f = urllib.urlopen(link);
myfile = f.read();
print myfile;
I guess the question becomes: Why is this particular website automatically formatting dates when accessed through non-standard web browser means? I've changed the question accordingly to reflect this.

Answer 1

It's because you need to set the Accept-Language header on the HTTP request.
The Accept-Language request HTTP header advertises which languages the
client is able to understand, and which locale variant is preferred.
Using content negotiation, the server then selects one of the
proposals, uses it and informs the client of its choice with the
Content-Language response header. (https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Accept-Language)
So it seems like if you don't set the header, the server hosting the website returns a variant that doesn't use your preferred local time settings that you see in your browser.
public static void main(String[] args) throws IOException {
URL link = new URL("https://www.ncdc.noaa.gov/gibbs/availability/1979-01-01");
Document doc = Jsoup.connect(link.toString())
.header("Accept-Language", "en-GB").get();
Elements links = doc.select(".availableChannels > a");
System.out.println(links.get(0));
}
Output:
IR