403 response with HttpClient but not with browser - httpclient

Question

I’m having a problem with the HttpClient library in java.
The target web site is on SSL (https://www.betcris.com), and I can load the index page from that site just fine .
However, the different pages showing odds for the different sports returns a 403 response code with HttpClient, but loading the same pages in a browser works just fine.
Here is such a page : https://www.betcris.com/en/live-lines/soccer.
I started troubleshooting this page with the information gathered by HttpFox (a Firefox add-on that resembles LiveHttpHeaders), making sure I had all the correct request headers and cookies, but I couldn’t get it to load using HttpClient. I also determined that cookies have nothing to do with the problem, as I can remove all cookies for that web site within my browser, and then hit the page directly and it will load.
I confirmed that there’s something special going on with these pages by using the online tool at http://www.therightapi.com/test. This tool allows you to input the url of a page along with any Request header you want, and shows you the response you get from the target web site. Using that tool, I can load https://www.google.com just fine, but I get the same 403 error when trying to load https://www.betcris.com/en/live-lines/soccer.
Here's my setup at therightapi :
And the response :
Does anyone know what’s going on here ?
Thanks.
EDIT : I've created a test project, here's the java code, followed by the maven dependency you should have in your pom :
package com.yourpackage;
import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStreamReader;
import org.apache.http.HttpResponse;
import org.apache.http.client.ClientProtocolException;
import org.apache.http.client.HttpClient;
import org.apache.http.client.methods.HttpGet;
import org.apache.http.impl.client.HttpClientBuilder;
public class TestHttpClient {
public static void main(String[] args) {
String url = "https://www.betcris.com/en/live-lines/soccer";
HttpClient client = HttpClientBuilder.create().build();
HttpGet request = new HttpGet(url);
// add request header
request.addHeader("User-Agent", "Mozilla/5.0 (Windows NT 10.0; WOW64; rv:55.0) Gecko/20100101 Firefox/55.0");
try {
HttpResponse response = client.execute(request);
System.out.println("Response Code : "
+ response.getStatusLine().getStatusCode());
BufferedReader rd = new BufferedReader(
new InputStreamReader(response.getEntity().getContent()));
StringBuffer result = new StringBuffer();
String line = "";
while ((line = rd.readLine()) != null) {
result.append(line);
}
} catch (ClientProtocolException e) {
// TODO Auto-generated catch block
e.printStackTrace();
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
}
<!-- https://mvnrepository.com/artifact/org.apache.httpcomponents/httpclient -->
<dependency>
<groupId>org.apache.httpcomponents</groupId>
<artifactId>httpclient</artifactId>
<version>4.5.3</version>
</dependency>

Answer 1

I have solved this problem (avoiding 403) by setting up User-Agent property while making a request as like follow:
If you use HttpClient
HttpGet httpGet = new HttpGet(URL_HERE);
httpGet.setHeader("User-Agent", "Mozilla/5.0 (Windows NT 6.1; WOW64)
AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.95 Safari/537.11");
If you use HttpURLConnection
HttpURLConnection connection = (HttpURLConnection) url.openConnection();
connection.setRequestProperty("User-Agent", "Mozilla/5.0 (Windows NT 6.1; WOW64)
AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.95 Safari/537.11");

Answer 2

I use the following code to consume HTTPS Urls:
import org.apache.http.HttpResponse;
import org.apache.http.client.methods.CloseableHttpResponse;
import org.apache.http.client.methods.HttpGet;
import org.apache.http.conn.ssl.NoopHostnameVerifier;
import org.apache.http.impl.client.CloseableHttpClient;
import org.apache.http.impl.client.HttpClients;
import org.apache.http.ssl.SSLContextBuilder;
...
SSLContext sslContext =
new SSLContextBuilder().loadTrustMaterial(null, (certificate, authType) -> true).build();
try (CloseableHttpClient httpClient = HttpClients.custom().setSSLContext(sslContext)
.setSSLHostnameVerifier(new NoopHostnameVerifier()).build()) {
HttpGet httpGet = new HttpGet("YOUR_HTTPS_URL");
httpGet.setHeader("Accept", "application/xml");
httpGet.setHeader("User-Agent",
"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.95 Safari/537.11");
HttpResponse response = httpClient.execute(httpGet);
logger.info("Response: " + response);
}
pom.xml:
<dependency>
<groupId>org.apache.httpcomponents</groupId>
<artifactId>httpclient</artifactId>
<version>4.5.3</version>
</dependency>

Answer 3

In my case, the web server does not use a proxy to communicate with APIs.
I just disbaled the defaultproxy under system.net in web.config.
<system.net>
<defaultProxy enabled="false" />
</system.net>

Answer 4

403 Forbidden is used to signal an authentication requirement. In fact, the full 403 response should tell you exactly that. Luckily, HttpClient can do authentication.