Scala WSClient cannot connect to https with no httpProxy provided - scala

Question

I'm facing a little issue with Scala WSClient.
I'm working on a VM with a squid proxy, and I want to only allow HTTPS proxying (blocking HTTP).
I tried a java HTTPS connection (using URLConnection), and it works fine with https proxy and https url. If I try mixing configs (proxy and url), I get this :
HTTP with http proxy : 302 answer (redirection to https)
HTTP with https proxy : UnknownHostException
HTTP with https proxy and ip resolution in /etc/hosts: SocketTimeoutException
HTTPS with https proxy : 200
HTTPS with http proxy : UnknownHostException
So we have dns issues when url protocol and proxy opened route are not matching (not sure why), but if I use HTTPS with https proxy only, I'm fine : expected result.
I tried the same with a Scala object (called from java), using also a URLConnection.
Same result, works fine (same errors when mixing proxy and protocols).
#throws(classOf[java.io.IOException])
#throws(classOf[java.net.SocketTimeoutException])
def doHttps(url: String,
connectTimeout: Int = 5000,
readTimeout: Int = 5000,
requestMethod: String = "GET") = {
import java.net.URL
import javax.net.ssl.HttpsURLConnection
val connection = (new URL("https://"+url)).openConnection.asInstanceOf[HttpsURLConnection]
connection.setConnectTimeout(connectTimeout)
connection.setReadTimeout(readTimeout)
connection.setRequestMethod(requestMethod)
connection.connect()
val headers = connection.getHeaderFields
for ((k,v) <- headers.asScala) {
if (k == null) {
println(v)
} else {
println(s"$k: $v")
}
}
}
But I need to use scala's WSClient, so I tested with it (tried both AhcWSClient and StandaloneAhcWsClient, and a problem appears. With this tool, I cannot connect to an https endpoint (here test.salesforce.com) if I don't have httpProxy allowed.
Here is the code (both ws clients) :
def wsClientTest(clientKey: String) {
implicit val ec: scala.concurrent.ExecutionContext = scala.concurrent.ExecutionContext.global
implicit val system: ActorSystem = ActorSystem("test", ConfigFactory.defaultReference())
implicit val materializer: ActorMaterializer = ActorMaterializer(ActorMaterializerSettings(system))
implicit val ws: AhcWSClient = AhcWSClient(AhcWSClientConfig())
val tokenEndpoint: String = s"https://test.salesforce.com/services/oauth2/token"
val query = ws.url(tokenEndpoint).withRequestTimeout(Duration.apply(5L, TimeUnit.SECONDS))
.withRequestFilter(AhcCurlRequestLogger())
.post( Map("response_type" -> "device_code", "client_id" -> clientKey));
val results = Await.result(query, Duration.apply(5L, TimeUnit.SECONDS))
println(Json.parse(results.body))
ws.close()
}
def standaloneWSClientTest(clientKey: String) {
implicit val ec: scala.concurrent.ExecutionContext = scala.concurrent.ExecutionContext.global
implicit val system: ActorSystem = ActorSystem("test", ConfigFactory.defaultReference())
implicit val materializer: ActorMaterializer = ActorMaterializer(ActorMaterializerSettings(system))
implicit val ws: StandaloneWSClient = StandaloneAhcWSClient()
val tokenEndpoint: String = s"https://test.salesforce.com/services/oauth2/token"
val query = ws.url(tokenEndpoint).withRequestTimeout(Duration.apply(5L, TimeUnit.SECONDS))
.withRequestFilter(AhcCurlRequestLogger())
.post( Map("response_type" -> "device_code", "client_id" -> clientKey));
val results = Await.result(query, Duration.apply(5L, TimeUnit.SECONDS))
println(Json.parse(results.body))
ws.close()
}
With that client, with only httpsProxy allowed, I get dns errors, and if I add the url/ip pair in /etc/hosts, I get a timeout.
Scala Using URLConnection (HTTPS with only httpsProxy defined)
--------------------------------------------------------------
Transfer-Encoding: [chunked]
[HTTP/1.1 200 OK]
X-Content-Type-Options: [nosniff]
X-FRAME-OPTIONS: [DENY]
Date: [Tue, 08 Feb 2022 10:03:19 GMT]
Strict-Transport-Security: [max-age=31536000; includeSubDomains]
Cache-Control: [no-cache,must-revalidate,max-age=0,no-store,private]
Content-Security-Policy: [frame-ancestors 'none', upgrade-insecure-requests]
Vary: [Accept-Encoding]
Set-Cookie: [QCQQ=bzfE6xRGxLR; path=/; secure, BrowserId=VzwVGIjGEey_0Qe3-91VQQ; domain=.salesforce.com; path=/; expires=Wed, 08-Feb-2023 10:03:19 GMT; Max-Age=31536000, LSKey-c$CookieConsentPolicy=0:0; domain=test.salesforce.com; path=/; expires=Wed, 08-Feb-2023 10:03:19 GMT; Max-Age=31536000, CookieConsentPolicy=0:0; domain=test.salesforce.com; path=/; expires=Wed, 08-Feb-2023 10:03:19 GMT; Max-Age=31536000]
Expires: [Thu, 01 Jan 1970 00:00:00 GMT]
X-XSS-Protection: [1; mode=block]
Content-Type: [text/html; charset=UTF-8]
Scala Using WSclient (HTTPS with only httpsProxy defined)
---------------------------------------------------------
INFO [main] AhcCurlRequestLogger - curl \
--verbose \
--request POST \
--header 'Content-Type: application/x-www-form-urlencoded' \
--data 'response_type=device_code&client_id=test-client-id' \
'https://test.salesforce.com/services/oauth2/token'
DEBUG[main] AbstractByteBuf - -Dplay.shaded.ahc.io.netty.buffer.checkAccessible: true
DEBUG[main] AbstractByteBuf - -Dplay.shaded.ahc.io.netty.buffer.checkBounds: true
DEBUG[main] ResourceLeakDetectorFactory - Loaded default ResourceLeakDetector: play.shaded.ahc.io.netty.util.ResourceLeakDetector#3e7634b9
DEBUG[main] NettyRequestSender - Aborting Future NettyResponseFuture{currentRetry=0,
isDone=0,
isCancelled=0,
asyncHandler=play.api.libs.ws.ahc.StandaloneAhcWSClient$$anon$1#2b5825fa,
nettyRequest=play.shaded.ahc.org.asynchttpclient.netty.request.NettyRequest#53d1b9b3,
future=java.util.concurrent.CompletableFuture#2cae1042[Not completed],
uri=https://test.salesforce.com/services/oauth2/token,
keepAlive=true,
redirectCount=0,
timeoutsHolder=play.shaded.ahc.org.asynchttpclient.netty.timeout.TimeoutsHolder#163d04ff,
inAuth=0,
touch=1644314479717}
DEBUG[main] NettyRequestSender - test.salesforce.com: Name or service not known
java.net.UnknownHostException: test.salesforce.com: Name or service not known
at java.net.Inet4AddressImpl.lookupAllHostAddr(Native Method) ~[na:1.8.0_281]
at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:929) ~[na:1.8.0_281]
at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1324) ~[na:1.8.0_281]
at java.net.InetAddress.getAllByName0(InetAddress.java:1277) ~[na:1.8.0_281]
at java.net.InetAddress.getAllByName(InetAddress.java:1193) ~[na:1.8.0_281]
at java.net.InetAddress.getAllByName(InetAddress.java:1127) ~[na:1.8.0_281]
at play.shaded.ahc.io.netty.util.internal.SocketUtils$9.run(SocketUtils.java:161) ~[scalatest-1.0-SNAPSHOT.jar:na]
at play.shaded.ahc.io.netty.util.internal.SocketUtils$9.run(SocketUtils.java:158) ~[scalatest-1.0-SNAPSHOT.jar:na]
at java.security.AccessController.doPrivileged(Native Method) ~[na:1.8.0_281]
at play.shaded.ahc.io.netty.util.internal.SocketUtils.allAddressesByName(SocketUtils.java:158) ~[scalatest-1.0-SNAPSHOT.jar:na]
at play.shaded.ahc.io.netty.resolver.DefaultNameResolver.doResolveAll(DefaultNameResolver.java:52) ~[scalatest-1.0-SNAPSHOT.jar:na]
at play.shaded.ahc.io.netty.resolver.SimpleNameResolver.resolveAll(SimpleNameResolver.java:81) ~[scalatest-1.0-SNAPSHOT.jar:na]
at play.shaded.ahc.io.netty.resolver.SimpleNameResolver.resolveAll(SimpleNameResolver.java:73) ~[scalatest-1.0-SNAPSHOT.jar:na]
at play.shaded.ahc.org.asynchttpclient.resolver.RequestHostnameResolver.resolve(RequestHostnameResolver.java:50) ~[scalatest-1.0-SNAPSHOT.jar:na]
at play.shaded.ahc.org.asynchttpclient.netty.request.NettyRequestSender.resolveAddresses(NettyRequestSender.java:357) [scalatest-1.0-SNAPSHOT.jar:na]
at play.shaded.ahc.org.asynchttpclient.netty.request.NettyRequestSender.sendRequestWithNewChannel(NettyRequestSender.java:300) [scalatest-1.0-SNAPSHOT.jar:na]
at play.shaded.ahc.org.asynchttpclient.netty.request.NettyRequestSender.sendRequestWithCertainForceConnect(NettyRequestSender.java:142) [scalatest-1.0-SNAPSHOT.jar:na]
at play.shaded.ahc.org.asynchttpclient.netty.request.NettyRequestSender.sendRequest(NettyRequestSender.java:113) [scalatest-1.0-SNAPSHOT.jar:na]
at play.shaded.ahc.org.asynchttpclient.DefaultAsyncHttpClient.execute(DefaultAsyncHttpClient.java:241) [scalatest-1.0-SNAPSHOT.jar:na]
at play.shaded.ahc.org.asynchttpclient.DefaultAsyncHttpClient.executeRequest(DefaultAsyncHttpClient.java:210) [scalatest-1.0-SNAPSHOT.jar:na]
at play.api.libs.ws.ahc.StandaloneAhcWSClient.execute(StandaloneAhcWSClient.scala:90) [scalatest-1.0-SNAPSHOT.jar:na]
at play.api.libs.ws.ahc.StandaloneAhcWSRequest.$anonfun$execute$1(StandaloneAhcWSRequest.scala:216) [scalatest-1.0-SNAPSHOT.jar:na]
Scala Using WSclient and DNS in /etc/hosts (HTTPS with only httpsProxy defined)
-------------------------------------------------------------------------------
DEBUG[AsyncHttpClient-timer-1-1] TimeoutTimerTask - Request timeout to test.salesforce.com:443 after 5000 ms for NettyResponseFuture{currentRetry=0,
isDone=0,
isCancelled=0,
asyncHandler=play.api.libs.ws.ahc.StandaloneAhcWSClient$$anon$1#156a2245,
nettyRequest=play.shaded.ahc.org.asynchttpclient.netty.request.NettyRequest#1badde98,
future=java.util.concurrent.CompletableFuture#1f924935[Not completed],
uri=https://test.salesforce.com/services/oauth2/token,
keepAlive=true,
redirectCount=0,
timeoutsHolder=play.shaded.ahc.org.asynchttpclient.netty.timeout.TimeoutsHolder#5a90d4a3,
inAuth=0,
touch=1644314381722} after 5042 ms
DEBUG[AsyncHttpClient-timer-1-1] NettyRequestSender - Aborting Future NettyResponseFuture{currentRetry=0,
isDone=0,
isCancelled=0,
asyncHandler=play.api.libs.ws.ahc.StandaloneAhcWSClient$$anon$1#156a2245,
nettyRequest=play.shaded.ahc.org.asynchttpclient.netty.request.NettyRequest#1badde98,
future=java.util.concurrent.CompletableFuture#1f924935[Not completed],
uri=https://test.salesforce.com/services/oauth2/token,
keepAlive=true,
redirectCount=0,
timeoutsHolder=play.shaded.ahc.org.asynchttpclient.netty.timeout.TimeoutsHolder#5a90d4a3,
inAuth=0,
touch=1644314381722}
Exception in thread "main" java.util.concurrent.TimeoutException: Future timed out after [5 seconds]
at scala.concurrent.impl.Promise$DefaultPromise.tryAwait0(Promise.scala:212)
at scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:225)
at scala.concurrent.Await$.$anonfun$result$1(package.scala:200)
at scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:62)
at scala.concurrent.Await$.result(package.scala:124)
at DeviceVerificationClient$.standaloneWSClientTest(DeviceVerification.scala:59)
at DeviceVerificationClient.standaloneWSClientTest(DeviceVerification.scala)
at ScalaTest.main(ScalaTest.java:17)
DEBUG[AsyncHttpClient-timer-1-1] NettyRequestSender - Request timeout to test.salesforce.com:443 after 5000 ms
I tried disabling hostname verification (disableHostnameVerification = true), SNI, and commented inet-address dns resolver to use async-dns (provider-object = "akka.io.dns.internal.AsyncDnsProvider") but none of them work.
I'd be glad to know if I'm missing something here, a parameter to configure, an option to set...
Any help would be greatly appreciated !
Regards
Frederic Esnault