using SBT for first time - issue retrieving dependencies behind proxy - scala

Question

I've downloaded SBT (sbt-0.13.15.zip) and unzipped it, but I'm facing issue when running sbt.bat .
Initially, I had proxy issues that I fixed when I found I needed to specify only the https config, and not the http one - when providing both, it was taking the http one and it was failing.
So now I don't have proxy issues anymore, but it's failing when trying to fetch the jansi dependencies :
setting 'basedir' to 'C:\Users\vfuchs070114\.sbt\boot'
:: resolving dependencies :: org.scala-sbt#boot-jansi;1.0
confs: [default]
validate = true
refresh = false
resolving dependencies for configuration 'default'
== resolving dependencies for org.scala-sbt#boot-jansi;1.0 [default]
loadData of org.scala-sbt#boot-jansi;1.0 of rootConf=default
== resolving dependencies org.scala-sbt#boot-jansi;1.0->org.fusesource.jansi#jansi;1.11 [default->default(compile)]
loadData of org.fusesource.jansi#jansi;1.11 of rootConf=default
using redefined-public to resolve org.fusesource.jansi#jansi;1.11
redefined-public: Checking cache for: dependency: org.fusesource.jansi#jansi;1.11 {default=[default(compile)]}
redefined-public: no namespace defined: using system
no ivy file in cache for org.fusesource.jansi#jansi;1.11: tried C:\Users\vfuchs070114\.ivy2\cache\org.fusesource.jansi\jansi\ivy-1.11.xml
redefined-public: no latest strategy defined: using default
local: no namespace defined: using system
no ivy file in cache for org.fusesource.jansi#jansi;1.11: tried C:\Users\vfuchs070114\.ivy2\cache\org.fusesource.jansi\jansi\ivy-1.11.xml
trying C:\Users\vfuchs070114\.ivy2\local\org.fusesource.jansi\jansi\1.11\ivys\ivy.xml
tried C:\Users\vfuchs070114\.ivy2\local\org.fusesource.jansi\jansi\1.11\ivys\ivy.xml
local: resource not reachable for org.fusesource.jansi#jansi;1.11: res=C:\Users\vfuchs070114\.ivy2\local\org.fusesource.jansi\jansi\1.11\ivys\ivy.xml
trying C:\Users\vfuchs070114\.ivy2\local\org.fusesource.jansi\jansi\1.11\jars\jansi.jar
tried C:\Users\vfuchs070114\.ivy2\local\org.fusesource.jansi\jansi\1.11\jars\jansi.jar
local: resource not reachable for org.fusesource.jansi#jansi;1.11: res=C:\Users\vfuchs070114\.ivy2\local\org.fusesource.jansi\jansi\1.11\jars\jansi.jar
local: no ivy file nor artifact found for org.fusesource.jansi#jansi;1.11
local-preloaded-ivy: no namespace defined: using system
no ivy file in cache for org.fusesource.jansi#jansi;1.11: tried C:\Users\vfuchs070114\.ivy2\cache\org.fusesource.jansi\jansi\ivy-1.11.xml
trying file:/C:/Users/vfuchs070114/.sbt/preloaded/org.fusesource.jansi/jansi/1.11/ivys/ivy.xml
tried file:/C:/Users/vfuchs070114/.sbt/preloaded/org.fusesource.jansi/jansi/1.11/ivys/ivy.xml
local-preloaded-ivy: resource not reachable for org.fusesource.jansi#jansi;1.11: res=file:/C:/Users/vfuchs070114/.sbt/preloaded/org.fusesource.jansi/jansi/1.11/ivys/ivy.xml
local-preloaded-ivy: no ivy file found for org.fusesource.jansi#jansi;1.11
local-preloaded: no namespace defined: using system
no ivy file in cache for org.fusesource.jansi#jansi;1.11: tried C:\Users\vfuchs070114\.ivy2\cache\org.fusesource.jansi\jansi\ivy-1.11.xml
trying file:/C:/Users/vfuchs070114/.sbt/preloaded/org/fusesource/jansi/jansi/1.11/jansi-1.11.pom
tried file:/C:/Users/vfuchs070114/.sbt/preloaded/org/fusesource/jansi/jansi/1.11/jansi-1.11.pom
local-preloaded: resource not reachable for org/fusesource/jansi#jansi;1.11: res=file:/C:/Users/vfuchs070114/.sbt/preloaded/org/fusesource/jansi/jansi/1.11/jansi-1.11.pom
trying file:/C:/Users/vfuchs070114/.sbt/preloaded/org/fusesource/jansi/jansi/1.11/jansi-1.11.jar
tried file:/C:/Users/vfuchs070114/.sbt/preloaded/org/fusesource/jansi/jansi/1.11/jansi-1.11.jar
local-preloaded: resource not reachable for org/fusesource/jansi#jansi;1.11: res=file:/C:/Users/vfuchs070114/.sbt/preloaded/org/fusesource/jansi/jansi/1.11/jansi-1.11.jar
local-preloaded: no ivy file nor artifact found for org.fusesource.jansi#jansi;1.11
Maven Central: no namespace defined: using system
no ivy file in cache for org.fusesource.jansi#jansi;1.11: tried C:\Users\vfuchs070114\.ivy2\cache\org.fusesource.jansi\jansi\ivy-1.11.xml
trying https://repo1.maven.org/maven2/org/fusesource/jansi/jansi/1.11/jansi-1.11.pom
tried https://repo1.maven.org/maven2/org/fusesource/jansi/jansi/1.11/jansi-1.11.pom
problem occurred while resolving dependency: org.fusesource.jansi#jansi;1.11 {default=[default(compile)]} with Maven Central: java.lang.RuntimeException: java.util.NoSuchElementException
at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1453)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1439)
at sun.net.www.protocol.http.HttpURLConnection.getHeaderField(HttpURLConnection.java:2965)
at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:489)
at sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:338)
at org.apache.ivy.util.url.BasicURLHandler.checkStatusCode(BasicURLHandler.java:130)
at org.apache.ivy.util.url.BasicURLHandler.getURLInfo$57a0216e(BasicURLHandler.java:73)
at org.apache.ivy.util.url.BasicURLHandler.getURLInfo(BasicURLHandler.java:54)
at org.apache.ivy.plugins.repository.url.URLResource.init(URLResource.java:65)
at org.apache.ivy.plugins.repository.url.URLResource.exists(URLResource.java:81)
at org.apache.ivy.plugins.resolver.RepositoryResolver.findResourceUsingPattern(RepositoryResolver.java:97)
at org.apache.ivy.plugins.resolver.AbstractPatternsBasedResolver.findResourceUsingPatterns(AbstractPatternsBasedResolver.java:96)
at org.apache.ivy.plugins.resolver.IBiblioResolver.findIvyFileRef(IBiblioResolver.java:102)
at org.apache.ivy.plugins.resolver.BasicResolver.getDependency(BasicResolver.java:228)
at org.apache.ivy.plugins.resolver.IBiblioResolver.getDependency(IBiblioResolver.java:512)
at org.apache.ivy.plugins.resolver.ChainResolver.getDependency(ChainResolver.java:104)
at org.apache.ivy.core.resolve.IvyNode.loadData(IvyNode.java:169)
at org.apache.ivy.core.resolve.VisitNode.loadData(VisitNode.java:292)
at org.apache.ivy.core.resolve.ResolveEngine.fetchDependencies(ResolveEngine.java:714)
at org.apache.ivy.core.resolve.ResolveEngine.doFetchDependencies(ResolveEngine.java:799)
at org.apache.ivy.core.resolve.ResolveEngine.fetchDependencies(ResolveEngine.java:722)
at org.apache.ivy.core.resolve.ResolveEngine.getDependencies(ResolveEngine.java:594)
at org.apache.ivy.core.resolve.ResolveEngine.resolve(ResolveEngine.java:234)
at xsbt.boot.Update.xsbt$boot$Update$$lockedApply(Update.scala:105)
at xsbt.boot.Update$$anon$4.call(Update.scala:99)
at xsbt.boot.Locks$GlobalLock.withChannel$1(Locks.scala:93)
at xsbt.boot.Locks$GlobalLock.xsbt$boot$Locks$GlobalLock$$withChannelRetries$1(Locks.scala:78)
at xsbt.boot.Locks$GlobalLock$$anonfun$withFileLock$1.apply(Locks.scala:97)
at xsbt.boot.Using$.withResource(Using.scala:10)
at xsbt.boot.Using$.apply(Using.scala:9)
at xsbt.boot.Locks$GlobalLock.ignoringDeadlockAvoided(Locks.scala:58)
at xsbt.boot.Locks$GlobalLock.withLock(Locks.scala:48)
at xsbt.boot.Locks$.apply0(Locks.scala:31)
at xsbt.boot.Locks$.apply(Locks.scala:28)
at xsbt.boot.Update.apply(Update.scala:100)
at xsbt.boot.Launch.update(Launch.scala:352)
at xsbt.boot.Launch$$anonfun$jansiLoader$1.apply(Launch.scala:178)
at scala.Option.getOrElse(Option.scala:120)
at xsbt.boot.Launch.jansiLoader$2f324eef(Launch.scala:173)
at xsbt.boot.Launch.<init>(Launch.scala:150)
at xsbt.boot.Launcher$.apply(Launch.scala:366)
at xsbt.boot.Launch$.apply(Launch.scala:18)
at xsbt.boot.Boot$.runImpl(Boot.scala:41)
at xsbt.boot.Boot$.main(Boot.scala:17)
at xsbt.boot.Boot.main(Boot.scala)
Caused by: java.util.NoSuchElementException
at java.util.StringTokenizer.nextToken(StringTokenizer.java:349)
at sun.net.www.protocol.http.HttpURLConnection.doTunneling(HttpURLConnection.java:2016)
at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:183)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1511)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1439)
at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480)
... 41 more
It tries to fetch data from https://repo1.maven.org/maven2/org/fusesource/jansi/jansi/1.11/jansi-1.11.pom that exists but it seems it's not able to parse it or something (NoSuchElementException thrown by StringTokenizer).. I've tried to place the pom and jar files in local where it tries to find them : it finds them but it doesn't make a big difference at the end.
similar stuff happens for typesafe-ivy-releases and sbt-ivy-snapshots, for which it's trying to get some ivy informations from remote URLs that don't exist :
https://repo.typesafe.com/typesafe/ivy-releases/org.fusesource.jansi/jansi/1.11/ivys/ivy.xml
https://repo.scala-sbt.org/scalasbt/ivy-snapshots/org.fusesource.jansi/jansi/1.11/ivys/ivy.xml
So in that case, it's not a major surprise that it fails.
Any idea of what could be wrong ?
Thanks
============
Edit
I see here https://github.com/typesafehub/activator/issues/18 that it could be an issue with Ipv4 / Ipv6 . I tried adding the recommended property to force to IPv4 (-Djava.net.preferIPv4Stack=true), but it still doesn't work.
However, I have the feeling the root cause could be something like this, or some encoding issue, that makes the remote file "unparsable"
Edit 2
I spent my whole day yesterday fetching the dependencies manually and at the end it was still not working.. then I tried on my computer at home without a proxy, and it worked perfectly. So now I'm trying again, focusing solely on running sbt.bat : I've added an "echo" in sbt.bat just before the java call to make sure it has what I expect, and it does :
"C:\Toolbox\apps\jdk\jdk1.8.0_25-windows-x64\bin\java.exe" -Dhttps.proxyHost=theProxyURL
-Dhttps.proxyPort=8080 -Dhttps.proxyUser=myId -Dhttps.proxyPassword=myPassword
-cp "C:\Toolbox\apps\sbt-0.13.15_2ndTry\bin\sbt-launch.jar" xsbt.boot.Boot
but I'm still getting the issue.. If it was a certificate issue or something similar, I should get a clearer message, right ?
Edit 3
What I observe is that even if the proxy is not configured properly, we don't get any relevant message in the logs. For instance, by providing intentionally an incorrect proxy settings (a non existing proxy host for instance), we don't get a different message than when configuring correct values. This makes it very inconvenient to debug.
One other I found, is that even when configuring password properly, a modified version may be passed to SBT launcher when password contains an exclamation mark. I've created an issue for this : https://github.com/sbt/sbt/issues/3139 . But even after solving that issue, I still face problems.
There are actually a couple of proxy related open issues : https://github.com/sbt/sbt/issues?utf8=%E2%9C%93&q=is%3Aissue%20is%3Aopen%20proxy
One thing I'm now looking out now is to configure a proxy repository : http://www.scala-sbt.org/0.13/docs/Proxy-Repositories.html (we use Nexus)

Answer 1

Even though I haven't been able to find the definitive answer to my initial question, I've learned a few things :
the workaround for using SBT behind the corporate internet proxy was to use the internal Nexus proxy : Why is sbt trying to download non-existing dependencies?
SBT logging is not very helpful, and the same error message could have dozens of actual reasons for failure.
Depending on the dependency repositories configured (http://www.scala-sbt.org/0.13/docs/Proxy-Repositories.html), SBT will try all kinds of UR, based on various naming conventions (Ivy and Maven). That's why sometimes the URL don't exist at all.

Answer 2

If you are behind the proxy(corporate network)
For windows :
In sbt installation folder, go to conf/sbtconfig.txt file just add the following lines and save and try.
-Dhttp.proxyHost=proxy
-Dhttp.proxyPort=portno
-Dhttp.proxyUser=UserId
-Dhttp.proxyPassword=Password
-Dhttps.proxyHost=proxyhost
-Dhttps.proxyPort=portno
-Dhttps.proxyUser=UserId
-Dhttps.proxyPassword=Password
For Linux/Unix:
export JAVA_OPTS="$JAVA_OPTS -Dhttp.proxyHost=proxyHost -Dhttp.proxyPort=proxyPortNumber -Dhttp.proxyUser=UserId -Dhttp.proxyPassword=Password -Dhttps.proxyHost=proxyHost -Dhttps.proxyPort=proxyPortNumber -Dhttps.proxyUser=UserId -Dhttps.proxyPassword=Password"
It worked for me.

Answer 3

Make sure you don't have any proxy environment variables (http_proxy, https_proxy etc.) set on the box your sbt build is being executed. In my experience, it could be problematic for SBT if you've set it up to download dependencies from a proxy repo over https (e.g. Nexus) and have https_proxy env var set at the same time. As explained in the docs those proxy settings are automatically picked up by SBT so unset https_proxy first before your run your sbt build.

Answer 4

For those behind a Corporate proxy and still trying to figure out how to set up HTTP proxy in IntelliJ, this worked for me. I hope this helps someone else new to IntelliJ like myself.
(Note: specify your network username and password in the corresponding boxes):-