Logback dependency preventing SBT running offline - scala

This is a refinement of a previous question with closer attribution to the problem.
I'm trying to confirm I can run my SBT project offline. I can, unless logback is one of my libraryDependencies. Why is logback triggering a problem? I've made a simple example consisting of just a build.sbt and a 'hello world' app...
build.sbt with a bunch of random dependencies to prove the point:
name := "testProject"
version := "0.0.1"
scalaVersion := "2.10.3"
libraryDependencies ++= Seq(
"org.apache.commons" % "commons-math3" % "3.2",
"com.amazonaws" % "aws-java-sdk" % "1.4.0.1",
"ch.qos.logback" % "logback-classic" % "1.1.1",
"com.typesafe.akka" %% "akka-actor" % "2.3.1",
"com.typesafe.akka" %% "akka-remote" % "2.3.1",
"org.scalaz" %% "scalaz-core" % "7.0.3"
)
My steps
Delete ~/.ivy2, ~/.sbt, and target/ inside my project, just to be sure
Ensure connected to the internet, and do sbt run. My hello world runs successfully
Disconnect from internet
sbt clean
sbt run. Build fails with logback as an unresolved dependency
Remove the logback dependency and sbt run again. Hello world runs.
This is the error from step 5:
[info] Resolving ch.qos.logback#logback-classic;1.1.1 ...
[warn] Host repo1.maven.org not found. url=http://repo1.maven.org/maven2/ch/qos/logback/logback-classic/1.1.1/logback-classic-1.1.1.pom
[info] You probably access the destination server through a proxy server that is not well configured.
[warn] module not found: ch.qos.logback#logback-classic;1.1.1
[warn] ==== local: tried
[warn] /home/user/.ivy2/local/ch.qos.logback/logback-classic/1.1.1/ivys/ivy.xml
[warn] ==== public: tried
[warn] http://repo1.maven.org/maven2/ch/qos/logback/logback-classic/1.1.1/logback-classic-1.1.1.pom
[info] Resolving org.fusesource.jansi#jansi;1.4 ...
[warn] ::::::::::::::::::::::::::::::::::::::::::::::
[warn] :: UNRESOLVED DEPENDENCIES ::
[warn] ::::::::::::::::::::::::::::::::::::::::::::::
[warn] :: ch.qos.logback#logback-classic;1.1.1: not found
[warn] ::::::::::::::::::::::::::::::::::::::::::::::
sbt.ResolveException: unresolved dependency: ch.qos.logback#logback-classic;1.1.1: not found
Update:
Using the last command in SBT there is a lot of debug info for dependency resolution. Here's an example of Akka resolving successfully:
[info] Resolving com.typesafe.akka#akka-actor_2.10;2.3.1 ...
[debug] sbt-chain: Checking cache for: dependency: com.typesafe.akka#akka-actor_2.10;2.3.1 {compile=[default(compile)]}
[debug] sbt-chain: module revision found in cache: com.typesafe.akka#akka-actor_2.10;2.3.1
[debug] local: revision in cache: com.typesafe.akka#akka-actor_2.10;2.3.1
[debug] found com.typesafe.akka#akka-actor_2.10;2.3.1 in sbt-chain
[debug] == resolving dependencies testproject#testproject_2.10;0.0.1->com.typesafe.akka#akka-actor_2.10;2.3.1 [compile->runtime]
[debug] == resolving dependencies testproject#testproject_2.10;0.0.1->com.typesafe.akka#akka-actor_2.10;2.3.1 [compile->compile]
[debug] == resolving dependencies com.typesafe.akka#akka-actor_2.10;2.3.1->org.scala-lang#scala-library;2.10.3 [compile->master(*)]
[debug] == resolving dependencies com.typesafe.akka#akka-actor_2.10;2.3.1->org.scala-lang#scala-library;2.10.3 [compile->compile(*)]
[debug] == resolving dependencies com.typesafe.akka#akka-actor_2.10;2.3.1->com.typesafe#config;1.2.0 [compile->master(*)]
Here's the failure for Logback:
[info] Resolving ch.qos.logback#logback-classic;1.1.1 ...
[debug] sbt-chain: Checking cache for: dependency: ch.qos.logback#logback-classic;1.1.1 {compile=[default(compile)]}
[debug] tried /home/user/.ivy2/local/ch.qos.logback/logback-classic/1.1.1/ivys/ivy.xml
[debug] local: no ivy file found for ch.qos.logback#logback-classic;1.1.1
[debug] tried http://repo1.maven.org/maven2/ch/qos/logback/logback-classic/1.1.1/logback-classic-1.1.1.pom
[warn] Host repo1.maven.org not found. url=http://repo1.maven.org/maven2/ch/qos/logback/logback-classic/1.1.1/logback-classic-1.1.1.pom
[info] You probably access the destination server through a proxy server that is not well configured.
[debug] public: no ivy file found for ch.qos.logback#logback-classic;1.1.1
[warn] module not found: ch.qos.logback#logback-classic;1.1.1
[warn] ==== local: tried
[warn] /home/user/.ivy2/local/ch.qos.logback/logback-classic/1.1.1/ivys/ivy.xml
[warn] ==== public: tried
[warn] http://repo1.maven.org/maven2/ch/qos/logback/logback-classic/1.1.1/logback-classic-1.1.1.pom
I can confirm the following files in the cache (not the '~/.ivy/local' folder):
~/.ivy2/cache/ch.qos.logback/logback-classic$ ls
ivy-1.1.1.xml ivy-1.1.1.xml.original ivydata-1.1.1.properties jars

When I try it, the file is indeed resolved from ~/.ivy2/cache. I have no ~/.ivy2/local because I do not publish locally. Is it possible that you have an ivysettings.xml file (on Linux, try locate ivysettings.xml) that is messing with some Ivy cache attribute, like resolutionCacheDir? It would still be a mystery why you only have problems with that one jar....
UPDATE: This appears to be a regression in SBT 0.13.1; it worked in 0.13.0, and as the OP himself discovered (see his answer), it was fixed in 0.13.2.

It can't find it in your local ivy cache, fix that and you've solved your problem.
Try clearing out your ivy cache, maybe it's corrupt.

SBT 0.13.2 was released 3 days after this question, and seems to resolve the problem. I don't know if this was a known issue, but the changes listed for SBT 0.13.2 include
Fix issue where ivy's .+ dependency ranges were not correctly
translated to maven.

Related

Add a maven dependency to SBT project

I am trying to reference a maven project dependency in my build.sbt file. I understand I need to add an additional resolver to my file since the project is hosted in an internal artifactory store
build.sbt
libraryDependencies ++= Seq(
"org.apache.spark" %% "spark-core" % "2.4.4",
"org.apache.spark" %% "spark-sql" % "2.4.4",
"com.<companyname>" %% "<libraryname>" % "2.3.0"
)
resolvers += "<library name>" at "http://artifactory.<internal url>.io:80/dsc-mvn"
However, it turns out that SBT ends up searching for a path that has _2.11 version appended to it. This is the error message that I see in IntelliJ
[info] Loading settings for project sbt-demo from build.sbt ...
[warn] Discarding 1 session setting. Use 'session save' to persist session settings.
[info] Set current project to SparkExample (in build file:<project_path>)
[info] Defining Global / sbtStructureOptions
[info] The new value will be used by Global / ssOptions
[info] Reapplying settings...
[info] Set current project to SparkExample (in build file:<project_path>)
[info] Updating ...
[warn] module not found: com.<companyname>#<libraryname>_2.11;2.3.0
[warn] ==== local: tried
[warn] /Users/vshah/.ivy2/local/com.<companyname>/<libraryname>_2.11/2.3.0/ivys/ivy.xml
[warn] ==== public: tried
[warn] https://repo1.maven.org/maven2/com/<companyname>/<libraryname>_2.11/2.3.0/<libraryname>_2.11-2.3.0.pom
[warn] ==== local-preloaded-ivy: tried
[warn] /Users/vshah/.sbt/preloaded/com.<companyname>/<libraryname>_2.11/2.3.0/ivys/ivy.xml
[warn] ==== local-preloaded: tried
[warn] file:////Users/vshah/.sbt/preloaded/com/<companyname>/<libraryname>_2.11/2.3.0/<libraryname>_2.11-2.3.0.pom
[warn] ==== <libraryname>: tried
[warn] http://artifactory.<internal url>.io:80/dsc-mvn/com/<companyname>/<libraryname>_2.11/2.3.0/<libraryname>_2.11-2.3.0.pom
[warn] ::::::::::::::::::::::::::::::::::::::::::::::
[warn] :: UNRESOLVED DEPENDENCIES ::
[warn] ::::::::::::::::::::::::::::::::::::::::::::::
[warn] :: com.<companyname>#<libraryname>_2.11;2.3.0: not found
[warn] ::::::::::::::::::::::::::::::::::::::::::::::
[warn]
Invalid Path (as the project is trying to search above)-
http://artifactory.<internal url>.io/dsc-mvn/com/<companyname>/<libraryname>_2.11/2.3.0/<libraryname>_2.11-2.3.0.pom
Valid path -
http://artifactory.<internal url>.io/dsc-mvn/com/<companyname>/<libraryname>/2.3.0/<libraryname>_2.11-2.3.0.pom
My question is how do I let SBT know that I am referencing a maven project at the above mentioned valid path?
This should work:
"com.<companyname>" % "<libraryname>" % "2.3.0"
The %% in the definition automatically appends the project scala version to the library name. Check out the official sbt documentation for more info.
refer to https://mvnrepository.com/artifact/org.scala-sbt to get the right Build tool for your project, and kindly check if the version in the dependency matches the version of your scala .
Can you please tell me more about the kind of dependency you need?

SBT module not found module not found: com.microsoft.sqlserver#sqljdbc4;4.0

I have a Scala app developed using IntelliJ.
When I try to package it using sbt, I get the following error and it complains about the sqljdbc4 which I have included in my build.sbt file.
> package
[info] Updating {file:/C:/Users/jthomas1/IdeaProjects/AvroToParquet/}avrotoparquet...
[info] Resolving com.microsoft.sqlserver#sqljdbc4;4.0 ...
[warn] module not found: com.microsoft.sqlserver#sqljdbc4;4.0
[warn] ==== local: tried
[warn] C:\Users\jthomas1\.ivy2\local\com.microsoft.sqlserver\sqljdbc4\4.0\ivys\ivy.xml
[warn] ==== public: tried
[warn] https://repo1.maven.org/maven2/com/microsoft/sqlserver/sqljdbc4/4.0/sqljdbc4-4.0.pom
[warn] ==== local-preloaded-ivy: tried
[warn] C:\Users\jthomas1\.sbt\preloaded\com.microsoft.sqlserver\sqljdbc4\4.0\ivys\ivy.xml
[warn] ==== local-preloaded: tried
[warn] file:/C:/Users/jthomas1/.sbt/preloaded/com/microsoft/sqlserver/sqljdbc4/4.0/sqljdbc4-4.0.pom
[info] Resolving jline#jline;2.11 ...
[warn] ::::::::::::::::::::::::::::::::::::::::::::::
[warn] :: UNRESOLVED DEPENDENCIES ::
[warn] ::::::::::::::::::::::::::::::::::::::::::::::
[warn] :: com.microsoft.sqlserver#sqljdbc4;4.0: not found
[warn] ::::::::::::::::::::::::::::::::::::::::::::::
[warn]
[warn] Note: Unresolved dependencies path:
[warn] com.microsoft.sqlserver:sqljdbc4:4.0 (C:\Users\jthomas1\IdeaProjects\AvroToParquet\build.sbt#L13-14)
[warn] +- default:avrotoparquet_2.11:0.1
[trace] Stack trace suppressed: run last *:update for the full output.
[error] (*:update) sbt.ResolveException: unresolved dependency: com.microsoft.sqlserver#sqljdbc4;4.0: not found
[error] Total time: 4 s, completed Nov 2, 2017 5:18:51 PM
Following from by build.sbt file
name := "AvroToParquet"
version := "0.1"
scalaVersion := "2.11.1"
libraryDependencies += "com.microsoft.sqlserver"
% "sqljdbc4" % "4.0"
% "runtime"
libraryDependencies ++= Seq(
"org.apache.spark" %% "spark-core" % "1.6.1",
"org.apache.spark" %% "spark-sql" % "1.6.1",
"org.apache.spark" %% "spark-hive" % "1.6.1"
)
Appreciate your help!
You can add
resolvers += "Clojars" at "http://clojars.org/repo/"
to your build.sbt and it should solve the problem.
Note, that there are similar questions already and in the answers people write
Unfortunately Microsoft doesn't make this artifact available via any maven repository. You need to download the jar from the Microsoft website, and then manually install it into your local maven repository.
So if you don't want to use Clojars repository from above, you can just download this jar from Microsoft and place it in the lib/ folder in your sbt project.
Also see this answer which refers to another artifact for this driver available in Maven Central.
A Microsoft Sql Server JDBC is listed in maven central here.
There is a tab on that page listing sbt info for this jar:
libraryDependencies += "com.microsoft.sqlserver" % "mssql-jdbc" % "7.4.1.jre12"

Fail to upload to sonatype with sbt 0.13 with xsbt-gpg-plugin unresolved dependency: com.jsuereth#xsbt-gpg-plugin;0.6

my plugins.sbt has this:
logLevel := Level.Warn
addSbtPlugin("com.earldouglas" % "xsbt-web-plugin" % "1.0.0")
resolvers += Resolver.url("sbt-plugin-releases", new URL("http://scalasbt.artifactoryonline.com/scalasbt/sbt-plugin-releases")) (Resolver.ivyStylePatterns)
addSbtPlugin("com.jsuereth" % "xsbt-gpg-plugin" % "0.6")
when i run sbt publish i get:
[warn] ==== sbt-plugin-releases: tried [warn]
http://scalasbt.artifactoryonline.com/scalasbt/sbt-plugin-releases/com.jsuereth/xsbt-gpg-plugin/scala_2.10/sbt_0.13/0.6/ivys/ivy.xml
[warn] :::::::::::::::::::::::::::::::::::::::::::::: [warn] ::
UNRESOLVED DEPENDENCIES :: [warn]
:::::::::::::::::::::::::::::::::::::::::::::: [warn] ::
com.jsuereth#xsbt-gpg-plugin;0.6: not found [warn]
:::::::::::::::::::::::::::::::::::::::::::::: [warn] [warn] Note:
Some unresolved dependencies have extra attributes. Check that these
dependencies exist with the requested attributes. [warn]
com.jsuereth:xsbt-gpg-plugin:0.6 (sbtVersion=0.13,
scalaVersion=2.10) [warn] sbt.ResolveException: unresolved
dependency: com.jsuereth#xsbt-gpg-plugin;0.6: not found at
sbt.IvyActions$.sbt$IvyActions$$resolve(IvyActions.scala:217) at
sbt.IvyActions$$anonfun$update$1.apply(IvyActions.scala:126)
So I tried updating the addSbtPlugin to be:
addSbtPlugin("com.jsuereth" % "xsbt-gpg-plugin" % "0.6",
sbtVersion = "0.12", // SBT version
scalaVersion = "2.9.2" )
Now I get
[warn] The global sbt directory is now versioned and is located at /home//.sbt/0.13.
[warn] You are seeing this warning because there is global configuration in /home//.sbt but not in /home//.sbt/0.13.
[warn] The global sbt directory may be changed via the sbt.global.base system property.
[info] Loading project definition from /home//dev/projects/myproj/project
[warn] Multiple resolvers having different access mechanism configured with same name 'sbt-plugin-releases'. To avoid conflict, Remove duplicate project resolvers (`resolvers`) or rename publishing resolver (`publishTo`).
**java.lang.NoClassDefFoundError: sbt/Scoped$ListSetting**
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:270)
anyway to fix it please?
thanks
Not sure, but I think you use a rather old version of that plugin?
I use the following with latest sbt 0.13.6. I have this globally installed in ~/.sbt/0.13/plugins/build.sbt:
addSbtPlugin("com.typesafe.sbt" % "sbt-pgp" % "0.8.1") // sign Sonatype artifacts
I also left my credentials in ~/.sbt/0.13/sonatype.sbt:
credentials += Credentials("Sonatype Nexus Repository Manager",
"oss.sonatype.org",
"user-name", "pass-word")
This way I can do sbt publish-signed.
Here is the plugin documentation. (As you can see, there is already an even newer version available)

SBT 0.13.1 offline

I'm trying to understand the use of SBT 0.13.1 offline. My goal is to bring whatever is necesary to an offline environment on a USB stick. I thought that if the ivy cache was copied then SBT could pull everything it needs out of it, but that doesn't seem to be the case.
For example...
I'll start by deleting the .ivy2 folder, just to be sure. Now, while online I use the following build.sbt and project/plugins.sbt files to create my eclipse project. from SBT 0.13.1
build.sbt:
name := "TestProject"
version := "0.0.1"
scalaVersion := "2.10.3"
libraryDependencies ++= Seq(
"org.apache.commons" % "commons-math3" % "3.2",
"ch.qos.logback" % "logback-classic" % "1.1.1"
)
project/plugins.sbt:
addSbtPlugin("com.typesafe.sbteclipse" % "sbteclipse-plugin" % "2.4.0")
Sill on the same computiner I go offline, and now sbt clean eclipse gives lots of unresolved dependency errors. E.g.
[warn] Host repo1.maven.org not found. url=http://repo1.maven.org/maven2/org/scala-lang/scala-library/2.10.3/scala-library-2.10.3.pom
[info] You probably access the destination server through a proxy server that is not well configured.
[warn] module not found: org.scala-lang#scala-library;2.10.3
[warn] ==== local: tried
[warn] /home/user/.ivy2/local/org.scala-lang/scala-library/2.10.3/ivys/ivy.xml
[warn] ==== public: tried
[warn] http://repo1.maven.org/maven2/org/scala-lang/scala-library/2.10.3/scala-library-2.10.3.pom
[info] Resolving ch.qos.logback#logback-classic;1.1.1 ...
[warn] Host repo1.maven.org not found. url=http://repo1.maven.org/maven2/ch/qos/logback/logback-classic/1.1.1/logback-classic-1.1.1.pom
[info] You probably access the destination server through a proxy server that is not well configured.
[warn] module not found: ch.qos.logback#logback-classic;1.1.1
[warn] ==== local: tried
[warn] /home/user/.ivy2/local/ch.qos.logback/logback-classic/1.1.1/ivys/ivy.xml
[warn] ==== public: tried
[warn] http://repo1.maven.org/maven2/ch/qos/logback/logback-classic/1.1.1/logback-classic-1.1.1.pom
[info] Resolving org.scala-lang#scala-reflect;2.10.3 ...
[warn] Host repo1.maven.org not found. url=http://repo1.maven.org/maven2/org/scala-lang/scala-reflect/2.10.3/scala-reflect-2.10.3.pom
[info] You probably access the destination server through a proxy server that is not well configured.
[warn] module not found: org.scala-lang#scala-reflect;2.10.3
[warn] ==== local: tried
[warn] /home/user/.ivy2/local/org.scala-lang/scala-reflect/2.10.3/ivys/ivy.xml
[warn] ==== public: tried
[warn] http://repo1.maven.org/maven2/org/scala-lang/scala-reflect/2.10.3/scala-reflect-2.10.3.pom
[info] Resolving org.fusesource.jansi#jansi;1.4 ...
[warn] ::::::::::::::::::::::::::::::::::::::::::::::
[warn] :: UNRESOLVED DEPENDENCIES ::
[warn] ::::::::::::::::::::::::::::::::::::::::::::::
[warn] :: org.scala-lang#scala-library;2.10.3: not found
[warn] :: ch.qos.logback#logback-classic;1.1.1: not found
[warn] :: org.scala-lang#scala-reflect;2.10.3: not found
[warn] ::::::::::::::::::::::::::::::::::::::::::::::
Is it really impossible to clean when working offline?
I notice that SBT is looking in .ivy2/local, but there is no such folder. Only '.ivy2/cache'.
Update:
Suggestion to add .ivy2/local didn't work. It gives many MalformedURLException
[warn] :: org.scala-lang#scala-library;2.10.3: ivy2 cache: unable to get resource for org/scala-lang#scala-library;2.10.3: res=/home/user/.ivy2/cache/org/scala-lang/scala-library/2.10.3/scala-library-2.10.3.pom: java.net.MalformedURLException: no protocol: /home/user/.ivy2/cache/org/scala-lang/scala-library/2.10.3/scala-library-2.10.3.pom
It's looking for a directory org/scala-lang etc, but thr directories in cache are of the form org.scala-lang
Update 2
It seems that the problem is caused with some interaction with logback. I've created a clearer version of my question here, and am accepting the answer that indicated that this 'should' run out of the box.
If you build the project online, then you can go offline after.
Basically, you'd just want to copy the ~/.ivy2 folder (and possibly ~/.sbt folder - unsure) into your environment from a location that has done the building and dependency resolution before.
Removing target/ in a project, then going offline, I can see that this does cause the resources to be resolved.
[info] Resolving org.fusesource.jansi#jansi;1.4 ...
[info] Done updating.
[info] Compiling 4 Scala sources to [...]
I'm not sure this will work but I guess you need to add another resolver in your Build.scala (or build.sbt) like this:
resolvers += "ivy2 cache" at "/home/user/.ivy2/cache"
Let me know if this works, I'm also curious about it.
Edit:
I tried it and sbt does look in your ivy2 cache, go in your build.sbt and add this line:
resolvers += Resolver.file("ivy2 cache", file("/path/to/your/ivy2/cache/"))(Resolver.ivyStylePatterns)
Now SBT should be looking in that directory too.

sbt and scct .... module not found: reaktor#sbt-scct;0.2-SNAPSHOT

scala is version 2.10.2 and sbt is 0.13.0
I am trying to setup scct for scala unit test code coverage so in build.sbt
I added
seq(ScctPlugin.instrumentSettings : _*)
and in project/plugins.sbt I added
resolvers += Classpaths.typesafeResolver
resolvers += "scct-github-repository" at "http://mtkopone.github.com/scct/maven-repo"
addSbtPlugin("reaktor" %% "sbt-scct" % "0.2-SNAPSHOT")
I get errors like ....
module not found: reaktor#sbt-scct;0.2-SNAPSHOT
Last message is
[warn] ==== scct-github-repository: tried
[warn] http://mtkopone.github.com/scct/maven-repo/reaktor/sbt-scct_2.10_0.13/0.2-SNAPSHOT/sbt-scct-0.2-SNAPSHOT.pom
[info] Resolving org.fusesource.jansi#jansi;1.4 ...
[warn] ::::::::::::::::::::::::::::::::::::::::::::::
[warn] :: UNRESOLVED DEPENDENCIES ::
[warn] ::::::::::::::::::::::::::::::::::::::::::::::
[warn] :: reaktor#sbt-scct;0.2-SNAPSHOT: not found
[warn] ::::::::::::::::::::::::::::::::::::::::::::::
[warn]
[warn] Note: Some unresolved dependencies have extra attributes. Check that these dependencies exist with the requested attributes.
[warn] reaktor:sbt-scct:0.2-SNAPSHOT (sbtVersion=0.13, scalaVersion=2.10)
I look in http://mtkopone.github.io/scct/maven-repo/reaktor/
and I see the one I seem to need is missing e.g. I see
sbt-scct_2.9.2_0.13/
scct_2.10/
and a bunch of others but no ... sbt-scct_2.10_0.13
Can you help ? (I am a sbt newbie)
SCCT has a new source code repository: https://github.com/SCCT/scct
As a result you can fetch a release version from maven central with addSbtPlugin("com.github.scct" %% "sbt-scct" % "0.2") in your plugins.sbt file.
My project to reproduce is on GitHub.
Update:
The repository has been relocated at https://github.com/sqality/scct
and you need addSbtPlugin("com.sqality.scct" % "sbt-scct" % "version") in your plugins.sbt file.
The original SCCT is inactive now.
Here are the two most active forks.
http://github.com/sqality
http://github.com/scoverage