How to specify a different resolver for certain dependencies - scala

I am in a situation where I need to specify a custom resolver for my SBT project, but only to download 1 or 2 dependencies. I want all the other dependencies to be fetched from Maven repository.
Here is my build.sbt file:
...Project definition...
resolvers := Seq(
"Maven" at "https://repo1.maven.org/"
)
//Akka dependencies
libraryDependencies ++= Seq(
"com.typesafe.akka" %% "akka-actor" % akkaActorsVersion,
"com.typesafe.akka" %% "akka-testkit" % akkaActorsVersion % Test,
"com.typesafe.akka" %% "akka-stream" % akkaStreamsVersion,
"com.typesafe.akka" %% "akka-stream-testkit" % akkaStreamsVersion % Test,
"com.typesafe.akka" %% "akka-http" % akkaHttpVersion,
"com.typesafe.akka" %% "akka-http-testkit" % akkaHttpVersion % Test,
"com.datastax.cassandra" % "cassandra-driver-core" % "3.3.0",
"com.typesafe.akka" %% "akka-http-spray-json" % akkaHttpVersion,
"io.spray" %% "spray-json" % "1.3.5",
"de.heikoseeberger" %% "akka-http-circe" % "1.23.0",
"io.circe" %% "circe-generic" % "0.10.0",
"com.pauldijou" %% "jwt-core" % "0.13.0",
"com.pauldijou" %% "jwt-circe" % "0.13.0",
"org.slf4j" % "slf4j-simple" % "1.6.4",
"com.microsoft.azure" % "azure-storage" % "8.4.0",
"com.datastax.cassandra" % "cassandra-driver-extras" % "3.1.4",
"io.jvm.uuid" %% "scala-uuid" % "0.3.0",
"org.scalatest" %% "scalatest" % "3.0.5" % "test",
"org.cassandraunit" % "cassandra-unit" % "3.1.1.0" % "test",
"io.monix" %% "monix" % "3.0.0-8084549",
"org.bouncycastle" % "bcpkix-jdk15on" % "1.48"
)
resolvers := Seq("Artifactory" at "http://10.3.1.6:8081/artifactory/libs-release-local/")
Credentials += Credentials("Artifactory Realm", "10.3.1.6", ARTIFACTORY_USER, ARTIFACTORY_PASSWORD)
libraryDependencies ++=
Seq(
"com.org" % "common-layer_2.11" % "0.3",
)
However the build fails with errors that say that SBT is trying to fetch libraries from Artifactory instead of from Maven.
For example the Cassandra driver dependency
unresolved dependency: com.datastax.cassandra#cassandra-driver-extras;3.1.4: Artifactory: unable to get resource for com/datastax/cassandra#cassandra-driver-extras;3.1.4: res=http://10.3.1.6:8081/artifactory/libs-release-local/com/datastax/cassandra/cassandra-driver-extras/3.1.4/cassandra-driver-extras-3.1.4.pom
I have searched the internet and the documentation and I don't see a clear way to handle this, even though I'm surprised because this seems like a common problem.
Any ideas about how I could enforce the priorities/ordering of resolvers in SBT?

Please note that when you are doing
resolvers := Seq("resolver" at "https://path")
You are overriding the existing user-defined additional resolvers. Therefore if you are doing:
resolvers := Seq("resolver1" at "https://path1")
resolvers := Seq("resolver2" at "https://path2")
You are ending up only with resolver2.
In order to have both resolvers, you need to do something like:
resolvers ++= Seq(
"resolver1" at "https://path1",
"resolver2" at "https://path2"
)
SBT search the dependencies according to the order of the given resolvers. This means that in the given example, it will search first at resolver1, and only if it doesn't find, it will go to resolver2.
Another thing you need to know, is that SBT has predefined resolvers.
You can read more about sbt resolvers at: https://www.scala-sbt.org/1.x/docs/Resolvers.html

Related

New sbt application using the akka-http template, how to determine resolvers and add maven central?

I have a new sbt application that I built using the akka http g8 template.
I am trying to add reactivemongo 1.0 to my build and I am getting this error:
not found: https://repo1.maven.org/maven2/org/reactivemongo/reactivemongo_2.13/1.0/reactivemongo_2.13-1.0.pom
The documentation says this library is in maven central.
How can I determine which resolver my project is using by default currently in sbt?
Is it possible that this library is not built for scala 2.13.3 or 2.13.1?
How can I debug this type of error.
Thanks!
build.sbt:
import Dependencies._
lazy val akkaHttpVersion = "10.2.1"
lazy val akkaVersion = "2.6.10"
lazy val root = (project in file("."))
.settings(
inThisBuild(
List(
organization := "com.example",
scalaVersion := "2.13.3"
)
),
name := "akka-http",
libraryDependencies ++= Seq(
"com.typesafe.akka" %% "akka-http" % akkaHttpVersion,
"com.typesafe.akka" %% "akka-http-spray-json" % akkaHttpVersion,
"com.typesafe.akka" %% "akka-actor-typed" % akkaVersion,
"com.typesafe.akka" %% "akka-stream" % akkaVersion,
"ch.qos.logback" % "logback-classic" % "1.2.3",
"com.softwaremill.macwire" %% "macros" % "2.3.3" % "provided",
"com.softwaremill.macwire" %% "util" % "2.3.3" % "provided",
"com.github.blemale" %% "scaffeine" % "3.1.0" % "compile",
"org.typelevel" %% "cats-core" % "2.1.1",
"com.lihaoyi" %% "scalatags" % "0.8.2",
"com.github.pureconfig" %% "pureconfig" % "0.13.0",
"org.reactivemongo" %% "reactivemongo" % "1.0",
"com.typesafe.akka" %% "akka-http-testkit" % akkaHttpVersion % Test,
"com.typesafe.akka" %% "akka-actor-testkit-typed" % akkaVersion % Test,
"org.scalatest" %% "scalatest" % "3.0.8" % Test
)
)
.enablePlugins(JavaAppPackaging)
Can you try replacing "org.reactivemongo" %% "reactivemongo" % "1.0" with "org.reactivemongo" %% "reactivemongo" % "1.0.0" % "provided"?
I copy it from Maven Repository https://mvnrepository.com/artifact/org.reactivemongo/reactivemongo_2.13/1.0.0

sbt different libraryDependencies in test than in normal mode

Because of conflicting / transitive (elasticsearch / lucene / jackrabbit) dependencies i want to have different libraryDependencies in test than i have when normally running the app. I solved it with the setup below, but this requires running activator with -Dtest and this will prevent my app from running normally when i'm done testing. The other way around, i.e. running just activator, will run my app but will not run my test. So, not very convenient and i think this can be done much better (btw i'm very new to sbt/scala)
name := """example"""
version := "0.1"
lazy val root = (project in file(".")).enablePlugins(PlayJava)
scalaVersion := "2.11.1"
// fork in Test := true
javaOptions in Test += "-Dconfig.file=conf/application.test.conf"
javaOptions in Test += "-Dlogger.file=conf/test-logger.xml"
// run activator -Dtest
if (sys.props.contains("test")) {
Seq[Project.Setting[_]](
libraryDependencies ++= {
Seq(
javaJdbc,
javaEbean,
cache,
javaWs,
"org.webjars" %% "webjars-play" % "2.3.0-2",
"org.webjars" % "bootstrap" % "3.3.6",
"org.webjars" % "font-awesome" % "4.5.0",
"be.objectify" %% "deadbolt-java" % "2.3.3",
"org.apache.lucene" % "lucene-core" % "3.6.0",
"org.elasticsearch" % "elasticsearch" % "1.7.4" exclude("org.apache.lucene", "lucene-core"),
"javax.jcr" % "jcr" % "2.0",
"org.apache.jackrabbit" % "jackrabbit-core" % "2.11.2",
"org.apache.jackrabbit" % "jackrabbit-jcr2dav" % "2.11.2",
"org.apache.tika" % "tika-parsers" % "1.11",
"org.apache.tika" % "tika-core" % "1.11",
"commons-io" % "commons-io" % "2.4",
"com.typesafe.akka" % "akka-testkit_2.11" % "2.3.14" % "test"
)
}
)
} else {
Seq[Project.Setting[_]](
libraryDependencies ++= {
Seq(
javaJdbc,
javaEbean,
cache,
javaWs,
"org.webjars" %% "webjars-play" % "2.3.0-2",
"org.webjars" % "bootstrap" % "3.3.6",
"org.webjars" % "font-awesome" % "4.5.0",
"be.objectify" %% "deadbolt-java" % "2.3.3",
"org.elasticsearch" % "elasticsearch" % "1.7.4",
"javax.jcr" % "jcr" % "2.0",
"org.apache.jackrabbit" % "jackrabbit-core" % "2.11.2",
"org.apache.jackrabbit" % "jackrabbit-jcr2dav" % "2.11.2",
"org.apache.tika" % "tika-parsers" % "1.11",
"org.apache.tika" % "tika-core" % "1.11",
"commons-io" % "commons-io" % "2.4",
"com.typesafe.akka" % "akka-testkit_2.11" % "2.3.14" % "test"
)
}
)
}
//.. our private nexus repo left out here
resolvers += "JBoss Repository" at "https://repository.jboss.org/nexus/content/repositories"
resolvers += "JBoss Third-Party Repository" at "https://repository.jboss.org/nexus/content/repositories/thirdparty-releases"
resolvers += "Typesafe repository" at "http://repo.typesafe.com/typesafe/releases/"
resolvers += Resolver.url("Objectify Play Repository", url("http://deadbolt.ws/releases/"))(Resolver.ivyStylePatterns)
I don't have a setup where I can really test whether this works, but from how I understand sbt dependencies it should:
Dependencies can have a kind of scope called a configuration. Typically, this is used to define test only dependencies:
"com.typesafe.akka" % "akka-testkit_2.11" % "2.3.14" % "test"
But you should also be able to define compile time and run time only dependencies using "compile" and "runtime" instead.
sbt prints me a warning if I use dependencies with different versions. The problem is, that this will use a different version of a dependency to compile it and then to run it with tests. So it will be run against a different version than it was compiled with. There are of course libraries, where this will work, especially, if you run with a newer version that what you use to compile.
If you really need to compile your application twice with different dependencies and use one build for running and one for testing, I fear, there won't be a solution without extending sbt or something like that.
You could try to make two modules, one with the main code and one for testing and then try to cross-build two different versions of the first module. Sbt can easily cross-build over multiple Scala versions, but I don't think it can do it out of the box for multiple versions of a library.
Thanks #dth, you put me on the right track. The settings below worked for me:
libraryDependencies ++= {
Seq(
javaJdbc,
javaEbean,
cache,
javaWs,
"org.webjars" %% "webjars-play" % "2.3.0-2",
"org.webjars" % "bootstrap" % "3.3.6",
"org.webjars" % "font-awesome" % "4.5.0",
"be.objectify" %% "deadbolt-java" % "2.3.3",
"org.apache.lucene" % "lucene-core" % "3.6.0" % "compile,test",
"org.elasticsearch" % "elasticsearch" % "1.7.4" % "compile,runtime",
"org.elasticsearch" % "elasticsearch" % "1.7.4" % "test" exclude("org.apache.lucene", "lucene-core"),
"javax.jcr" % "jcr" % "2.0",
"org.apache.jackrabbit" % "jackrabbit-core" % "2.11.2",
"org.apache.jackrabbit" % "jackrabbit-jcr2dav" % "2.11.2",
"org.apache.tika" % "tika-parsers" % "1.11",
"org.apache.tika" % "tika-core" % "1.11",
"commons-io" % "commons-io" % "2.4",
"com.typesafe.akka" % "akka-testkit_2.11" % "2.3.14" % "test"
)
}

sbt not setting javaOptions in a forked setting

I'm trying to instrument my server with Kamon, which requires Aspectj weaver. I'm using sbt 0.13.8
However, the options aren't being passed to the forked process.
I've looked here:
https://github.com/eigengo/activator-akka-aspectj/blob/master/build.sbt
and here:
http://www.scala-sbt.org/0.13/docs/Forking.html
And this is my build.sbt:
import sbt.Keys._
name := """myApp"""
version := "0.0.1"
lazy val root = (project in file(".")).enablePlugins(PlayScala)
scalaVersion := "2.11.6"
libraryDependencies ++= Seq(
//jdbc, don not enable this when using slick
cache,
ws,
specs2 % Test,
"com.typesafe.akka" %% "akka-contrib" % "2.4.+",
"org.scalatest" % "scalatest_2.11" % "2.2.4" % "test",
"org.scalatestplus" %% "play" % "1.4.0-M3" % "test",
"com.github.seratch" %% "awscala" % "0.5.+",
"com.typesafe.play" %% "play-slick" % "1.1.1",
"com.typesafe.play" %% "play-slick-evolutions" % "1.1.1",
"mysql" % "mysql-connector-java" % "5.1.+",
"commons-net" % "commons-net" % "3.3",
"net.sourceforge.htmlcleaner" % "htmlcleaner" % "2.15",
"io.strongtyped" %% "active-slick" % "0.3.3",
"org.aspectj" % "aspectjweaver" % "1.8.8",
"org.aspectj" % "aspectjrt" % "1.8.8",
"io.kamon" %% "kamon-core" % "0.5.+",
// "io.kamon" %% "kkamon-system-metrics" % "0.5.+",
"io.kamon" %% "kamon-scala" % "0.5.+",
// "io.kamon" %% "kamon-akka" % "0.5.+",
"io.kamon" %% "kamon-datadog" % "0.5.+"
)
resolvers ++= Seq(
"scalaz-bintray" at "http://dl.bintray.com/scalaz/releases"
)
// Play provides two styles of routers, one expects its actions to be injected, the
// other, legacy style, accesses its actions statically.
routesGenerator := InjectedRoutesGenerator
javaOptions in run += "-javaagent:" + System.getProperty("user.home") + "/.ivy2/cache/org.aspectj/aspectjweaver/jars/aspectjweaver-1.8.8.jar -Xmx:2G"
fork in run := true
connectInput in run := true
I've tried running the app using ./activator start as well as ./activator stage and then running the script.
What am I doing wrong?
Thanks!
The production application should be configurable during deployment. This is my example of start script:
PARAMETERS="-Dconfig.file=conf/production.conf -Dlogger.file=conf/prod-logger.xml"
PARAMETERS="$PARAMETERS -Dhttp.port=9000"
PARAMETERS="$PARAMETERS -J-Xmx8g -J-Xms8g -J-server -J-verbose:gc -J-Xloggc:../logs/portal/gc.log -J-XX:+PrintGCDateStamps"
nohup bin/myApp $PARAMETERS &
For more details see Production Configuration

Why does IntelliJ IDEA debugger jump to wrong Scala version library?

I am using IntelliJ IDEA 13.1.2 with the Scala plugin version 0.36.431 on Windows 7 with sbt 0.13.1.
The following project definition build.sbt has no references to any Scala version other than 2.9.3.
import sbt._
import Keys._
import AssemblyKeys._
import NativePackagerKeys._
name := "simplews"
version := "0.1.0-SNAPSHOT"
val sparkVersion = "0.8.1-incubating"
scalaVersion := "2.9.3"
val akkaVersion = "2.0.5"
libraryDependencies ++= Seq(
"org.apache.spark" % "spark-core_2.9.3" % sparkVersion % "compile->default" withSources(),
"org.apache.spark" % "spark-examples_2.9.3" % sparkVersion % "compile->default" withSources(),
"org.apache.spark" % "spark-tools_2.9.3" % sparkVersion % "compile->default" withSources(),
"org.scalatest" % "scalatest_2.9.3" % "1.9.2" % "test" withSources(),
"org.apache.spark" % "spark-repl_2.9.3" % sparkVersion % "compile->default" withSources(),
"org.apache.kafka" % "kafka" % "0.7.2-spark",
"com.thenewmotion.akka" % "akka-rabbitmq_2.9.2" % "0.0.2" % "compile->default" withSources(),
"com.typesafe.akka" % "akka-actor" % akkaVersion % "compile->default" withSources(),
"com.typesafe.akka" % "akka-testkit" % akkaVersion % "compile->default" withSources(),
"com.rabbitmq" % "amqp-client" % "3.0.1" % "compile->default" withSources(),
"org.specs2" % "specs2_2.9.3" % "1.12.4.1" % "compile->default" withSources(),
"com.nebhale.jsonpath" % "jsonpath" % "1.2" % "compile->default" withSources(),
"org.mockito" % "mockito-all" % "1.8.5",
"junit" % "junit" % "4.11"
)
packagerSettings
packageArchetype.java_application
resolvers ++= Seq(
"Apache repo" at "https://repository.apache.org/content/repositories/releases",
"Cloudera repo" at "https://repository.cloudera.com/artifactory/repo/org/apache/kafka/kafka/0.7.2-spark/",
"akka rabbitmq" at "http://nexus.thenewmotion.com/content/repositories/releases-public",
"Local Repo" at Path.userHome.asFile.toURI.toURL + "/.m2/repository",
Resolver.mavenLocal
)
However as seen in the screenshot the debugger has jumped to scala 2.10.2. Note: the debugger is correctly going to 2.9.3 for some other debugging.
Here is project/plugins.sbt:
resolvers += "sbt-plugins" at "http://scalasbt.artifactoryonline.com/scalasbt/sbt-plugin-releases"
addSbtPlugin("com.typesafe.sbt" % "sbt-native-packager" % "0.7.0-RC2")
addSbtPlugin("com.github.mpeltonen" % "sbt-idea" % "1.6.0")
addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.11.2")
EDIT In order to reproduce it is necessary to do a mvn local install on one or two libraries that are not available in any public repo.
mvn org.apache.maven.plugins:maven-install-plugin:2.5.1:install-file -Dfile=c:\shared\kafka-0.7.2-spark.jar -DgroupId=org.apache.kafka -DartifactId=kafka -Dversion=0.7.2-spark -Dpackaging=jar
I had in any case not considered someone (om-nom-nom !) would attempt an exact reproduction - so had also omitted otherwise extraneous items like mergeStrategy and assemblyKeys.
A fully independent reproducible setup may be a bit in coming - I have been under rather quite heavy demands here.

IDEA and scalariform configuration - unresolved symbols even if scalariform works from command line

This question is partially related to sbt-scalariform plugin - can't resolve settings. I managed to run scalariform from command line as SBT task.
Now the problem is with IDEA. When I open my build.sbt, which looks like this:
import scalariform.formatter.preferences._
name := """scheduling-backend"""
version := "1.0"
scalaVersion := "2.10.2"
resolvers += "spray repo" at "http://repo.spray.io"
resolvers += "spray nightlies" at "http://nightlies.spray.io"
resolvers += "SpringSource Milestone Repository" at "http://repo.springsource.org/milestone"
resolvers += "Neo4j Cypher DSL Repository" at "http://m2.neo4j.org/content/repositories/releases"
libraryDependencies ++= Seq(
"com.typesafe.akka" %% "akka-actor" % "2.3.0",
"com.typesafe.akka" %% "akka-slf4j" % "2.3.0",
"com.typesafe.akka" %% "akka-testkit" % "2.3.0" % "test",
"com.typesafe.akka" %% "akka-persistence-experimental" % "2.3.0",
"io.spray" % "spray-can" % "1.3.0",
"io.spray" % "spray-routing" % "1.3.0",
"io.spray" % "spray-testkit" % "1.3.0" % "test",
"io.spray" %% "spray-json" % "1.2.5",
"ch.qos.logback" % "logback-classic" % "1.0.13",
"org.specs2" %% "specs2" % "1.14" % "test",
"org.springframework.scala" % "spring-scala" % "1.0.0.M2",
"org.springframework.data" % "spring-data-neo4j" % "3.0.0.RELEASE",
"org.springframework.data" % "spring-data-neo4j-rest" % "3.0.0.RELEASE",
"javax.validation" % "validation-api" % "1.1.0.Final",
"com.github.nscala-time" %% "nscala-time" % "0.8.0",
"org.neo4j" % "neo4j-kernel" % "2.0.1" % "test" classifier "tests",
"com.sun.jersey" % "jersey-core" % "1.9",
"org.mockito" % "mockito-all" % "1.9.5"
)
scalacOptions ++= Seq(
"-unchecked",
"-deprecation",
"-Xlint",
"-Ywarn-dead-code",
"-language:_",
"-target:jvm-1.7",
"-encoding", "UTF-8"
)
org.scalastyle.sbt.ScalastylePlugin.Settings
scalariformSettings
ScalariformKeys.preferences := ScalariformKeys.preferences.value
.setPreference(AlignParameters, true)
.setPreference(CompactControlReadability, true)
IDEA reports problems with my file.
I am getting Cannot resolve symbol scalariformSettings and Cannot resolve symbol ScalariformKeyseven if I everything works from terminal.
adding import com.typesafe.sbt.SbtScalariform._ to build.sbt seems to fix the error on 13.1.1 with scala plugin 0.33.403, but I have to admit it ignored the import at first and then randomly started to see it.