I have a simple Scala application built using spray.io. I am using Scala 2.11.2 and SBT 0.13.0. All of my dependencies are listed here:
libraryDependencies ++= Seq(
"com.typesafe.akka" %% "akka-actor" % "2.3.6",
"com.typesafe.akka" %% "akka-slf4j" % "2.3.6",
"io.spray" % "spray-can_2.11" % "1.3.2",
"io.spray" % "spray-routing_2.11" % "1.3.2",
"io.spray" % "spray-json_2.11" % "1.3.1"
)
libraryDependencies += "org.mongodb" %% "casbah" % "2.7.2"
libraryDependencies += "com.stormpath.sdk" % "stormpath-sdk-api" % "1.0.RC4.2"
libraryDependencies += "com.stormpath.sdk" % "stormpath-sdk-httpclient" % "1.0.RC4.2"
While building and running it locally everything is fine but when i try to push / deploy it to PaaS platforms like cloudControl or heroku i am getting below dependency issue:
[info] Resolving org.apache.httpcomponents#httpclient;${httpclient.version} ...
[warn] module not found: org.apache.httpcomponents#httpclient;${httpclient.version}
[warn] ==== local: tried
[warn] /tmp/scala_buildpack_build_dir/.sbt_home/.ivy2/local/org.apache.httpcomponents/httpclient/${httpclient.version}/ivys/ivy.xml
[warn] ==== public: tried
[warn] http://repo1.maven.org/maven2/org/apache/httpcomponents/httpclient/${httpclient.version}/httpclient-${httpclient.version}.pom
[warn] ==== spray repo: tried
[warn] http://repo.spray.io/org/apache/httpcomponents/httpclient/${httpclient.version}/httpclient-${httpclient.version}.pom
[warn] ==== spray nightlies: tried
[warn] http://nightlies.spray.io/org/apache/httpcomponents/httpclient/${httpclient.version}/httpclient-${httpclient.version}.pom
[warn] ==== public: tried
[warn] http://repo1.maven.org/maven2/org/apache/httpcomponents/httpclient/${httpclient.version}/httpclient-${httpclient.version}.pom
...
[warn] ::::::::::::::::::::::::::::::::::::::::::::::
[warn] :: UNRESOLVED DEPENDENCIES ::
[warn] ::::::::::::::::::::::::::::::::::::::::::::::
[warn] :: org.apache.httpcomponents#httpclient;${httpclient.version}: not found
[warn] ::::::::::::::::::::::::::::::::::::::::::::::
[warn] ::::::::::::::::::::::::::::::::::::::::::::::
[warn] :: FAILED DOWNLOADS ::
[warn] :: ^ see resolution messages for details ^ ::
[warn] ::::::::::::::::::::::::::::::::::::::::::::::
[warn] :: com.stormpath.sdk#stormpath-sdk-api;1.0.RC4.2!stormpath-sdk-api.jar
[warn] ::::::::::::::::::::::::::::::::::::::::::::::
sbt.ResolveException: unresolved dependency: org.apache.httpcomponents#httpclient;${httpclient.version}: not found
...
[error] (*:update) sbt.ResolveException: unresolved dependency: org.apache.httpcomponents#httpclient;${httpclient.version}: not found
[error] download failed: com.stormpath.sdk#stormpath-sdk-api;1.0.RC4.2!stormpath-sdk-api.jar
[error] Total time: 59 s, completed May 4, 2015 1:05:37 PM
Inspecting my dependencies using sbt-dependency-graph shows me that not resolved org.apache.httpcomponents:httpclient is a nested dependency of com.stormpath.sdk:stormpath-sdk-httpclient:1.0.RC4.2.
For some reason variable ${httpclient.version} is not resolved during built - but that is all I can figure it out here. Please help!!!
I am not sure what is going on. But excluding the transitive dependency and adding it to SBT explicitily seems to fix the problem, at least the unresolved dependency problem:
libraryDependencies += "com.stormpath.sdk" % "stormpath-sdk-httpclient" % "1.0.RC4.2" exclude("org.apache.httpcomponents","httpclient")
libraryDependencies += "org.apache.httpcomponents" % "httpclient" % "4.2.2"
But keep in mind that we don't really know which version of apache's httpclient stormpath-sdk-httpclient used in its compilation, so you might run into runtime classpath/linking related exceptions. If that happens I recommend asking the developers of stormpath-sdk-httpclient.
EDIT:
you can see the version they used at:
https://github.com/stormpath/stormpath-sdk-java/blob/master/pom.xml#L98
Related
My dependencies in build.sbt is as follows
scalaVersion := "2.11.8"
ibraryDependencies ++= Seq(
"org.apache.spark" % "spark-core_2.11" % "2.1.0" %"provided" ,
"org.apache.spark" % "spark-sql_2.11" % "2.1.0" %"provided" ,
"org.scalatest" %%"scalatest" % "2.0" %"test"
}
when i build project i got below error
[warn] ::::::::::::::::::::::::::::::::::::::::::::::
[warn] :: UNRESOLVED DEPENDENCIES ::
[warn] ::::::::::::::::::::::::::::::::::::::::::::::
[warn] :: org.scalatest#scalatest_2.11;2.0: not found
[warn] ::::::::::::::::::::::::::::::::::::::::::::::
[warn]
[warn] Note: Unresolved dependencies path:
[warn] org.scalatest:scalatest_2.11:2.0
[warn] +- transformer:transformer_2.11:1.0
sbt.ResolveException: unresolved dependency: org.scalatest#scalatest_2.11;2.0: not found
at sbt.IvyActions$.sbt$IvyActions$$resolve(IvyActions.scala:313)
at sbt.IvyActions$$anonfun$updateEither$1.apply(IvyActions.scala:191)
at sbt.IvyActions$$anonfun$updateEither$1.apply(IvyActions.scala:168)
at sbt.IvySbt$Module$$anonfun$withModule$1.apply(Ivy.scala:156)
though it work fine if i remove the provided keyword from spark dependencies
and also work if i use 3.0 version of scalatest but i need to use 2.0 version only
can someone help how could i resolve this issue?
Error reason: "scalatest" version "2.0" absent in Maven repository:
http://search.maven.org/#search%7Cga%7C1%7Cg%3A%22org.scalatest%22%20AND%20a%3A%22scalatest_2.11%22
Please try existed, for example "2.2.0".
Guess, keyword "provided" is not related to error.
I have a Scala app developed using IntelliJ.
When I try to package it using sbt, I get the following error and it complains about the sqljdbc4 which I have included in my build.sbt file.
> package
[info] Updating {file:/C:/Users/jthomas1/IdeaProjects/AvroToParquet/}avrotoparquet...
[info] Resolving com.microsoft.sqlserver#sqljdbc4;4.0 ...
[warn] module not found: com.microsoft.sqlserver#sqljdbc4;4.0
[warn] ==== local: tried
[warn] C:\Users\jthomas1\.ivy2\local\com.microsoft.sqlserver\sqljdbc4\4.0\ivys\ivy.xml
[warn] ==== public: tried
[warn] https://repo1.maven.org/maven2/com/microsoft/sqlserver/sqljdbc4/4.0/sqljdbc4-4.0.pom
[warn] ==== local-preloaded-ivy: tried
[warn] C:\Users\jthomas1\.sbt\preloaded\com.microsoft.sqlserver\sqljdbc4\4.0\ivys\ivy.xml
[warn] ==== local-preloaded: tried
[warn] file:/C:/Users/jthomas1/.sbt/preloaded/com/microsoft/sqlserver/sqljdbc4/4.0/sqljdbc4-4.0.pom
[info] Resolving jline#jline;2.11 ...
[warn] ::::::::::::::::::::::::::::::::::::::::::::::
[warn] :: UNRESOLVED DEPENDENCIES ::
[warn] ::::::::::::::::::::::::::::::::::::::::::::::
[warn] :: com.microsoft.sqlserver#sqljdbc4;4.0: not found
[warn] ::::::::::::::::::::::::::::::::::::::::::::::
[warn]
[warn] Note: Unresolved dependencies path:
[warn] com.microsoft.sqlserver:sqljdbc4:4.0 (C:\Users\jthomas1\IdeaProjects\AvroToParquet\build.sbt#L13-14)
[warn] +- default:avrotoparquet_2.11:0.1
[trace] Stack trace suppressed: run last *:update for the full output.
[error] (*:update) sbt.ResolveException: unresolved dependency: com.microsoft.sqlserver#sqljdbc4;4.0: not found
[error] Total time: 4 s, completed Nov 2, 2017 5:18:51 PM
Following from by build.sbt file
name := "AvroToParquet"
version := "0.1"
scalaVersion := "2.11.1"
libraryDependencies += "com.microsoft.sqlserver"
% "sqljdbc4" % "4.0"
% "runtime"
libraryDependencies ++= Seq(
"org.apache.spark" %% "spark-core" % "1.6.1",
"org.apache.spark" %% "spark-sql" % "1.6.1",
"org.apache.spark" %% "spark-hive" % "1.6.1"
)
Appreciate your help!
You can add
resolvers += "Clojars" at "http://clojars.org/repo/"
to your build.sbt and it should solve the problem.
Note, that there are similar questions already and in the answers people write
Unfortunately Microsoft doesn't make this artifact available via any maven repository. You need to download the jar from the Microsoft website, and then manually install it into your local maven repository.
So if you don't want to use Clojars repository from above, you can just download this jar from Microsoft and place it in the lib/ folder in your sbt project.
Also see this answer which refers to another artifact for this driver available in Maven Central.
A Microsoft Sql Server JDBC is listed in maven central here.
There is a tab on that page listing sbt info for this jar:
libraryDependencies += "com.microsoft.sqlserver" % "mssql-jdbc" % "7.4.1.jre12"
Newbie in scala here, going through a scala tutorial.
I am trying to use scalatest as mentioned in the tutorial but getting this error:
[error] (*:update) sbt.ResolveException: unresolved dependency: org.scalatest#scalatest;3.0.1: not found
> scalaVersion
[info] 2.10.6
My build.sbt file looks like this:
name := "tasky"
version := "0.1.0"
libraryDependencies += "org.scalactic" %% "scalactic" % "3.0.1"
libraryDependencies += "org.scalatest" % "scalatest" % "3.0.1" % "test"
Also as mentioned here, I have created the global.sbt and plugins.sbt file at the respective locations with the required details. But, I am still getting the error:
[info] Updating {file:/home/tushar/lsbt/tasky/}tasky...
[info] Resolving org.scalatest#scalatest;3.0.1 ...
[warn] module not found: org.scalatest#scalatest;3.0.1
[warn] ==== local: tried
[warn] /home/tushar/.ivy2/local/org.scalatest/scalatest/3.0.1/ivys/ivy.xml
[warn] ==== public: tried
[warn] https://repo1.maven.org/maven2/org/scalatest/scalatest/3.0.1/scalatest-3.0.1.pom
[warn] ==== local-preloaded-ivy: tried
[warn] /home/tushar/.sbt/preloaded/org.scalatest/scalatest/3.0.1/ivys/ivy.xml
[warn] ==== local-preloaded: tried
[warn] file:////home/tushar/.sbt/preloaded/org/scalatest/scalatest/3.0.1/scalatest-3.0.1.pom
[warn] ==== Artima Maven Repository: tried
[warn] http://repo.artima.com/releases/org/scalatest/scalatest/3.0.1/scalatest-3.0.1.pom
[info] Resolving org.fusesource.jansi#jansi;1.4 ...
[warn] ::::::::::::::::::::::::::::::::::::::::::::::
[warn] :: UNRESOLVED DEPENDENCIES ::
[warn] ::::::::::::::::::::::::::::::::::::::::::::::
[warn] :: org.scalatest#scalatest;3.0.1: not found
[warn] ::::::::::::::::::::::::::::::::::::::::::::::
[warn]
[warn] Note: Unresolved dependencies path:
[warn] org.scalatest:scalatest:3.0.1 (/home/tushar/lsbt/tasky/built.sbt#L4-5)
[warn] +- default:tasky_2.10:0.1.0
[trace] Stack trace suppressed: run last *:update for the full output.
[error] (*:update) sbt.ResolveException: unresolved dependency: org.scalatest#scalatest;3.0.1: not found
Any pointers to resolve this issue?
you should use %% instead of a single % as shown below. the %% changes the artifact from scalatest to scalatest_2.10 since the version of scala used in the project is 2.10.
libraryDependencies += "org.scalatest" %% "scalatest" % "3.0.1" % "test"
You can either use %% or specify the artifact_id explicitly as below
libraryDependencies += "org.scalatest" % "scalatest_2.10" % "3.0.1" % "test"
the former approach is recommended over the later.
Explanation
Scala's minor version's are not backward compatible. so a library compiled with 2.10 version cannot be used in a project using scala 2.11. So a library has to be published by compiling it for each versions separately (2.10, 2.11 and 2.12 for eg).
thus all Scala artifacts are published with scala version suffixed at the end. Thus scalatest has a artifact scalatest_2.10,scalatest_2.11 and scalatest_2.12.
In order to make it easier to select the right version of the artifact in your SBT build depending on your scala version , SBT provides a convient symbol %% that will change the artifact id by appending the scala version for you. Thus if "org.scalatest" %% "scalatest" % "3.0.1" would translate to org.scalatest#scalatest_2.10;3.0.1 on your build dynamically.
If you are using maven repository then it should be
libraryDependencies += "org.scalatest" % "scalatest_2.11" % "3.0.1"
And your scalactic should be
libraryDependencies += "org.scalactic" % "scalactic_2.11" % "3.0.1"
I'm trying to build a Scala jar file to run it in spark.
I'm following this tutorial.
when trying to build jar file using sbt as here, i'm facing with following error
[info] Resolving org.apache.spark#spark-core_2.10.4;1.0.2 ...
[warn] module not found: org.apache.spark#spark-core_2.10.4;1.0.2
[warn] ==== local: tried
[warn] /home/hduser/.ivy2/local/org.apache.spark/spark-core_2.10.4/1.0.2/ivys/ivy.xml
[warn] ==== Akka Repository: tried
[warn] http://repo.akka.io/releases/org/apache/spark/spark-core_2.10.4/1.0.2/spark-core_2.10.4-1.0.2.pom
[warn] ==== public: tried
[warn] http://repo1.maven.org/maven2/org/apache/spark/spark-core_2.10.4/1.0.2/spark-core_2.10.4-1.0.2.pom
[warn] ::::::::::::::::::::::::::::::::::::::::::::::
[warn] :: UNRESOLVED DEPENDENCIES ::
[warn] ::::::::::::::::::::::::::::::::::::::::::::::
[warn] :: org.apache.spark#spark-core_2.10.4;1.0.2: not found
[warn] ::::::::::::::::::::::::::::::::::::::::::::::
[error] {file:/home/prithvi/scala/asd/}default-d57abf/*:update: sbt.ResolveException: unresolved dependency: org.apache.spark#spark-core_2.10.4;1.0.2: not found
[error] Total time: 2 s, completed 13 Aug, 2014 5:24:24 PM
what's the issue and how to solve it.
Dependency issue has been resolved. Thank you "om-nom-nom"but new error arised
[warn] ::::::::::::::::::::::::::::::::::::::::::::::
[warn] :: FAILED DOWNLOADS ::
[warn] :: ^ see resolution messages for details ^ ::
[warn] ::::::::::::::::::::::::::::::::::::::::::::::
[warn] :: org.eclipse.jetty.orbit#javax.transaction;1.1.1.v201105210645!javax.transaction.orbit
[warn] :: org.eclipse.jetty.orbit#javax.servlet;3.0.0.v201112011016!javax.servlet.orbit
[warn] :: org.eclipse.jetty.orbit#javax.mail.glassfish;1.4.1.v201005082020!javax.mail.glassfish.orbit
[warn] :: org.eclipse.jetty.orbit#javax.activation;1.1.0.v201105071233!javax.activation.orbit
[warn] ::::::::::::::::::::::::::::::::::::::::::::::
[error] {file:/home/prithvi/scala/asd/}default-c011e4/*:update: sbt.ResolveException: download failed: org.eclipse.jetty.orbit#javax.transaction;1.1.1.v201105210645!javax.transaction.orbit
[error] download failed: org.eclipse.jetty.orbit#javax.servlet;3.0.0.v201112011016!javax.servlet.orbit
[error] download failed: org.eclipse.jetty.orbit#javax.mail.glassfish;1.4.1.v201005082020!javax.mail.glassfish.orbit
[error] download failed: org.eclipse.jetty.orbit#javax.activation;1.1.0.v201105071233!javax.activation.orbit
[error] Total time: 855 s, completed 14 Aug, 2014 12:28:33 PM
You have your dependency defined as
"org.apache.spark" %% "spark-core" % "1.0.2"
That %% instructs sbt to substitute current scala version to artifact name. Apparently, spark was build for the whole family of 2.10 scala, without specific jars for 2.10.1, 2.10.2 ...
So all you have to do is to redefine it as:
"org.apache.spark" % "spark-core_2.10" % "1.0.2"
I had the same issue.
Looks like that some bugs are in different versions/compilations/etc.
For me the following build.sbt worked fine
name := "My Project"
version := "1.0"
scalaVersion := "2.11.8"
libraryDependencies += "org.apache.spark" %% "spark-sql" % "2.3.2"
Hope it helps
spark-core_2.10.4;1.0.2 means that it is build on top of scala 2.10 vesion.
so you have to specified this scalaVersion := "2.10.4" in your build file.
Please check your .sbt file and change accordingly.
libraryDependencies ++= Seq(
"org.apache.spark" %% "spark-core" % "1.1.0",
"org.eclipse.jetty.orbit" % "javax.servlet" % "3.0.0.v201112011016",
"org.eclipse.jetty.orbit" % "javax.transaction" % "1.1.1.v201105210645",
"org.eclipse.jetty.orbit" % "javax.mail.glassfish" % "1.4.1.v201005082020"
)
How can you change the current dependencies? I mean, when you type sbt package for a build file like:
name := "Simple Project"
version := "1.0"
scalaVersion := "2.10.4"
libraryDependencies += "org.apache.spark" %% "spark-core" % "1.2.0"
SBT will start resolving and downloading all kinds of dependencies. But if you see that it is failing on a dependency that is no longer inthe maven repo, what to do? Where can you change the dpencies it tries.
#OP: The problem is that your SBT is outdated. If you downloaded it using apt, you can use apt to remove it as well. In any case, download the latest .tgz (not the .deb) and simply unpack it, after that add the /sbt/bin/ folder to your .bashrc I noticed that older SBT's (the .deb and apt-get versions) work with older scala versions. You either need to manually add or change the dependencies that the older SBT is trying to find or simply change to the latest (not sooo)SBT.
What is the repository for the current version of scrooge-sbt-plugin? Or are the setup instructions outdated?
According to the documentation, I added this to a Play Framework project:
In project/plugins.sbt
addSbtPlugin("com.twitter" %% "scrooge-sbt-plugin" % "3.3.2")
In build.sbt:
com.twitter.scrooge.ScroogeSBT.newSettings
libraryDependencies ++= Seq(
"org.apache.thrift" % "libthrift" % "0.8.0",
"com.twitter" %% "scrooge-core" % "3.3.2",
"com.twitter" %% "finagle-thrift" % "6.5.0"
)
After play clean-all and play-compile I get this output:
[warn] module not found: com.twitter#scrooge-sbt-plugin;3.3.2
[warn] ==== typesafe-ivy-releases: tried
[warn] http://repo.typesafe.com/typesafe/ivy-releases/com.twitter/scrooge-sbt-plugin/scala_2.10/sbt_0.13/3.3.2/ivys/ivy.xml
[warn] ==== sbt-plugin-releases: tried
[warn] http://repo.scala-sbt.org/scalasbt/sbt-plugin-releases/com.twitter/scrooge-sbt-plugin/scala_2.10/sbt_0.13/3.3.2/ivys/ivy.xml
[warn] ==== local: tried
[warn] /opt/play-2.2.0/repository/local/com.twitter/scrooge-sbt-plugin/scala_2.10/sbt_0.13/3.3.2/ivys/ivy.xml
[warn] ==== Maven2 Local: tried
[warn] file:/home/fernando/.m2/repository/com/twitter/scrooge-sbt-plugin_2.10_0.13/3.3.2/scrooge-sbt-plugin-3.3.2.pom
[warn] ==== sonatype-oss-snapshots: tried
[warn] http://oss.sonatype.org/content/repositories/snapshots/com/twitter/scrooge-sbt-plugin_2.10_0.13/3.3.2/scrooge-sbt-plugin-3.3.2.pom
[warn] ==== public: tried
[warn] http://repo1.maven.org/maven2/com/twitter/scrooge-sbt-plugin_2.10_0.13/3.3.2/scrooge-sbt-plugin-3.3.2.pom
[warn] ==== Typesafe repository: tried
[warn] http://repo.typesafe.com/typesafe/releases/com/twitter/scrooge-sbt-plugin_2.10_0.13/3.3.2/scrooge-sbt-plugin-3.3.2.pom
[warn] ::::::::::::::::::::::::::::::::::::::::::::::
[warn] :: UNRESOLVED DEPENDENCIES ::
[warn] ::::::::::::::::::::::::::::::::::::::::::::::
[warn] :: com.twitter#scrooge-sbt-plugin;3.3.2: not found
[warn] ::::::::::::::::::::::::::::::::::::::::::::::
[warn]
[warn] Note: Some unresolved dependencies have extra attributes. Check that these dependencies exist with the requested attributes.
[warn] com.twitter:scrooge-sbt-plugin:3.3.2 (sbtVersion=0.13, scalaVersion=2.10)
[warn]
sbt.ResolveException: unresolved dependency: com.twitter#scrooge-sbt-plugin;3.3.2: not found
There seems to be a version 3.3.1 at maven.twttr.com. What about version 3.3.2? I couldn't find it at mvnrepository.com or oss.sonatype.org.
The repository is on https://oss.sonatype.org/content/groups/public.
With a look into the Build.scala you can find out to which repository they publish.
If you look into https://oss.sonatype.org/content/groups/public/com/twitter/ and search for "scrooge-sbt-plugin" you fill find folders that ends with "_0.12", so it is published there as SBT 0.12.x plugin. You probably can't use this plugin for Play 2.2.x since it uses SBT 0.13.x.
Version 3.3.2 is not the latest release and I had problems to resolve all files. Using 3.9.2 works:
project/build.properties mus contain SBT 0.12 (example):
sbt.version=0.12.2
project/plugins.sbt must contain the resolver:
resolvers += "sonatype" at "https://oss.sonatype.org/content/groups/public"
addSbtPlugin("com.twitter" %% "scrooge-sbt-plugin" % "3.9.2")
And finally build.sbt or Build.scala must contain:
com.twitter.scrooge.ScroogeSBT.newSettings
scalaVersion := "2.10.1"
libraryDependencies ++= Seq(
"org.apache.thrift" % "libthrift" % "0.8.0",
"com.twitter" %% "scrooge-core" % "3.9.2",
"com.twitter" %% "finagle-thrift" % "6.5.0"
)
First of all, version 3.16.3 is the latest version for sbt 0.13.x
What do you mean by repository?
The code repository is on GitHub, all of the Scrooge stuff including the sbt-plugin stuff are all there.
As for the artifact repository, I am pretty sure it's on maven central or some other standard repo. You should not have to add a resolver to your sbt build. But if you do for some reason than the sonatype one mentioned by #Schleichardt seems about right.