I have a project that's main purpose is to provide a collection of algorithms to client. However, it secondarily handles a HTTP interfaces to those algorithms. Since I don't want my clients to inherent the HTTP libraries (unfiltered and dispatch) I have them scoped as provided in sbt. How can I set up my project so that when I sbt assembly, these dependencies (along with a logging implementation) are compiled but when I sbt package or sbt publish they are not?
I think the answer is as simple as telling the assembly plugin to use the compile configuration of jars instead of the runtime dependency (since provided are not on the runtime classpath).
sbt 0.13 syntax:
fullClasspath in assembly := (fullClasspath in Compile).value
Related
I'm using SBT as a build tool for spark projects. I'm able to create a fat jar of my dependencies using the sbt-assembly plugin.
However, this produces a ~120M jar, mostly of dependencies which I need to keep uploading to S3 to run my code -- this takes 3-5 minutes to do. Not a lot of time, but fairly annoying.
What would improve things a lot would be to have SBT produce a jar of the dependencies (which changes rarely), and a small jar of my application code which I should be able to upload in a few seconds.
Is this possible? I'm pretty new to SBT.
sbt-assembly supports out-of-the-box splitting project application jar from dependency jar. To produce just the dependency jar execute
sbt assemblyPackageDependency
To produce jar with just your project application code define assemblyOption as follows
assemblyOption in assembly := (assemblyOption in assembly).value.copy(includeScala = false, includeDependency = false)
and execute sbt assembly as usual.
I have this line of code in my build.sbt file:
libraryDependencies ++= Seq("com.foo" %% "lib" % "1.2.3")
Imagine that this library depends on "com.bar.lib" lib. Now in my code I can import com.bar.lib._ and it'll work. But I don't want this to compile, so maybe there is SBT plugin out there just for this purpose?
One of libraries I'm using depends on old cats version. I spent really long time to understand why mapN method not works... I just never imported a newer version of cats in the subproject.
SBT offers the intransitive and exclude features to deal with issues like this, as #earldouglas points out. See: https://www.scala-sbt.org/1.x/docs/Library-Management.html
You replied:
I tried to do so, but intransitive() don't import transitive dependencies (so I need to import all of them by hand to make it compile).
Yes, that is what it is for
What I want is something that will warn me about using libraries not directly imported in SBT file.
So you want transitive dependencies on your classpath, but you want the compiler to reject uses of transitive classes in your project code while allowing them in library code?
That is not a sensible approach: at runtime, these transitive dependencies will be on the classpath. The JVM classpath does not distinguish between different kinds of dependencies; such distinction only exists in SBT at build time.
You would be much better served by either
including a newer version of the cats library, overriding the transitive dep or
excluding the transitively included cats library, if it is broken.
However, I think you probably could achieve what you want by setting different dependencies at different build stages:
at Compile stage, include the dependency with intransitive. Your code should compile against your direct dependencies, but fail if you referenced any transitive dependencies
at Runtime stage, include the dependency with its transitive deps
the SBT code might look like this (untested):
(libraryDependencies in Compile) ++= Seq("com.foo" %% "lib" % "1.2.3" intransitive())
(libraryDependencies in Runtime) ++= Seq("com.foo" %% "lib" % "1.2.3")
I am trying to generate, with sbt assembly, from a single project several jars. Each containing some of the dependencies.
So far I have found only this QA that is close to what I am looking for. However I don't need to have separate configs, basically when I run assembly, I just want to generate all the different jars.
To be more concrete. I want to generate:
One jar with my code and some general dependencies
One jar with hadoop dependencies <- this is the problem, as I don't know how to say, generate another jar that has only those dependencies.
One jar with scala
Without going deep into complex sbt configurations, you could try another approach. The hadoop dependencies being standard, you could mark them as provided in your build to exclude them.
"org.apache.hadoop" % "hadoop-client" % "2.6.0" % "provided"
For Scala, the library jar is also standard and can be downloaded separately by your "user". To remove it from the fat jar, use the following setting (assembly 0.13.0):
assemblyOption in assembly := (assemblyOption in assembly).value.copy(includeScala = false)
The user of your fat jar is then aked to provide both Scala and Hadoop libraries in the classpath.
For example, when using Spark this is the correct approach as these two libraries are both provided by the Spark running environment. The same logic applies for the Hadoop MapReduce environment.
This is for Scala 2.11.1 and sbt 0.13.5.
Say I have a Scala/sbt project with the following directory structure:
root/
build.sbt
src/ ..
project/
plugins.sbt
build.properties
LolUtils.scala
and I want to use some external library in LolUtils.scala. How is this generally accomplished in sbt?
If I simply add the libs I need into build.sbt via libraryDependencies += .. then it doesn't find them and fails on the import line with not found: object ...
If I add a separate project/build.sbt, for some reason it starts failing to resolve my plugins, plus I need to manually specify the Scala version in the nested project/build.sbt, which is unnecessary duplication.
What's the best way to accomplish this?
sbt is recursive which means that it uses itself to compile a build definition, i.e. *.sbt files and *.scala files under project directory. To add extra dependencies to use them in the build definition you have to declare them in a project/build.sbt.
There is one caveat to that. You can set any scalaVersion to your project, that is in build.sbt, but you should not modify scalaVersion in the project/build.sbt as it might conflict with the version sbt itself uses (that may or may not lead to binary incompatibility for plugins).
Sbt 0.13.5 is using Scala 2.10.4, and the library you're going to use must be compatible with that particular version of Scala.
> about
[info] This is sbt 0.13.5
...
[info] sbt, sbt plugins, and build definitions are using Scala 2.10.4
In a SBT build.sbt project file, is it possible to retrieve library dependencies which are not bundled as jar?
In my case, I am trying to use QTSampledSP which requires .dll and .jnilib libraries.
To download the artifact, you need to make Ivy (and hence sbt) explicitly aware of the DLL artifact. Add the following to build.sbt in your project.
lazy val QtSampledJniLibArt = Artifact("qtsampledsp-osx", "jnilib", "jnilib")
libraryDependencies += "com.tagtraum" % "qtsampledsp-osx" % "0.9.6" artifacts(QtSampledJniLibArt)
resolvers += "beatunes" at "http://www.beatunes.com/repo/maven2"
Then you need to tell sbt to pay attention to these artifacts (again build.sbt):
classpathTypes ++= Set("jnilib", "dll")
By default, sbt will only add a few types into the classpath (and jnilib and dll are not amongst them).
[sbt-0-13-1]> help classpathTypes
Artifact types that are included on the classpath.
[sbt-0-13-1]> show classpathTypes
[info] Set(eclipse-plugin, bundle, hk2-jar, orbit, jar)
Since these DLLs/jnilibs are needed on the classpath to run correctly, the above setting classpathTypes where you add the additional types will correct things as you can see below (don't forget to reload when in sbt console).
[sbt-0-13-1]> show classpathTypes
[info] Set(eclipse-plugin, bundle, hk2-jar, jnilib, orbit, jar, dll)
If you need to look in more detail at these files, check out the update report (from the update task) where you can inspect all configurations/modules/artifacts. Run show update in sbt console and look at the files in target/resolution-cache/reports.