I am using a library say A in Scala which is dependent on version x.11 of another library say Z.
Now, I am also using a library say B which is dependent on version x.31 of Z.
This leads to compile error because we will have two versions of library Z, how can I use both libraries A and B in scala's sbt? Is there any way to specify it.
If completely replacing one dependency with a newer version happens to work, then Sparko's solution works. However, that isn't always the case.
If you want to include both versions of a library in the uber-jar produced by sbt-assembly, you'll need to use shading. See this post for an overview of what shading is, and some of the drawbacks associated with it.
Shading is covered in sbt-assembly's documentation here, but if you're anything like me, their way of explaining it will leave you more confused than you started. There's a good blog post, Spark, Uber Jars and Shading with sbt-assembly (waybackmachine link), that helps to demystify it a bit. Here's the relevant section:
I can shade over my typesafe config version, giving it a different
name so Spark won’t get confused between the versions. I quickly went
to my build.sbt file, and added the following code:
assemblyShadeRules in assembly := Seq(
ShadeRule.rename("com.typesafe.config.**" -> "my_conf.#1")
.inLibrary("com.typesafe" % "config" % "1.3.0")
.inProject )
According to the documentation, this should place any class under
com.typesafe.config under the new package my_conf.
For your case, the solution would be adding something like this to your build.sbt file:
assemblyShadeRules in assembly := Seq(
ShadeRule.rename("com.somecompany.**" -> "my_conf.#1")
.inLibrary("com.somecompany" % "libraryZ" % "0.11")
.inProject
)
In sbt, conflicts between libraries are configured using the conflict manager. By default, the latest revision is selected but this can also be overridden in you .sbt file:
conflictManager := ConflictManager.strict
If you're using sbt 0.13.6 or greater you will be warned when you have an incompatible binary version between your dependencies. In this situation, you could configure an override in your sbt file for the specific library:
dependencyOverrides += "org.raman" % "Z" % "x.11"
This will force the resolved version of Z to x.11 but not pull a direct dependency in.
Related
Let's say I am creating an SBT project called Y which is using the package X in an SBT build:
libraryDependencies ++= Seq(
"org.something" %% "X" % XVersion
)
I've learned that X contains a minor bug, which the developer is working on (thank you!). The code of X (built with SBT) is available to clone and I know how to hot-fix this issue while the developer works on it.
Here's what I would love to be able to do:
Clone X to ~/code/X
Create the hot-fix by just editing the code
In the build.sbt for my own project Y, replace "org.something" %% "X" % XVersion with a link to ~/code/X so that my own code is compiled with the hot-fix.
Is there a general way to do this? There's a good chance this is described in the SBT Docs, I'd be happy for a pointer - there's a lot of docs with new terminology.
Example in point (but the question is about the general case): This build file for a package I'm trying to hot-fix is complex, and I'd love to have a way of hot-fixing without requiring to understand this build file fully.
You can edit the dependency code and then run sbt publishLocal which will create a new local version of the dependency with the changes applied, you can then edit your build.sbt to use that new version and that is all you need for a local setup
If you have some CI pipeline or you deeply your code you would need to do more things but this is the basic idea.
I have created basic Scala Play application with https://www.playframework.com/getting-started play-scala-seed. This project compiles and runs with sbt run. But I have another Scala project that compiles and runs and which I have submitted to my local Ivy repository with command sbt publishLocal. This other project was saved at C:\Users\tomr\.ivy2\local\com.agiintelligence\scala-isabelle_2.13\master-SNAPSHOT as a result of this command.
Then I imported (exactly so - imported, no just opened) my Play project in IntelliJ and I used Project - Open Module Settings - Project Settings - Libraries to add com.agiintelligence jar from my ivy2 location. After such operations IntelliJ editor recognizes com.agiintelligence classes. That is fine.
But when I am trying to run my Play application with sbt run, I experience the error message not found: object com that is exactly when compiling import com.agiintelligence line in my Scala controller file of Play application.
Of course - such error has been reported and resolved with, e.g. object play not found in scala application
But that solution suggests to append build.sbt file. My build.sbt file is pretty bare:
name := """agiintelligence"""
organization := "com.agiintelligence"
version := "1.0-SNAPSHOT"
lazy val root = (project in file(".")).enablePlugins(PlayScala)
scalaVersion := "2.13.5"
libraryDependencies += guice
libraryDependencies += "org.scalatestplus.play" %% "scalatestplus-play" % "5.0.0" % Test
// Adds additional packages into Twirl
//TwirlKeys.templateImports += "com.skaraintelligence.controllers._"
// Adds additional packages into conf/routes
// play.sbt.routes.RoutesKeys.routesImport += "com.skaraintelligence.binders._"
My Play application contains (as can bee seen from the IntelliJ project pane) some tens of 'external libraries' (it shows my com.agiintelligence jar as well), but why should I add my own ivy2 library in build.sbt file if no other libraries are listed here? What is different with my library? It is on my computer, in the repository as expected already?
Of course, I can try to add it build.sbt and issue sbt update and see what happens, but I can not understand this logic? Can someone explain it and provide some clue to intelligible solution of my error message?
My Play application contains (as can bee seen from the IntelliJ project pane) some tens of 'external libraries'
Those are probably just transitive dependencies of your Play dependency, that is why sbt downloaded all of them and put them in your classpath so you could use them without you needing to tell it about them; because the pom of Play already did.
It is not that the build tool or the IDE magically added all those dependencies for you because they read your mind and magically understood you wanted them. And that for some reason the magic stopped working for your own library.
Why it is not sufficient to list it Project-Setting--External Libraries in IntelliJ only?
That is sufficient for the IDE to work, but not for the build tool. The build tool is independent of the IDE; it doesn't know about it. sbt just knows about the dependencies you configured in your definition file.
Even more, you should always configure your dependencies on your build tool and then import that in the IDE; rather than the opposite. IDEs are graphical tools, so their state can not be committed, can not be shared, can not keep track of changes, can not be used in CI / CD environments; additionally, different teammates may want to use different IDEs.
I resolved the error message by adding line in build.sbt file
libraryDependencies += "de.unruh" %% "scala-isabelle" % "master-SNAPSHOT"
and by subsequent run of sbt update.
Error is solved, but the main question still stand - why I had to do this? Why there are tens of dependencies that are not listed in build.sbt and why should I list my dependency in build.sbt and why it is not sufficient to list it Project-Setting--External Libraries in IntelliJ only?
OK, comment by #Luis_Miguel_Mejía_Suárez gave the explanation, that comment is the actual and expected answer to my question.
I am following a tutorial to perform object detection in scala. I am
having issues adding the tensorFlow dependency. I have followed the instructions on the official Tensorflow for Scala website http://platanios.org/tensorflow_scala/installation.html, but that doesn't seem to work. I also made sure to use the Java 11 JDK for the project. However, whenever I try to add the sbt dependency
libraryDependencies += "org.platanios" % "tensorflow" % "0.4.0" classifier "linux-cpu-x86_64", I get a "No dependencies found for given import" error in IntelliJ. Any idea on how to set this up properly ?
Try to replace one % in your dependency line to twice %%:
libraryDependencies += "org.platanios" %% "tensorflow" % "0.4.0" classifier "linux-cpu-x86_64"
On top of what the previous answer already suggested, I believe it's probably worth mentioning that (until 2.12) libraries in the 2.x are not binary-compatible across versions. The convention for Scala libraries is to append a _2.x to the published library JAR's artifact identifier. Since SBT was built around Scala (and it's its de facto standard build tool) it acknowledges this conventions and the %% operator will automatically append that extra "qualifier" based on the Scala version you are using.
Notice here on mvnrepository.com how the artifact identifier changes between the Maven and the SBT dependency declaration (in Maven, the artifact identifier is tensorflow_2.12, in SBT the %% allows you to not have to specify that).
The single % is generally used for Java dependencies (that are not affected by the aforementioned convention).
As an alternative (that I would suggest just to play around and see that there's no magic involved), you can also use % to specify a Scala dependency and explicitly mention the Scala version in the artifact identifier, as follows:
libraryDependencies += "org.platanios" % "tensorflow_2.12" % "0.4.0" classifier "linux-cpu-x86_64"
The good news is that starting from Scala 2.13 this issue was tackled at the very root using an intermediate representation that was also introduced to make sure the interoperability between Scala 2.13 and Scala 3.x compiled code.
EDIT
What you have found was actually an issue in the documentation that was already reported, I opened a PR to fix it.
So I'm having trouble importing a package in scala. I downloaded the package, breeze, from github, because I wanted to sample from probability distributions.
I'm used to Python, where I can just download a package, include it in the path, and then import it in the code. So I'm very new to the idea of using a separate "build tool" to use 3rd party packages.
So I downloaded the "breeze" source code from github, installed sbt, and then within the source code for breeze, I ran sbt, and then I used the "assembly" command to get a .jar for breeze.
If I want to use the scala interpreter, I can import the package just fine with
scala -cp breeze-master/target/scala-2.11/breeze-parent-assembly-0.8.jar
The problem is that I want to use this package in a separate piece of code that I'm writing in a file called Chromosome.scala. And when I try to import the package (as seen below), I get an error:
error: not found: object breeze
Here's my code:
// Chromosome.scala
import breeze.stats.distributions._
class Chromosome(s:Int, bitstring:Array[Boolean]) {
val size:Int = s;
val dna:Array[Boolean] = bitstring;
var fitness:Int = 0;
def mutate(prob:Float):Unit = {
// This method will randomly "mutate" the dna sequence by flipping a bit.
// Any individual bit may be flipped with probability 'pm', usually small.
val pm:Float = prob;
// The variable bern is an instance of a Bernoulli random variable,
// whose probability parameter is equal to 'pm'.
var bern = new Bernoulli(pm);
//Loop through the 'dna' array and flip each bit with probability pm.
for (i <- 0 to (size - 1)) {
var flip = bern.draw();
if (flip) {
dna(i) = !(dna(i));
}
}
}
“A script?” What is this and what is its connection to your SBT project? Scala scripts include their own launch command for the Scala interpreter / compiler ( / REPL…). If you want to access things beyond the standard library, you'll have to inclulde them there. Alternately, you can use the SBT Start Script plug-in to produce a launcher script that will include the project dependencies. It will only work locally, though you can write some text processing and other shell scripting to produce a portable launch bundle.
It looks like there's some understandable confusion about what sbt is supposed to do for you.
First off, you generally don't need to download a package from github and build it from source. In the rare cases that you do (such as when you require features that have not made it into a release of the library), sbt can handle the grunt work.
Instead, you tell sbt a little about the project you're building (including what its dependencies are), and sbt will download them, compile your code, and set up the runtime classpath for the scala interpreter (amongst myriad other build-related tasks).
Just follow the directions on the breeze wiki. Specifically, create a build.sbt file in your project's root folder and copy this into it:
libraryDependencies ++= Seq(
// other dependencies here
"org.scalanlp" % "breeze_2.10" % "0.7",
// native libraries are not included by default. add this if you want them (as of 0.7)
// native libraries greatly improve performance, but increase jar sizes.
"org.scalanlp" % "breeze-natives_2.10" % "0.7",
)
resolvers ++= Seq(
// other resolvers here
// if you want to use snapshot builds (currently 0.8-SNAPSHOT), use this.
"Sonatype Snapshots" at "https://oss.sonatype.org/content/repositories/snapshots/",
"Sonatype Releases" at "https://oss.sonatype.org/content/repositories/releases/"
)
// Scala 2.9.2 is still supported for 0.2.1, but is dropped afterwards.
// Don't use an earlier version of 2.10, you will probably get weird compiler crashes.
scalaVersion := "2.10.3"
Put your source into the appropriate folder (by default, src/main/scala) and run sbt console. This command will download the dependencies, compile your code, and launch the Scala interpreter. At this point you should be able to interact with your class.
how to use JavaCV from SBT (Simple Build Tool) in Scala? I need to use JavaCV so I can write a Scala application using this.
Simple Build Tool is here: http://www.scala-sbt.org/
JavaCV is here: http://code.google.com/p/javacv/
Add the following to your build.sbt file.
resolvers += "JavaCV maven repo" at "http://maven2.javacv.googlecode.com/git/"
libraryDepedencies += "com.googlecode.javacv" % "javacv" % "0.2"
This should pull down javacv for you, as well as any dependencies. Note you will still need to have all the native libraries set up like javacv details, this only gets the jars for your project.
I created an SBT plugin that solves most of this for you in 1 line; no need to set up native libraries etc because this will do it all for you.
https://github.com/lloydmeta/sbt-opencv
Usage:
Add the following in project/plugins.sbt:
addSbtPlugin("com.beachape" % "sbt-opencv" % "1.2")