How to rename and move an uberjar generated with SBT assembly plugin?
addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.14.6")
My assemblyMergeStrategy(for META-INF removal):
assemblyMergeStrategy in assembly := {
case PathList("META-INF", xs # _*) => MergeStrategy.discard
case x => MergeStrategy.first
}
It generates something like :
target/scala-2.12/my-project-assembly-0.1.jar
which I would like to able to automatically rename (and generate in another directory) with a consistent name (without the need of a separate script).
You can find a bit of documentation in project's page. There, you can find the keys you can rewrite for the assembly task.
The ones you are searching for are assemblyJarName and assemblyOutputPath. Then, your project build should look something like:
lazy val myProject = (project in file(".")).
settings(
...
assemblyJarName in assembly := "myName.jar",
assemblyOutputPath in assembly := "...",
...
)
Related
where i use sbt assembly my scala project, i met a error like that: deduplicate: different file contents found in the following: and below is the picture:
and my build.sbt:
sbt version: 0.13.15
scala version: 2.8.11
jdk: 1.8
You need to set the assembly merge strategy to either take one of the files, concat them or remove them completely:
assemblyMergeStrategy in assembly := {
case PathList("org", "springframework", xs#_*) => MergeStrategy.last
case x => MergeStrategy.defaultMergeStrategy(x)
}
I am using jdk 8(tried in 11 also) / sbt 1.4.7 / scala 2.13.3 and the following 2 plugins in my rootprojectdir/project/plugins.sbt
addSbtPlugin("com.typesafe.sbt" % "sbt-native-packager" % "1.8.0")
addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.15.0")
assemblyMergeStrategy in assembly := {
case PathList(ps # _*) if ps.contains("module-info.class") => MergeStrategy.discard
case x => MergeStrategy.defaultMergeStrategy(x)
}
Also I have added the above strategy in my build.sbt, but still getting the same error. The sbt-assembly plugin ticket is still open in github.
Any workaround will be much appreciated
I am using sbtassembly from https://github.com/sbt/sbt-assembly to package my project.
I'm wondering is there anyway to exclude the resource files?
You can specify files (and paths) to exclude by customizing the mergeStrategy:
https://github.com/sbt/sbt-assembly#excluding-specific-files
So for discarding specific file you can do something like this:
// build.sbt
assemblyMergeStrategy in assembly := {
case PathList("about.html") => MergeStrategy.discard
case x =>
val oldStrategy = (assemblyMergeStrategy in assembly).value
oldStrategy(x)
}
Here's the documentaion for all available strategies:
https://github.com/sbt/sbt-assembly#merge-strategy
Using Dani's approach with sbt 0.13.13, the config files were still included in my jar. This worked, though:
excludeFilter in Compile := "myconfig.conf",
In my case, all the files have the same name, myconfig.conf, but exist within a tree structure under src/main/resources/config. I tried:
unmanagedResourceDirectories in Compile += { baseDirectory.value / "src/main/resources/config" },
But it removed the directories from the jar, leaving the files.
It's documented here: http://www.scala-sbt.org/0.13/docs/Howto-Customizing-Paths.html
I need a Fat Jar with Spark because I'm creating a custom node for Knime. Basically it's a self-contained jar executed inside Knime and I assume a Fat Jar is the only way to spawn a local Spark Job. Eventually we will go on submitting a job to a remote cluster but for now I need it to spawn this way.
That said, I made a Fat Jar using this: https://github.com/sbt/sbt-assembly
I made an empty sbt project, included Spark-core in the dependencies and assembled the Jar. I added it to the manifest of my custom Knime node and tried to spawn a simple job (pararellize a collection, collect it and print it). It starts but I get this error:
No configuration setting found for key 'akka.version'
I have no idea how to solve it.
Edit: this is my build.sbt
name := "SparkFatJar"
version := "1.0"
scalaVersion := "2.11.6"
libraryDependencies ++= Seq(
"org.apache.spark" %% "spark-core" % "1.3.0"
)
libraryDependencies += "com.typesafe.akka" %% "akka-actor" % "2.3.8"
assemblyJarName in assembly := "SparkFatJar.jar"
assemblyMergeStrategy in assembly := {
case PathList("META-INF", xs # _*) => MergeStrategy.discard
case x => MergeStrategy.first
}
I've found this mergestrategy for Spark somewhere on the internet but I can't find the source right now.
I think the issue is with how you've setup assemblyMergeStrategy. Try this:
assemblyMergeStrategy in assembly := {
case PathList("META-INF", xs # _*) => MergeStrategy.discard
case "application.conf" => MergeStrategy.concat
case "reference.conf" => MergeStrategy.concat
case x =>
val baseStrategy = (assemblyMergeStrategy in assembly).value
baseStrategy(x)
}
I'm using SBT (within IntelliJ IDEA) to build a simple Scala project.
I would like to know what is the simplest way to build an Uber JAR file (aka Fat JAR, Super JAR).
I'm currently using SBT but when I'm submiting my JAR file to Apache Spark I get the following error:
Exception in thread "main" java.lang.SecurityException: Invalid
signature file digest for Manifest main attributes
Or this error during compilation time:
java.lang.RuntimeException: deduplicate: different file contents found
in the following:
PATH\DEPENDENCY.jar:META-INF/DEPENDENCIES
PATH\DEPENDENCY.jar:META-INF/MANIFEST.MF
It looks like it is because some of my dependencies include signature files (META-INF) which needs to be removed in the final Uber JAR file.
I tried to use the sbt-assembly plugin like that:
/project/assembly.sbt
addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.12.0")
/project/plugins.sbt
logLevel := Level.Warn
/build.sbt
lazy val commonSettings = Seq(
name := "Spark-Test"
version := "1.0"
scalaVersion := "2.11.4"
)
lazy val app = (project in file("app")).
settings(commonSettings: _*).
settings(
libraryDependencies ++= Seq(
"org.apache.spark" %% "spark-core" % "1.2.0",
"org.apache.spark" %% "spark-streaming" % "1.2.0",
"org.apache.spark" % "spark-streaming-twitter_2.10" % "1.2.0"
)
)
When I click "Build Artifact..." in IntelliJ IDEA I get a JAR file. But I end up with the same error...
I'm new to SBT and not very experimented with IntelliJ IDE.
Thanks.
Finally I totally skip using IntelliJ IDEA to avoid generating noise in my global understanding :)
I started reading the official SBT tutorial.
I created my project with the following file structure :
my-project/project/assembly.sbt
my-project/src/main/scala/myPackage/MyMainObject.scala
my-project/build.sbt
Added the sbt-assembly plugin in my assembly.sbt file. Allowing me to build a fat JAR :
addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.12.0")
My minimal build.sbt looks like :
lazy val root = (project in file(".")).
settings(
name := "my-project",
version := "1.0",
scalaVersion := "2.11.4",
mainClass in Compile := Some("myPackage.MyMainObject")
)
val sparkVersion = "1.2.0"
libraryDependencies ++= Seq(
"org.apache.spark" %% "spark-core" % sparkVersion % "provided",
"org.apache.spark" %% "spark-streaming" % sparkVersion % "provided",
"org.apache.spark" %% "spark-streaming-twitter" % sparkVersion
)
// META-INF discarding
mergeStrategy in assembly <<= (mergeStrategy in assembly) { (old) =>
{
case PathList("META-INF", xs # _*) => MergeStrategy.discard
case x => MergeStrategy.first
}
}
Note: The % "provided" means not to include the dependency in the final fat JAR (those libraries are already included in my workers)
Note: META-INF discarding inspired by this answser.
Note: Meaning of % and %%
Now I can build my fat JAR using SBT (how to install it) by running the following command in my /my-project root folder:
sbt assembly
My fat JAR is now located in the new generated /target folder :
/my-project/target/scala-2.11/my-project-assembly-1.0.jar
For those who wants to embeed SBT within IntelliJ IDE: How to run sbt-assembly tasks from within IntelliJ IDEA?
3 Step Process For Building Uber JAR/Fat JAR in IntelliJ Idea:
Uber JAR/Fat JAR : JAR file having all external libraray dependencies in it.
Adding SBT Assembly plugin in IntelliJ Idea
Go to ProjectName/project/target/plugins.sbt file and add this line addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.12.0")
Adding Merge,Discard and Do Not Add strategy in build.sbt
Go to ProjectName/build.sbt file and add the Strategy for Packaging of an Uber JAR
Merge Strategy : If there is conflict in two packages about a version of library then which one to pack in Uber JAR.
Discard Strategy : To remove some files from library which you do not want to package in Uber JAR.
Do not Add Strategy : Do not add some package to Uber JAR.For ex: spark-core will be already present at your Spark Cluster.So we should not package this in Uber JAR
Merge Strategy and Discard Strategy Basic Code :
assemblyMergeStrategy in assembly := {
case PathList("META-INF", xs # _*) => MergeStrategy.discard
case x => MergeStrategy.first
}
So you are asking to discard META-INF files using this command MergeStrategy.discard and for rest of the files you are taking the first occurrence of library file if there is any conflict by using this command MergeStrategy.first.
Do not Add Strategy Basic Code :
libraryDependencies += "org.apache.spark" %% "spark-core" % "1.4.1" %"provided"
If we do not want to add the spark-core to our Uber JAR file as it will be already on our clutser, so we are adding the % "provided" at end of it library dependency.
Building Uber JAR with all its dependencies
In terminal type sbt assembly for building up the package
Voila!!! Uber JAR is built. JAR will be in ProjectName/target/scala-XX
Add the following line to your project/plugins.sbt
addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.12.0")
Add the following to your build.sbt
mainClass in assembly := some("package.MainClass")
assemblyJarName := "desired_jar_name_after_assembly.jar"
val meta = """META.INF(.)*""".r
assemblyMergeStrategy in assembly := {
case PathList("javax", "servlet", xs # _*) => MergeStrategy.first
case PathList(ps # _*) if ps.last endsWith ".html" => MergeStrategy.first
case n if n.startsWith("reference.conf") => MergeStrategy.concat
case n if n.endsWith(".conf") => MergeStrategy.concat
case meta(_) => MergeStrategy.discard
case x => MergeStrategy.first
}
The Assembly merge strategy is used to resolve conflicts occurred when creating fat jar.
I need to package my test classes, resources and also test dependencies with sbt assembly.
This question sbt-assembly : including test classes didn't help - test:assembly still didn't generate a jar with any of the desired classes included.
Note that my setup currently looks like this:
FooBuild.scala:
lazy val cucumberAssemblySettings = assemblySettings ++ Seq(
mainClass in assembly := Some("cucumber.api.cli.Main"),
jarName in assembly := "cucumber.jar",
mergeStrategy in assembly := {
case "application.conf" => MergeStrategy.concat
case "logback.xml" => MergeStrategy.last
case x => defaultMergeStrategy(x)
}
)
And it's about the subproject foo-cucumber:
lazy val foo_cucumber = Project("foo-cucumber", file("foo-cucumber"))
.settings(defaultSettings: _*)
.settings(cucumberAssemblySettings: _*)
.dependsOn(
foo_test_server % "compile->compile;test->test",
foo_test_utils % "compile->compile;test->test"
)
even if I append (Test, assembly) in the settings above, I only get a jar (whose name is not the one specified, but the full name, with version) that does not contain test classes or dependencies, no matter whether I invoke sbt foo-cucumber/test:assembly or sbt foo-cucumber/assembly
How do I get a jar that has everything in it (compile and test classes and dependencies)
The key for the multi-module project is to set the settings this way:
.settings(inConfig(Test)(cucumberAssemblySettings): _*)
and then not have any in (Test, assembly) in the settings