The code I'm writing will be run in AWS Lambda which only has the Java 8 runtime installed so I need the scala libraries to be included in my jar. When I give it the jar I built with sbt-assembly I'm getting java.lang.NoClassDefFoundError: scala/Function3.
This is all I have in build.sbt for the assembly plugin:
assemblyOption in assembly := (assemblyOption in assembly).value.copy(includeScala = true)
assemblyMergeStrategy in assembly := {
case PathList("META-INF", _ # _*) => MergeStrategy.discard
case _ => MergeStrategy.first
}
This still happens whether or not I have the line that discards the META-INF files.
I used brew to install scala and I have tried setting my $SCALA_HOME to /usr/local/opt/scala/idea (caveats section of brew info scala) and /usr/local/bin/scala (output of which scala)
:: EDIT ::
I unpacked the jar and found that the class in question was actually included in the jar here: scala/Function3_scala-library-2.12.7_scala-library-2.12.7_scala-library-2.12.7.class
I added scala-library.jar to my classpath in Intellij, removed the target folder and ran sbt assembly and that worked.
Related
I am using Databricks cluster to execute spark application..
My application having some dependency on few libraries but now these libraries are not available via the Databricks install new library option.
I came to know the through the Fat jar or Uber jar I can add multiple libraries and pass this to the cluster.
I also came to know that to create a fat jar you have to provide a main class so I have written a simple program in my local system and added the dependencies to the build.sbt file.
I am using the 'sbt assembly' command to create fat jar.
Please note that I am not using the library in my sample program.
My aim is to create a fat jar that inherits all the required jar in it so that my other Spark based application can access the libraries via this fat jar..
I did the following steps.
'Sample Program'
def main(args: Array[String]): Unit = {
print("Hello World")
}
}
'Build.sbt'
name := "CrealyticsFatJar"
version := "0.1"
scalaVersion := "2.11.12"
// https://mvnrepository.com/artifact/com.crealytics/spark-excel
libraryDependencies += "com.crealytics" %% "spark-excel" % "0.12.0"
assemblyMergeStrategy in assembly := {
case PathList("META-INF", xs # _*) => MergeStrategy.discard
case x => MergeStrategy.first
}
'assembly.sbt'
addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.14.6")
But I am not sure whatever I am doing is correct and will help to execute my spark programs in Databricks cluster.
Q1) There might be some possibility that one library having dependency on the other library so If I have mention the library name in SBT then would it load other dependent libraries?
Q2) If I am not using the libraries to the existing program would it be available to the other program of the cluster.
Q3) After the installation of Fat jar in cluster- how do I access the libraries.. I mean by which name I would access the libraries..the import command...
Apologies if my questions are so silly.
Thanks
I'd like to figure out how to excluded the Python files from the JAR file generated by the sbt package command.
The Delta lake project uses SBT version 0.13.18 and the JAR file is built with sbt package.
The directory structure is as follows:
python/
delta/
testing/
utils.py
tests/
test_sql.py
src/
main/
...scala files
build.sbt
run-tests.py
It follows the standard SBT project structure with a couple of Python files added in.
The Python files are included in the JAR file when sbt package is run and I'd like to figure out how to exclude all the Python files.
Here's what I tried adding to the build.sbt file:
mappings in (Compile, packageBin) ~= { _.filter(!_._1.getName.endsWith("py")) } per this answer
excludeFilter in Compile := "*.py" per this answer
Neither of these worked.
Haven't tested it, but I think something like this when you make a fat jar.
assemblyMergeStrategry in assembly := {
case PathList(parts # _*) if parts.last.endsWith(".py") => MergeStrategy.discard
case _ => MergeStrategy.first // or whatever you currently have for your other merges
}
This question already has answers here:
How to add "provided" dependencies back to run/test tasks' classpath?
(4 answers)
Closed 3 years ago.
I have a Spark project which I usually package with sbt-assembly. All spark dependencies are marked provided and not included in my fat jar.
I want to have another command to build a really fat jar with all dependencies, spark included.
I am trying the following with no luck:
lazy val standalone = project
.dependsOn(mainProj % "compile->compile;test->test;provided->compile")
.settings(
logLevel in assembly := Level.Debug,
assemblyOption in assembly := (assemblyOption in assembly).value.copy(includeScala = true, includeDependency = true)
)
Note, the answers to How to add "provided" dependencies back to run/test tasks' classpath? explain how to add provided dependencies to the runtime classpath, however my question is about how to have them end up in the packaged artifact after executing sbt assembly.
To build a truly fat jar which packages everything including provided dependencies, we could redefine fullClasspath in assembly like so
assembly / fullClasspath := (Compile / fullClasspath).value
If we put this in a separate command like so
commands += Command.command("assemblyTrulyFatJar") { state =>
"""set assembly / fullClasspath := (Compile / fullClasspath).value""" :: "assembly" :: state
}
then executing sbt assemblyTrulyFatJar should package everything, while sbt assembly keeps its default behaviour.
where i use sbt assembly my scala project, i met a error like that: deduplicate: different file contents found in the following: and below is the picture:
and my build.sbt:
sbt version: 0.13.15
scala version: 2.8.11
jdk: 1.8
You need to set the assembly merge strategy to either take one of the files, concat them or remove them completely:
assemblyMergeStrategy in assembly := {
case PathList("org", "springframework", xs#_*) => MergeStrategy.last
case x => MergeStrategy.defaultMergeStrategy(x)
}
I am using jdk 8(tried in 11 also) / sbt 1.4.7 / scala 2.13.3 and the following 2 plugins in my rootprojectdir/project/plugins.sbt
addSbtPlugin("com.typesafe.sbt" % "sbt-native-packager" % "1.8.0")
addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.15.0")
assemblyMergeStrategy in assembly := {
case PathList(ps # _*) if ps.contains("module-info.class") => MergeStrategy.discard
case x => MergeStrategy.defaultMergeStrategy(x)
}
Also I have added the above strategy in my build.sbt, but still getting the same error. The sbt-assembly plugin ticket is still open in github.
Any workaround will be much appreciated
I need to package my test classes, resources and also test dependencies with sbt assembly.
This question sbt-assembly : including test classes didn't help - test:assembly still didn't generate a jar with any of the desired classes included.
Note that my setup currently looks like this:
FooBuild.scala:
lazy val cucumberAssemblySettings = assemblySettings ++ Seq(
mainClass in assembly := Some("cucumber.api.cli.Main"),
jarName in assembly := "cucumber.jar",
mergeStrategy in assembly := {
case "application.conf" => MergeStrategy.concat
case "logback.xml" => MergeStrategy.last
case x => defaultMergeStrategy(x)
}
)
And it's about the subproject foo-cucumber:
lazy val foo_cucumber = Project("foo-cucumber", file("foo-cucumber"))
.settings(defaultSettings: _*)
.settings(cucumberAssemblySettings: _*)
.dependsOn(
foo_test_server % "compile->compile;test->test",
foo_test_utils % "compile->compile;test->test"
)
even if I append (Test, assembly) in the settings above, I only get a jar (whose name is not the one specified, but the full name, with version) that does not contain test classes or dependencies, no matter whether I invoke sbt foo-cucumber/test:assembly or sbt foo-cucumber/assembly
How do I get a jar that has everything in it (compile and test classes and dependencies)
The key for the multi-module project is to set the settings this way:
.settings(inConfig(Test)(cucumberAssemblySettings): _*)
and then not have any in (Test, assembly) in the settings