Cannot run jar file created from Scala file - scala

This the code that I have written in Scala.
object Main extends App {
println("Hello World from Scala!")
}
This is my build.sbt.
name := "hello-world"
version := "1.0"
scalaVersion := "2.11.5"
mainClass := Some("Main")
This is the command that I have run to create the jar file.
sbt package
My problem is that a jar file named hello-world_2.11-1.0.jar has been created at target/scala-2.11. But I cannot run the file. It is giving me an error saying NoClassDefFoundError.
What am I doing wrong?

It also says what class is not found. Most likely you aren't including scala-library.jar. You can run scala target/scala-2.11/hello-world_2.11-1.0.jar if you have Scala 2.11 available from the command line or java -cp "<path to scala-library.jar>:target/scala-2.11/hello-world_2.11-1.0.jar" Main (use ; instead of : on Windows).

The procedure depicted proves valid up to the way the jar file is executed. From target/scala-2.11 try running it with
scala hello-world_2.11-1.0.jar
Check whether it is runnable also from the project root folder with sbt run.

To run the jar file(containing scala code) with multiple main classes use following approach
scala -cp "<jar-file>.jar;<other-dependencies>.jar" com.xyz.abc.TestApp
This command will take care of including scala-library.jar in dependency and will also identify TestApp as main class if it has a def main(args:Array[String]) method. Please note that multiple jar files should be separated by semi-colon(";")

We can use sbt-assembly to package and run the application.
First, create or add the plugin to project/plugins.sbt
addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.14.9")
The sample build.sbt looks like below:
name := "coursera"
version := "0.1"
scalaVersion := "2.12.10"
mainClass := Some("Main")
val sparkVersion = "3.0.0-preview2"
val playVersion="2.8.1"
val jacksonVersion="2.10.1"
libraryDependencies ++= Seq(
"org.scala-lang" % "scala-library" % scalaVersion.toString(),
"org.apache.spark" %% "spark-streaming" % sparkVersion,
"org.apache.spark" %% "spark-core" % sparkVersion,
"org.apache.spark" %% "spark-sql" % sparkVersion,
"com.typesafe.play" %% "play-json" % playVersion,
// https://mvnrepository.com/artifact/org.apache.spark/spark-streaming-kafka-0-10
"org.apache.spark" %% "spark-streaming-kafka-0-10" % sparkVersion,
// https://mvnrepository.com/artifact/org.mongodb/casbah
"org.mongodb" %% "casbah" % "3.1.1" pomOnly(),
// https://mvnrepository.com/artifact/com.typesafe/config
"com.typesafe" % "config" % "1.2.1"
)
assemblyMergeStrategy in assembly := {
case PathList("META-INF", xs # _*) => MergeStrategy.discard
case x => MergeStrategy.first
}
From console, we can run sbt assembly and the jar file gets created in target/scala-2.12/ path.
sbt assembly will create a fat jar. Here is an excerpt from the documentation :
sbt-assembly is a sbt plugin originally ported from codahale's assembly-sbt, which I'm guessing was inspired by Maven's assembly plugin. The goal is simple: Create a fat JAR of your project with all of its dependencies.

Related

Exception in thread "main" java.lang.NoClassDefFoundError: org/rogach/scallop/ScallopConf

As I want to execute the jar generated by my scala project in the Command Line Interface, I get the following problem:
Exception in thread "main" java.lang.NoClassDefFoundError: org/rogach/scallop/ScallopConf
Although in the dependency file I mentionned scallop dependency as follow
import sbt._
object Dependencies {
lazy val betterFiles = "com.github.pathikrit" %% "better-files" % "3.7.0"
lazy val scalaz = "org.scalaz" %% "scalaz-core" % "7.2.27"
lazy val scallop = "org.rogach" %% "scallop" % "3.1.5"
// -- Logging
lazy val scalaLogging = "com.typesafe.scala-logging" %% "scala-logging" % "3.9.2"
lazy val slf4jBackend = "org.slf4j" % "slf4j-simple" % "1.7.26"
// -- Testing
lazy val scalaTest = "org.scalatest" %% "scalatest" % "3.0.5"
}
My build.sbt file is the following:
import Dependencies._
ThisBuild / scalaVersion := "2.12.5"
ThisBuild / sbtVersion := "1.2.6"
ThisBuild / version := "0.1.0-SNAPSHOT"
lazy val root = (project in file("."))
.settings(
name := "phenix-challenge",
libraryDependencies ++= Seq(
betterFiles,
scalaz,
scallop,
scalaLogging,
slf4jBackend % Runtime,
scalaTest % Test
)
)
If you have an Idea that could resolve my Issue please HELP!
Many thanks in advance
To execute the jar generated by your scala project in the Command Line Interface you can use sbt plugin to assembly a fat-jar including your libraries/dependencies. Having such jar you would be able to run your app via java -jar ...
There are several SBT plugins for build a fat-jar. Perhaps the easiest one would be the sbt-assembly.
Add this plugin to file project/plugins.sbt (create this file if needed):
addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.14.9")
Now use sbt to build a fat-jar:
sbt assembly
Then run via java -jar YouMainClass
Another option is to use pure sbt to run Main class using command sbt run, then you do really need to build a fat-jar.

Building Scala JAR file Spark

My build.sbt file has (Im using IntelliJ)
scalaVersion := "2.11.8"
resolvers += "MavenRepository" at "http://central.maven.org/maven2"
resolvers += "spark-packages" at "https://dl.bintray.com/spark-packages/maven/"
libraryDependencies ++= {
val sparkVersion = "2.2.1"
Seq( "org.apache.spark" %% "spark-core" % sparkVersion )
}
Im trying to build a JAR and deploy it into Spark. I have issued following commands
sbt compile
sbt assembly
Compilation was successful but assembly failed with following error message
java.lang.RuntimeException: Please add any Spark dependencies by supplying the sparkVersion and sparkComponents. Please remove: org.apache.spark:spark-core:2.2.1
I tried to add "provided" to keep it out that time compilation itself fails as "provided" key word
doesnt include those JARs
What is the mistake am doing?
You first need to add plugin and dependencies for assembly which will create jar for you.
In plugins.sbt
addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.14.5")
Add this in your build.sbt
mainClass := Some("name of jar")
assemblyMergeStrategy in assembly := {
case PathList("META-INF", xs # _*) => MergeStrategy.discard
case x => MergeStrategy.first
}
You can refer my github to create jar and deploy

Spark with IntelliJ or Eclipse

I am trying to setup IntelliJ for spark 2.11 but it is very daunting and after days I have not been able to compile a simple instruction such as with "spark.read.format" which is not found in main core and sql spark libraries.
I have seen a few posts on the subject but with none resolved. Does anyone have some experience with perhaps a working sample program I can start with?
Could it be that it would be easier with Eclipse?
Many thanks in advance for your answers,
EZ
build project in Intellij using with scala 2.11 and sbt 0.13: then ensure that your plugins.sbt contains as below:
logLevel := Level.Warn
addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.13.0")
then your build.sbt must contain as below:
scalaVersion := "2.11.8"
val sparkVersion = "2.1.0"
libraryDependencies += "org.apache.spark" %% "spark-core" % sparkVersion %"provided"
libraryDependencies += "org.apache.spark" %% "spark-sql" % sparkVersion %"provided"
Then write your code, click Terminal in Intellij and type sbt assembly: you can ship that jar to remote cluster, otherwise run from Intelij locally, let me know how it goes

How to build an Uber JAR (Fat JAR) using SBT within IntelliJ IDEA?

I'm using SBT (within IntelliJ IDEA) to build a simple Scala project.
I would like to know what is the simplest way to build an Uber JAR file (aka Fat JAR, Super JAR).
I'm currently using SBT but when I'm submiting my JAR file to Apache Spark I get the following error:
Exception in thread "main" java.lang.SecurityException: Invalid
signature file digest for Manifest main attributes
Or this error during compilation time:
java.lang.RuntimeException: deduplicate: different file contents found
in the following:
PATH\DEPENDENCY.jar:META-INF/DEPENDENCIES
PATH\DEPENDENCY.jar:META-INF/MANIFEST.MF
It looks like it is because some of my dependencies include signature files (META-INF) which needs to be removed in the final Uber JAR file.
I tried to use the sbt-assembly plugin like that:
/project/assembly.sbt
addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.12.0")
/project/plugins.sbt
logLevel := Level.Warn
/build.sbt
lazy val commonSettings = Seq(
name := "Spark-Test"
version := "1.0"
scalaVersion := "2.11.4"
)
lazy val app = (project in file("app")).
settings(commonSettings: _*).
settings(
libraryDependencies ++= Seq(
"org.apache.spark" %% "spark-core" % "1.2.0",
"org.apache.spark" %% "spark-streaming" % "1.2.0",
"org.apache.spark" % "spark-streaming-twitter_2.10" % "1.2.0"
)
)
When I click "Build Artifact..." in IntelliJ IDEA I get a JAR file. But I end up with the same error...
I'm new to SBT and not very experimented with IntelliJ IDE.
Thanks.
Finally I totally skip using IntelliJ IDEA to avoid generating noise in my global understanding :)
I started reading the official SBT tutorial.
I created my project with the following file structure :
my-project/project/assembly.sbt
my-project/src/main/scala/myPackage/MyMainObject.scala
my-project/build.sbt
Added the sbt-assembly plugin in my assembly.sbt file. Allowing me to build a fat JAR :
addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.12.0")
My minimal build.sbt looks like :
lazy val root = (project in file(".")).
settings(
name := "my-project",
version := "1.0",
scalaVersion := "2.11.4",
mainClass in Compile := Some("myPackage.MyMainObject")
)
val sparkVersion = "1.2.0"
libraryDependencies ++= Seq(
"org.apache.spark" %% "spark-core" % sparkVersion % "provided",
"org.apache.spark" %% "spark-streaming" % sparkVersion % "provided",
"org.apache.spark" %% "spark-streaming-twitter" % sparkVersion
)
// META-INF discarding
mergeStrategy in assembly <<= (mergeStrategy in assembly) { (old) =>
{
case PathList("META-INF", xs # _*) => MergeStrategy.discard
case x => MergeStrategy.first
}
}
Note: The % "provided" means not to include the dependency in the final fat JAR (those libraries are already included in my workers)
Note: META-INF discarding inspired by this answser.
Note: Meaning of % and %%
Now I can build my fat JAR using SBT (how to install it) by running the following command in my /my-project root folder:
sbt assembly
My fat JAR is now located in the new generated /target folder :
/my-project/target/scala-2.11/my-project-assembly-1.0.jar
For those who wants to embeed SBT within IntelliJ IDE: How to run sbt-assembly tasks from within IntelliJ IDEA?
3 Step Process For Building Uber JAR/Fat JAR in IntelliJ Idea:
Uber JAR/Fat JAR : JAR file having all external libraray dependencies in it.
Adding SBT Assembly plugin in IntelliJ Idea
Go to ProjectName/project/target/plugins.sbt file and add this line addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.12.0")
Adding Merge,Discard and Do Not Add strategy in build.sbt
Go to ProjectName/build.sbt file and add the Strategy for Packaging of an Uber JAR
Merge Strategy : If there is conflict in two packages about a version of library then which one to pack in Uber JAR.
Discard Strategy : To remove some files from library which you do not want to package in Uber JAR.
Do not Add Strategy : Do not add some package to Uber JAR.For ex: spark-core will be already present at your Spark Cluster.So we should not package this in Uber JAR
Merge Strategy and Discard Strategy Basic Code :
assemblyMergeStrategy in assembly := {
case PathList("META-INF", xs # _*) => MergeStrategy.discard
case x => MergeStrategy.first
}
So you are asking to discard META-INF files using this command MergeStrategy.discard and for rest of the files you are taking the first occurrence of library file if there is any conflict by using this command MergeStrategy.first.
Do not Add Strategy Basic Code :
libraryDependencies += "org.apache.spark" %% "spark-core" % "1.4.1" %"provided"
If we do not want to add the spark-core to our Uber JAR file as it will be already on our clutser, so we are adding the % "provided" at end of it library dependency.
Building Uber JAR with all its dependencies
In terminal type sbt assembly for building up the package
Voila!!! Uber JAR is built. JAR will be in ProjectName/target/scala-XX
Add the following line to your project/plugins.sbt
addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.12.0")
Add the following to your build.sbt
mainClass in assembly := some("package.MainClass")
assemblyJarName := "desired_jar_name_after_assembly.jar"
val meta = """META.INF(.)*""".r
assemblyMergeStrategy in assembly := {
case PathList("javax", "servlet", xs # _*) => MergeStrategy.first
case PathList(ps # _*) if ps.last endsWith ".html" => MergeStrategy.first
case n if n.startsWith("reference.conf") => MergeStrategy.concat
case n if n.endsWith(".conf") => MergeStrategy.concat
case meta(_) => MergeStrategy.discard
case x => MergeStrategy.first
}
The Assembly merge strategy is used to resolve conflicts occurred when creating fat jar.

How can I ignore scala library while sbt assembly

I am using sbt to build my scala project.
This is my build.sbt:
name := "My Spark App"
version := "1.0"
libraryDependencies += "org.apache.spark" %% "spark-core" % "1.2.0" % "provided"
libraryDependencies += "org.apache.spark" %% "spark-sql" % "1.2.0" % "provided"
I am running sbt assembly to create an assembly jar, but I found a scala directory containing scala library class codes.
Is it possible to take scala library as a provided dependency, since the run-time environment already contains scala?
From docs, this might help
assemblyOption in assembly := (assemblyOption in assembly).value.copy(includeScala = false)