I installed Spark to C:\Spark1_6\spark-1.6.0-bin-hadoop2.6. After navigating to this path I am entering sbt assembly command and I am getting the following error message:
[error] Not a valid command: assembly
[error] Not a valid project ID: assembly
[error] Expected ':'
[error] Not a valid key: assembly
[error] assembly
[error] ^
Here is my sbt project structure.
-Project101
-project
-build.properties
-plugins.sbt
-src
-build.sbt
Here is my build.sbt:
name := "Project101"
version := "1.0"
scalaVersion := "2.10.2"
libraryDependencies ++= Seq(
"org.apache.spark" % "spark-core_2.10" % "1.6.0" exclude ("org.apache.hadoop","hadoop-yarn-server-web-proxy"),
"org.apache.spark" % "spark-sql_2.10" % "1.6.0" exclude ("org.apache.hadoop","hadoop-yarn-server-web-proxy"),
"org.apache.spark" %% "spark-hive" % "1.6.0",
"org.apache.spark" %% "spark-streaming" % "1.6.0",
"org.apache.spark" %% "spark-streaming-kafka" % "1.6.0"
)
resolvers in Global ++= Seq(
"Sbt plugins" at "https://dl.bintray.com/sbt/sbt-plugin-releases",
"Maven Central Server" at "http://repo1.maven.org/maven2",
"TypeSafe Repository Releases" at "http://repo.typesafe.com/typesafe/releases/",
"TypeSafe Repository Snapshots" at "http://repo.typesafe.com/typesafe/snapshots/"
)
Here is the plugins.sbt:
addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.12.0")
sbt package command is working and able to create the jar file. But I had to execute sbt assembly command too but is not working.
Not a valid command: assembly
Whenever you face the error message, please make sure you're in the top-level directory of the project with the sbt-assembly plugin installed.
If you have a project in Project101 directory, make sure that project/plugins.sbt has the line in:
addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.12.0")
With that, you should again be in Project101 directory and execute sbt assembly. That should execute the plugin to create an uber-jar.
As correctly stated in another answer, the issue is caused by running sbt assembly not from the project directory.
If you getting that issue within Docker container, make sure that your mounts are set correctly.
For example:
docker run ... -v $$PWD/:/<your Scala project directory here> -w /<your Scala project directory here> <your build image> bash -c "sbt assembly"
Related
I'm developing a Spark process in Scala (Eclipse IDE) and runs fine in my local cluster, but when I try to compiled it with SBT that I installed on my pc I got a error (see picture).
My first doubt is why SBT try to compile with scala 2.12 if I explicitly set scalaVersion to 2.11.11 in my build.sbt. I tried installing other SBT versions with the same results, also in other PCs but not works. I need help to fix it.
scala_version(Spark) :2.11.11
sbt_version : 1.0.2
spark: 2.2
build.sbt
name := "Comple"
version := "1.0"
organization := "com.antonio.spark"
scalaVersion := "2.11.11"
libraryDependencies ++= Seq(
"org.apache.spark" %% "spark-core" % "2.2.0" % "provided",
"org.apache.spark" %% "spark-sql" % "2.2.0" % "provided"
)
assembly.sbt
addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "1.0.2")
Error:
ResolveException: unresolved dependency: sbt_assembly;1.0.2: not found
Try changing your assembly.sbt file to:
addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.14.5")
as stated in the documentation here: https://github.com/sbt/sbt-assembly
I recently used that with spark-core_2.11 version 2.2.0 and it worked.
I'm new to Scala and working with IntelliJ Community Edition.
I open a new project and edit the build.sbt file to contain the following:
name := "test"
version := "1.0"
scalaVersion := "2.11.8"
libraryDependencies ++= Seq(
"org.locationtech.geotrellis" %% "geotrellis-raster" % "1.0.0" % "1.1.0" ,
"org.locationtech.geotrellis" % "geotrellis-shapefile_2.11" % "1.1.0"
).map(_
.exclude("com.azavea.geotrellis", "geotrellis-proj4_2.11")
.exclude("com.azavea.geotrellis", "geotrellis-raster_2.11")
)
After that, I refresh the project, and I get the following errors in the error file:
[error] (*:ssExtractDependencies) java.lang.IllegalArgumentException: Cannot add dependency 'org.locationtech.geotrellis#geotrellis-raster_2.11;1.0.0' to configuration '1.1.0' of module default#test_2.11;1.0 because this configuration doesn't exist!
[error] (*:update) java.lang.IllegalArgumentException: Cannot add dependency 'org.locationtech.geotrellis#geotrellis-raster_2.11;1.0.0' to configuration '1.1.0' of module default#test_2.11;1.0 because this configuration doesn't exist!
What am I doing wrong here?
Main issue seems to be two different version numbers on one of your dependencies.
"org.locationtech.geotrellis" %% "geotrellis-raster" % "1.0.0" % "1.1.0"
Should be
"org.locationtech.geotrellis" %% "geotrellis-raster" % "1.0.0"
or
"org.locationtech.geotrellis" %% "geotrellis-raster" % "1.1.0"
if you want the 1.1.0 version
This the code that I have written in Scala.
object Main extends App {
println("Hello World from Scala!")
}
This is my build.sbt.
name := "hello-world"
version := "1.0"
scalaVersion := "2.11.5"
mainClass := Some("Main")
This is the command that I have run to create the jar file.
sbt package
My problem is that a jar file named hello-world_2.11-1.0.jar has been created at target/scala-2.11. But I cannot run the file. It is giving me an error saying NoClassDefFoundError.
What am I doing wrong?
It also says what class is not found. Most likely you aren't including scala-library.jar. You can run scala target/scala-2.11/hello-world_2.11-1.0.jar if you have Scala 2.11 available from the command line or java -cp "<path to scala-library.jar>:target/scala-2.11/hello-world_2.11-1.0.jar" Main (use ; instead of : on Windows).
The procedure depicted proves valid up to the way the jar file is executed. From target/scala-2.11 try running it with
scala hello-world_2.11-1.0.jar
Check whether it is runnable also from the project root folder with sbt run.
To run the jar file(containing scala code) with multiple main classes use following approach
scala -cp "<jar-file>.jar;<other-dependencies>.jar" com.xyz.abc.TestApp
This command will take care of including scala-library.jar in dependency and will also identify TestApp as main class if it has a def main(args:Array[String]) method. Please note that multiple jar files should be separated by semi-colon(";")
We can use sbt-assembly to package and run the application.
First, create or add the plugin to project/plugins.sbt
addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.14.9")
The sample build.sbt looks like below:
name := "coursera"
version := "0.1"
scalaVersion := "2.12.10"
mainClass := Some("Main")
val sparkVersion = "3.0.0-preview2"
val playVersion="2.8.1"
val jacksonVersion="2.10.1"
libraryDependencies ++= Seq(
"org.scala-lang" % "scala-library" % scalaVersion.toString(),
"org.apache.spark" %% "spark-streaming" % sparkVersion,
"org.apache.spark" %% "spark-core" % sparkVersion,
"org.apache.spark" %% "spark-sql" % sparkVersion,
"com.typesafe.play" %% "play-json" % playVersion,
// https://mvnrepository.com/artifact/org.apache.spark/spark-streaming-kafka-0-10
"org.apache.spark" %% "spark-streaming-kafka-0-10" % sparkVersion,
// https://mvnrepository.com/artifact/org.mongodb/casbah
"org.mongodb" %% "casbah" % "3.1.1" pomOnly(),
// https://mvnrepository.com/artifact/com.typesafe/config
"com.typesafe" % "config" % "1.2.1"
)
assemblyMergeStrategy in assembly := {
case PathList("META-INF", xs # _*) => MergeStrategy.discard
case x => MergeStrategy.first
}
From console, we can run sbt assembly and the jar file gets created in target/scala-2.12/ path.
sbt assembly will create a fat jar. Here is an excerpt from the documentation :
sbt-assembly is a sbt plugin originally ported from codahale's assembly-sbt, which I'm guessing was inspired by Maven's assembly plugin. The goal is simple: Create a fat JAR of your project with all of its dependencies.
I am using sbt 0.13.7 and Scala 2.11.4 on a Windows machine to compile my code into a fat jar, that I eventually want to run on a Linux machine.
Below is my build.sbt file:
import AssemblyKeys._
name := "Simple Project"
version := "1.0"
organization := "com.myorg"
scalaVersion := "2.11.4"
libraryDependencies ++= Seq(
// Spark dependency
"org.apache.spark" % "spark-core_2.10" % "1.2.0" % "provided",
// Third party libraries
"net.sf.jopt-simple" % "jopt-simple" % "4.3",
"joda-time" % "joda-time" % "2.0"
)
libraryDependencies += Defaults.sbtPluginExtra("com.eed3si9n" % "sbt-assembly" % "0.7.2", "0.11.2", "2.9.1")
// This statement includes the assembly plugin capabilities
assemblySettings
// Configure jar named used with the assembly plug-in
jarName in assembly := "my-project-assembly.jar"
// A special option to exclude Scala itself form our assembly jar, since Spark
// already bundles Scala.
assemblyOption in assembly := (assemblyOption in assembly).value.copy(includeScala = false)
The error I am facing is:
build.sbt:16: error: type mismatch;
found : Seq[sbt.Project.Setting[_]]
required: sbt.internals.DslEntry
assemblySettings
^
[error] Type error in expression
Project loading failed: (r)etry, (q)uit, (l)ast, or (i)gnore?
Are you using sbt-assembly 0.12.0? If so, you don't need assemblySettings any more since it's an auto plugin.
Edit:
You have to include
addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.12.0")
in project/*.sbt like project/assembly.sbt, not build.sbt.
I have been fighting with this for the past few hours and I haven't made any headway at all. It seems no matter what I do, I keep getting the same error.
java.lang.NoClassDefFoundError: play/Project$
My build.sbt
name := "appname"
version := "1.0-SNAPSHOT"
libraryDependencies ++= Seq(
jdbc,
anorm,
cache,
filters,
"org.postgresql" % "postgresql" % "9.3-1100-jdbc4",
"org.mindrot" % "jbcrypt" % "0.3m",
"org.webjars" %% "webjars-play" % "2.3.0-2",
"org.webjars" % "foundation" % "5.3.0",
"org.scalaj" %% "scalaj-http" % "0.3.16"
)
lazy val root = (project in file(".")).enablePlugins(PlayScala)
scalaVersion := "2.10.4"
I have also tried this build.sbt
object ApplicationBuild extends Build {
val appName = "appname"
val appVersion = "1.0-SNAPSHOT"
val appDependencies ++= Seq(
jdbc,
anorm,
cache,
filters,
"org.postgresql" % "postgresql" % "9.3-1100-jdbc4",
"org.mindrot" % "jbcrypt" % "0.3m",
"org.webjars" %% "webjars-play" % "2.3.0-2",
"org.webjars" % "foundation" % "5.3.0",
"org.scalaj" %% "scalaj-http" % "0.3.16"
)
val main = Project(appName, file(".")).enablePlugins(play.PlayScala).settings(
scalaVersion := "2.10.4",
version := appVersion,
libraryDependencies ++= appDependencies
)
}
As well as different tweaks and modifications. Always the same error. The build.properties is set to sbt.version=0.13.5
plugins.sbt
logLevel := Level.Debug
// The Typesafe repository
resolvers += "Typesafe repository" at "http://repo.typesafe.com/typesafe/releases/"
// Use the Play sbt plugin for Play projects
addSbtPlugin("com.typesafe.play" % "sbt-plugin" % "2.3.4")
After running Patrick Mahoney's suggestion and making sure I removed the import.
Errors:
[error] java.lang.NoClassDefFoundError: play/Project$
[error] Use 'last' for the full log.
Project loading failed: (r)etry, (q)uit, (l)ast, or (i)gnore? q
me#me-desktop:~/Projects/appname$ sbt --version
sbt launcher version 0.13.6
me#me-desktop:~/Projects/appname$ find . | grep -r "play.Project"
me#me-desktop:~/Projects/appname$ find . | grep -r "play/Project"
Try cleaning your build project compiled outputs:
rm -rf project/target
or
$> sbt "reload plugins" clean
(in addition to applying James' answer)
Thanks to James and Patrick, but I found the problem. The problem was in the activator-sbt-echo-play-shim.sbt file in the /project directory.
It contained the following:
addSbtPlugin("com.github.mpeltonen" % "sbt-idea" % "0.1.1.3")
I created a second blank 2.3.4 app using Activator and it's activator-sbt-echo-play-shim.sbt listed:
addSbtPlugin("com.github.mpeltonen" % "sbt-idea" % "1.5")
So I updated the one in my project to 1.5 and deleted the idea and eclipse .sbts for good measure and the app was finally able to compile. It auotmatically updated the above to:
addSbtPlugin("com.github.mpeltonen" % "sbt-idea" % "1.5.2")
Now I have other, more expected, issues I need to work out (like anorm). But I'm now able to compile it and get those errors to start working them out. I did not think these files were part of the activator/sbt compile, but I guess they were. I thought they were for debugging. But they made the difference and the changes above worked.
Thanks again to everyone for their help.
Make sure you've updated project/build.properties to have sbt version 0.13.5. Then, remove import play.Project._, it's not needed. That should work.