I need to create fat jars for multiple version of scala using sbt assembly.
When I target a single version, I write in simple.sbt:
scalaVersion := "2.11.12"
And the fat jar is output to target/scala-2.11/Kernalytics-assembly-1.0.jar. Now I would like to also target Scala 2.12. I could edit the sbt file to change scalaVersion, but I would like the assembly process to be automated over a range of versions of Scala when I call sbt assembly.
If I use crossScalaVersions:
name := "Kernalytics"
version := "1.0"
crossScalaVersions := Seq("2.11.12", "2.12.4")
libraryDependencies ++= Seq(
"org.scalanlp" %% "breeze" % "0.13.2",
"org.scalanlp" %% "breeze-natives" % "0.13.2",
"org.scalanlp" %% "breeze-viz" % "0.13.2"
)
libraryDependencies += "commons-io" % "commons-io" % "2.6"
resolvers += "Sonatype Releases" at "https://oss.sonatype.org/content/repositories/releases/"
libraryDependencies += "org.scalactic" %% "scalactic" % "3.0.4"
libraryDependencies += "org.scalatest" %% "scalatest" % "3.0.4" % "test"
The only output is target/scala-2.12/Kernalytics-assembly-1.0.jar
If you use crossScalaVersions I think you need to prefix the command with a '+' if you want to build for all versions.
From Cross-Building a Project:
To build against all versions listed in crossScalaVersions, prefix the action to run with +
Related
When I tray using spark with Scala in SBT build system to read a Json file a have the error:
and my SBT file is:
ThisBuild / version := "0.1.0-SNAPSHOT"
ThisBuild / scalaVersion := "2.13.10"
// https://mvnrepository.com/artifact/org.apache.spark/spark-core
libraryDependencies += "org.apache.spark" %% "spark-core" % "3.3.1"
// https://mvnrepository.com/artifact/org.apache.spark/spark-sql
libraryDependencies += "org.apache.spark" %% "spark-sql" % "3.3.1"
// https://mvnrepository.com/artifact/org.mongodb.spark/mongo-spark-connector
libraryDependencies += "org.mongodb.spark" % "mongo-spark-connector" % "10.0.5"
I tray to change all "3.3.1" and "10.0.5" to "3.0.1" and it still the same problem
You should use a 2.12 version of Scala, since those dependencies do not all have 2.13 built.
ThisBuild / scalaVersion := "2.12.11"
Look at the screenshot from Maven to see that 2.13 isn't listed for those versions (or at all).
I'm working on a back-end project using for the first time Scala and the Play Framework with IntelliJ IDEA.
I've been reading a lot of documentation and topics:
SBT Library dependencies doc
IntelliJ support post
Another post from stackoverflow
I can't understand how to import dependencies.
When I try to append a new dependency, IntelliJ underlines it and shows either "Unknown artifact. Not resolved or indexed" or a log shows up and says "SBT unknown import".
Every dependency I add is from the Maven Repository.
Here is the build.sbt file:
name := "server"
version := "1.0"
lazy val `server` = (project in file(".")).enablePlugins(PlayScala)
resolvers += "scalaz-bintray" at "https://dl.bintray.com/scalaz/releases"
resolvers += "Akka Snapshot Repository" at "http://repo.akka.io/snapshots/"
resolvers := ("Atlassian Releases" at "https://maven.atlassian.com/public/") +: resolvers.value
scalaVersion := "2.12.2"
libraryDependencies ++= Seq(
jdbc,
ehcache,
ws,
specs2 % Test,
guice,
"org.mongodb.scala" %% "mongo-scala-driver" % "2.1.0",
"com.mohiva" %% "play-silhouette" % "5.0.2",
"com.mohiva" %% "play-silhouette-password-bcrypt" % "5.0.2",
"com.mohiva" %% "play-silhouette-crypto-jca" % "5.0.2",
"com.mohiva" %% "play-silhouette-persistence" % "5.0.2",
"com.mohiva" %% "play-silhouette-testkit" % "5.0.2" % "test"
)
unmanagedResourceDirectories in Test <+= baseDirectory ( _ /"target/web/public/test" )
These dependencies were specified using a shorthand that is defined by the Play plugin:
jdbc,
ehcache,
ws,
specs2 % Test,
guice
So for them to work, be sure to include a line like the following in an .sbt file in the project directory:
addSbtPlugin("com.typesafe.play" % "sbt-plugin" % "2.6.10")
That should fix your problem.
Bonus tip: No need for the following lines, and because each additional resolver slows down SBT, you should delete them:
resolvers += "scalaz-bintray" at "https://dl.bintray.com/scalaz/releases"
resolvers += "Akka Snapshot Repository" at "http://repo.akka.io/snapshots/"
resolvers := ("Atlassian Releases" at "https://maven.atlassian.com/public/") +: resolvers.value
I am trying to execute spark project from eclipse tool.
In build.sbt I have added below
name := "simple-spark-scala"
version := "1.0"
scalaVersion := "2.11.8"
libraryDependencies += "org.apache.spark" % "spark-core_2.11" % "1.6.2"
When I am importing this project I am getting error -
project is missing library around 100 such error
Description Resource Path Location Type
Project 'simple-spark' is missing required library: '/root/.ivy2/cache/aopalliance/aopalliance/jars/aopalliance-1.0.jar' simple-spark Build path Build Path Problem
However I am able to see all the jars under the mentioned path in missing jar files
Any idea how to resolve ?
Add dependency resolvers as below in your built.sbt file
resolvers += "MavenRepository" at "https://mvnrepository.com/"
Because, you dont link Spark Packages Repo. You can see my built.sbt below:
name := "spark"
version := "1.0"
scalaVersion := "2.11.8"
resolvers += "Spark Packages Repo" at "http://dl.bintray.com/spark-packages/maven"
libraryDependencies ++= Seq(
"org.apache.spark" % "spark-core_2.11" % "2.1.0",
"org.apache.spark" % "spark-sql_2.11" % "2.1.0",
"org.apache.spark" % "spark-graphx_2.11" % "2.1.0",
"org.apache.spark" % "spark-mllib_2.11" % "2.1.0",
"neo4j-contrib" % "neo4j-spark-connector" % "2.0.0-M2"
)
I made a following dependency in build.sbt file for apache-spark 2.11.
name := "Project1"
version := "1.0"
scalaVersion := "2.11.8"
libraryDependencies += "org.apache.spark" % "spark-core_2.11" % "2.0.1"
libraryDependencies ++= Seq(
"org.scala-lang" % "scala-compiler" % "2.11.8",
"org.scala-lang" % "scala-reflect" % "2.11.8",
"org.scala-lang.modules" % "scala-parser-combinators_2.11" % "1.0.4",
"org.scala-lang.modules" % "scala-xml_2.11" % "1.0.4"
)
However Intellij could not resolve spark-core_2.11 dependency . I tried multiple times but could not succeed. Thanks in Advance.
I had the same problem in IntelliJ 2016.3.2 with almost the same Scala/Spark versions:
name := "some-project"
version := "1.0"
scalaVersion := "2.11.8"
libraryDependencies += "org.apache.spark" % "spark-core_2.11" % "2.1.0"
To get it to work I had to manually add the spark-core jar to my project libraries, ie:
Right click on the project -> Open Module Settings
Under Project Settings -> Libraries click + and select the 'Java' option.
Browse for the jar. I found it in my Ivy cache - I assume it got there because I had run the 'update' task from the sbt console previously.
The below sbt file doesn't resolve the spark-xml databricks package from within intelliJ Idea where as works fine with command line
name := "dataframes"
version := "1.0"
scalaVersion := "2.11.8"
val sparkVersion="1.4.0"
libraryDependencies ++= Seq(
"org.apache.spark" %% "spark-core" % sparkVersion % "provided",
"org.apache.spark" %% "spark-sql" % sparkVersion,
"com.databricks" %% "spark-xml" % "0.3.3"
)
resolvers ++= Seq(
"Apache HBase" at "https://repository.apache.org/content/repositories/releases",
"Typesafe repository" at "http://repo.typesafe.com/typesafe/releases/"
)
resolvers += Resolver.mavenLocal
The sbt was set to both bundled and other pointing to the locally installed sbt, but did not work either way.
The below package resolves and works perfectly from commandline
import com.databricks.spark._