How to include test dependencies in sbt-assembly jar? - scala

I am unable to package my test dependencies in my test assembly jar. Here is an excerpt from my build.sbt:
...
name := "project"
scalaVersion := "2.10.6"
assemblyOption in (Compile, assembly) := (assemblyOption in (Compile, assembly)).value.copy(includeScala = false)
fork in Test := true
parallelExecution in IntegrationTest := false
lazy val root = project.in(file(".")).configs(IntegrationTest.extend(Test)).settings(Defaults.itSettings: _ *)
Project.inConfig(Test)(baseAssemblySettings)
test in (Test, assembly) := {}
assemblyOption in (Test, assembly) := (assemblyOption in (Test, assembly)).value.copy(includeScala = false, includeDependency = true)
assemblyJarName in (Test, assembly) := s"${name.value}-test.jar"
fullClasspath in (Test, assembly) := {
val cp = (fullClasspath in Test).value
cp.filter{ file => (file.data.name contains "classes") || (file.data.name contains "test-classes")} ++ (fullClasspath in Runtime).value
}
libraryDependencies ++= Seq(
...
"com.typesafe.play" %% "play-json" % "2.3.10" % "test" excludeAll ExclusionRule(organization = "joda-time"),
...
)
...
When I assemble my fat jar using sbt test:assembly, is produces the fat jar project-test.jar, but the play-json dependencies aren't being packaged in:
$ jar tf /path/to/project-test.jar | grep play
$
However, if I remove the "test" configuration from the play-json dep (i.e. "com.typesafe.play" %% "play-json" % "2.3.10" excludeAll ExclusionRule(organization = "joda-time")), I can see it being included:
$ jar tf /path/to/project-test.jar | grep play
...
play/libs/Json.class
...
$
Am I doing anything wrong and/or missing anything? My goal here is to include the play-json library in ONLY the test:assembly jar and NOT the assembly jar

I have left out a crucial part in the original build.sbt excerpt I posted above which turned out to be the cause of the issuse:
fullClasspath in (Test, assembly) := {
val cp = (fullClasspath in Test).value
cp.filter{ file => (file.data.name contains "classes") || (file.data.name contains "test-classes")} ++ (fullClasspath in Runtime).value
}
This code block was essentially filter out deps from the test classpath. We include this to avoid painful merge conflicts. I fixed this by adding logic to include the play-json dep that was needed:
fullClasspath in (Test, assembly) := {
val cp = (fullClasspath in Test).value
cp.filter{ file =>
(file.data.name contains "classes") ||
(file.data.name contains "test-classes") ||
// sorta hacky
(file.data.name contains "play")
} ++ (fullClasspath in Runtime).value
}

Related

Spark / scala : publish a single fat jar with sbt

I am trying to deploy a single Fat Jar from a spark/scala project to a private nexus repository. The publish works fine but I get a large amount of files instead of one single assembly Jar :
*-assembly.jar
*-assembly.jar.md5
*-assembly.jar.sha1
*-javadoc.jar
*-javadoc.jar.md5
*-javadoc.jar.sha1
*-source.jar
*-source.jar.md5
*-source.jar.sha1
*.jar
*.jar.md5
*.jar.sha1
*-pom.jar
*-pom.jar.md5
*-pom.jar.sha1
My built.sbt is :
name := "SampleApp"
version := "0.1-SNAPSHOT"
scalaVersion := "2.12.14"
ThisBuild / useCoursier := false
libraryDependencies ++= Seq(
"com.github.scopt" %% "scopt" % "4.0.1",
"org.apache.spark" %% "spark-sql" % "3.1.1" % "provided"
)
resolvers ++= Seq(
"confluent" at "https://packages.confluent.io/maven/"
)
assembly / assemblyMergeStrategy := {
case PathList("META-INF","services",xs # _*) => MergeStrategy.filterDistinctLines
case PathList("META-INF",xs # _*) => MergeStrategy.discard
case "application.conf" => MergeStrategy.concat
case _ => MergeStrategy.first
}
assembly / assemblyExcludedJars := {
val cp = (assembly / fullClasspath).value
cp filter { f =>
f.data.getName.contains("hadoop-hdfs-2")
f.data.getName.contains("hadoop-client")
}
}
assembly / artifact := {
val art = (assembly / artifact).value
art.withClassifier(Some("assembly"))
}
assembly / assemblyJarName := s"${name.value}-${version.value}.jar"
addArtifact(assembly / artifact, assembly)
resolvers += ("Sonatype Nexus Repository Manager" at "http://localhost:8081/repository/app/").withAllowInsecureProtocol(true)
credentials += Credentials("Sonatype Nexus Repository Manager", "localhost:8081", "user", "pass")
publishTo := {
val nexus = "http://localhost:8081/repository/app/"
if (isSnapshot.value) {
Some("snapshots" at nexus + "test-SNAPSHOT")
} else
Some("releases" at nexus + "test-RELEASE")
}
Is there a way to filter files with sbt before the publish in order to get only the *-assembly.jar ? Thanks a lot.
This should disable the publish of unnecessary files in my case :
publishArtifact in (Compile, packageBin) := false
publishArtifact in (Compile, packageDoc) := false
publishArtifact in (Compile, packageSrc) := false

ScalaJS - SBT crossproject configuration for testing part of the `JS`-module on the JVM

The build.sbt below is for a project where there 3 modules:
Shared-module
compiles to the JS-platform and to the JVM-platform
visible to the JVM module, to the JS module and to itself
JVM-module
compiles to JVM-platform,
not visible to JS-module, nor to Shared-module, only visible to itself
JS-module
compiles to JS-platform
not visible to JVM-module, nor to Shared-module, only visible to itself
I would like to have a fourth module TestJSonJVM which compiles to both the JVM-platform and to the JS-platform and it is visible only to the JS-module and to itself.
The purpose of the TestJSonJVM-module is to extract part of the JS-modul's logic/code into TestJSonJVM-module which makes it possible to test the extracted part on the JVM-platform (which has better tooling - (debuging, IDE integration, etc..) than the JS-platform for ScalaJS).
How do I need to modify the build.sbt file below ? Such that this becomes possible ?
lazy val root = project
.in(file("."))
.aggregate(imJS, imJVM)
.settings(
scalaVersion := Settings.versions.scala,
publish := {},
publishLocal := {}
)
lazy val im = crossProject
.in(file("."))
.settings(
libraryDependencies ++= Settings.sharedDependencies.value,
addCompilerPlugin(
"org.scalamacros" % "paradise" % "2.1.0" cross CrossVersion.full),
scalaVersion := Settings.versions.scala,
name := "im",
version := "0.1-SNAPSHOT"
)
.jvmSettings(
libraryDependencies ++= Settings.jvmDependencies.value,
mainClass in Test := Some("app.server.rest.TestHttpServerApp"),
mainClass in Compile := Some("app.server.rest.TestHttpServerApp")
)
.jsSettings(
mainClass in Compile := Some("app.client.Main"),
libraryDependencies ++= Settings.scalajsDependencies.value,
persistLauncher in Compile := true,
persistLauncher in Test := false,
jsDependencies += RuntimeDOM,
scalaJSOptimizerOptions ~= { _.withDisableOptimizer(true) }
)
lazy val imJVM = im.jvm
lazy val imJS = im.js
persistLauncher in Compile := true
persistLauncher in Test := false
cancelable in Global := true

SBT assembly falis

I am running a spark job through intellij. Job executes and gives me output. i need to take this job as jar file to server and run, but when i try to do sbt assembly it throws below error:
[error] Not a valid command: assembly
[error] Not a valid project ID: assembly
[error] Expected ':' (if selecting a configuration)
[error] Not a valid key: assembly
[error] assembly
my sbt version is 0.13.8
below is my build.sbt file:
import sbt._, Keys._
name := "mobilewalla"
version := "1.0"
scalaVersion := "2.11.7"
libraryDependencies ++= Seq("org.apache.spark" %% "spark-core" % "2.0.0",
"org.apache.spark" %% "spark-sql" % "2.0.0")
i added a file assembly.sbt under project dir. it contains:
addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.14.3")
what am i missing here
Add these lines in your build.sbt
assemblyMergeStrategy in assembly := {
case PathList("META-INF", xs # _*) => MergeStrategy.discard
case x => MergeStrategy.first
}
mainClass in assembly := Some("com.SparkMain")
resolvers += "spray repo" at "http://repo.spray.io"
assemblyJarName in assembly := "streaming-api.jar"
and include these lines in your plugins.sbt file
addSbtPlugin("io.spray" % "sbt-revolver" % "0.7.2")
addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.13.0")
To assemble the multiple jars to one u need add below plugin in plugins.sbt under project directory.
addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.14.3")
If u need to customize the assembled jar to trigger specific MainClass take example assembly.sbt
import sbtassembly.Plugin.AssemblyKeys._
Project.inConfig(Compile)(baseAssemblySettings)
mainClass in (Compile, assembly) := Some("<main application name with package path>")
jarName in (Compile, assembly) := s"${name.value}-${version.value}-dist.jar"
//below is merge strategy to make what all file need to exclude or include
mergeStrategy in (Compile, assembly) <<= (mergeStrategy in (Compile, assembly)) {
(old) => {
case PathList(ps # _*) if ps.last endsWith ".html" =>MergeStrategy.first
case "META-INF/MANIFEST.MF" => MergeStrategy.discard
case x => old(x)
}
}

How to reference a custom SBT Setting in sub-projects

Somewhat similar to this question, how can reference a custom setting in a sub project.
In build.sbt:
import sbt.Keys._
val finagleVersion = settingKey[String]("Defines the Finagle version")
val defaultSettings = Defaults.coreDefaultSettings ++ Seq(
finagleVersion in ThisBuild := "6.20.0",
organization in ThisBuild := "my.package",
scalaVersion in ThisBuild := "2.10.4",
version in ThisBuild := "0.1-SNAPSHOT"
)
lazy val root = project.in(file(".")).aggregate(thrift).settings(
publishArtifact in (Compile, packageBin) := false,
publishArtifact in (Compile, packageDoc) := false,
publishArtifact in (Compile, packageSrc) := false
)
lazy val thrift = project.in(file("thrift"))
In thrift/build.sbt:
name := "thrift"
// doesn't work
libraryDependencies ++= Seq(
"com.twitter" %% "finagle-thriftmux" % (finagleVersion in LocalRootProject).value
)
.sbt files cannot see the definitions (e.g., vals) in other .sbt files, even if they are part of the same build.
However, all .sbt files in a build can see/import the content of project/*.scala files. So you'll have to declare your val finagleVersion in a .scala file:
project/CustomKeys.scala:
import sbt._
import Keys._
object CustomKeys {
val finagleVersion = settingKey[String]("Defines the Finagle version")
}
Now, in your .sbt files, just
import CustomKeys._
and you're good to go.

What is the proper play-mini project setup for a proper production deployment

Play mini doesn't work like a play project at all really unless i am missing something. it runs within sbt and you cannot use play commands.
https://github.com/typesafehub/play2-mini
So how do you deploy this stuff to production? I've tried one-jar and assembly too and it just doesn't work for me
Ive stried the start-script/stage approach but it cannto seem to find my mainclass:
sbt
>add-start-script-tasks
>stage
[info] Wrote start script for mainClass := None to /Users/rmedlin/rtbv2/target/start
This is my Build.scala. Ive also tried: mainClass in (Compile, stage, run) and many other combinations
object Build extends Build {
override lazy val settings = super.settings
lazy val root = Project(id = "rtbv2",
base = file("."), settings = Project.defaultSettings).settings(
resolvers += "Typesafe Repo" at "http://repo.typesafe.com/typesafe/releases/",
resolvers += "Typesafe Snapshot Repo" at "http://repo.typesafe.com/typesafe/snapshots/",
libraryDependencies += "com.typesafe" %% "play-mini" % "2.0.1",
mainClass in (Compile, run) := Some("play.core.server.NettyServer"))
}
my Build.scala was incorrect and i was able to get the assembly command working:
trait ConfigureScalaBuild {
lazy val typesafe = "Typesafe Repository" at "http://repo.typesafe.com/typesafe/releases/"
lazy val typesafeSnapshot = "Typesafe Snapshots Repository" at "http://repo.typesafe.com/typesafe/snapshots/"
val netty = Some("play.core.server.NettyServer")
def scalaMiniProject(org: String, name: String, buildVersion: String, baseFile: java.io.File = file(".")) = Project(id = name, base = baseFile, settings = Project.defaultSettings ++ assemblySettings).settings(
version := buildVersion,
organization := org,
resolvers += typesafe,
resolvers += typesafeSnapshot,
logManager <<= extraLoggers(com.typesafe.util.Sbt.logger),
libraryDependencies += "com.typesafe" %% "play-mini" % "2.0.1",
mainClass in (Compile, run) := netty,
mainClass in assembly := netty,
ivyXML := <dependencies> <exclude org="org.springframework"/> </dependencies>
)
}