sbt-assembly | NoSuchMethodError at runtime when shading - scala

I'm trying to use sbt-assembly for shading some dependencies.
I've shaded and published a jar for project X as mentioned in the sbt-assembly github readme file.
When I'm trying to use that published jar in project Y, I'm getting NoSuchMethodError error at runtime.
Below are my shade rules used for project X
val shadePackage = "mycompany.spark.v1.shaded"
Seq(
ShadeRule.rename("cats.**" -> s"$shadePackage.cats.#1").inAll,
ShadeRule.rename("com.mycompany.sdk.scala.**" -> s"$shadePackage.sdk.scala.#1").inAll,
ShadeRule.rename("com.google.protobuf.**" -> s"$shadePackage.com.google.protobuf.#1").inAll,
ShadeRule.rename("fs2.**" -> s"$shadePackage.fs2.#1").inAll,
ShadeRule.rename("io.circe.**" -> s"$shadePackage.io.circe.#1").inAll,
ShadeRule.rename("jawn.**" -> s"$shadePackage.jawn.#1").inAll, <==============
ShadeRule.rename("shapeless.**" -> s"$shadePackage.shapeless.#1").inAll,
ShadeRule.rename("sttp.client3.**" -> s"$shadePackage.sttp.client3.#1").inAll,
)
For jawn, When I use the below shade rule,
ShadeRule.rename("jawn.**" -> s"$shadePackage.jawn.#1").inAll
it gives me the below error at runtime for project Y which includes project X jar as a dependency.
java.lang.NoSuchMethodError: org.typelevel.jawn.Facade.$init$(Lorg/typelevel/jawn/Facade;)V
But when I use the shade rule as below, it works fine.
ShadeRule.rename("org.typelevel.jawn.**" -> s"$shadePackage.org.typelevel.jawn.#1").inAll,
In project Y, I'm using the below dependencies which have transitive dependencies with jawn
val CirceVersion = "0.14.0"
val Http4sVersion = "0.22.2"
"org.http4s" %% "http4s-blaze-client" % Http4sVersion,
"org.http4s" %% "http4s-circe" % Http4sVersion,
"io.circe" %% "circe-generic" % CirceVersion,
"io.circe" %% "circe-derivation" % CirceVersion,
"io.circe" %% "circe-literal" % CirceVersion,
"io.circe" %% "circe-generic-extras" % CirceVersion,
I would appreciate if anyone could explain whats going on here.

I was able to fix it by shading as below
assemblyShadeRules := {
val shadePackage = "shaded"
Seq(
ShadeRule.rename("cats.**" -> s"$shadePackage.cats.#1").inAll,
ShadeRule.rename("com.google.protobuf.**" -> s"$shadePackage.com.google.protobuf.#1").inAll,
ShadeRule.rename("fs2.**" -> s"$shadePackage.fs2.#1").inAll,
ShadeRule.rename("io.circe.**" -> s"$shadePackage.io.circe.#1").inAll,
ShadeRule.rename("org.typelevel.jawn.**" -> s"$shadePackage.org.typelevel.jawn.#1").inAll,
ShadeRule.rename("shapeless.**" -> s"$shadePackage.shapeless.#1").inAll,
ShadeRule.rename("sttp.client3.**" -> s"$shadePackage.sttp.client3.#1").inAll,
)
},
and making a new project in sbt as,
lazy val fatJar = project.settings(
commonSettings,
name := "my-project-fat",
Compile / packageBin := (library / assembly).value
)
sbt packageBin will create run the assembly phase and create the fat-jar
Details can also be found here.

Related

How to create WAR instead JAR?

I would like to compile via sbt to WAR instead of JAR file and I followed this guide.
I have changed the build.sbt to:
libraryDependencies ++= Seq(
"org.http4s" %% "http4s-jetty" % Http4sVersion,
"org.http4s" %% "http4s-jetty-client" % Http4sVersion,
"org.http4s" %% "http4s-circe" % Http4sVersion,
"org.http4s" %% "http4s-dsl" % Http4sVersion,
"io.circe" %% "circe-generic" % CirceVersion,
"org.specs2" %% "specs2-core" % Specs2Version % "test",
"ch.qos.logback" % "logback-classic" % LogbackVersion
),
addCompilerPlugin("org.typelevel" %% "kind-projector" % "0.10.3"),
addCompilerPlugin("com.olegpy" %% "better-monadic-for" % "0.3.1"),
// disable .jar publishing
Compile / packageBin / publishArtifact := false,
// create an Artifact for publishing the .war file
Compile / packageWar / artifact := {
val prev: Artifact = (Compile / packageWar / artifact).value
prev.withType("war").withExtension("war")
},
// add the .war file to what gets published
addArtifact(Compile / packageWar / artifact, packageWar),
)
and the compile has complained:
/home/developer/scala/user-svc/build.sbt:27: error: not found: value packageWar
Compile / packageWar / artifact := {
^
/home/developer/scala/user-svc/build.sbt:28: error: not found: value packageWar
val prev: Artifact = (Compile / packageWar / artifact).value
^
/home/developer/scala/user-svc/build.sbt:33: error: not found: value packageWar
addArtifact(Compile / packageWar / artifact, packageWar),
^
/home/developer/scala/user-svc/build.sbt:33: error: not found: value packageWar
addArtifact(Compile / packageWar / artifact, packageWar),
What am I doing wrong?
There is a bug in documentation - see https://github.com/sbt/sbt/issues/4490
Try using dedicated sbt plugin - xsbt-web-plugin - instead of reading that website. According to current docs you need to add to project/plugins.sbt
addSbtPlugin("com.earldouglas" % "xsbt-web-plugin" % "4.2.0")
then enable plugin in build.sbt e.g. Jetty
enablePlugins(JettyPlugin)
then you could build WAR with package command.

Scala Spark code for creating GCP Publisher throws: java.lang.NoSuchMethodError: com.google.common.base.Preconditions.checkArgument

Iam trying to publish messages to a topic in GCP's Pub/Sub using Spark Scala using IntelliJ. Here is the code:
GcpPublish.scala
val publisher = Publisher.newBuilder(s"projects/projectid/topics/test")
.setCredentialsProvider(FixedCredentialsProvider
.create(ServiceAccountCredentials
.fromStream(new FileInputStream("gs://credsfiles/projectid.json"))))
.build()
publisher.publish(PubsubMessage
.newBuilder
.setData(ByteString.copyFromUtf8(JSONData.toString()))
.build())
And this is the build.sbt:
name := "TryingSomething"
version := "1.0"
scalaVersion := "2.11.12"
val sparkVersion = "2.3.2"
libraryDependencies ++= Seq(
"org.apache.spark" %% "spark-core" % "2.3.2" % "provided",
"org.apache.spark" %% "spark-sql" % "2.3.2" ,
"com.google.cloud" % "google-cloud-bigquery" % "1.106.0",
"org.apache.beam" % "beam-sdks-java-core" % "2.19.0" ,
"org.apache.beam" % "beam-runners-google-cloud-dataflow-java" % "2.19.0",
"com.typesafe.scala-logging" %% "scala-logging" % "3.1.0" ,
"org.apache.beam" % "beam-sdks-java-extensions-google-cloud-platform-core" % "2.19.0" ,
"org.apache.beam" % "beam-sdks-java-io-google-cloud-platform" % "2.19.0" ,
"com.google.apis" % "google-api-services-bigquery" % "v2-rev456-1.25.0" ,
"com.google.cloud" % "google-cloud-pubsub" % "1.102.1",
"com.google.guava" % "guava" % "28.2-jre",
"org.apache.httpcomponents" % "httpclient" % "4.5.11"
)
assemblyMergeStrategy in assembly := {
case PathList("META-INF", xs # _*) => MergeStrategy.discard
case _ => MergeStrategy.first
}
but when I create the fat jar and run it on the Dataprocs cluster I get the below error:
Exception in thread "main" java.lang.NoSuchMethodError: com.google.common.base.Preconditions.checkArgument(ZLjava/lang/String;I)V
at com.google.api.gax.grpc.InstantiatingGrpcChannelProvider$Builder.setPoolSize(InstantiatingGrpcChannelProvider.java:527)
at com.google.api.gax.grpc.InstantiatingGrpcChannelProvider$Builder.setChannelsPerCpu(InstantiatingGrpcChannelProvider.java:546)
at com.google.api.gax.grpc.InstantiatingGrpcChannelProvider$Builder.setChannelsPerCpu(InstantiatingGrpcChannelProvider.java:535)
at com.google.cloud.pubsub.v1.Publisher$Builder.<init>(Publisher.java:633)
at com.google.cloud.pubsub.v1.Publisher$Builder.<init>(Publisher.java:588)
at com.google.cloud.pubsub.v1.Publisher.newBuilder(Publisher.java:584)
I followed the solutions stated here and added the guava and httpcomponents dependencies but I still get the same exception.
I even changed the code to instantiate the Publisher to:
val publisher = Publisher.newBuilder(s"projects/projectid/topics/test").build()
But this gives the same error as well.
Any suggestions what could cause this error?
The problem was that both Spark and Hadoop injected their own versions of guava that is also present in the Google Pubsub package. I got around this by adding shade rules in the build.sbt file:
assemblyShadeRules in assembly := Seq(
ShadeRule.rename("com.google.common.**" -> "repackaged.com.google.common.#1").inAll,
ShadeRule.rename("com.google.protobuf.**" -> "repackaged.com.google.protobuf.#1").inAll,
ShadeRule.rename("io.netty.**" -> "repackaged.io.netty.#1").inAll
)
The shade rule for com.google.common and com.google.protobuf is the one which solves the guava dependencies. I added the others for later dependency conflicts I encountered on the way.

Dependency duplication when creating a fat jar for playframework with play-ws

I noticed that when trying to build fat jara for playframework 2.7 with play-ws (sbt assembly), dependency duplications occur. I get a lot of errors related to javax.activation-api and shaded-asynchttpclient e.g.
[error] deduplicate: different file contents found in the following:
[error] /home/user/.cache/coursier/v1/https/repo1.maven.org/maven2/javax/activation/javax.activation-api/1.2.0/javax.activation-api-1.2.0.jar:javax/activation/UnsupportedDataTypeException.class
[error] /home/user/.cache/coursier/v1/https/repo1.maven.org/maven2/com/typesafe/play/shaded-asynchttpclient/2.0.6/shaded-asynchttpclient-2.0.6.jar:javax/activation/UnsupportedDataTypeException.class
The problem turns out to be the play-ws without which sbt assembly is carried out correctly. The only place in my code where I explicitly use javax is dependency injection. Using guice instead gives the same result. Here is my build.sbt (which is based on https://www.playframework.com/documentation/2.7.x/Deploying)
name := "My-App"
version := "1.0"
scalaVersion := "2.11.12"
lazy val root = (project in file(".")).enablePlugins(PlayScala)
scalaSource in ThisScope := baseDirectory.value
mainClass in assembly := Some("play.core.server.ProdServerStart")
fullClasspath in assembly += Attributed.blank(PlayKeys.playPackageAssets.value)
libraryDependencies ++= Seq(
ws,
specs2 % Test,
"com.typesafe.play" %% "play-json" % "2.7.4",
"com.typesafe.play" %% "play-slick" % "4.0.2",
"com.typesafe.play" %% "play-slick-evolutions" % "4.0.2",
"com.typesafe" % "config" % "1.4.0",
"org.mindrot" % "jbcrypt" % "0.4",
"mysql" % "mysql-connector-java" % "8.0.17",
"org.mindrot" % "jbcrypt" % "0.4",
"com.iheart" %% "ficus" % "1.4.7",
"com.typesafe.scala-logging" % "scala-logging_2.11" % "3.9.0"
)
assemblyMergeStrategy in assembly := {
case manifest if manifest.contains("MANIFEST.MF") =>
// We don't need manifest files since sbt-assembly will create
// one with the given settings
MergeStrategy.discard
case referenceOverrides if referenceOverrides.contains("reference-overrides.conf") =>
// Keep the content for all reference-overrides.conf files
MergeStrategy.concat
case x =>
// For all the other files, use the default sbt-assembly merge strategy
val oldStrategy = (assemblyMergeStrategy in assembly).value
oldStrategy(x)
}

Sbt 0.13 to 1.0 - What's the replacement for onLoad in Global for multiprojects?

I am trying to create a new project, this time with sbt 1.0.2 instead of 0.13.x which I used for quite some time now.
There I had a multi-project setup comparable to https://github.com/vmunier/akka-http-with-scalajs-example/blob/master/build.sbt
My problem now is that I always get [error] (projectname/compile:bgRun) No main class detected. when I try to run sbt run
Here is my current build.sbt file:
lazy val generalSettings = Seq(
name := "awesomeproject.tld",
version := "0.1",
scalaVersion := "2.12.3"
)
lazy val client = (project in file("modules/client"))
.settings(generalSettings: _*)
.settings(
name := "client",
libraryDependencies ++= Seq(
"com.lihaoyi" %%% "scalatags" % "0.6.5",
"org.scala-js" %%% "scalajs-dom" % "0.9.2"
)
, scalaJSUseMainModuleInitializer := true
)
.enablePlugins(ScalaJSPlugin, ScalaJSWeb)
.dependsOn(sharedJS)
lazy val server = (project in file("modules/server"))
.settings(generalSettings: _*)
.settings(
name := "server",
scalaJSProjects := Seq(client),
pipelineStages in Assets := Seq(scalaJSPipeline),
// triggers scalaJSPipeline when using compile or continuous compilation
//compile in Compile := ((compile in Compile) dependsOn scalaJSPipeline).value,
WebKeys.packagePrefix in Assets := "public/",
//managedClasspath in Runtime += (packageBin in Assets).value,
libraryDependencies ++= Seq(
"com.typesafe.akka" %% "akka-actor" % "2.5.4",
"com.typesafe.akka" %% "akka-testkit" % "2.5.4" % Test,
"com.typesafe.akka" %% "akka-stream" % "2.5.4",
"com.typesafe.akka" %% "akka-stream-testkit" % "2.5.4" % Test,
"com.typesafe.akka" %% "akka-http" % "10.0.10",
"com.typesafe.akka" %% "akka-http-testkit" % "10.0.10" % Test,
"ch.qos.logback" % "logback-classic" % "1.2.3",
"com.typesafe.scala-logging" %% "scala-logging" % "3.7.2",
"com.lihaoyi" %% "scalatags" % "0.6.5",
"com.vmunier" %% "scalajs-scripts" % "1.1.0"
)
, mainClass := Some("tld.awesomeproject.Main")
)
.dependsOn(sharedJVM)
lazy val shared = (crossProject.crossType(CrossType.Pure) in file("modules/shared"))
.settings(generalSettings: _*)
.settings(
name := "shared"
)
lazy val sharedJS = shared.js
lazy val sharedJVM = shared.jvm
As you can see I tried to solve the problem with setting it explicitly with mainClass := Some("tld.awesomeproject.Main") in the subproject. I also tried to set a root project explicitly, make it dependOn the server but no luck.
I guess the real problem here is, that
onLoad in Global := (Command.command("project server", _: State)) compose (onLoad in Global) does not work in sbt 1.0.2 anymore.
I checked the Command class, but I am not wiser after. There simply is no more method that gives back a state.
Can anyone shed like on this? What I want is to run a server that sends some javascript to the client... that shouldn't be black magic, in fact everything worked like a charm in my 0.13. project.
This should work:
onLoad in Global ~= (_ andThen ("project server" :: _))
Reference: https://github.com/sbt/sbt/issues/1224#issuecomment-331840364

java.lang.NoSuchMethodError in Scalatra using Scalate with Markdown

So I have a Scalatra app (using Scalatra 2.2.1). I'm building views using Scalate; I've decided to go with the Jade/Markdown one-two. Only one problem: if I try to use markdown in a jade template (started with the :markdown tag), I get this:
scala.Predef$.any2ArrowAssoc(Ljava/lang/Object;)Lscala/Predef$ArrowAssoc;
java.lang.NoSuchMethodError: scala.Predef$.any2ArrowAssoc(Ljava/lang/Object;)Lscala/Predef$ArrowAssoc;
at org.fusesource.scalamd.Markdown$.<init>(md.scala:119)
at org.fusesource.scalamd.Markdown$.<clinit>(md.scala:-1)
at org.fusesource.scalate.filter.ScalaMarkdownFilter$.filter(ScalaMarkdownFilter.scala:32)
at org.fusesource.scalate.RenderContext$class.filter(RenderContext.scala:276)
at org.fusesource.scalate.DefaultRenderContext.filter(DefaultRenderContext.scala:30)
at org.fusesource.scalate.RenderContext$class.value(RenderContext.scala:235)
at org.fusesource.scalate.DefaultRenderContext.value(DefaultRenderContext.scala:30)
at templates.views.$_scalate_$about_jade$.$_scalate_$render(about_jade.scala:37)
at templates.views.$_scalate_$about_jade.render(about_jade.scala:48)
at org.fusesource.scalate.DefaultRenderContext.capture(DefaultRenderContext.scala:92)
at org.fusesource.scalate.layout.DefaultLayoutStrategy.layout(DefaultLayoutStrategy.scala:45)
at org.fusesource.scalate.TemplateEngine$$anonfun$layout$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(TemplateEngine.scala:559)
at org.fusesource.scalate.TemplateEngine$$anonfun$layout$1$$anonfun$apply$mcV$sp$1.apply(TemplateEngine.scala:559)
at org.fusesource.scalate.TemplateEngine$$anonfun$layout$1$$anonfun$apply$mcV$sp$1.apply(TemplateEngine.scala:559)
So that's pretty cool. The error vanishes as soon as I remove the :markdown flag, and beyond that everything compiles (beyond markdown not getting rendered correctly).
Things I know and have found so far:
There's some thought that this error is the biproduct of incompatible Scala versions somewhere in the build. My build.scala defines Scala version as 2.10.0, which Scalatra is explicitly compatible with.
...that said, I have no idea which version of Scalate Scalatra pulls in, and my attempts to override it have not worked so far. I know the current stable of Scalate (1.6.1) is only compatible up to Scala 2.10.0 -- but that's what I'm using.
I am, however, sure that my classpath is clean. I have no conflicting Scala versions. Everything is 2.10.0 in the dependencies.
Has anybody worked with this one before? Any ideas?
EDIT
Per request, here are my build definitions:
//build.sbt
libraryDependencies += "org.scalatest" %% "scalatest" % "2.0.M5b" % "test"
libraryDependencies += "org.twitter4j" % "twitter4j-core" % "3.0.3"
libraryDependencies += "org.fusesource.scalamd" % "scalamd" % "1.5"
//build.properties
sbt.version=0.12.3
//build.scala
import sbt._
import Keys._
import org.scalatra.sbt._
import org.scalatra.sbt.PluginKeys._
import com.mojolly.scalate.ScalatePlugin._
import ScalateKeys._
object TheRangeBuild extends Build {
val Organization = "com.gastove"
val Name = "The Range"
val Version = "0.1.0-SNAPSHOT"
val ScalaVersion = "2.10.0"
val ScalatraVersion = "2.2.1"
lazy val project = Project (
"the-range",
file("."),
settings = Defaults.defaultSettings ++ ScalatraPlugin.scalatraWithJRebel ++ scalateSettings ++ Seq(
organization := Organization,
name := Name,
version := Version,
scalaVersion := ScalaVersion,
resolvers += Classpaths.typesafeReleases,
libraryDependencies ++= Seq(
"org.scalatra" %% "scalatra" % ScalatraVersion,
"org.scalatra" %% "scalatra-scalate" % ScalatraVersion,
"org.scalatra" %% "scalatra-specs2" % ScalatraVersion % "test",
"ch.qos.logback" % "logback-classic" % "1.0.6" % "runtime",
"org.eclipse.jetty" % "jetty-webapp" % "8.1.8.v20121106" % "compile;container",
"org.eclipse.jetty.orbit" % "javax.servlet" % "3.0.0.v201112011016" % "compile;container;provided;test" artifacts (Artifact("javax.servlet", "jar", "jar"))
),
scalateTemplateConfig in Compile <<= (sourceDirectory in Compile){ base =>
Seq(
TemplateConfig(
base / "webapp" / "WEB-INF" / "templates",
Seq.empty, /* default imports should be added here */
Seq(
Binding("context", "_root_.org.scalatra.scalate.ScalatraRenderContext", importMembers = true, isImplicit = true)
), /* add extra bindings here */
Some("templates")
)
)
}
) ++ seq(com.typesafe.startscript.StartScriptPlugin.startScriptForClassesSettings: _*)
)
}
When using the markdown filter, you need to add the scalamd library as runtime dependency:
"org.fusesource.scalamd" %% "scalamd" % "1.6"
The most recent version can be found on Maven Central
Also you can delete the build.sbt file and put the dependencies into build.scala file which makes things a bit simpler.
import sbt._
import Keys._
import org.scalatra.sbt._
import org.scalatra.sbt.PluginKeys._
import com.mojolly.scalate.ScalatePlugin._
import ScalateKeys._
object TheRangeBuild extends Build {
val Organization = "com.gastove"
val Name = "The Range"
val Version = "0.1.0-SNAPSHOT"
val ScalaVersion = "2.10.0"
val ScalatraVersion = "2.2.1"
lazy val project = Project(
"the-range",
file("."),
settings = Defaults.defaultSettings ++ ScalatraPlugin.scalatraWithJRebel ++ scalateSettings ++ Seq(
organization := Organization,
name := Name,
version := Version,
scalaVersion := ScalaVersion,
resolvers += Classpaths.typesafeReleases,
libraryDependencies ++= Seq( // adding this Seq to the libraryDependencies
"org.scalatra" %% "scalatra" % ScalatraVersion,
"org.scalatra" %% "scalatra-scalate" % ScalatraVersion,
"org.scalatra" %% "scalatra-specs2" % ScalatraVersion % "test",
"ch.qos.logback" % "logback-classic" % "1.0.6" % "runtime",
"org.eclipse.jetty" % "jetty-webapp" % "8.1.8.v20121106" % "compile;container",
"org.scalatest" %% "scalatest" % "2.0.M5b" % "test",
"org.twitter4j" % "twitter4j-core" % "3.0.3",
"org.fusesource.scalamd" % "scalamd" % "1.6",
"org.eclipse.jetty.orbit" % "javax.servlet" % "3.0.0.v201112011016" % "compile;container;provided;test" artifacts (Artifact("javax.servlet", "jar", "jar"))
),
scalateTemplateConfig in Compile <<= (sourceDirectory in Compile){ base =>
Seq(
TemplateConfig(
base / "webapp" / "WEB-INF" / "templates",
Seq.empty, /* default imports should be added here */
Seq(
Binding("context", "_root_.org.scalatra.scalate.ScalatraRenderContext", importMembers = true, isImplicit = true)
), /* add extra bindings here */
Some("templates")
)
)
}
) ++ seq(com.typesafe.startscript.StartScriptPlugin.startScriptForClassesSettings: _*)
)
}
This comes from:
libraryDependencies += "org.fusesource.scalamd" % "scalamd" % "1.5"
Looking at the scalamd-1.5 pom.xml, it is built against Scala 2.8.1, which is not compatible with 2.10.
Dependency resolution keeps 2.10 and discard the 2.8.1 dependency, and you end up with this classpath issue.
The only solution you have is to try and build a new scalamd version against Scala 2.10, potentially fix a few things to get it there, and then publish it (at least locally).