i am using scala 2.13 and i have to use Gatling load tests, Gatling does not support Scala-2.13 for that i looked into sbt cross build and how to cross build library dependency
i have two Gatling dependencies
"io.gatling.highcharts" % "gatling-charts-highcharts" % "3.3.1",
"io.gatling" % "gatling-test-framework" % "3.3.1"
these libraries support Scala 2.12 for that i am doing something like this
build.sbt
lazy val scala212 = "2.12.10"
lazy val scala213 = "2.13.1"
scalaVersion := scala213
lazy val supportedScalaVersions = List(scala213, scala212)
lazy val root = (project in file("."))
.settings(
crossScalaVersions := supportedScalaVersions,
)
libraryDependencies ++= Seq(
"org.mongodb.scala" %% "mongo-scala-driver" % "2.8.0",
"com.typesafe.akka" %% "akka-actor" % "2.6.3",
"com.typesafe.akka" %% "akka-stream" % "2.6.3"
)
libraryDependencies ++= (scalaBinaryVersion.value match {
case "2.12.10" => Seq(
"io.gatling.highcharts" % "gatling-charts-highcharts" % "3.3.1",
"io.gatling" % "gatling-test-framework" % "3.3.1"
)
case _ => Seq()
}
)
Sbt is not downloading the Gatling dependencies and sbt update command does not show any error it looks like the case 2.12.10 match part is not even executed
what is the right way to do this ? i want to write Gatling simulations in test directory
You got it almost right but scalaBinaryVersion is 2.12 in your case. scalaVersion.value is what you are looking for.
Related
I'm using Scala 2.11 and Spark 2.4.3 for our AWS glue jobs. Recently, I got the error message below in our the build pipeline.
Cause: com.fasterxml.jackson.databind.JsonMappingException: Scala module 2.8.11 requires Jackson Databind version >= 2.8.0 and < 2.9.0
I've tried -
Change the the jackson-module-scala version from 2.8.11 to 2.12.0. This fixed the build pipeline, but I get a different error message in the glue job
Exception in User Class: java.lang.ExceptionInInitializerError.
Change the jackson-module-scala version from 2.8.11 to 2.13.1. I refactored code to get the unit tests, and fix the build pipeline, but in the glue job get the error message below
Exception in User Class: java.lang.VerifyError : Bad return type
I've tried added dependencyOverrides with jackson-module-scala version 2.12.0. The build pipeline would work, but the glue job would fail.
dependencyOverrides += "com.fasterxml.jackson.core" % "jackson-core" % "2.8.7"
dependencyOverrides += "com.fasterxml.jackson.core" % "jackson-databind" % "2.8.7"
dependencyOverrides += "com.fasterxml.jackson.module" % "jackson-module-scala_2.11" % "2.8.7"
Got any ideas what I'm doing wrong or how I can fix my issue?
See below for the build.sbt file
name in ThisBuild := "etl"
organization in ThisBuild := "XXXXXXXXXX"
scalaVersion in ThisBuild := "2.11.12"
version in ThisBuild := "0.2"
addCommandAlias("sanity", ";clean ;compile ;test ;scalafmtAll ;scalastyle ;assembly")
lazy val framework = project.settings(settings, libraryDependencies ++= commonDependencies)
lazy val scripts = project
.settings(settings, libraryDependencies ++= commonDependencies)
.dependsOn(framework % "compile->compile;test->test")
lazy val settings = Seq(
test in assembly := {},
scalacOptions ++= Seq(),
resolvers ++= Seq(
Resolver.sonatypeRepo("releases"),
"aws-glue-etl-artifacts" at "https://aws-glue-etl-artifacts.s3.amazonaws.com/release/"
),
assemblyMergeStrategy in assembly := {
case PathList("META-INF", "io.netty.versions.properties", xs # _*) => MergeStrategy.singleOrError
case "module-info.class" => MergeStrategy.discard
case x: String if x.contains("UnusedStubClass") => MergeStrategy.first
case y =>
val oldStrategy = (assemblyMergeStrategy in assembly).value
oldStrategy(y)
}
)
lazy val commonDependencies = Seq(
"com.amazonaws" % "AWSGlueETL" % "1.0.0" % Provided,
"com.databricks" %% "spark-xml" % "0.8.0",
"org.scalatest" %% "scalatest" % "3.1.1" % Test,
"org.scalamock" %% "scalamock" % "4.4.0" % Test,
"com.fasterxml.jackson.module" %% "jackson-module-scala" % "2.8.11",
"io.netty" % "netty-all" % "4.1.17.Final" % Test,
"org.apache.spark" %% "spark-avro" % "2.4.3",
"com.crealytics" %% "spark-excel" % "0.13.6"
)
fork in ThisBuild := true
parallelExecution in ThisBuild := true
testForkedParallel in ThisBuild := false
logBuffered in ThisBuild := false
testOptions in ThisBuild += Tests.Argument(TestFrameworks.ScalaTest, "-oDFG")
javaOptions ++= Seq(
"-XX:+CMSClassUnloadingEnabled",
"-XX:MaxMetaspaceSize=512M",
"-XX:MetaspaceSize=256M",
"-Xms512M",
"-Xmx2G",
"-XX:MaxPermSize=2048M"
)
The function below uses the Jackson module scala 2.8.11 (up to 2.12.0)
import com.fasterxml.jackson.databind.ObjectMapper
import com.fasterxml.jackson.module.scala.DefaultScalaModule
import com.fasterxml.jackson.module.scala.experimental.ScalaObjectMapper
object JsonUtils {
private val mapper = new ObjectMapper() with ScalaObjectMapper
mapper.registerModule(DefaultScalaModule)
def fromJson[T](json: String)(implicit m: Manifest[T]): T = {
mapper.readValue[T](json)
}
}
U need to override it, like this:
dependencyOverrides ++= Seq(
"com.fasterxml.jackson.core" % "jackson-databind" % versions("jackson"),
"com.fasterxml.jackson.core" % "jackson-core" % versions("jackson"))
I've resolved my issue.
In the function, I changed the library import from
import com.fasterxml.jackson.module.scala.experimental.ScalaObjectMapper
to
import com.fasterxml.jackson.module.scala.ScalaObjectMapper
In the build.sbt, I added the following lines
libraryDependencies += "com.fasterxml.jackson.module" % "jackson-module-scala_2.11" % "2.12.0"
dependencyOverrides += "com.fasterxml.jackson.module" % "jackson-module-scala_2.11" % "2.6.7.1"
I have sbt multi projects and try to set the envVars in the build.sbt for subprojects as following:
envVars in Test := Map("KAFKA_SERVER" -> "localhost:9092")
the test abort with following message:
[info] java.util.NoSuchElementException: None.get
[info] at scala.None$.get(Option.scala:349)
[info] at scala.None$.get(Option.scala:347)
[info] at io.khinkali.auth.AppSpec.<init>(AppSpec.scala:23)
[info] at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
In the test file, I tried to get the value as following:
sys.env.get("KAFKA_SERVER").get
Intellj provides the set the environment variable as following:
How to set an environment variable in sbt for subprojects also?
Update
The root build.sbt looks as following:
name := "bary"
scalacOptions += "-Ypartial-unification"
scalacOptions += "-feature"
addCompilerPlugin("org.scalamacros" % "paradise" % "2.1.0" cross CrossVersion.full)
val Cats = "1.0.0"
val Shiro = "1.4.0"
val Logback = "1.2.3"
val CatsEffect = "0.5"
val Kafka = "1.0.0"
val Bean = "1.9.3"
val Circe = "0.9.0-M3"
val Log4j = "1.7.25"
val ScalaCheck = "1.13.4"
val Scalactic = "3.0.4"
val Scalatest = "3.0.4"
val JavaJwt = "3.3.0"
val Simulacrum = "0.11.0"
val Http4s = "0.18.0-M7"
lazy val commonSettings = Seq(
organization := "io.khinkali",
version := "0.1.0-SNAPSHOT",
scalaVersion := "2.12.4",
envVars in Test := Map("KAFKA_SERVER" -> "localhost:9092"),
fork in Test := true,
libraryDependencies ++= Seq(
"org.slf4j" % "slf4j-simple" % Log4j,
"ch.qos.logback" % "logback-core" % Logback,
"org.apache.shiro" % "shiro-all" % Shiro,
"org.typelevel" %% "cats-core" % Cats,
"org.typelevel" %% "cats-effect" % CatsEffect,
"org.apache.kafka" % "kafka-streams" % Kafka,
"org.apache.kafka" % "kafka-clients" % Kafka,
"commons-beanutils" % "commons-beanutils" % Bean,
"io.circe" %% "circe-core" % Circe,
"io.circe" %% "circe-generic" % Circe,
"io.circe" %% "circe-parser" % Circe,
"io.circe" %% "circe-literal" % Circe,
"com.github.mpilquist" %% "simulacrum" % Simulacrum,
"org.scalactic" %% "scalactic" % Scalactic,
"org.scalatest" %% "scalatest" % Scalatest % "test",
"org.scalacheck" %% "scalacheck" % ScalaCheck % "test",
),
resolvers ++= Seq(
"Sonatype OSS Snapshots" at "https://oss.sonatype.org/content/repositories/snapshots"
),
fork in run := true,
)
lazy val root = (project in file("."))
.settings(commonSettings)
.settings(
name := "bary",
organization := "io.khinkali",
moduleName := "bary"
).
aggregate(
kafka_api,
auth_stream,
rest)
lazy val kafka_api = (project in file("kafka-api")).
settings(commonSettings).
settings(
name := "kafka-api",
moduleName := "kafka-api"
)
lazy val auth_stream = (project in file("auth-stream")).
settings(commonSettings).
settings(
name := "auth-stream",
moduleName := "auth-stream",
libraryDependencies ++= Seq(
"com.auth0" % "java-jwt" % JavaJwt,
)
).dependsOn(kafka_api)
lazy val rest = (project in file("rest")).
settings(commonSettings).
settings(
name := "rest",
moduleName := "rest",
libraryDependencies ++= Seq(
"org.http4s" %% "http4s-dsl" % Http4s,
"org.http4s" %% "http4s-blaze-server" % Http4s,
"org.http4s" %% "http4s-blaze-client" % Http4s,
"org.http4s" %% "http4s-circe" % Http4s,
)
).dependsOn(kafka_api, auth_stream)
I still got the exception.
Any setting you want to apply to a subproject can either be specified just for that subproject or in commonSettings, as shown in build.sbt below.
You did not show your multiproject definitions, so here is a short example. There are many ways of setting up these types of projects, and I am not going to elaborate on the possible ways; this is a complicated topic, and it has evolved a lot over the years, especially recently.
lazy val commonSettings = Seq(
envVars in Test := Map("KAFKA_SERVER" -> "localhost:9092"),
fork in Test := true, // required for envVars task to work
javacOptions ++= Seq(
"-Xlint:deprecation",
"-Xlint:unchecked",
"-source", "1.8",
"-target", "1.8",
"-g:vars"
),
licenses += ("Apache-2.0", url("https://www.apache.org/licenses/LICENSE-2.0.html")),
version := "0.5.0"
)
lazy val demo = project
.settings(commonSettings:_*)
.settings(
name := "demo"
).dependsOn(root)
lazy val root = (project in file("root"))
.settings(commonSettings:_*)
.settings(
name := "root"
)
One more important fact: The environment setting for test will not be respected when run with the IntelliJ IDEA test runner. As a workaround, you can set environment variables in the Run/Debug Configurations -> Environment variables window. When you run sbt test, however, the environment variable specified in build.sbt will be set.
Normally, the only way to apply an environment variable to a process is to start the process with that environment variable. If you want SBT to run your program in an environment with a certain environment variable, it will need to be launched in a new environment. This is called forking.
I am trying to create a new project, this time with sbt 1.0.2 instead of 0.13.x which I used for quite some time now.
There I had a multi-project setup comparable to https://github.com/vmunier/akka-http-with-scalajs-example/blob/master/build.sbt
My problem now is that I always get [error] (projectname/compile:bgRun) No main class detected. when I try to run sbt run
Here is my current build.sbt file:
lazy val generalSettings = Seq(
name := "awesomeproject.tld",
version := "0.1",
scalaVersion := "2.12.3"
)
lazy val client = (project in file("modules/client"))
.settings(generalSettings: _*)
.settings(
name := "client",
libraryDependencies ++= Seq(
"com.lihaoyi" %%% "scalatags" % "0.6.5",
"org.scala-js" %%% "scalajs-dom" % "0.9.2"
)
, scalaJSUseMainModuleInitializer := true
)
.enablePlugins(ScalaJSPlugin, ScalaJSWeb)
.dependsOn(sharedJS)
lazy val server = (project in file("modules/server"))
.settings(generalSettings: _*)
.settings(
name := "server",
scalaJSProjects := Seq(client),
pipelineStages in Assets := Seq(scalaJSPipeline),
// triggers scalaJSPipeline when using compile or continuous compilation
//compile in Compile := ((compile in Compile) dependsOn scalaJSPipeline).value,
WebKeys.packagePrefix in Assets := "public/",
//managedClasspath in Runtime += (packageBin in Assets).value,
libraryDependencies ++= Seq(
"com.typesafe.akka" %% "akka-actor" % "2.5.4",
"com.typesafe.akka" %% "akka-testkit" % "2.5.4" % Test,
"com.typesafe.akka" %% "akka-stream" % "2.5.4",
"com.typesafe.akka" %% "akka-stream-testkit" % "2.5.4" % Test,
"com.typesafe.akka" %% "akka-http" % "10.0.10",
"com.typesafe.akka" %% "akka-http-testkit" % "10.0.10" % Test,
"ch.qos.logback" % "logback-classic" % "1.2.3",
"com.typesafe.scala-logging" %% "scala-logging" % "3.7.2",
"com.lihaoyi" %% "scalatags" % "0.6.5",
"com.vmunier" %% "scalajs-scripts" % "1.1.0"
)
, mainClass := Some("tld.awesomeproject.Main")
)
.dependsOn(sharedJVM)
lazy val shared = (crossProject.crossType(CrossType.Pure) in file("modules/shared"))
.settings(generalSettings: _*)
.settings(
name := "shared"
)
lazy val sharedJS = shared.js
lazy val sharedJVM = shared.jvm
As you can see I tried to solve the problem with setting it explicitly with mainClass := Some("tld.awesomeproject.Main") in the subproject. I also tried to set a root project explicitly, make it dependOn the server but no luck.
I guess the real problem here is, that
onLoad in Global := (Command.command("project server", _: State)) compose (onLoad in Global) does not work in sbt 1.0.2 anymore.
I checked the Command class, but I am not wiser after. There simply is no more method that gives back a state.
Can anyone shed like on this? What I want is to run a server that sends some javascript to the client... that shouldn't be black magic, in fact everything worked like a charm in my 0.13. project.
This should work:
onLoad in Global ~= (_ andThen ("project server" :: _))
Reference: https://github.com/sbt/sbt/issues/1224#issuecomment-331840364
I need to cross-build a project with both Scala 2.10 and 2.11 (and ultimately 2.12 also). Some packages that were part of 2.10, for example parser-combinators, are packaged independently in 2.11. Thus they are not required to be mentioned in the 2.10 build but are required in 2.11. Furthermore, there may be more than one such package, which are required to use a common version.
I found the documentation re: cross-building on the SBT site to be somewhat lacking in helpfulness here. And, although, there are several StackOverflow Q&As which relate to this subject, I could not find one that addressed this specific point.
The solution is as follows (showing only the relevant part of build.sbt):
scalaVersion := "2.10.6"
crossScalaVersions := Seq("2.10.6","2.11.8")
val scalaModules = "org.scala-lang.modules"
val scalaModulesVersion = "1.0.4"
val akkaGroup = "com.typesafe.akka"
lazy val akkaVersion = SettingKey[String]("akkaVersion")
lazy val scalaTestVersion = SettingKey[String]("scalaTestVersion")
akkaVersion := (scalaBinaryVersion.value match {
case "2.10" => "2.3.15"
case "2.11" => "2.4.1"
})
scalaTestVersion := (scalaBinaryVersion.value match {
case "2.10" => "2.2.6"
case "2.11" => "3.0.1"
})
libraryDependencies ++= (scalaBinaryVersion.value match {
case "2.11" => Seq(
scalaModules %% "scala-parser-combinators" % scalaModulesVersion,
scalaModules %% "scala-xml" % scalaModulesVersion,
"com.typesafe.scala-logging" %% "scala-logging" % "3.4.0"
)
case _ => Seq()
}
)
libraryDependencies ++= Seq(
akkaGroup %% "akka-actor" % akkaVersion.value % "test",
"org.scalatest" %% "scalatest" % scalaTestVersion.value % "test"
)
Note that this solution also addresses the problem of how to set the version of a dependency(ies) according to the binary version. I think this may be addressed elsewhere in Stackoverflow but here it is all in the same place.
I have dependencies in my build.sbt that don't have specific builds for the Scala 2.9.0-1 I'm using, instead I'm supposed to use the build for 2.9.0. How to configure the build so that it determines that without specifying the exact version for each dependency? For instance subcut doesn't have a build for 2.9.0-1.
Some lines off my build.sbt:
...
scalaVersion := "2.9.0-1"
libraryDependencies ++= Seq(
"org.scala-tools" %% "subcut" % "0.8"
)
...
I'd rather avoid this:
"org.scala-tools" % "subcut_2.9.0" % "0.8"
Something along the lines of specifying multiple versions it'd try in the specified order.
Here's what I've done:
libraryDependencies <++= (scalaVersion) { (v) =>
val scalaVersionString = v match {
case "2.9.0-1" => "2.9.0"
case _ => v
}
Seq(
"org.scala-tools.testing" % ("scalacheck_" + scalaVersionString) % "1.8" % "test" withSources,
"org.specs2" %% "specs2" % "1.3" % "test" withSources,
"com.github.dmlap" %% "sizeof" % "0.1" % "test" from "http://cloud.github.com/downloads/dmlap/jvm-sizeof/jvm-sizeof-0.1.jar"
)
}