I am transitioning from Java to Scala and trying to write some functional/regression tests. Although, initially, I am trying to fix some pre-existing tests as it seems when tests are running, they are using actual application.config instead of testApplication.conf.
This is the way the way the tests are written:
class ClientValidationTests extends PlaySpec with OneAppPerSuite with MockitoSugar {
val myConfigFile = new File("test/resources/testApplication.conf")
val parsedConfig = ConfigFactory.parseFile(myConfigFile)
val configuration = ConfigFactory.load(parsedConfig)
override lazy val app: Application = new GuiceApplicationBuilder()
//.configure(Configuration.apply(ConfigFactory.parseFile(new File("test/resources/testApplication.conf")).resolve()))
.overrides(bind[Configuration].toInstance(Configuration(configuration)))
.overrides(bind[ApprovalsClient].to[MockApprovalsClient])
.overrides(bind[ClientDAO].to[MockClientDAO])
.build()
"The validation in the getListForClientWeb method for the web service" must {
"throw a UserNotPermittedForActionException when an client does not belong to the provided org" in {
val clientWebService = app.injector.instanceOf[ClientWebService]
val clientDao = app.injector.instanceOf[ClientDAO]
val clientId = -1
//Verify that mock dao routes do as expected
clientDao.getId(clientId) mustBe List(-1)
}
}
}
This is the file structure where application.conf and testApplication.conf exists:
project-root
- app
- conf
-application.conf
- test
- resources
- testApplication.conf
These are libararies being used here:
scalaVersion := "2.11.6"
ivyScala := ivyScala.value map { _.copy(overrideScalaVersion = true) }
libraryDependencies ++= Seq(
jdbc,
//anorm dependency
"com.typesafe.play" %% "anorm" % "2.5.3",
"org.scalatestplus" %% "play" % "1.4.0-M4" % "test",
"org.mockito" % "mockito-core" % "1.10.19" % "test",
"com.microsoft.sqlserver" % "mssql-jdbc" % "6.4.0.jre8"
)
Any insight?
You can replace a real conf/application.conf with a conf/application.test.conf file by putting this in your sbt build.sbt file:
javaOptions in Test += "-Dconfig.file=conf/application.test.conf"
Related
It work fine when using generic class.
But get java.lang.NoClassDefFoundError: scala/Product$class error after change class to case class.
Not sure is sbt packaging problem or code problem.
When I'm using:
sbt
scala: 2.11.12
java: 8
sbt assembly to package
package example
import java.util.Properties
import java.nio.charset.StandardCharsets
import org.apache.flink.api.scala._
import org.apache.flink.streaming.util.serialization.{DeserializationSchema, SerializationSchema}
import org.apache.flink.streaming.api.scala.{DataStream, StreamExecutionEnvironment}
import org.apache.flink.streaming.connectors.kafka.{FlinkKafkaConsumer, FlinkKafkaProducer}
import org.apache.flink.streaming.api.watermark.Watermark
import org.apache.flink.streaming.api.functions.AssignerWithPunctuatedWatermarks
import org.apache.flink.api.common.typeinfo.TypeInformation
import Config._
case class Record(
id: String,
startTime: Long
) {}
class RecordDeSerializer extends DeserializationSchema[Record] with SerializationSchema[Record] {
override def serialize(record: Record): Array[Byte] = {
return "123".getBytes(StandardCharsets.UTF_8)
}
override def deserialize(b: Array[Byte]): Record = {
Record("1", 123)
}
override def isEndOfStream(record: Record): Boolean = false
override def getProducedType: TypeInformation[Record] = {
createTypeInformation[Record]
}
}
object RecordConsumer {
def main(args: Array[String]): Unit = {
val config : Properties = {
var p = new Properties()
p.setProperty("zookeeper.connect", Config.KafkaZookeeperServers)
p.setProperty("bootstrap.servers", Config.KafkaBootstrapServers)
p.setProperty("group.id", Config.KafkaGroupID)
p
}
val env = StreamExecutionEnvironment.getExecutionEnvironment
env.enableCheckpointing(1000)
var consumer = new FlinkKafkaConsumer[Record](
Config.KafkaTopic,
new RecordDeSerializer(),
config
)
consumer.setStartFromEarliest()
val stream = env.addSource(consumer).print
env.execute("record consumer")
}
}
Error
2020-08-05 04:07:33,963 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Discarding checkpoint 1670 of job 4de8831901fa72790d0a9a973cc17dde.
java.lang.NoClassDefFoundError: scala/Product$class
...
build.SBT
First idea is that maybe version is not right.
But every thing work fine if use normal class
Here is build.sbt
ThisBuild / resolvers ++= Seq(
"Apache Development Snapshot Repository" at "https://repository.apache.org/content/repositories/snapshots/",
Resolver.mavenLocal
)
name := "deedee"
version := "0.1-SNAPSHOT"
organization := "dexterlab"
ThisBuild / scalaVersion := "2.11.8"
val flinkVersion = "1.8.2"
val flinkDependencies = Seq(
"org.apache.flink" %% "flink-scala" % flinkVersion % "provided",
"org.apache.flink" %% "flink-streaming-scala" % flinkVersion % "provided",
"org.apache.flink" %% "flink-streaming-java" % flinkVersion % "provided",
"org.apache.flink" %% "flink-connector-kafka" % flinkVersion,
)
val thirdPartyDependencies = Seq(
"com.github.nscala-time" %% "nscala-time" % "2.24.0",
"com.typesafe.play" %% "play-json" % "2.6.14",
)
lazy val root = (project in file(".")).
settings(
libraryDependencies ++= flinkDependencies,
libraryDependencies ++= thirdPartyDependencies,
libraryDependencies += "org.scala-lang" % "scala-compiler" % scalaVersion.value,
)
assembly / mainClass := Some("dexterlab.TelecoDataConsumer")
// make run command include the provided dependencies
Compile / run := Defaults.runTask(Compile / fullClasspath,
Compile / run / mainClass,
Compile / run / runner
).evaluated
// stays inside the sbt console when we press "ctrl-c" while a Flink programme executes with "run" or "runMain"
Compile / run / fork := true
Global / cancelable := true
// exclude Scala library from assembly
assembly / assemblyOption := (assembly / assemblyOption).value.copy(includeScala = false)
autoCompilerPlugins := true
Finally success after I add this line in build.sbt
assembly / assemblyOption := (assemblu / assemblyOption).value.copy(includeScala = true)
To include scala library when running sbt assembly
I am reading this sample which shows me how to generate source code using Slick-CodeGen
https://github.com/slick/slick-codegen-example/blob/master/build.sbt
And while this sample, is good, I want to modify it so that it reads the database config from application.conf using typesafe config.
Otherwise I will have to replicate the database connection configuration here and also in application.conf file.
Does anyone know, how can this sample be modified so that we can use the typesafe config to read the config values from application.conf?
Edit: Based on the suggestion below, I tried the following
I created a file called build.sbt in the project folder
libraryDependencies += "com.typesafe" % "config" % "1.3.1"
modified my main build.sbt file (in project root) as
val slickVersion = "3.1.1"
lazy val mainProject = Project(
id = "FooBar",
base=file("."),
settings = Defaults.coreDefaultSettings ++ Seq(
scalaVersion := "2.11.8",
libraryDependencies ++= Seq(
"com.typesafe.slick" %% "slick" % slickVersion,
"com.typesafe.slick" %% "slick-codegen" % slickVersion,
"mysql" % "mysql-connector-java" % "5.1.35",
"com.typesafe" % "config" % "1.3.1"
),
myConf := {
ConfigFactory.parseFile(new File("src/main/resources/application.conf"))
},
slick <<= slickCodeGenTask,
sourceGenerators in Compile <+= slickCodeGenTask
)
)
lazy val slick = TaskKey[Seq[File]]("gen-tables")
lazy val myConf = settingKey[Config]("The application properties")
lazy val slickCodeGenTask = (sourceManaged, dependencyClasspath in Compile, runner in Compile, streams) map {(dir, cp, r, s) =>
val outputDir = (dir / "slick").getPath
val username = myConf.value.getString("mysql.username")
val password = myConf.value.getString("mysql.password")
val port = myConf.value.getInt("mysql.port")
val db = myConf.value.getString("mysql.db")
val server = myConf.value.getString("mysql.server")
val url = s"jdbc:mysql://$server:$port/$db?username=$username&password=$password"
val jdbcDriver = myConf.value.getString("mysql.jdbcDriver")
val slickDriver = myConf.value.getString("mysql.slickDriver")
val pkg = "sql"
val fname = outputDir + "/db/Tables.scala"
toError(r.run("slick.codegen.SourceCodeGenerator", cp.files, Array(slickDriver, jdbcDriver, url, outputDir, pkg), s.log))
Seq(file(fname))
}
But it cannot resolve the Config and the ConfigFactory classes.
Declare a dependency on Typesafe Config in project/build.sbt:
libraryDependencies += "com.typesafe" % "config" % "1.3.1"
And define a setting holding your config file in build.sbt:
lazy val myConf = settingKey[Config]("The application properties")
myConf := {
ConfigFactory.parseFile(new File("src/main/resources/application.conf"))
}
Now you can use myConf.value.getString("xyz") to get hold of your configuration values in other tasks or settings.
I'm having difficulty implementing a custom test reporter in Scala using SBT.
I have a basic SBT project set up with this build.sbt file:
name := "Test Framework"
version := "0.1"
scalaVersion := "2.11.7"
scalacOptions += "-deprecation"
scalacOptions += "-feature"
libraryDependencies += "org.scalatest" % "scalatest_2.11" % "2.2.4" % "test"
libraryDependencies += "org.scalacheck" %% "scalacheck" % "1.12.4" % "test"
testFrameworks += new TestFramework("custom.framework.MyTest")
testListeners += new MyTest()
My MyTest.scala class is located in the projectsfolder under projects/custom/framework/MyTest.scala and looks like this:
import sbt.TestsListener._
import sbt.TestReportListener._
class MyTest extends TestReportListener {
def doInit(): Unit = {
println("Starting Test")
}
def testEvent(event: TestEvent): Unit = {
println(event.result.get)
}
def startGroup(name: String): Unit = {
println("Group Started")
}
}
The documentation here is sparse, and i'm obviously missing something. It states that i need to
Specify the test reporters you want to use by overriding the testListeners setting in your project definition. Where customTestListener is of type sbt.TestReportListener.
This should be possible by doing testListeners += customTestListener.
Since my class MyTest extends TestReportListener i thought i could just do testListeners += custom.framework.MyTest or testListeners += new custom.framework.MyTest, but this clearly is not the case.
I'm running sbt test to execute the tests, and the output is
error: not found: value custom
testListeners += new custom.framework.MyTest
I'm not sure how this is supposed to work.
Does anyone know how this is done correctly?
I did not figure out why stating the package did not work, but moving the MyTest.scala file out to the project directory, and removing the custom.frameworkpackage made it work.
I'm trying to get SBT to build a project that could have more than one unmanaged directory. If I had a single directory, I could easily do it like this:
unmanagedBase := file( "custom-libs" ).getAbsoluteFile
But since I have two directories with unmanaged jars, I need to be able to add them all. I have found some information in here, but still doesn't seem useful for my full .scala file build.
I have created a simple project that shows the issue in here. And below is my Build.scala file.
UPDATE
I've got some help form the sbt-users list and have been able to define the unmanaged-jars correctly, but the code still doesn't compile (but sbt show unmanaged-jars shows the files correctly).
import sbt._
import com.github.siasia._
import PluginKeys._
import Keys._
object Build extends sbt.Build {
import Dependencies._
val unmanagedListing = unmanagedJars := {
Dependencies.listUnmanaged( file(".").getAbsoluteFile )
}
lazy val myProject = Project("spray-template", file("."))
.settings(WebPlugin.webSettings: _*)
.settings(port in config("container") := 8080)
.settings(
organization := "com.example",
version := "0.9.0-RC1",
scalaVersion := "2.9.1",
scalacOptions := Seq("-deprecation", "-encoding", "utf8"),
resolvers ++= Dependencies.resolutionRepos,
libraryDependencies ++= Seq(
Compile.akkaActor,
Compile.sprayServer,
Test.specs2,
Container.jettyWebApp,
Container.akkaSlf4j,
Container.slf4j,
Container.logback
),
unmanagedListing
)
}
object Dependencies {
val resolutionRepos = Seq(
ScalaToolsSnapshots,
"Typesafe repo" at "http://repo.typesafe.com/typesafe/releases/",
"spray repo" at "http://repo.spray.cc/"
)
def listUnmanaged( base : RichFile ) : Keys.Classpath = {
val baseDirectories = (base / "custom-libs") +++ ( base / "custom-libs2" )
(baseDirectories ** "*.jar").classpath
}
object V {
val akka = "1.3"
val spray = "0.9.0-RC1"
val specs2 = "1.7.1"
val jetty = "8.1.0.v20120127"
val slf4j = "1.6.4"
val logback = "1.0.0"
}
object Compile {
val akkaActor = "se.scalablesolutions.akka" % "akka-actor" % V.akka % "compile"
val sprayServer = "cc.spray" % "spray-server" % V.spray % "compile"
}
object Test {
val specs2 = "org.specs2" %% "specs2" % V.specs2 % "test"
}
object Container {
val jettyWebApp = "org.eclipse.jetty" % "jetty-webapp" % V.jetty % "container"
val akkaSlf4j = "se.scalablesolutions.akka" % "akka-slf4j" % V.akka
val slf4j = "org.slf4j" % "slf4j-api" % V.slf4j
val logback = "ch.qos.logback" % "logback-classic" % V.logback
}
}
I just post the fragment from my build.sbt file, using sbt 0.11.x. It could probably be refactored a bit.
unmanagedJars in Compile <++= baseDirectory map { base =>
val libs = base / "lib"
val dirs = (libs / "batik") +++ (libs / "libtw") +++ (libs / "kiama")
(dirs ** "*.jar").classpath
}
You can add additional paths to the list of folders to scan for unmanaged dependencies. For example, to look in a folder called "config" in addition to "lib" for the run task, you can add the following. For the compile task change Runtime to Compile.
unmanagedClasspath in Runtime <+= (baseDirectory) map {
bd => Attributed.blank(bd / "config")
}
As answered by Eugene Vigdorchik, what made it work as the following code:
import sbt._
import com.github.siasia._
import PluginKeys._
import Keys._
object Build extends sbt.Build {
import Dependencies._
var unmanagedListing = unmanagedJars in Compile := {
Dependencies.listUnmanaged( file(".").getAbsoluteFile )
}
lazy val myProject = Project("spray-template", file("."))
.settings(WebPlugin.webSettings: _*)
.settings(port in config("container") := 8080)
.settings(
organization := "com.example",
version := "0.9.0-RC1",
scalaVersion := "2.9.1",
scalacOptions := Seq("-deprecation", "-encoding", "utf8"),
resolvers ++= Dependencies.resolutionRepos,
libraryDependencies ++= Seq(
C.akkaActor,
C.sprayServer,
Test.specs2,
Container.jettyWebApp,
Container.akkaSlf4j,
Container.slf4j,
Container.logback
),
unmanagedListing
)
}
object Dependencies {
val resolutionRepos = Seq(
ScalaToolsSnapshots,
"Typesafe repo" at "http://repo.typesafe.com/typesafe/releases/",
"spray repo" at "http://repo.spray.cc/"
)
def listUnmanaged( base : RichFile ) : Keys.Classpath = {
val baseDirectories = (base / "custom-libs") +++ ( base / "custom-libs2" )
(baseDirectories ** "*.jar").classpath
}
object V {
val akka = "1.3"
val spray = "0.9.0-RC1"
val specs2 = "1.7.1"
val jetty = "8.1.0.v20120127"
val slf4j = "1.6.4"
val logback = "1.0.0"
}
object C {
val akkaActor = "se.scalablesolutions.akka" % "akka-actor" % V.akka % "compile"
val sprayServer = "cc.spray" % "spray-server" % V.spray % "compile"
}
object Test {
val specs2 = "org.specs2" %% "specs2" % V.specs2 % "test"
}
object Container {
val jettyWebApp = "org.eclipse.jetty" % "jetty-webapp" % V.jetty % "container"
val akkaSlf4j = "se.scalablesolutions.akka" % "akka-slf4j" % V.akka
val slf4j = "org.slf4j" % "slf4j-api" % V.slf4j
val logback = "ch.qos.logback" % "logback-classic" % V.logback
}
}
Source repo with the full example available on Github.
Here's a general solution to recursive loading of unmanaged JARs (for sbt 0.13.x): https://stackoverflow.com/a/29357699/1348306
I'm trying to set up a scala sbt project with the lift web framework. I'm using
scala 2.9.0-1
sbt 0.10.1
lift 2.3
xsbt-web-plugin 0.1.1 (which is only on scala 2.8.1, see end of question)
(quite recent versions I know).
I followed http://d.hatena.ne.jp/k4200/20110711/1310354698 and https://github.com/siasia/xsbt-web-plugin/blob/master/README.md to obtain the following sbt configuration files:
project/build.properties
sbt.version=0.10.1
project/plugins/build.sbt
resolvers += "Web plugin repo" at "http://siasia.github.com/maven2"
libraryDependencies <+= sbtVersion(v => "com.github.siasia" % "xsbt-web-plugin_2.8.1" % ("0.1.1-"+v))
project/Build.scala
import sbt._
import Keys._
object BuildSettings {
val buildOrganization = "xbaz"
val buildScalaVersion = "2.9.0-1"
val buildVersion = "0.0.1"
val buildSettings = Defaults.defaultSettings ++ Seq (
organization := buildOrganization,
scalaVersion := buildScalaVersion,
version := buildVersion)
}
object Resolvers {
val webPluginRepo = "Web plugin repo" at "http://siasia.github.com/maven2"
val jettyRepo = "Jetty Repo" at "http://repo1.maven.org/maven2/org/mortbay/jetty"
}
object Dependencies {
// web plugin
val webPluginDeps = Seq(
"org.mortbay.jetty" % "jetty" % "6.1.26" % "jetty", // The last part is "jetty" not "test".
"javax.servlet" % "servlet-api" % "2.5" % "provided->default"
)
val liftDeps = {
val liftVersion = "2.3" // I'll switch to 2.3 soon!
Seq(
"net.liftweb" % "lift-webkit_2.8.1" % liftVersion % "compile->default",
"net.liftweb" % "lift-mapper_2.8.1" % liftVersion % "compile->default"
)
}
val scalaTest = "org.scalatest" % "scalatest_2.9.0" % "1.6.1" % "test"
val apacheHttpClient = "org.apache.httpcomponents" % "httpclient" % "4.1.1"
val apacheHttpCore = "org.apache.httpcomponents" % "httpcore" % "4.1.1"
// Logging
lazy val grizzled = "org.clapper" % "grizzled-slf4j_2.8.1" % "0.5"
lazy val junit = "junit" % "junit" % "4.8" % "test"
lazy val logback_core = "ch.qos.logback" % "logback-core" % "0.9.24" % "compile" //LGPL 2.1
lazy val logback_classic = "ch.qos.logback" % "logback-classic" % "0.9.24" % "compile" //LGPL 2.1
lazy val log4j_over_slf4j = "org.slf4j" % "log4j-over-slf4j" % "1.6.1"
val logDeps = Seq(grizzled, log4j_over_slf4j, logback_core, logback_classic)
}
object MyBuild extends Build {
import com.github.siasia.WebPlugin._ // web plugin
import BuildSettings._
import Dependencies._
import Resolvers._
//End dependencies
lazy val root = Project("root", file(".") , settings = buildSettings ++
Seq( name := "foo")
) aggregate(core, cli, web)
// mainClass:= Some("Main"))
lazy val core : Project = Project("core", file("core"), delegates = root :: Nil, settings = buildSettings ++
Seq(
name := "foo-core",
libraryDependencies ++= logDeps ++ Seq(scalaTest, apacheHttpClient, apacheHttpCore)
)
)
lazy val cli: Project = Project("cli", file("cli"), settings = buildSettings ++
Seq(
name := "foo-cli",
libraryDependencies ++= logDeps ++ Seq(apacheHttpClient),
fork in run := true,
javaOptions in run += "-Djava.library.path=/home/jolivier/Projets/asknow/lib/jnotify-lib-0.93"
)) dependsOn(core) settings(
)
lazy val web: Project = Project("web", file("web"), settings = buildSettings ++
Seq (resolvers := Seq(webPluginRepo, jettyRepo),
name := "foo-http",
libraryDependencies ++= logDeps ++ webPluginDeps ++ liftDeps
) ++
webSettings
) dependsOn(core)
}
When I try sbt jetty-run I get the following error message:
[error] Not a valid command: jetty-run
[error] Not a valid project ID: jetty-run
[error] Not a valid configuration: jetty-run
[error] Not a valid key: jetty-run (similar: jetty-port, jetty-context, run)
[error] jetty-run
[error]
So I noticed that some jetty-* commands do exist, but not the one I want, so I printed webSettings which is supposed to contain all these new settings and it contains jetty-context and jetty-port, as well as jetty-configuration and others, but not jetty-run :s.
What did I go wrong to not have jetty-run?
I tried switching to scala-2.8.1 since the web plugin is currently only on scala 2.8.1, by changing my buildScalaVersion variable but that didn't change anything. Do you have any idea?
Thanks in advance for your help
Tasks are aggregated; commands are not.
jetty-run is a command. It is only available in the context of the sub-project with the web plugin settings.
> project web
> jetty-run
Once it is running, you can use the prepare-webapp task to redeploy the webapp. This can be run from the context of the root project, because it aggregates the web project.