dependency resolution with sbt for continuous integration - scala

I'm an open-source Scala developer who wants to minimize the hassle of dependencies when pushing to GitHub and triggering a check with continuous integration (CircleCI). I have two projects where one (A) is dependent on the other (B). B is under development at the same time (as a snapshot). My project A build.sbt file has a dependency on this (snapshot) version of B and of course all works fine on my local machine. When I push to GitHub, it naturally fails as that snapshot file is not available to CircleCI.
I have generally worked around this by putting the jar file into my lib directory (and removing the dependency from build.sbt). I believe this is known as an unmanaged dependency.
My question is this: is there any way of setting up my lib directory so that CircleCI can resolve the (managed) dependency from the lib directory? I have tried putting the ivy structure into lib starting with the top level com.phasmidsoftware, with b_2.13 under that and under that 1.0.4-SNAPSHOT and so on down. That doesn't work. I've attached the build.sbt for project A (called Numeric).
organization := "com.phasmidsoftware"
name := "Number"
version := "1.0.9"
scalaVersion := "2.13.6"
scalacOptions ++= Seq( "-target:jvm-1.8", "-encoding", "UTF-8", "-unchecked", "-deprecation", "-Ywarn-dead-code", "-Ywarn-value-discard", "-Ywarn-unused" )
val scalaTestVersion = "3.2.3"
libraryDependencies += "org.scalatest" %% "scalatest" % scalaTestVersion % "test"
resolvers += "Typesafe Repository" at "https://repo.typesafe.com/typesafe/releases/"
libraryDependencies ++= Seq(
"com.phasmidsoftware" %% "matchers" % "1.0.4-SNAPSHOT",
"org.scala-lang.modules" %% "scala-parser-combinators" % "1.2.0-M1",
"org.apache.commons" % "commons-math3" % "3.6.1",
"org.slf4j" % "slf4j-api" % "1.7.31",
"ch.qos.logback" % "logback-classic" % "1.2.3" % "test",
"org.scalacheck" %% "scalacheck" % "1.14.1" % "test"
)

The answer described in How can sbt pull dependency artifacts from git? is indeed the right answer.
I will just add a couple of caveats.
Make sure that you use the git protocol in the URI of your git repository (as is shown in the other answer);
The mechanism works by cloning the repository (the branch is defined by ...#branch) but, once you've cloned it, sbt won't fetch if appropriate -- you do have to do that explicitly yourself.
The other thing to keep in mind is that the clone(s) are placed in ~/.sbt/1.0/staging/... where 1.0 is based on the sbt version number.
And, of course, don't forget to remove the reference to the other project if you have it in your libraryDependencies.
Here's the relevant part of my build.sbt file:
lazy val root = (project in file(".")).dependsOn(matchers)
lazy val matchers = RootProject(uri("git://github.com/rchillyard/Matchers#V1_0_5"))

Related

How to define Scala API for Kafka Streams as dependency in build.sbt?

I am trying to start a new SBT Scala project and have the following in build.sbt file:
name := "ScalaKafkaStreamsDemo"
version := "1.0"
scalaVersion := "2.12.1"
libraryDependencies += "javax.ws.rs" % "javax.ws.rs-api" % "2.1" artifacts(Artifact("javax.ws.rs-api", "jar", "jar"))
libraryDependencies += "org.apache.kafka" %% "kafka" % "2.0.0"
libraryDependencies += "org.apache.kafka" % "kafka-streams" % "2.0.0"
So according to the GitHub repo, in 2.0.0 I should see the Scala classes/functions etc etc that I want to use, however they just don't seem to be available. Within IntelliJ I can open up the kafka-streams-2.0.0.jar, but I don't see any Scala classes.
Is there another JAR I need to include?
Just while we are on the subject of extra JARs, does anyone know what JAR I need to include to be able to use the EmbeddedKafkaCluster?
The artifact you need is kafka-streams-scala:
libraryDependencies += "org.apache.kafka" %% "kafka-streams-scala" % "2.0.1"
(please use 2.0.1, or even better 2.1.0, as 2.0.0 has some scala API bugs)
To answer your latter question, it's in the test-jar, which you can address using a classifier:
libraryDependencies += "org.apache.kafka" %% "kafka-streams" % "2.0.1" % "test" classifier "test"
But note that this is an internal class and subject to change (or removal) without notice. If at all possible, it's highly recommended that you use the TopologyTestDriver in the test-utils instead:
libraryDependencies += "org.apache.kafka" %% "kafka-streams-test-utils" % "2.0.1" % "test"
It looks like you face the issue of unresolved javax.ws.rs-api dependency that happens with some Java projects that are direct or transitive dependencies of Scala projects that use sbt. I've faced it with Scala projects that use Apache Spark and recently with Kafka Streams (with and without the Scala API).
A workaround that has worked fine for me is to simply exclude the dependency and define it again explicitly.
excludeDependencies += ExclusionRule("javax.ws.rs", "javax.ws.rs-api")
libraryDependencies += "javax.ws.rs" % "javax.ws.rs-api" % "2.1.1"
Make sure that you use the latest and greatest of sbt (i.e. 1.2.7 as of the time of this writing).
With that said, the dependencies in build.sbt should be as follows:
scalaVersion := "2.12.8"
val kafkaVer = "2.1.0"
libraryDependencies += "org.apache.kafka" % "kafka-streams" % kafkaVer
libraryDependencies += "org.apache.kafka" %% "kafka-streams-scala" % kafkaVer
excludeDependencies += ExclusionRule("javax.ws.rs", "javax.ws.rs-api")
libraryDependencies += "javax.ws.rs" % "javax.ws.rs-api" % "2.1.1"
Within IntelliJ I can open up the kafka-streams-2.0.0.jar, but I don't see any Scala classes. Is there another JAR I need to include?
The following dependency is all you need:
libraryDependencies += "org.apache.kafka" %% "kafka-streams-scala" % kafkaVer
You can also use following workaround, that works in my case - more details
here
import sbt._
object PackagingTypePlugin extends AutoPlugin {
override val buildSettings = {
sys.props += "packaging.type" -> "jar"
Nil
}
}

How to attach sources to scala sbt project at Intellij Idea?

I am new at scala. And for start I want to use Intellij 13.1.5 IDE.
However IDE can't attach sources. Here is how it looks for AnyVal:
Search at internet can't find any source.
I tried Attach sources and attach unpacked scala archive. It doesn't work either.
UPDATE:
Here is sbt configuration:
name := "scalatest-selenium"
version := "1.0"
scalaVersion := "2.11.1"
libraryDependencies ++= Seq(
"net.sourceforge.htmlunit" % "htmlunit" % "2.14",
"org.seleniumhq.selenium" % "selenium-java" % "2.42.2",
"org.scalacheck" % "scalacheck_2.10" % "1.11.4" % "test",
"org.scalatest" % "scalatest_2.11" % "2.2.0" % "test"
)
testOptions in Test += Tests.Argument(TestFrameworks.ScalaTest, "-u", "target/test-reports")
How to solve this trouble?
I get rid of this trouble at the following way:
removed the .sbt directory in your Home Folder.
When you run sbt again, the new folder is created in the correct format and the error goes away.

sbt multi-module project global version setting

I am new to SBT and trying to set up a multi-module project. I come across to a situation where I would like to have a single place where I could have defined versions for libs used accross modules. I tried following with creating a custom settingKey - in the root project:
val akkaVersion = SettingKey[String]("Akka version used in our project")
name := "hello-app"
version in ThisBuild := "1.0.0"
organization in ThisBuild := "com.jaksky.hello"
scalaVersion := "2.10.4"
akkaVersion in ThisBuild:= "2.3.4"
// Common settings/definitions for the build
def OurProject(name: String): Project = (
Project(name, file(name))
)
lazy val common = (
OurProject("common")
)
lazy val be_services = (
OurProject("be-services")
dependsOn(common)
)
In project be-services I tried following:
libraryDependencies ++= Seq(
"com.typesafe.akka" %% "akka-actor" % akkaVersion.value,
"com.typesafe.akka" %% "akka-cluster" % akkaVersion.value,
"com.typesafe.akka" %% "akka-kernel" % akkaVersion.value,
"com.typesafe.akka" %% "akka-remote" % akkaVersion.value,
"com.typesafe.akka" %% "akka-slf4j" % akkaVersion.value,
"ch.qos.logback" % "logback-classic" % "1.0.13"
)
The point here is akkaVersion is not visible (akkaVersion is not found - that is the error message).
My questins:
Is settings map shared accross modules? Described issue probably answers -> NO
What is best practice in this situation?
I found following possibilities:
Scala object holding string constants. Seems to me a bit akward as project version is specified in build.sbt so why dependant libs should be hidden somewhere in project/GlobalVersions.scala or so.
Crating a libDepenDency seq which can be reused. That limits flexibility and I do not always want to be dependant on those libs mentioned.
Custom setting seems to be a bit heavy weighed but seems to me as a claean way but wasn't able to make it work.
Just to complete the picture - using SBT 0.13.5
I believe that this is so fundamental problem that I am not the first one facing this issue.
Thanks for helping me out
The definitions in .sbt files are not visible in other .sbt files. In order to share code between .sbt files, define one or more Scala files in the project/ directory of the build root.
[from documentation http://www.scala-sbt.org/]

Can't compile 2.2 to 2.3 Migration

I have been fighting with this for the past few hours and I haven't made any headway at all. It seems no matter what I do, I keep getting the same error.
java.lang.NoClassDefFoundError: play/Project$
My build.sbt
name := "appname"
version := "1.0-SNAPSHOT"
libraryDependencies ++= Seq(
jdbc,
anorm,
cache,
filters,
"org.postgresql" % "postgresql" % "9.3-1100-jdbc4",
"org.mindrot" % "jbcrypt" % "0.3m",
"org.webjars" %% "webjars-play" % "2.3.0-2",
"org.webjars" % "foundation" % "5.3.0",
"org.scalaj" %% "scalaj-http" % "0.3.16"
)
lazy val root = (project in file(".")).enablePlugins(PlayScala)
scalaVersion := "2.10.4"
I have also tried this build.sbt
object ApplicationBuild extends Build {
val appName = "appname"
val appVersion = "1.0-SNAPSHOT"
val appDependencies ++= Seq(
jdbc,
anorm,
cache,
filters,
"org.postgresql" % "postgresql" % "9.3-1100-jdbc4",
"org.mindrot" % "jbcrypt" % "0.3m",
"org.webjars" %% "webjars-play" % "2.3.0-2",
"org.webjars" % "foundation" % "5.3.0",
"org.scalaj" %% "scalaj-http" % "0.3.16"
)
val main = Project(appName, file(".")).enablePlugins(play.PlayScala).settings(
scalaVersion := "2.10.4",
version := appVersion,
libraryDependencies ++= appDependencies
)
}
As well as different tweaks and modifications. Always the same error. The build.properties is set to sbt.version=0.13.5
plugins.sbt
logLevel := Level.Debug
// The Typesafe repository
resolvers += "Typesafe repository" at "http://repo.typesafe.com/typesafe/releases/"
// Use the Play sbt plugin for Play projects
addSbtPlugin("com.typesafe.play" % "sbt-plugin" % "2.3.4")
After running Patrick Mahoney's suggestion and making sure I removed the import.
Errors:
[error] java.lang.NoClassDefFoundError: play/Project$
[error] Use 'last' for the full log.
Project loading failed: (r)etry, (q)uit, (l)ast, or (i)gnore? q
me#me-desktop:~/Projects/appname$ sbt --version
sbt launcher version 0.13.6
me#me-desktop:~/Projects/appname$ find . | grep -r "play.Project"
me#me-desktop:~/Projects/appname$ find . | grep -r "play/Project"
Try cleaning your build project compiled outputs:
rm -rf project/target
or
$> sbt "reload plugins" clean
(in addition to applying James' answer)
Thanks to James and Patrick, but I found the problem. The problem was in the activator-sbt-echo-play-shim.sbt file in the /project directory.
It contained the following:
addSbtPlugin("com.github.mpeltonen" % "sbt-idea" % "0.1.1.3")
I created a second blank 2.3.4 app using Activator and it's activator-sbt-echo-play-shim.sbt listed:
addSbtPlugin("com.github.mpeltonen" % "sbt-idea" % "1.5")
So I updated the one in my project to 1.5 and deleted the idea and eclipse .sbts for good measure and the app was finally able to compile. It auotmatically updated the above to:
addSbtPlugin("com.github.mpeltonen" % "sbt-idea" % "1.5.2")
Now I have other, more expected, issues I need to work out (like anorm). But I'm now able to compile it and get those errors to start working them out. I did not think these files were part of the activator/sbt compile, but I guess they were. I thought they were for debugging. But they made the difference and the changes above worked.
Thanks again to everyone for their help.
Make sure you've updated project/build.properties to have sbt version 0.13.5. Then, remove import play.Project._, it's not needed. That should work.

LIFT setup for web project: generate with mvn, and manage with sbt

I am new to LIFT, and I am trying to find a reliable instructions how to generate and manage LIFT web project with maven and sbt respectively. Can someone please direct me (or provide here) to the up to date instructions how to setup sbt for the maven generated project? From every post what I've red, it looks like the best setup for the LIFT projects: generate with mvn, manage with sbt. Will you agree? (I cannot generate LIFT/web project with sbt. Right? SBT is only good for managing it. Right? ) Every instructions I tried are out of date though. (I obviously can simply download and un-tar the archetype project, but I want to find a more fundamental approach for managing the environment ) Thanks.
While I'm using lift I don't need a maven at all, just SBT. So, it is very useful to read SBT Getting Started section. Also lift wiki contains some information. But be sure that you read material related to the proper SBT version. And finally, you can pay attention to my lift project template on github.
Good Luck with Lift! It's awesome ;)
By following question in you comment I put here some common config from my projects.
So, that is ./project/build.scala as alternative to ./build.sbt
import sbt._
import Keys._
import com.github.siasia._
import PluginKeys._
import WebPlugin._
import WebappPlugin._
object LiftProjectBuild extends Build {
override lazy val settings = super.settings ++ buildSettings
lazy val buildSettings = Seq(
organization := "com.yourorganization",
version := "0.1-SNAPSHOT",
scalaVersion := "2.9.1")
def yourWebSettings = webSettings ++ Seq(
// If you are using jrebel
scanDirectories in Compile := Nil
)
lazy val shade = Project(
id = "project-name",
base = file("."),
settings = defaultSettings ++ yourWebSettings)
lazy val defaultSettings = Defaults.defaultSettings ++ Seq(
name := "project-name",
resolvers ++= Seq(
"Typesafe Repo" at "http://repo.typesafe.com/typesafe/releases",
"Java.net Maven2 Repository" at "http://download.java.net/maven/2/"),
libraryDependencies ++= {
val liftVersion = "2.4-M5"
Seq(
"net.liftweb" %% "lift-webkit" % liftVersion % "compile",
"org.eclipse.jetty" % "jetty-webapp" % "7.5.4.v20111024" % "container",
"org.squeryl" %% "squeryl" % "0.9.5-SNAPSHOT" % "compile",
"ch.qos.logback" % "logback-classic" % "1.0.0" % "compile",
"org.scalatest" %% "scalatest" % "1.6.1" % "test",
"junit" % "junit" % "4.10" % "test")
},
// compile options
scalacOptions ++= Seq("-encoding", "UTF-8", "-deprecation", "-unchecked"),
javacOptions ++= Seq("-Xlint:unchecked", "-Xlint:deprecation"),
// show full stack traces
testOptions in Test += Tests.Argument("-oF")
)
}
./project/build.properties
#Project properties
sbt.version=0.11.1
./project/plugins.sbt
resolvers += Classpaths.typesafeResolver
addSbtPlugin("com.typesafe.sbteclipse" % "sbteclipse" % "1.5.0")
libraryDependencies <+= sbtVersion(v => "com.github.siasia" %% "xsbt-web-plugin" % (v+"-0.2.10"))
Having these three files are enough to configure sbt.
And of course you can run your application by calling container:start
Your mileage may vary, but my experience with sbt has been less than stellar (out-of-date docs, breakage on version changes, etc). The recent improvements to the eclipse scala IDE and corresponding maven and jrebel plugins make it a clear winner compared to using sbt.
If you follow the instructions, you'll get the ability to build/package at the command line, but superior support for eclipse features and fast development.
See the sample project setup at:
https://github.com/awkay/lift_squeryl_demo