I have defined a minimal build.sbt with two custom profiles ‘dev’ and ‘staging’ (what SBT seems to call Configurations). However, when I run SBT with the Configuration that was defined first in the file (dev), both Configuration blocks are executed - and if both modify the same setting, the last one wins (staging).
This seems to break any notion of conditional activation, so what am I doing wrong with SBT?
For reference, I want to emulate the conditionally activated Profiles concept of Maven e.g. mvn test -P staging.
SBT version: 1.2.1
build.sbt:
name := "example-project"
scalaVersion := "2.12.6"
...
fork := true
// Environment-independent JVM property (always works)
javaOptions += "-Da=b"
// Environment-specific JVM property (doesn’t work)
lazy val Dev = config("dev") extend Test
lazy val Staging = config("staging") extend Test
val root = (project in file("."))
.configs(Dev, Staging)
.settings(inConfig(Dev)(Seq(javaOptions in Test += "-Dfoo=bar")))
.settings(inConfig(Staging)(Seq(javaOptions in Test += "-Dfoo=qux")))
Command:
# Bad
sbt test
=> foo=qux
a=b
# Bad
sbt clean dev:test
=> foo=qux
a=b
# Good
sbt clean staging:test
=> foo=qux
a=b
Notice that despite of the inConfig usage you're still setting javaOptions in Test, i.e. in the Test config. If you remove in Test, it works as expected:
...
.settings(inConfig(Dev)(javaOptions += "-Dfoo=bar"))
.settings(inConfig(Staging)(javaOptions += "-Dfoo=qux"))
(also Seq(...) wrapping is unnecessary)
Now in sbt:
> show Test/javaOptions
[info] *
> show Dev/javaOptions
[info] * -Dfoo=bar
> show Staging/javaOptions
[info] * -Dfoo=qux
You can achieve the same result by scoping each setting explicitly (without inConfig wrapping):
.settings(
Dev/javaOptions += "-Dfoo=bar",
Staging/javaOptions += "-Dfoo=qux",
...
)
(here Conf/javaOptions is the same as javaOptions in Conf)
In my SBT (0.13.16) build, I have the following task:
startThing := {
var bin_path = s"${file(".").getAbsolutePath}/bin"
val result = s"$bin_path/start_thing".!
if (result != 0)
throw new RuntimeException("Could not start Thing..")
true
}
And start_thing contains:
(run_subprocess &)
and my build hangs.
I can see that start_thing exits (the process table does not have it as an entry) but adding some printlns to the task shows that it's stuck on val result = s"$bin_path/start_thing".!.
If I kill the run_subprocess process then SBT unblocks and runs normally.
In this particular case, run_subprocess has set up some Kubernetes port-forwarding that needs to be there in order for subsequent tests to work.
Try daemonising the background process like so
(run_subprocess >/dev/null 2>&1 &)
The issue could be output from run_subprocess still going to sbt parent as suggested here.
I was able to replicate the issue in both sbt 0.13.17 and 1.0.2. Daemonising worked in both.
Regardless of my comment, in my case the reason for the hanging was ACTUALLY the potential leaving of open STDOUT , STDERR handles in a daemon started by the script , that is OK:
/usr/local/bin/minio server "$minio_data_dir" > /dev/null 2>&1 & # and start the server
and NOT ok:
/usr/local/bin/minio server "$minio_data_dir" 2>&1 & # and start the server
So the hanging occurred randomly EVEN with the accepted answers start in the background ...
Thus this solution needed NOT, any wrapper bash scripts ... This is how the code looked in the build.sbt
lazy val startLocalS3 = inputKey[Unit]("localS3")
lazy val startLocalS3Task = TaskKey[Unit]("localS3", "create local s3")
lazy val core: Project = project
.in(file("."))
.settings(
name := "rfco-core",
startLocalS3Task := {
val cmd: Seq[String] = Seq("bash" , "-c" , "./CI/start-s3-svr.sh")
import sys.process._
cmd.mkString(" ").!!
},
fork in startLocalS3Task := true,
compile.in(Compile) := (compile in Compile).dependsOn((startLocalS3Task)).value
// you might want to use Test scope ^^ here
)
I have a subproject in my build.sbt with a rather long setting for initialCommands, comprising a list of imports and some definitions. I'd like to test this as part of regular CI, because otherwise I won't notice breaking changes after refactoring code. It is not clear to me how to do so.
Just running sbt console doesn't seem to cut it, because there is always a "successful" exit code even when code doesn't compile.
Moving the code out into an object defined in a special source file won't help because I need the list of imports to be present (and I don't want to cakeify my whole code base).
Moving the code out into a source file and then loading that with :load also always gives a successful exit code.
I found out about scala -e but that does strange things on my machine (see the error log below).
This is Scala 2.12.
$ scala -e '1'
cat: /release: No such file or directory
Exception in thread "main" java.net.UnknownHostException: <my-host-name-here>: <my-host-name-here>: Name or service not known
You could generate a file and run it like any other test file:
(sourceGenerators in Test) += Def.task {
val contents = """object TestRepl {
{{}}
}""".replace("{{}}", (initialCommands in console).value)
val file = (sourceManaged in Test).value / "repltest.scala"
IO.write(file, contents)
Seq(file)
}.taskValue
I am trying to call sbt assembly from the command line passing it a scalac compiler flag to elides (elide-below 1).
I have managed to get the flag working in the build.sbt by adding this line to the build.sbt
scalacOptions ++= Seq("-Xelide-below", "1")
And also it's working fine when I start sbt and run the following:
$> sbt
$> set scalacOptions in ThisBuild ++=Seq("-Xelide-below", "0")
But I would like to know how to pass this in when starting sbt, so that my CI jobs can use it while doing different assembly targets (ie. dev/test/prod).
One way to pass the elide level as a command line option is to use system properties
scalacOptions ++= Seq("-Xelide-below", sys.props.getOrElse("elide.below", "0"))
and run sbt -Delide.below=20 assembly. Quick, dirty and easy.
Another more verbose way to accomplish the same thing is to define different commands for producing test/prod artifacts.
lazy val elideLevel = settingKey[Int]("elide code below this level.")
elideLevel in Global := 0
scalacOptions ++= Seq("-Xelide-below", elideLevel.value.toString)
def assemblyCommand(name: String, level: Int) =
Command.command(s"${name}Assembly") { s =>
s"set elideLevel in Global := $level" ::
"assembly" ::
s"set elideLevel in Global := 0" ::
s
}
commands += assemblyCommand("test", 10)
commands += assemblyCommand("prod", 1000)
and you can run sbt testAssembly prodAssembly. This buys you a cleaner command name in combination with the fact that you don't have to exit an active sbt-shell session to call for example testAssembly. My sbt-shell sessions tend to live for a long time so I personally prefer the second option.
I'm just learning scala coming out of the groovy/java world. My first script requires a 3rd party library TagSoup for XML/HTML parsing, and I'm loath to have to add it the old school way: that is, downloading TagSoup from its developer website, and then adding it to the class path.
Is there a way to resolve third party libraries in my scala scripts? I'm thinking Ivy, I'm thinking Grape.
Ideas?
The answer that worked best for me was to install n8:
curl https://raw.github.com/n8han/conscript/master/setup.sh | sh
cs harrah/xsbt --branch v0.11.0
Then I could import tagsoup fairly easily example.scala
/***
libraryDependencies ++= Seq(
"org.ccil.cowan.tagsoup" % "tagsoup" % "1.2.1"
)
*/
def getLocation(address:String) = {
...
}
And run using scalas:
scalas example.scala
Thanks for the help!
While the answer is SBT, it could have been more helpful where scripts are regarded. See, SBT has a special thing for scripts, as described here. Once you get scalas installed, either by installing conscript and then running cs harrah/xsbt --branch v0.11.0, or simply by writing it yourself more or less like this:
#!/bin/sh
java -Dsbt.main.class=sbt.ScriptMain \
-Dsbt.boot.directory=/home/user/.sbt/boot \
-jar sbt-launch.jar "$#"
Then you can write your script like this:
#!/usr/bin/env scalas
!#
/***
scalaVersion := "2.9.1"
libraryDependencies ++= Seq(
"net.databinder" %% "dispatch-twitter" % "0.8.3",
"net.databinder" %% "dispatch-http" % "0.8.3"
)
*/
import dispatch.{ json, Http, Request }
import dispatch.twitter.Search
import json.{ Js, JsObject }
def process(param: JsObject) = {
val Search.text(txt) = param
val Search.from_user(usr) = param
val Search.created_at(time) = param
"(" + time + ")" + usr + ": " + txt
}
Http.x((Search("#scala") lang "en") ~> (_ map process foreach println))
You may also be interested in paulp's xsbtscript, which creates an xsbtscript shell that has the same thing as scalas (I guess the latter was based on the former), with the advantage that, without either conscript or sbt installed, you can get it ready with this:
curl https://raw.github.com/paulp/xsbtscript/master/setup.sh | sh
Note that it installs sbt and conscript.
And there's also paulp's sbt-extras, which is an alternative "sbt" command line, with more options. Note that it's still sbt, just the shell script that starts it is more intelligent.
SBT (Simple Build Tool) seems to be the build tool of choice in the Scala world. It supports a number of different dependency resolution mechanisms: https://github.com/harrah/xsbt/wiki/Library-Management
Placed as an answer cause it doesn't fit in comment length constraint.
In addition to #Chris answer, I would like to recommend you some commons for sbt (which I personally think is absolutely superb). Although sbt denote Simple Build Tool, sometimes it is not so easy for first-timers to setup project with sbt (all this things with layouts, configs, and so on).
Use giter (g8) to create new project with predefined template (which g8 fetches from github.com). There are templates for Android app, unfiltered and more. Sometimes they are include some of the dependencies by default.
To create layout just type:
g8 gseitz/android-sbt-project
(An example for Android app)
Alternatively, use np pluggin for sbt, which provides interactive type-through way to create new project and basic layout.
A corrected and simplified version of the current main answer: use scalas.
You have to compose your script of 3 parts. One would be sbt, another would be a very simple wrapper around SBT called scalas, the last one is your custom script. Note that the first two scripts can be installed either globally (/usr/bin/, ~/bin/) or locally (in the same directory).
the first part is sbt. If you already have it installed then good. If not, you can either install it, or use a very cool script from paulp: https://github.com/paulp/sbt-extras/blob/master/sbt BTW, that thing is a charming way to use sbt on Unix. Although not available on windows. Anyways...
the second part is scalas. It's just an entrypoint to SBT.
#!/bin/sh
exec /path/to/sbt -Dsbt.main.class=sbt.ScriptMain -sbt-create \
-Dsbt.boot.directory=$HOME/.sbt/boot \
"$#"
the last part is your custom script. Example:
#!/usr/bin/env scalas
/***
scalaVersion := "2.11.0"
libraryDependencies ++= Seq(
"org.joda" % "joda-convert" % "1.5",
"joda-time" % "joda-time" % "2.3"
)
*/
import org.joda.time._
println(DateTime.now())
//println(DateTime.now().minusHours(12).dayOfMonth())
What Daniel said. Although it's worth mentioning that the sbt docs carefully label this functionality "experimental".
Indeed, if you try to run the embedded script with scalaVersion := "2.10.3", you'll get not found: value !#
Luckily, the !# script header-closer is unnecessary here, so you can leave it out.
Under scalaVersion := "2.10.3", the script will need to have the file extension ".scala"; using the bash shell script file extension, ".sh", won't work.
Also, it isn't clear to me that the latest version of Dispatch (0.11.0) supports dispatch-twitter, which is used in the example.
For more about header-closers in this context, see Alvin Alexander's blog post on Scala scripting, or section 14.10 of his Scala Cookbook.
I have a build.gradle file with the following task:
task classpath(dependsOn: jar) << {
println "CLASSPATH=${tasks.jar.archivePath}:${configurations.runtime.asPath}"
}
Then, in my Scala script:
#!
script_dir=$(cd $(dirname "$0") >/dev/null; pwd -P)
classpath=$(cd ${script_dir} && ./gradlew classpath | grep '^CLASSPATH=' | sed -e 's|^CLASSPATH=||')
PATH=${SCALA_HOME}/bin:${PATH}
JAVA_OPTS="-Xmx4g -XX:MaxPermSize=1g" exec scala -classpath ${classpath} "$0" "$0" "$#"
!#
Note that we don't need a separate scalas executable in our PATH, since we can use the self-executing shell script trick.
Here's an example script, which reads its own content (via the $0 variable), chops off everything before an arbitrary marker (__BEGIN_SCRIPT__) and runs sbt on the result. We use process substitution to pretend this calculated content is a real file. One problem with this approach is that sbt will seek within the given file, i.e. it doesn't read it sequentially. That stops it working with the <(foo) form of process substitution, as found in bash; however zsh has a =(foo) form which is seekable.
#!/usr/bin/env zsh
set -e
# Find the line # in this file ($0) after the line beginning __BEGIN_SCRIPT__
LINENUM=$(awk '/^__BEGIN_SCRIPT__/ {print NR + 1; exit 0; }' "$0")
sbtRun() {
# Run the sbt command, such that it will fetch dependencies and execute a
# script
sbt -Dsbt.main.class=sbt.ScriptMain \
-sbt-create \
-Dsbt.boot.directory="$HOME/.sbt/boot" \
"$#"
}
# Run SBT on the contents of this file, starting at LINENUM
sbtRun =(tail -n+"$LINENUM" "$0")
exit 0
__BEGIN_SCRIPT__
/***
scalaVersion := "2.11.0"
libraryDependencies ++= Seq(
"org.joda" % "joda-convert" % "1.5",
"joda-time" % "joda-time" % "2.3"
)
*/
import org.joda.time._
println(DateTime.now())