Forking and ordering tests in Sbt 0.13.x - scala

Here is how it was configured for Sbt 0.12.x:
parallelExecution in test := false
testGrouping in Test <<= definedTests in Test map { tests =>
tests.map { test =>
import Tests._
import scala.collection.JavaConversions._
new Group(
name = test.name,
tests = Seq(test),
runPolicy = SubProcess(javaOptions = Seq(
"-server", "-Xms4096m", "-Xms4096m", "-XX:NewSize=3584m",
"-Xss256k", "-XX:+UseG1GC", "-XX:+TieredCompilation",
"-XX:+UseNUMA", "-XX:+UseCondCardMark",
"-XX:-UseBiasedLocking", "-XX:+AlwaysPreTouch") ++
System.getProperties.toMap.map {
case (k, v) => "-D" + k + "=" + v
}))
}.sortWith(_.name < _.name)
}
During migration to Sbt 0.13.x I get the following error:
[error] Could not accept connection from test agent: class java.net.SocketException: socket closed
java.net.SocketException: socket closed
at java.net.DualStackPlainSocketImpl.accept0(Native Method)
at java.net.DualStackPlainSocketImpl.socketAccept(DualStackPlainSocketImpl.java:131)
at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398)
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:199)
at java.net.ServerSocket.implAccept(ServerSocket.java:530)
at java.net.ServerSocket.accept(ServerSocket.java:498)
at sbt.ForkTests$$anonfun$mainTestTask$1$Acceptor$2$.run(ForkTests.scala:48)
at java.lang.Thread.run(Thread.java:745)
Migration changes are just updates in sbt & plugin versions.
Are there any other approaches to forking and ordering of tests in Sbt 0.13.x to overcome that exception?

Works fine on Linux and Mac OS.
Got error on Windows because of limit of classpath length that prevents launching of test agent instance with following error in System.err:
Error: Could not find or load main class sbt.ForkMain

I also got this error when moving to Scala repo to sbt version sbt.version = 1.3.8 (previously 1.2.8 was ok). Strangely worked fine on my mac, but failed on teamcity linux build agents.
Fix for me was to set
fork := false,
in build.sbt.
Not sure why repo had it previously set to fork := true (guess it was cut/paste from somewhere else as no strong reason for this in this repo), but this change resolved the issue. Locally on my mac also runs a few seconds faster now.
See here for background
https://www.scala-sbt.org/1.0/docs/Forking.html

Related

SBT hangs waiting for orphaned subprocess to complete

In my SBT (0.13.16) build, I have the following task:
startThing := {
var bin_path = s"${file(".").getAbsolutePath}/bin"
val result = s"$bin_path/start_thing".!
if (result != 0)
throw new RuntimeException("Could not start Thing..")
true
}
And start_thing contains:
(run_subprocess &)
and my build hangs.
I can see that start_thing exits (the process table does not have it as an entry) but adding some printlns to the task shows that it's stuck on val result = s"$bin_path/start_thing".!.
If I kill the run_subprocess process then SBT unblocks and runs normally.
In this particular case, run_subprocess has set up some Kubernetes port-forwarding that needs to be there in order for subsequent tests to work.
Try daemonising the background process like so
(run_subprocess >/dev/null 2>&1 &)
The issue could be output from run_subprocess still going to sbt parent as suggested here.
I was able to replicate the issue in both sbt 0.13.17 and 1.0.2. Daemonising worked in both.
Regardless of my comment, in my case the reason for the hanging was ACTUALLY the potential leaving of open STDOUT , STDERR handles in a daemon started by the script , that is OK:
/usr/local/bin/minio server "$minio_data_dir" > /dev/null 2>&1 & # and start the server
and NOT ok:
/usr/local/bin/minio server "$minio_data_dir" 2>&1 & # and start the server
So the hanging occurred randomly EVEN with the accepted answers start in the background ...
Thus this solution needed NOT, any wrapper bash scripts ... This is how the code looked in the build.sbt
lazy val startLocalS3 = inputKey[Unit]("localS3")
lazy val startLocalS3Task = TaskKey[Unit]("localS3", "create local s3")
lazy val core: Project = project
.in(file("."))
.settings(
name := "rfco-core",
startLocalS3Task := {
val cmd: Seq[String] = Seq("bash" , "-c" , "./CI/start-s3-svr.sh")
import sys.process._
cmd.mkString(" ").!!
},
fork in startLocalS3Task := true,
compile.in(Compile) := (compile in Compile).dependsOn((startLocalS3Task)).value
// you might want to use Test scope ^^ here
)

pgp signature failure on publishing artifact

This is the first time ever i am trying to publish an artifact, and it could not be harder.
I am using sbt 0.13.8 with plug-ins sbt-release 1.0.8, sbt-sonatype 2.0 sbt-pgp 1.1.1
relevant part of my build sbt looks like this:
pgpSecretRing := file("""C:\Users\kali\.sbt\gpg\secring.asc""")
pgpPublicRing := file("""C:\Users\kali\.sbt\gpg\pubring.asc""")
//pgpSecretRing := file("""C:\Users\kali\AppData\Local\lxss\home\kali\.gnupg\secring.gpg""")
//pgpPublicRing := file("""C:\Users\kali\AppData\Local\lxss\home\kali\.gnupg\pubring.gpg""")
usePgpKeyHex("c500a525a2efcb99")
name := "project-name"
organization := "com.mehmetyucel"
version := "0.0.2-SNAPSHOT"
scalaVersion := "2.12.2"
crossScalaVersions := Seq("2.11.11", "2.12.2")
lazy val projectName = project in file(".")
homepage := Some(url("https://some-github-url"))
scmInfo := Some(
ScmInfo(url(
"some-github-url"),
"some-github-url.git"))
developers := List(
Developer(
"mehmetyucel",
"Mehmet Yucel",
"mehmet#mehmetyucel.com",
url("some-github-url")))
licenses += ("MIT", url("http://opensource.org/licenses/MIT"))
publishMavenStyle := true
publishTo := Some(
if (isSnapshot.value)
Opts.resolver.sonatypeSnapshots
else
Opts.resolver.sonatypeStaging
)
releaseCrossBuild := true
releaseProcess := Seq[ReleaseStep](
checkSnapshotDependencies,
inquireVersions,
runClean,
runTest,
setReleaseVersion,
commitReleaseVersion,
tagRelease,
// For non cross-build projects, use releaseStepCommand("publishSigned")
releaseStepCommandAndRemaining("+publishSigned"),
setNextVersion,
commitNextVersion,
releaseStepCommand("sonatypeReleaseAll"),
pushChanges
)
The first 5 lines here I added out of desperation because when I do sbt release "I think" it signs my packages with a key that i am not aware of.
the error message i get is:
[info] Evaluate: signature-staging
[info] Failed: signature-staging, failureMessage:No public key: Key with id: (5b09423c9d5fbb5d) was not able to be located on http://keyserver.ubuntu.com:11371/. Upload your public key and try the operation again.
but my key is already uploaded. I can go and find my key on keyserver.ubuntu.com (http://keyserver.ubuntu.com/pks/lookup?op=get&search=0xC500A525A2EFCB99) unfortunately you will realize that the key id here is different. And I have no idea where this "5b09423c9d5fbb5d" is coming from.
I tried cloning the repository into 3 differnt systems (macos, ubuntu and win10) create a brand new key upload the key to ubuntu keyserver and try to release it again. The id in the error message is always same (5b09423c9d5fbb5d) I have no idea where this is coming from and how it is same despite using a totally different key/system
I tried changing usePgpKeyHex("c500a525a2efcb99") to usePgpKeyHex("c500a525a2efcb91") (basically something that does not exist) and I get
[error] Not a valid command: failure-- (similar: onFailure)
[error] Not a valid project ID: failure--
[error] Expected ':' (if selecting a configuration)
[error] Not a valid key: failure--
[error] failure--
[error] ^
on signing stage, which is awesome, which actually means sbt is using my key to sign so, still where does 5b09423c9d5fbb5d come from?
Solved:
Apparently this key id was coming from the very first "publishSigned" i did for this project, which failed for another reason. I kept running publishSigned sonatypeRelease back to back quite a few times and this caused a lot of "open" staging repositories in sonatype.
I went to https://oss.sonatype.org/ and dropped all open staging repositories using nexus UI and restarted the release process and everything works.

ensime server fails to start from emacs

I am starting a new project in scala. I use emacs as editor.
So far it cannot be more simple. This is the build.sbt
name := "Sampler"
version := "1.0"
scalaVersion := "2.11.8"
I have run the ensimeConfig command in sbt (0.13 version)
When I start ensime on Emacs (M-ensime) I get the following error:
13:34:52.511 ERROR None Server - There was a problem parsing /home/jeronimo/prog/coursera/progfun2/circuits/.ensime
java.lang.NoSuchMethodError: shapeless.DataT$.genTraversableDataT(Lshapeless/Lazy;Lscala/collection/generic/CanBuildFrom;)Lshapeless/DataT;
at org.ensime.config.EnsimeConfigProtocol$anon$default$macro$337$1.inst$macro$320$lzycompute(EnsimeConfigProtocol.scala:61)
at org.ensime.config.EnsimeConfigProtocol$anon$default$macro$337$1.inst$macro$320(EnsimeConfigProtocol.scala:61)
at org.ensime.config.EnsimeConfigProtocol$anon$default$macro$337$1$$anonfun$inst$macro$319$1.apply(EnsimeConfigProtocol.scala:61)
at org.ensime.config.EnsimeConfigProtocol$anon$default$macro$337$1$$anonfun$inst$macro$319$1.apply(EnsimeConfigProtocol.scala:61)
at shapeless.Lazy$$anon$1.value$lzycompute(lazy.scala:121)
at shapeless.Lazy$$anon$1.value(lazy.scala:121)
at shapeless.EverywhereAux$$anonfun$default$2.apply(sybclass.scala:184)
at shapeless.Cases$Case1$$anon$1$$anonfun$1.apply(cases.scala:40)
at shapeless.Cases$Case1$$anon$1$$anonfun$1.apply(cases.scala:38)
at shapeless.PolyDefns$Case.apply(poly.scala:37)
at shapeless.DataT$$anon$12.gmapT(sybclass.scala:144)
at shapeless.DataT$$anon$12.gmapT(sybclass.scala:142)
at shapeless.DataT$$anon$12.gmapT(sybclass.scala:144)
at shapeless.DataT$$anon$12.gmapT(sybclass.scala:142)
at shapeless.DataT1$$anon$9.gmapT(sybclass.scala:116)
at shapeless.EverywhereAux$$anonfun$default$2.apply(sybclass.scala:184)
at shapeless.Cases$Case1$$anon$1$$anonfun$1.apply(cases.scala:40)
at shapeless.Cases$Case1$$anon$1$$anonfun$1.apply(cases.scala:38)
at shapeless.PolyDefns$Case.apply(poly.scala:39)
at org.ensime.core.Canonised$.apply(Canon.scala:19)
at org.ensime.config.EnsimeConfigProtocol$.validated(EnsimeConfigProtocol.scala:61)
at org.ensime.config.EnsimeConfigProtocol$$anonfun$3.apply(EnsimeConfigProtocol.scala:44)
at org.ensime.config.EnsimeConfigProtocol$$anonfun$3.apply(EnsimeConfigProtocol.scala:44)
at scala.collection.immutable.List.map(List.scala:273)
at org.ensime.config.EnsimeConfigProtocol$.validated(EnsimeConfigProtocol.scala:44)
at org.ensime.config.EnsimeConfigProtocol$.parse(EnsimeConfigProtocol.scala:32)
at org.ensime.server.Server$.main(Server.scala:120)
at org.ensime.server.Server.main(Server.scala)
Exception in thread "main" java.lang.NoSuchMethodError: shapeless.DataT$.genTraversableDataT(Lshapeless/Lazy;Lscala/collection/generic/CanBuildFrom;)Lshapeless/DataT;
at org.ensime.config.EnsimeConfigProtocol$anon$default$macro$337$1.inst$macro$320$lzycompute(EnsimeConfigProtocol.scala:61)
at org.ensime.config.EnsimeConfigProtocol$anon$default$macro$337$1.inst$macro$320(EnsimeConfigProtocol.scala:61)
at org.ensime.config.EnsimeConfigProtocol$anon$default$macro$337$1$$anonfun$inst$macro$319$1.apply(EnsimeConfigProtocol.scala:61)
at org.ensime.config.EnsimeConfigProtocol$anon$default$macro$337$1$$anonfun$inst$macro$319$1.apply(EnsimeConfigProtocol.scala:61)
at shapeless.Lazy$$anon$1.value$lzycompute(lazy.scala:121)
at shapeless.Lazy$$anon$1.value(lazy.scala:121)
at shapeless.EverywhereAux$$anonfun$default$2.apply(sybclass.scala:184)
at shapeless.Cases$Case1$$anon$1$$anonfun$1.apply(cases.scala:40)
at shapeless.Cases$Case1$$anon$1$$anonfun$1.apply(cases.scala:38)
at shapeless.PolyDefns$Case.apply(poly.scala:37)
at shapeless.DataT$$anon$12.gmapT(sybclass.scala:144)
at shapeless.DataT$$anon$12.gmapT(sybclass.scala:142)
at shapeless.DataT$$anon$12.gmapT(sybclass.scala:144)
at shapeless.DataT$$anon$12.gmapT(sybclass.scala:142)
at shapeless.DataT1$$anon$9.gmapT(sybclass.scala:116)
at shapeless.EverywhereAux$$anonfun$default$2.apply(sybclass.scala:184)
at shapeless.Cases$Case1$$anon$1$$anonfun$1.apply(cases.scala:40)
at shapeless.Cases$Case1$$anon$1$$anonfun$1.apply(cases.scala:38)
at shapeless.PolyDefns$Case.apply(poly.scala:39)
at org.ensime.core.Canonised$.apply(Canon.scala:19)
at org.ensime.config.EnsimeConfigProtocol$.validated(EnsimeConfigProtocol.scala:61)
at org.ensime.config.EnsimeConfigProtocol$$anonfun$3.apply(EnsimeConfigProtocol.scala:44)
at org.ensime.config.EnsimeConfigProtocol$$anonfun$3.apply(EnsimeConfigProtocol.scala:44)
at scala.collection.immutable.List.map(List.scala:273)
at org.ensime.config.EnsimeConfigProtocol$.validated(EnsimeConfigProtocol.scala:44)
at org.ensime.config.EnsimeConfigProtocol$.parse(EnsimeConfigProtocol.scala:32)
at org.ensime.server.Server$.main(Server.scala:120)
at org.ensime.server.Server.main(Server.scala)
Process *inferior-ensime-server-circuits* exited abnormally with code 1
After trying to delete the .ivy2 cache without success I fixed the issue re-installing the EMACS ensime plugin.

Scala Stdin.readLine() does not seem to work as expected

I am trying to write a simple console client application where i can present some options to the user, get their input and act accordingly. If i run the code through intellij or paste it into the scala console, it works. If i run it through sbt (which is how i really need it to run), i run into all sorts of problems.
I have sbt version 0.13.8, OS is Mac, my build.sbt contains:
scalaVersion := "2.11.6"
fork in run := true
EDIT I started with the minimum scala activator template in case that is useful info in this context
I have simplified the code to barebones,
import scala.io.StdIn._
object TestClient {
def main(args: Array[String]): Unit = {
join()
}
def join(): Unit = {
val name = readLine(s"Enter your name.${System.getProperty("line.separator")}")
name match {
case n: String => println(n)
case o => {
println(s"invalid name ${o}")
join()
};
}
}
}
When go into sbt and from the prompt enter run. one of the following seems to happen
1) I get this exception as soon as i run
Exception in thread "Thread-2" java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Arrays.java:3332)
at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:137)
at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:121)
at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:569)
at java.lang.StringBuffer.append(StringBuffer.java:369)
at java.io.BufferedReader.readLine(BufferedReader.java:370)
at java.io.BufferedReader.readLine(BufferedReader.java:389)
at sbt.BasicIO$$anonfun$processFully$1$$anonfun$apply$8.apply(ProcessImpl.scala:58)
at sbt.BasicIO$$anonfun$processFully$1$$anonfun$apply$8.apply(ProcessImpl.scala:58)
at sbt.BasicIO$.readFully$1(ProcessImpl.scala:63)
at sbt.BasicIO$.processLinesFully(ProcessImpl.scala:69)
at sbt.BasicIO$$anonfun$processFully$1.apply(ProcessImpl.scala:58)
at sbt.BasicIO$$anonfun$processFully$1.apply(ProcessImpl.scala:55)
at sbt.SimpleProcessBuilder$$anonfun$3.apply$mcV$sp(ProcessImpl.scala:354)
at sbt.Spawn$$anon$3.run(ProcessImpl.scala:17)
2)
No memory issues, but getting these messages in an infinite loop, so the readline does not seem to be waiting for any input
background log: info: Enter your name.
background log: info: invalid name null
background log: info: Enter your name.
background log: info: invalid name null
...
Either way I am unable to actually enter any input in the console. Not sure what I'm missing or doing wrong
Put
connectInput in run := true
in your build.sbt. See the official docs for more info on how to correctly handle forks in sbt.

sbt: suppressing logging prefix in stdout

When using sbt with forking (fork in run := true), every output from my application to stdout is prefixed by [info]; output to stderr is prefixed with [error].
This behavior is somewhat annoying when using a Java logging framework which outputs to stderr. The resulting debug messages typically look like this:
[error] [main] INFO MyClass ...
[error] [main] DEBUG MyClass ...
I would like to suppress these prefixes like when running the code without forking. What I tried:
setting sbt -Dsbt.log.noformat=true in the sbt launch script. But this only removes colored ANSI output; prefixes are still there just without color
setting logLevel in run := Level.Error in build.sbt. This does not seem to have any influence on logging with forking.
Is there any way to suppress the prefixes?
You need to set the output strategy of your project.
In my extended build I have the following settings:
settings = Project.defaultSettings ++ Seq(
fork := true, // Fork to separate process
connectInput in run := true, // Connects stdin to sbt during forked runs
outputStrategy := Some(StdoutOutput) // Get rid of output prefix
// ... other settings
)
Can do
sbt -error ...
and also
sbt -warn ...