I'm experiencing some perplexing behavior when running tests from SBT. I have a project with ~100 ScalaTest tests. They all ran in a few seconds. I added a dozen more larger tests that take around a second each in IntelliJ. When I run them in SBT via "test" they pass, but take on the order of 10 minutes. When I run them in SBT via "test-only" they pass in seconds, similar to when I run them in IntelliJ.
I tried tinkering with the heap memory, the parallel test running settings, and forking, but nothing I've done fixes the problem. Unfortunately it's not an open-source project, but any hypotheses would be greatly appreciated.
Related
While researching on the subject of automating my integration tests, I found out a nice plugin in the maven world called FailSafe. it gives me phases like pre-integration-test, post-integration-test and integration-test.
By tying into these phases, I can have other plugins which can start/stop and run docker images.
The plugin also has a nice way in which I can differentiate between UnitTests and IntegrationTests (even though both are written in JUNIT).
Well now the question is how can I do the same thing with Scala / SBT combination?
my requirement is
Write Integration tests in SpecFlow.
Integration tests are treated differently than unit tests.
First Unit Tests are run.
Then docker containers are created and run
then integration tests are run.
docker contains are shut-down.
test results are captured in files. (just like surefire/failsafe plugins).
Is this possible in Scala/sbt combo?
A simple solution is to run $ sbt "~ it:test" (make sure integration test are in a package named 'it') for integration test which will automatically run every time source code is updated. Furthermore, $sbt "~ test" for automated unit testing. If you are using a IDE such as IntelliJ IDEA, you can make it easier to run this in a custom configuration from the IDE. Hope this helps a little bit. I run these all the time when working.
I found the answer to the question. SBT provides means to do integration test and also setup and cleanup methods to do things like creation / destruction of docker containers
http://www.scala-sbt.org/0.13/docs/Testing.html
I have a new Scala/sbt project set up in IntelliJ IDEA with ScalaTest and jUnit interface successfully installed as dependencies. I have one test that passes when I hit run or debug.
My problem is that it doesn't rerun if I change something. I have to hit run or debug again and then it runs and gives the expected response.
How do I set up IntelliJ IDEA (with or without sbt) to run all tests every time the code changes? Is it possible to run the tests related to the files that were changed only?
As answered in comment by Boris the Spider you can run
> ~test
from sbt shell. Hopefully sbt and IntelliJ can integrate better via sbt server, but I don't think it's currently possible.
I'm having an issue with Gradle, JUnit and Scala test execution, in our Scala language extension for Vert.x. Recently we upgraded from Vert.x 2.1.RC1 to 2.1.RC2, we started seeing that in some environments such as our Cloudbees instance, test execution looks up a phantom method called classMethod in a test class, which does not exist, and the test times out. As a result of that, the rest of tests fail.
In this console instance, you can see:
org.vertx.scala.tests.core.http.HttpTest > classMethod FAILED
java.lang.AssertionError: Timed out waiting for test to complete
We have no idea what this classMethod is, where it's coming from...etc. Even more spooky is that I'm unable to replicate it neither in my OSX nor Linux (RHEL7) environments, both with JDK 1.7u45 and 1.7u51.
The only way we've found to be to fix it so far is to revert to Vert.x 2.1.RC1 (see console of run with RC1), but we're really unable to see the link between this and this misterious classMethod.
We've tried upgrading to Gradle 1.11 in case it's an issue with Gradle itself but no luck. We're currently using JUnit 4.11.
The only thing that looks slightly suspicious is how even though the project uses Scala 2.10.4, the Gradle Scala plugin seems to bring in Zinc which appears to need Scala 2.9.2. When I wiped out my .gradle/ folder locally, I didn't see any Scala 2.9.2 being downloaded. I wonder if this is messing up things?
Finally, for this last run when HttpCompressionTest seem to have that classMethod, I've compared the javap output locally with the one in Cloudbees, and they look identical.
I'm out of ideas, any suggestions?
UPDATE: As a last resort, I decided to upgrade to Vert.x 2.1.RC3-SNAPSHOT and it's all back to normal. I've absolutely no idea why, but I'll take it :)
I've seen this behavior caused by (accidentally) having two different versions of the same lib on the classpath. Also, when a particular project is packaged as several different JARs, one of them was down-level from all the rest.
I'm currently using IDEA's build mechanism with fsc for developing with Scala. It's still a bit slow and having to (re) start the compilation server is a pain. Many people here are suggesting SBT as a build tool together with IDEA.
What do you consider the pros and cons of each approach?
I tried both and in the end I prefer straight sbt for compiling.
Cons? I really miss being able to click through compile errors and fix the code directly, but... compiling in sbt is just much faster.
The nightly builds of the Idea Scala plugin can vary in quality/performance, but it's been getting better and better lately. The Scala plugin can now flag a number of errors that before I would have had to run compile to catch. (For example, I'm running nightly build 0.4.693 and the new method inspections have already been dead helpful.)
My advice for life with sbt on the command line: start sbt up and leave it running interactively as long as possible to take advantage of everything being loaded and JIT-ed.
sbt left running will fall over eventually but by giving it more memory in your sbt wrapper you can make that happen only rarely.
Here's the sbt launch wrapper that works for me.
java -Xms512M -Xmx1500M -XX:MaxPermSize=512m -jar `dirname $0`/sbt-launch.jar "$#"
My biggest issue with sbt 0.7 is that it frequently goes back and recompiles great swathes of files that seem only tangential to the code I was actually changing. (Even so, still faster than Idea and fsc!)
Good news: sbt 0.9 has some great incremental compile improvements. Unfortunately, the migration path from 0.7 to 0.9 is still in its early days. Mark Harrah's presentation at NEScala is online at http://www.nescala.org/2011/ if you're interested.
Useful plugins
http://github.com/mpeltonen/sbt-idea - easily create and keep your Idea project in sync with your sbt project
http://github.com/orfjackal/idea-sbt-plugin - lets you create run profiles for sbt from within Idea (I found this slower than running sbt at the command line last October - but I see orfjackal is still developing so I should give it another shot)
I'm developing Scala code using Eclipse, often when I run tests I get this error:
No tests found with test runner 'JUnit 3'.
Environment:
Eclipse for Java Developers, 3.5.1
Scala 2.7.7
JUnit 4.7
I'm currently writing my tests as JUnit3 tests, and invoking them by right clicking on a package in the project explorer, choosing Run As -> JUnit Test. (I was writing them as JUnit4 tests but ran into even more problems.)
If I fire up eclipse, the tests may not run unless I first open the source code file for the test. If I do open the source code file for the test, it will run. However, often then when I make any modification to the test file or any other source code file, Eclipse will refuse to run my tests, saying: "No tests found with test runner 'JUnit 3'."
I just repeated this just now:
Open eclipse
Open the .scala file with some tests
Invoke the tests by right clicking on the package for that file in the project explorer and choosing Run As -> JUnit Test
It ran the tests
One failed
I changed a string literal in the failing test to fix it
I then re-launced the test using the same method, and I get the dreaded "No tests found with test runner 'JUnit 3'."
I get this same method using other methods of launching the tests, e.g. JUnit buttons or menus to re-run all or some tests
To get the tests to run again I close and re-open Eclipse... So I end up relaunching Eclipse many many times a day.
Note: I do often use XML literals in my tests, I wonder if that has anything to do with it.
2nd Note: See my answer to this thread: What is the current state of the Scala Eclipse plugin? where I described some of the other problems I'm having with Scala+Eclipse. Most of the problems are just minor annoyances, but this test invocation problem is a real time waster, would love to find a way around it!
This Just works on trunk ... JUnit 3 and 4 unit test are detected and Run As => JUnit test does the right thing.
Oh, and BTW, the best way to get my attention on this sort of thing is to file a bug or enhancement ticket in Trac rather than posting messages to StackOverflow (even though it worked this time ;-)