I have this in my build.gradle:
test {
testLogging {
exceptionFormat 'full'
showExceptions true
showStackTraces true
}
}
This works with java ("plain" junit) tests, but when I running scalatest tests, even with -i on command line, all I get in case of a failure is something like this:
com.mypackage.mytests.ScalatestSpec > .apply should fail miserably FAILED
org.scalatest.exceptions.TestFailedException: 2 was not equal to 1
No traceback or even a line number is printed out, and I have to rerun the test manually to be able to see where it actually failed.
Is there another special flag I have to set to get it to stop following my output?
Related
I use NUnit and TeamCity to run my tests.
Some tests (not all) has actions made in test class constructor. I call these actions "pre-actioins" for validations. So in one test class I have for example 5 validations (tests) and a set of pre-actions.
I noticed that if a suite of tests if failed on the stage of pre-actions executing then TeamCity doesn't display these tests in its report at all (not under any status).
In build log I see error like:
SetUp Error : {test_name} + error code.
What I expect from TeamCity is to report these tests at least as Ignored.
To compare running tests using TeamCity with running tests using Visual Studio in Visual Studio the result of the same failure condition will be failure for all the test suite. Failure error will be the same for all the tests.
So what I want is just to know if some of my tests were not run at all because if TeamCity doesn't include then in test results then I don't even know about problems!
Configs: TeamCity 10.0, NUnit 3.0.
Command line params: --result=TestResult.xml --workers=4 --teamcity
Update: results of tests executing in log looks like:
[13:03:48][Step 1/1] Test Run Summary
[13:03:48][Step 1/1] Overall result: Failed
[13:03:48][Step 1/1] Tests run: 82, Passed: 0, Errors: 82, Failures: 0, Inconclusive: 0
[13:03:48][Step 1/1] Not run: 0, Invalid: 0, Ignored: 0, Explicit: 0, Skipped: 0
[13:03:48][Step 1/1] Start time: 2016-09-08 09:56:33Z
[13:03:48][Step 1/1] End time: 2016-09-08 10:03:48Z
[13:03:48][Step 1/1] Duration: 434,948 seconds
So NUnit marks such tests even not as failed but as "erros". Still I want them in test results.
Your tests are errors because you are throwing an exception in the constructor. Since the test fixture can't be constructed, the test is not really being run as far as NUnit is concerned. The fact that it's an NUnit assertion failure causing the exception is irrelevant in the context of constructing the object.
We have always advised people to keep their constructors very simple because NUnit makes no guarantees about when and how often your object will be constructed. Using assertions in the constructor is an extreme violation of that principal and, in fact, I've never seen anyone do it before.
The OneTimeSetUp attribute is there if you want some thing to happen every time your test is run as opposed to constructed. NUnit does make guarantees about when that method will be executed. :-)
None of this tells me for sure why TC is not recognizing the error but I'm guessing it's because once the constructor fails, the tests are never actually run. NUnit itself compensates for that by reporting the tests as errors but TC would not necessarily do the same.
I'm having a problem where jasmine-node silently fails if unhandled exceptions happen in a test.
If I run a single file, everything is OK and I get the expected jasmine output:
./node_modules/jasmine-node/bin/jasmine-node spec/unit/accessControlSpec.js
Finished in 0.011 seconds
4 tests, 6 assertions, 0 failures, 0 skipped
But, if I run all specs in a folder, it fails silently.
./node_modules/jasmine-node/bin/jasmine-node spec/unit
Tried --verbose and --captureExceptions but no luck.
In this specific case, some code inside a test was calling a method that didn't exist.
So, turns out the problem is that I'm not calling the correct command because I didn't install jasmine-node globally.
The correct way is:
node ./node_modules/jasmine-node/lib/jasmine-node/cli.js ./spec/unit
This is further described here: Command Line Usage
We have a Scala project and we use SBT as its build tool.
our CI tool is TeamCity, and we build the project using the command line custom script option with the following command:
call %system.SBT_HOME%\bin\sbt clean package
The build process works fine when the build succeeds, however, when compilation fails - TeamCity thinks that the script exited with exitCode 0 and not 1 as expected, this cause TeamCity build to succeed although the compilation failed.
when we run the same commands on local cmd we see that the errorLevel is 1.
the relevant part of the build log:
[11:33:44][Step 1/3] [error] trait ConfigurationDomain JsonSupport extends CommonFormats {
[11:33:44][Step 1/3] [error] ^
[11:33:44][Step 1/3] [error] one error found
[11:33:45][Step 1/3] [error] (compile:compile) Compilation failed
[11:33:45][Step 1/3] [error] Total time: 12 s, completed Jan 9, 2014 11:33:45 AM
[11:33:45][Step 1/3] Process exited with code 0
how can we make TeamCity recognize the failure of the build?
Try explicitly exit with:
call %system.SBT_HOME%\bin\sbt clean package
echo the exit code is %errorlevel%
exit /b
If you can't get the process to output a non-zero exit code then you could use a build failure condition based on specific text in the build log. See this page for the documentation but in essence you can get the build to fail if it finds the text error found in the build log.
I have a psake task that I run from my build script to verify that the expected artifacts are created in the expected location. It seems to work when I run it from the command line, however it is returning an error code of "-1" when the assert passes or fails. What does exit code -1 mean and why wouldn't I get an exit code of 0 when it passes the assertion?
task VerifyArtifacts {
Assert (Test-Path $package_file) "$package_file could not be found"
}
Command to execute:
psake .\default.ps1 VerifyArtifacts
If I create an SBT project, even a simple "hello world", compile (successfully) and then exec, the folowing error is thrown. WHat may the reason be and how to fix this?
java.lang.ArrayIndexOutOfBoundsException: 0
at java.lang.ProcessBuilder.start(ProcessBuilder.java:459)
at sbt.SimpleProcessBuilder.run(ProcessImpl.scala:381)
at sbt.AbstractProcessBuilder.run(ProcessImpl.scala:130)
at sbt.AbstractProcessBuilder.$bang(ProcessImpl.scala:158)
at sbt.ExecProject$$anonfun$execOut$1.apply(ScalaProject.scala:436)
at sbt.ExecProject$$anonfun$execOut$1.apply(ScalaProject.scala:435)
at sbt.TaskManager$Task.invoke(TaskManager.scala:62)
at sbt.impl.RunTask.doRun$1(RunTask.scala:77)
at sbt.impl.RunTask.runTask(RunTask.scala:85)
at sbt.impl.RunTask.run(RunTask.scala:32)
at sbt.impl.RunTask$.apply(RunTask.scala:17)
at sbt.impl.RunTask$.apply(RunTask.scala:16)
at sbt.Project$class.run(Project.scala:98)
at sbt.Project$class.call(Project.scala:93)
at sbt.BasicScalaProject.call(DefaultProject.scala:21)
at sbt.xMain$$anonfun$7.apply(Main.scala:512)
at sbt.xMain$$anonfun$7.apply(Main.scala:512)
at sbt.xMain.withAction(Main.scala:541)
at sbt.xMain.sbt$xMain$$handleAction(Main.scala:512)
at sbt.xMain.handleCommand(Main.scala:502)
at sbt.xMain.processAction(Main.scala:441)
at sbt.xMain.process$1(Main.scala:257)
at sbt.xMain$Continue$1.apply(Main.scala:132)
at sbt.xMain.run$1(Main.scala:136)
at sbt.xMain.processArguments(Main.scala:266)
at sbt.xMain.startProject(Main.scala:107)
at sbt.xMain.run(Main.scala:84)
at sbt.xMain.run0$1(Main.scala:35)
at sbt.xMain.run(Main.scala:42)
at xsbt.boot.Launch$.run(Launch.scala:53)
at xsbt.boot.Launch$$anonfun$explicit$1.apply(Launch.scala:42)
at xsbt.boot.Launch$$anonfun$explicit$1.apply(Launch.scala:42)
at xsbt.boot.Launch$.launch(Launch.scala:57)
at xsbt.boot.Launch$.explicit(Launch.scala:42)
at xsbt.boot.Launch$.initialized(Launch.scala:38)
at xsbt.boot.Launch$.parsed(Launch.scala:31)
at xsbt.boot.Launch$.configured(Launch.scala:21)
at xsbt.boot.Launch$.apply(Launch.scala:16)
at xsbt.boot.Launch$.apply(Launch.scala:13)
at xsbt.boot.Boot$.runImpl(Boot.scala:24)
at xsbt.boot.Boot$.run(Boot.scala:19)
at xsbt.boot.Boot$.main(Boot.scala:15)
at xsbt.boot.Boot.main(Boot.scala)
[info] == exec ==
[error] Error running exec: java.lang.ArrayIndexOutOfBoundsException: 0
The purpose of the build action exec is to execute a command on the underlying shell. As such it needs to be followed with a command. EG:
exec killall firefox
Under the covers, SBT calls java.lang.ProcessBuilder, which throws this Exception if the caller tries to start it but has not provided any parameters.
IndexOutOfBoundsException - If the command is an empty list (has size 0)
I reckon SBT should not be propagating this exception and that this is a bug. You should get an error message instead.
Perhaps you were looking for the build action run, which will invoke your main class.