We use ScalaTest to run tests and display the results in Jenkins. It works great, but for some reason ScalaTest` does not populate the test runtime.
I'm adding test options like this:
Tests.Argument("-oD", "-u", "target/test-reports")
But the XML file does not have timings (see time is always 0):
<testcase classname="beekeeper.warehouse.jdbc.JDBCWarehouseTest" name="testListTables" time="0.0">
</testcase>
<testcase classname="beekeeper.warehouse.jdbc.JDBCWarehouseTest" name="testGetDatabases" time="0.0">
</testcase>
<testcase classname="beekeeper.warehouse.jdbc.JDBCWarehouseTest" name="testSchema" time="0.0">
</testcase>
Do I need to do something else to have it report this, or is it not supported?
I can't reproduce this issue. Using sbt 1.1.6 and ScalaTest 3.0.5 I see non-zero time field:
<testcase classname="example.HelloSpec" name="The Hello object should say hello" time="0.016">
Related
How to send the XML report to SonarQube? I mean, while the TEST-report.xml file contains any failures, the import operation fails. I got an error:
Running SonarQube using SonarQube Runner.17:18:18.452 ERROR: Error during SonarScanner execution
java.lang.NullPointerException
at java.text.DecimalFormat.parse(DecimalFormat.java:2030)
at java.text.NumberFormat.parse(NumberFormat.java:383)
...
The TEST-report.xml file (JUnit) contains something like:
<?xml version='1.0' encoding='UTF-8'?>
<testsuites name='X UITests.xctest' tests='12' failures='3'>
<testsuite name='X.MoreViewTests' tests='1' failures='1'>
<testcase classname='X.MoreViewTests' name='testSucceed_allAction'>
<failure message='XCTAssertTrue failed'>X UITests/Cases/Bottom Navigation/MoreViewTests.swift:212</failure>
</testcase>
</testsuite>
<testsuite name='X.OrderViewTests' tests='1' failures='0'>
<testcase classname='X.OrderViewTests' name='testSucceed_allAction' time='68.570'/>
</testsuite>
</testsuites>
But when I removed the failure lines, becomes:
<?xml version='1.0' encoding='UTF-8'?>
<testsuites name='X UITests.xctest' tests='12' failures='3'>
<testsuite name='X.OrderViewTests' tests='1' failures='0'>
<testcase classname='X.OrderViewTests' name='testSucceed_allAction' time='68.570'/>
</testsuite>
</testsuites>
It works. Any idea? Thanks.
Your XML report is invalid (doesn't meet the Surefire XML format requirements).
testsuite required parameters:
name - Full class name of the test for non-aggregated testsuite documents. Class name without the package for aggregated testsuites documents
tests - The total number of tests in the suite
failures - The total number of tests in the suite that failed. A failure is a test which the code has explicitly failed by using the mechanisms for that purpose. e.g., via an assertEquals
errors - The total number of tests in the suite that errored. An errored test is one that had an unanticipated problem. e.g., an unchecked throwable; or a problem with the implementation of the test
skipped - The total number of ignored or skipped tests in the suite
The exception starts with DecimalFormat.parse
java.lang.NullPointerException
at java.text.DecimalFormat.parse(DecimalFormat.java:2030)
at java.text.NumberFormat.parse(NumberFormat.java:383)
which means that the XML report parser couldn't parse a number. You have two missing parameters: errors and skipped. I don't know how you generate these reports (which JUnit version etc.), but probably the used tool is outdated or miss-configured.
I am running 5 API test scenarios using karate. When I run the test in non-parallel mode using #RunWith(Karate.class) then in xml generated by surefire-reports, all scenarios are reported individually as .
<testcase classname="[healthCheck]" name="[1:3] Check health check API returns status code 200" time="2.846"/>
<testcase classname="[healthCheckHeader]" name="[1:6] Check health check API returns status code 200" time="0.285"/>
<testcase classname="[userLogin]" name="[1:3] Check User Login API returns status code 200" time="0.108"/>
<testcase classname="[requestChaining]" name="[1:7] chain request demo" time="0.521"/>
<testcase classname="[viewRequests]" name="[1:10] Check View Requests API returns status code 200" time="0.278"/>
However, when I use karate parallel runner, then each scenario is not reported individually.
<testcase classname="demoTest.AutomationSuiteParallelCucRunner" name="testParallel" time="10.917"/>
I want to have similar report for parallel runner as generated when tests executed in non-parallel mode.
Here is the code for running tests in non-parallel mode:
#RunWith(Karate.class)
public class AutomationSuiteTest {
}
Here is the code for running tests in parallel mode:
#CucumberOptions(tags = {"~#ignore"})
public class AutomationSuiteParallelCucRunner {
#Test
public void testParallel() {
String karateOutputPath = "target/surefire-reports";
KarateStats stats = CucumberRunner.parallel(getClass(), 3, karateOutputPath);
assertTrue("SCENARIO FAILURES!!", stats.getFailCount() == 0);
}
}
Having struggled with this myself, the solution was surprisingly simple, though not really covered in the karate docs (as of this time).
The point is: when using the parallel runner in this manner, there is from JUnit's perspective just one unit test - the single method annotated with #Test. So this single test is all that's listed in target/surefire-reports. Karate doesn't seem to hook into the JUnit execution or so, so that's it.
But - and that was the missing piece for me - the karate runner itself can generate JUnit XML files. You just have to call .outputJunitXml(true) when creating it. This causes it to generate JUnit XML files in the karate report dir along with the HTML report and (optionally) cucumber JSON files:
import com.intuit.karate.Results;
import com.intuit.karate.Runner;
public class KarateTestRunner {
#Test
public void runParallel() {
final Results results = Runner.path("classpath:api/karate")
.outputCucumberJson(true)
.outputJunitXml(true) # <----
.parallel(8);
assertEquals(0, results.getFailCount());
}
}
As mentioned in frisby official document (http://frisbyjs.com/) , I am using --junitreport something like following
jasmine-node ./demo/validation_spec.js/ --junitreport --output
C:\Users\Administrator\Documents\script/Reports
which is generating 15 xml files. As I am calling 15 time post using for loop. File contains data like following
<?xml version="1.0" encoding="UTF-8" ?>
<testsuites>
<testsuite name="Frisby Test:blank action " errors="0" tests="1" failures="0" time="0.284" timestamp="2017-03-21T13:55:15">
<testcase classname="Frisby Test: blank action " name="
[ POST https://localhost:8443/api/v2/settings/pointwise ]" time="0.282"></testcase>
</testsuite>
</testsuites>
so my difficulty is that I need to read each and every file to check if tests is pass or not. I want some kind of overview page like http://maven.apache.org/surefire/maven-surefire-report-plugin/surefire-report.html
which contain information how many test passed, fail etc. Is it possible to get that.
Because we started same tests with different interface, then take all reports and send them to jenkins. It's difficult to recognise with what interface we have errors. Before we use nose, it has parameter "--xunit-testsuite-name", has pytest analog?
I want to change name of test suite in reports
testsuite errors="0" failures="0" name="pytest" skips="0" tests="12" time="103.702"
to
testsuite errors="0" failures="0" name="inteface1" skips="0" tests="12" time="103.702"
You can add a junit_suite_name attribute to your Pytest config file since pytest 3.1:
[pytest]
junit_suite_name = my_suite
This can also be set in the command line:
pytest ... -o junit_suite_name=my_suite
I try to start my selenium test using command line but test are fired to http://mycomputer and not to http://mycomputer/myapplication
D:\projectsnet\Production\MyWebTests\tools\selenium-server>java -jar selenium-
server.jar -Dhttp.proxyHost=mycomputer -Dhttp.proxyport=4444 -htmlSuite "*firefox" "ht
tp://mycomputer/myapplication/" "D:\mypathtotestssuites\0001-testsuite.html" "c:\temp\result.html"
In test cases I have got link
<link rel="selenium.base" href="http://mycomputer/myapplication/" />
This is by design.
If you are doing development against http://mycomputer/myapplication/ then I would recommend that you do http://mycomputer as your baseURL and then add
open | /myapplication/
To the top of your tests.