I have 2 set of tests and have run then on a code base with bullseye enabled. Have got the merged coverage reports of all tests of set1 and set2.
Now is there a way to compare these 2 coverage reports to check what are the unique coverpoints being hit by set1 and unique coverpoints hit by set 2
Related
Background:
We have a regression test suite that tests the generation of some large xml files by comparing them field-by-field to the corresponding baseline files.
This is implemented using a junit4 parameterized test running the test for each file and assertj soft assertions to collect the field comparison errors.
Problem:
When I run this test from my IDE, I can see the assertion errors output after each test (each file), but when run from maven, surefire collects all the errors in memory and outputs them at the end (when all the tests for the class have finished). Now, running this for 2000+ files, comparing hundreds of fields in each and having a lot of differences results in OutOfMemoryError, even with 8GB of heap allocated.
Question:
I'm trying to find out if there's any option in surefire to either output the errors after each test or not collect & output them at all (we're logging them into a file and generating custom reports from there anyway).
I've tried <redirectTestOutputToFile>true</redirectTestOutputToFile> but this only redirects stdout (the logs produced during test execution), the assertion errors are still spit to console after the tests finish.
Other options I see are:
Split the test parameters into smaller batches and run the test suite
for each batch, then aggregate the results - this could be done in
the Jenkins job.
Remove the detailed error reporting using soft assertions and only have a single assertion at the end of the test. This is what we had
before and obviously didn't help in finding errors. I wouldn't like
to go back there.
Add an option to run the tests in two modes:
use soft assertions to provide detailed error reporting when run locally (with a smaller set of data)
use a single assertion when run on Jenkins (with the full set of data) - here we're only interested in the logs, not the console output
The third solution would result in some ifs in the code and would make the code less readable, that's why I'm trying to solve this from configuration first.
Any other ideas? Thanx!
My team uses VSTS with hosted agents and a Visual Studio Test build task to run all tests and produce code coverage. However the tab Code coverage remains empty afterwards, only showing a link to download the *.coverage file.
I actually expect the code coverage results to be shown, with tables and graphs which projects are tested and it's respective coverage.
We are using the Visual Studio Test build task to test net461 assemblies with the Code Coverage enabled checked.
Somewhat related are both this and this issue, but not entirely, because we are just using MSTest Framework with built-in Visual Studio Test build task, which (I understand) should also automatically publish the code coverage results.
Am I missing something to get this Code coverage view working? Thanks!!
In Code coverage Tab, you need to use Publish Code Coverage Results task to show tables and graphs there.
While for Publish Code Coverage Results task, it only supports Code Coverage data in Jacoco or Cobertura formats. So the result of the *.coverage file can not be shown by tables and graphs in Code coverage Tab.
More details, you can refer the blog Browse Code Coverage reports and the issue How to publish the code coverage result with a .coverage file.
I have a CakeBuild build script which compiles a Visual Studio 2015 solution then uses the Cake add in for NUnit 3 to run the unit tests and generate the TestResult.xml file. The tests pass successfully when run locally and the report is generated.
This same build script is being run by TeamCity (version 10.0.2), with the build agent also successfully running the tests (all passing) and the TestReport.xml file being generated.
The TeamCity build configuration has been configured via the XML Report Processing Build Feature to import the NUnit report. The build log suggests this has also occurred successfully. Example of the log:
[10:06:41]NUnit report watcher
[10:06:41][NUnit report watcher] 1 report found for paths:
[10:06:41][NUnit report watcher] **/TestResult.xml
[10:06:41][NUnit report watcher] Successfully parsed
[10:06:41][Successfully parsed] 1 report
[10:06:41][Successfully parsed] build\TestResult.xml
However, there is no reference to any of the tests in the build run - the Test tab is missing completely and no other references to the tests at all.
Is there anything else that needs to be configured to get the test results to show up?
We've had the same issue and found the solution in https://github.com/nunit/teamcity-event-listener/issues/39
Main point is to make sure to include the package NUnit.Extension.TeamCityEventListener which reports the results to TeamCity.
In addition the Nunit Runner Settings should include
NUnit3(path, new NUnit3Settings {
TeamCity = true
});
I'm trying to build my Perl project with Jenkins in such a way that I get a JUnit report and a test coverage report. I'm having trouble getting my code report when my build fails. Here's how the build is configured:
perl Build.PL; ./Build;
cover -delete;
PERL5OPT=-MDevel::Cover prove -b --formatter=TAP::Formatter::JUnit t/ > junit.xml;
cover;
The problem with this is when prove exits with a non-zero status (i.e. when some of my tests fail) the cover command never runs; the build aborts, reporting:
Build step 'Execute shell' marked build as failure
Is there a good way to get a test coverage report, even when some of the tests are currently failing?
You could put the cover part into a Post Build task so it always executed and you wouldn't need any additional logic to check status as you go along
i'am looking for tool which will make a parallelizing NUnit test in TeamCity! i wanna parallelize my tests (which goes near 12 hours) on some PC, not one, which plugin or tool can i use for it?!
If you have several unit test projects, you could create one build configuration per agent which will be running the unit tests. Each build step would run a subset of the unit tests. You could then tie the builds together through dependent builds. Here is an example of an open source project using it (you can visually picture how many agents can be simultaneously running build steps.