I am running nunit-console from a CI configured in TeamCity to run tests from various assemblies. Once one of the TestFixtures has a failing test, then the test execution will stop.
Currently i am able to see the first tests that failed, but am unaware if there are more testfixtures that might fail down the line.
I would like to get a summary that lists the failing tests and test fixtures, without all the details of the exceptions thrown.
Anyone have any ideas?
Thanks.
NUnit should run all of the unit tests in the specified assembly, regardless of the number of test failures. The first thing I would check is the raw xml output from the unit test run. You may find that the tests are being executed, but the build server is failing to display all of the results. If that is the case, there may be a faulty xslt that needs to be modified.
Another thing to try is running all of the tests on your box using the command-line tool, and see if it runs all of the tests. If they run on your box but not the server, you may have a configuration problem on the build box.
Yet another possibility is that the failure is a critical one (failure to load an assembly perhaps) which is causing NUnit itself to error out.
Related
I have a TeamCity build configuration that builds a C# project, runs some unit tests, and then does some extra things. My question is: Can I get information about my unit test run stored into build configuration variables (i.e. how many tests were run, how many were successful, how many failed, how many were skipped) so that I can then check these variables in a PowerShell script in later build steps and perform different actions depending on how many tests have passed?
AFAIK the best way is to ask these information directly to teamcity server using its REST API (pay attention, maybe the build locator could be a little be tricly to be found, if the build is still running).
By other hand, you can parse your NUnit test result file (or files if you run more than one NUnit test runner step in your build) inside your build agent machine.
I am trying to run some unit tests in teamcity build configuration. I am using NUnit.ConsoleRunner.3.6.1 . However It says "Has no TestFixture". But when I tried to run it in command prompt using same runner it completed successfuly. What are the possible reasons ?
Really, more info is needed to judge what your problem is. However, most errors of that type indicate that the nunit framework has not been deployed with the test assembly that uses it.
Is it possible to run opencover without running unit tests?
I have the TestResults.xml from NUnit and want to pass this to OpenCover without running the unit tests again.
Is this possible?
Q1. Is it possible to run opencover without running unit tests?
OpenCover can run against most .NET application that can be launched from the commandline. With a little effort you can get it to run against a service like IIS.
Q2. I have the TestResults.xml from NUnit and want to pass this to
OpenCover without running the unit tests again. Is this possible?
No, it will not be able to do what you want as the information in the TestResults.xml is about tests (pass/fail) and is not enough to determine what code was actually executed by those tests.
Just run your tests with OpenCover using the nunit-console.exe as the target - instructions exist in the documentation provided to help you.
I do not know OpenCover but from what I guess about dotCover it needs to go alongside the unit tests as they progress through your code line by line. Code coverage is then determined by what percentage of your code has been visited.
We use NUnit for unit tests and TeamCity as our CI server. After each commit all tests are executing. If some tests failed then e-mail notifications are sent.
All went well but today I noticed that many tests were ignored. Also I saw message which described the reason:
TestFixtureSetUp failed in MyApplicationTests
I was confused why these tests were ignored but not failed. My concern is that developers think all is going well but actualy tests were not run (ignored).
Question: how configure NUnit to fail tests (instead of ignore) if TestFixtureSetUp failed?
Maybe we can configure TeamCity to send e-mail notifications if tests are ignored. But it is not what I want because we have some tests marked with Ignore attribute. So notification will be send each time and becomes useless.
TeamCity cannot filter this event and report it differently to you. There seems to be no way of programmatically failing all the tests in a fixture from the TestFixtureSetUp callback.
So, in my opinion, you have no chance but closely monitoring the Ignores in your build results. There seems to be no automatic way of distinguishing them from the tests you actually are ignoring.
As a side note, whenever I or my colleagues marked tests with the Ignore attribute (in my career) it was never temporary. Always permanent. We should use the Ignore flag very carefully, in my opinion.
NUnit tests being aborted randomly (Involves ServiceStack & RavenDB)
We have a project where we use ServiceStack and RavenDB. Testing is done using NUnit.
When running the tests individually everything works fine.
When running more than one test a few will do their thing (pass/fail) but very often one of the tests will be aborted and all subsequent tests will not be run.
Which test aborts is seemingly random. The more tests that are being run the higher the chance that one will be aborted.
The test that gets aborted does seem to be able to run through all its actions though seeing from the test log.
Unfortunately I'm not able to give more info besides the following files which show the way our tests are set up.
IntegrationBaseTest.cs (Base test class)
GlobalSetupFixture.cs
AccountServiceTests.cs (Example file with tests)
test log (Log of aborted test, in this case DeleteAccount_DeletesAccount)
Result view of running all tests in AccountServiceTests.cs.
Which test gets aborted is completely random.
Does anyone have any idea of what I could try to fix this? :)
It turned out that when disabling the logging the tests ran normally without aborting.
I'm not sure what caused them to abort but I think it might be because the jetbrains taskrunner was running out of memory because of all the logs.