Run long-running NUnit/xUnit tests so that they are not blocking other tests - nunit

I'm running a set of integration tests and while most of them finish within reasonable timeline, there're two tests that are waiting for specific conditions (financial markets conditions to be precise) and they can last for 2-3 hours. So ideally I'd like to achieve two things:
Start those two tests after other tests are finished
Run them in parallel
Is there a way to achieve that in NUnit/XUnit (or other test runner)?

Start those two tests after other tests are finished
You could keep those two tests in a separate nunit test project, allowing you to run all the other tests separately.
For running tests in parallel, this blog has a nice article:
https://blog.sanderaernouts.com/running-unit-tests-in-parallel-with-nunit
Mark your test fixtures with the Parallelizable attribute and set the parallel scope to ParallelScope.All.
Create a private class called TestScope and implement IDisposable.
Put all startup and clean-up logic inside the TestScope constructor and .Dispose() method respectively.
Wrap your test code in a using (var scope = new TestScope) { ... } block
[TestFixture]
[Parallelizable(ParallelScope.All)]
public class MyClassTests {
[Test]
public void MyParallelTest() {
using(var scope = new TestScope()) {
scope.Sut.DoSomething();
scope.Repository.Received(1).Save();
}
}
private sealed class TestScope : IDisposable {
public IRepository Repository{get;}
public MyClass Sut {get;}
public TestScope() {
Repository = Substitute.For<IRepository>();
Sut = new MyClass(Repository);
}
public void Dispose() {
//clean-up code goes here
Repository?.Dispose()
}
}
}
You should take precautions to ensure that while running in parallel, your tests do not interfere with each other.
As the article states:
How to safely run tests in parallel
To allow tests to run in parallel
without them interfering with each other, I have been applying the
following pattern for a while:
Create a nested private TestScope class that implements IDisposable.
All initialization or startup code that would go into the SetUp method
goes into the constructor of the TestScope class.
Any clean-up or
teardown code that would go into the TearDown method goes into the
Dispose method All tests run inside a using block that handles the
creation and disposal of the TestScope.
[TestFixture]
[Parallelizable(ParallelScope.All)]
public class MyClassTests {
[Test]
public void MyParallelTest() {
using(var scope = new TestScope()) {
scope.Sut.DoSomething();
scope.Repository.Received(1).Save();
}
}
private sealed class TestScope : IDisposable {
public IRepository Repository{get;}
public MyClass Sut {get;}
public TestScope() {
Repository = Substitute.For<IRepository>();
Sut = new MyClass(Repository);
}
public void Dispose() {
//clean-up code goes here
Repository?.Dispose()
}
}
}
The article provides more valuable advice. I suggest reading it, and thanking the author.

Parallel test run depends on arguments of test runner, if you are using xUnit console test runner there is a -parallel argument or MSBuild Options, see: https://xunit.net/docs/running-tests-in-parallel. But in any case you have to split your long running tests on separate Test Classes.
It is harder to guarantee order of test running you could use TestCollection (however according to quide collection running sequentially). You could rename your long running test to place them at the end of list, i.e TestClass2 will be executed after TestClass1. You also could use category attribute parameter to separate tests and run them via 2 commands from dotnet test --filter=TestCategory=LongTests (one for long and another for others), see https://learn.microsoft.com/ru-ru/dotnet/core/testing/selective-unit-tests?pivots=mstest

Related

Define order of "Parallelizable" in NUnit

Im having 4 test classes inside one project (Lets call them Class A, Class B, Class C, Class D)
Each of these 3 classes have two [TestFixture("string")], which makes it to 8 tests in total.
All classes are having the [Parallelizable] parameter.
When i start the test all at once by clicking inside the Test Explorer on the name of the project and "Run", then it will start all 8 tests at the same time.
The problem here is, that it consumes a lot RAM and the tests fail because it takes too long to load and i get a timeout error (Im doing automation tests with selenium in chrome)
Now i want to define a order.
For example:
Class A and Class B should start parallel
Class C and Class D should start parallel when Class A and Class B is done
Is it possible?
I tried the parameter [Order(1)] for Class A and Class B and Order(2)] for Class C and Class D
But when i run the tests, all 8 tests start to load.
Example from my code:
[TestFixture("normalUser")]
[TestFixture("adminUser")]
[Parallelizable]
public class ImportTest
{
private IWebDriver webDriver;
private const int waitTimer = 60;
public WebDriverWait w;
public string userRole;
// Constructor
public ImportTest(string userRole)
{
this.userRole = userRole;
Console.WriteLine(userRole);
}
////-----------------------------
[SetUp]
{
}
//-------------------------------
[Test]
public void Test1()
{
Do Test
}
[Test]
public void Test2()
{
Do Test
}
//--------------------------
[TearDown]
public void CloseBrowser()
{
webDriver.Quit();
}
}
First, I'll describe what's happening...
The OrderAttribute was created in NUnit V2, before parallel tests existed. It defines the order in which tests are started. Since there was no parallelism at the time, one test had to finish before the next one started.
When parallel execution was introduced in NUnit 3, Order was not exactly broken, because it continued to start tests in the specified order. But many users perceptions were "broken", because they thought that one test would not start until the prior one finished.
Order could, of course, be changed to work like that. However, at this point, that would be a breaking change for some people, so you most likely won't see it happen until there's an NUnit 4.
So... what can you do as a workaround? I can see three options...
The simplest approach would be to make each fixture [NonParallelizable]. Then they would all run separately. You should try that first and see if the performance is acceptable to you. If you want the tests within each fixture to run in parallel, you could use [Parallelizable(ParallelScope.Children)] instead but that might break things if the tests change the state of the fixture or of any common references found in the fixture.
Alternatively, you could pick only some fixtures to mark as [NonParallelizable]. In that case, I'd do it for the ones that consume a lot of memory.
For the most effort required, you could implement ordering yourself for these classes. I'd do that by creating some sort of shared token... e.g. a lock... which each class had to acquire on startup. I'd grab the lock in the OneTimeSetUp for a fixture and release it in the onetime teardown. The locking code should be before any setup code, which acquires resources and should be released after your teardown releases those resources.
I made option 3 rather sketchy because (a) I don't know precisely how your application works and (b) I presume that you won't do it unless it's absolutely necessary.
Final advice: don't make any assumptions about the performance impact of any of these options, even the first. Measure first!

Why Nunit3 OneTimeSetUp() is called after [Test] and not before

I have my unit tests written in Nunit 2.6 but planning to upgrade to Nunit 3.6.1 , however I noticed a weird problem with Nunit 3.6.1 (or may be I did not understood it correctly). The problem is around OneTimeSetUp().
In Nunit 2.6.3, I had SetUpFixtureAttribute [SetUpFixture] and inside that SetUpAttribute [SetUp] and it worked as expected for me, the flow was
SetUpFixture.Setup
TestFixture.Setup
TestFixture.Test
TestFixture.TearDown
TestFixture.Setup
TestFixture.Test
TestFixture.TearDown
SetUpFixture.TearDown
When I upgraded to Nunit 3, I replaced the SetUp() inside SetUpFixture with OneTimeSetUp, and after running my code I got following flow
TestFixture.Setup
TestFixture.Test
TestFixture.TearDown
SetUpFixture.OneTimeSetUp
SetUpFixture.OneTimeTearDown
Following is the sample code which I tried on my machine and also the command line output
[SetUpFixture]
public class TestBase
{
[OneTimeSetUp]
//[SetUp]
public static void MyTestSetup()
{
Console.WriteLine(" ---------- Calling OneTimeSetUp ----------");
}
}
[TestFixture]
class TestClass : TestBase
{
[Test]
public void test()
{
Console.WriteLine("\n ....I'm inside TestClass.test() ....");
}
}
Console Output
=> TestSample.TestClass.test
....I'm inside TestClass.test() ....
=> TestSample.TestClass
---------- Calling OneTimeSetUp ----------
=> TestSpecflow.TestBase
---------- Calling OneTimeSetUp ----------
Can someone please suggest what am i missing here ?
I'm running the test via nunit-console
The issue is that the output is misleading and not in the order the code is executed. Because NUnit 3 supports parallel execution, it captures output and displays it on the console when that level of test execution is completed.
In your case, the fixture setup wraps the tests, so it finishes executing after the tests and outputs the captured text afterward.
If you debug your tests, or switch your Console.WriteLine calls to TestContext.Progress.WriteLine which outputs immediately, you will see that the code is executed in the order you expect.
If it is not in the order you expect, look at the namespaces. Remember that a [SetupFixture] is used to setup at the namespace level. If your tests are in a different namespace, they may be called in a different order. If you want a setup for all of your tests, put the class in your top level namespace, or if you have multiple namespaces, in no namespace.
Here is some test code,
namespace NUnitFixtureSetup
{
[SetUpFixture]
public class SetupClass
{
[OneTimeSetUp]
public void MyTestSetup()
{
TestContext.Progress.WriteLine("One time setup");
}
}
[TestFixture]
public class TestClass
{
[Test]
public void TestMethod()
{
TestContext.Progress.WriteLine("Test Method");
}
}
}
And here is the output from running with nunit3-console.exe
=> NUnitFixtureSetup.SetupClass
One time setup
=> NUnitFixtureSetup.TestClass.TestMethod
Test Method
Rob's answer is the fundamental reason you have a problem, but there is an additional one, which is present in your code although not in Rob's.
In your code, you are using TestBase twice: as a SetUpFixture and as the base class for your TestFixture.
That means the OneTimeSetUp method will be used twice... Once before all the fixtures in the namespace and once before any test fixture that inherits from it. Using a SetUpFixture in this way defeats it's purpose, which is to have some code that runs only once before all the fixtures in a namespace.
Use separate classes as a base class (if you need one) and as a setup fixture (if you need one of those).

Can a TestActionAttribute in NUnit run BeforeTest before the fixture's own SetUp method?

I have some old MbUnit code which looks like this:
public class MyFixture {
[SetUp]
public void SetUp() {
// Add data to database
}
[Test, Rollback]
public void DoTest() {
// Tests with the data
}
}
My new NUnit Rollback attribute looks a bit like this:
public class RollbackAttribute : TestActionAttribute
{
public override void BeforeTest(TestDetails testDetails)
{
// Begin transaction
}
public override void AfterTest(TestDetails testDetails)
{
// Abort transaction
}
}
The Rollback should roll back the new data added in the SetUp method as well as any modifications during the test itself. Unfortunately, it seems that NUnit's BeforeTest runs after the fixture's SetUp method, so the data added during SetUp is not rolled back.
Is there a way to run BeforeTest before SetUp?
One option would be a base class, and replace the existing Rollback attributes with additional code in SetUp and TearDown, however some of my tests require running outside a transaction (they create multiple transactions themselves during the test run), so adding transactions around all test cases would require a bit of care. I'd rather find a solution which can re-use the existing Rollback attributes.
Is there a way to run BeforeTest before SetUp?
I don't think so, see e.g. this related discussion on google groups. Issue being discussed there is very similar, as you can see, code in SetUp method would run even prior to BeforeTest method used on test fixture level (you have it on test level).
Workaround from my point of view would be to remove the SetUpAttribute from the SetUp method and call the SetUp method explicitly at the beginning of the each test, i.e.:
public class MyFixture
{
public void SetUp()
{
// Add data to database
}
[Test, Rollback]
public void DoTest()
{
SetUp();
// Tests with the data
}
}
Your question also reminded me of question that marc_s raised in this SO thread. Question is unrelated to your problem, but he used the same construct as I am proposing above so it is perhaps not that bad idea.
EDIT:
Here is an opened issue on NUnit's github. But still, requested order there is:
BeforeTest (BaseFixture)
BaseSetUp BeforeTest (Fixture)
SetUp
BeforeTest (Test)
Test AfterTest (Test)
TearDown AfterTest (Fixture)
BaseTearDown AfterTest (BaseFixture)
So not exactly what you desire, "BeforeTest (Test)" would be executed after SetUp.

Integration Testing Entity Framework CRD operations

I'm struggling with Integration Tests when using Entity Framework.
I seed my database with Test data in my Test project, but I am wondering how you manange to test the Create, Update and Delete operations.
Basicly I have my Test data which e.g. contains 5 customer entries... I can now write some unit tests to get the data based on these 5 entries. (e.g. get all will return a collection containing 5 items).
But what if I have a test which deletes 1 customer, this means the GetAll test will expect 5 customers, but only return 4 (if this test is executed after the delete test) and fails.
How do you work around this issue? Do you give a certain order to your tests or reseed the database before every test (but this sounds so bad?)...
Thanks !
An effective way to do this is to use Transaction Scope. This basically wraps all sql calls and rolls back changes if the scope is disposed without calling the Complete method. The basic test will look like this.
public class TransactionalTestsBase
{
private TransactionScope _scope;
[TestInitialize]
public void Initialize()
{
_scope = new TransactionScope();
}
[TestCleanup]
public void TestCleanup()
{
_scope.Dispose();
}
[TestMethod]
public void CrudAction()
{
var repo = new YourRepo();
var client = ; // Make client
repo.DeleteClient(client);
Assert.AreEqual(4,repo.GetClients().Count());
}
}
Ideally you would inherit from this base test class not write your tests in it.
There is some new heat in beta still that I think will help with this greatly in the future. Have a look at Effort

Execute TestSetup only some of the time?

Within one fixture is it possible to markup tests in such a way that the test-setup is only called for some tests and not for others?
[Test] public void TestWithoutSetup() { .. }
[Test] public void TestWithSetup() { .. }
What would I need to do make the above work?
There's no attribute to accomplish what you want.
I would suggest refactoring your test cases into two separate classes that inherit base functionality from an abstract test class. Each test class can have its own setup method.