I have my unit tests written in Nunit 2.6 but planning to upgrade to Nunit 3.6.1 , however I noticed a weird problem with Nunit 3.6.1 (or may be I did not understood it correctly). The problem is around OneTimeSetUp().
In Nunit 2.6.3, I had SetUpFixtureAttribute [SetUpFixture] and inside that SetUpAttribute [SetUp] and it worked as expected for me, the flow was
SetUpFixture.Setup
TestFixture.Setup
TestFixture.Test
TestFixture.TearDown
TestFixture.Setup
TestFixture.Test
TestFixture.TearDown
SetUpFixture.TearDown
When I upgraded to Nunit 3, I replaced the SetUp() inside SetUpFixture with OneTimeSetUp, and after running my code I got following flow
TestFixture.Setup
TestFixture.Test
TestFixture.TearDown
SetUpFixture.OneTimeSetUp
SetUpFixture.OneTimeTearDown
Following is the sample code which I tried on my machine and also the command line output
[SetUpFixture]
public class TestBase
{
[OneTimeSetUp]
//[SetUp]
public static void MyTestSetup()
{
Console.WriteLine(" ---------- Calling OneTimeSetUp ----------");
}
}
[TestFixture]
class TestClass : TestBase
{
[Test]
public void test()
{
Console.WriteLine("\n ....I'm inside TestClass.test() ....");
}
}
Console Output
=> TestSample.TestClass.test
....I'm inside TestClass.test() ....
=> TestSample.TestClass
---------- Calling OneTimeSetUp ----------
=> TestSpecflow.TestBase
---------- Calling OneTimeSetUp ----------
Can someone please suggest what am i missing here ?
I'm running the test via nunit-console
The issue is that the output is misleading and not in the order the code is executed. Because NUnit 3 supports parallel execution, it captures output and displays it on the console when that level of test execution is completed.
In your case, the fixture setup wraps the tests, so it finishes executing after the tests and outputs the captured text afterward.
If you debug your tests, or switch your Console.WriteLine calls to TestContext.Progress.WriteLine which outputs immediately, you will see that the code is executed in the order you expect.
If it is not in the order you expect, look at the namespaces. Remember that a [SetupFixture] is used to setup at the namespace level. If your tests are in a different namespace, they may be called in a different order. If you want a setup for all of your tests, put the class in your top level namespace, or if you have multiple namespaces, in no namespace.
Here is some test code,
namespace NUnitFixtureSetup
{
[SetUpFixture]
public class SetupClass
{
[OneTimeSetUp]
public void MyTestSetup()
{
TestContext.Progress.WriteLine("One time setup");
}
}
[TestFixture]
public class TestClass
{
[Test]
public void TestMethod()
{
TestContext.Progress.WriteLine("Test Method");
}
}
}
And here is the output from running with nunit3-console.exe
=> NUnitFixtureSetup.SetupClass
One time setup
=> NUnitFixtureSetup.TestClass.TestMethod
Test Method
Rob's answer is the fundamental reason you have a problem, but there is an additional one, which is present in your code although not in Rob's.
In your code, you are using TestBase twice: as a SetUpFixture and as the base class for your TestFixture.
That means the OneTimeSetUp method will be used twice... Once before all the fixtures in the namespace and once before any test fixture that inherits from it. Using a SetUpFixture in this way defeats it's purpose, which is to have some code that runs only once before all the fixtures in a namespace.
Use separate classes as a base class (if you need one) and as a setup fixture (if you need one of those).
Related
Im having 4 test classes inside one project (Lets call them Class A, Class B, Class C, Class D)
Each of these 3 classes have two [TestFixture("string")], which makes it to 8 tests in total.
All classes are having the [Parallelizable] parameter.
When i start the test all at once by clicking inside the Test Explorer on the name of the project and "Run", then it will start all 8 tests at the same time.
The problem here is, that it consumes a lot RAM and the tests fail because it takes too long to load and i get a timeout error (Im doing automation tests with selenium in chrome)
Now i want to define a order.
For example:
Class A and Class B should start parallel
Class C and Class D should start parallel when Class A and Class B is done
Is it possible?
I tried the parameter [Order(1)] for Class A and Class B and Order(2)] for Class C and Class D
But when i run the tests, all 8 tests start to load.
Example from my code:
[TestFixture("normalUser")]
[TestFixture("adminUser")]
[Parallelizable]
public class ImportTest
{
private IWebDriver webDriver;
private const int waitTimer = 60;
public WebDriverWait w;
public string userRole;
// Constructor
public ImportTest(string userRole)
{
this.userRole = userRole;
Console.WriteLine(userRole);
}
////-----------------------------
[SetUp]
{
}
//-------------------------------
[Test]
public void Test1()
{
Do Test
}
[Test]
public void Test2()
{
Do Test
}
//--------------------------
[TearDown]
public void CloseBrowser()
{
webDriver.Quit();
}
}
First, I'll describe what's happening...
The OrderAttribute was created in NUnit V2, before parallel tests existed. It defines the order in which tests are started. Since there was no parallelism at the time, one test had to finish before the next one started.
When parallel execution was introduced in NUnit 3, Order was not exactly broken, because it continued to start tests in the specified order. But many users perceptions were "broken", because they thought that one test would not start until the prior one finished.
Order could, of course, be changed to work like that. However, at this point, that would be a breaking change for some people, so you most likely won't see it happen until there's an NUnit 4.
So... what can you do as a workaround? I can see three options...
The simplest approach would be to make each fixture [NonParallelizable]. Then they would all run separately. You should try that first and see if the performance is acceptable to you. If you want the tests within each fixture to run in parallel, you could use [Parallelizable(ParallelScope.Children)] instead but that might break things if the tests change the state of the fixture or of any common references found in the fixture.
Alternatively, you could pick only some fixtures to mark as [NonParallelizable]. In that case, I'd do it for the ones that consume a lot of memory.
For the most effort required, you could implement ordering yourself for these classes. I'd do that by creating some sort of shared token... e.g. a lock... which each class had to acquire on startup. I'd grab the lock in the OneTimeSetUp for a fixture and release it in the onetime teardown. The locking code should be before any setup code, which acquires resources and should be released after your teardown releases those resources.
I made option 3 rather sketchy because (a) I don't know precisely how your application works and (b) I presume that you won't do it unless it's absolutely necessary.
Final advice: don't make any assumptions about the performance impact of any of these options, even the first. Measure first!
I'm running a set of integration tests and while most of them finish within reasonable timeline, there're two tests that are waiting for specific conditions (financial markets conditions to be precise) and they can last for 2-3 hours. So ideally I'd like to achieve two things:
Start those two tests after other tests are finished
Run them in parallel
Is there a way to achieve that in NUnit/XUnit (or other test runner)?
Start those two tests after other tests are finished
You could keep those two tests in a separate nunit test project, allowing you to run all the other tests separately.
For running tests in parallel, this blog has a nice article:
https://blog.sanderaernouts.com/running-unit-tests-in-parallel-with-nunit
Mark your test fixtures with the Parallelizable attribute and set the parallel scope to ParallelScope.All.
Create a private class called TestScope and implement IDisposable.
Put all startup and clean-up logic inside the TestScope constructor and .Dispose() method respectively.
Wrap your test code in a using (var scope = new TestScope) { ... } block
[TestFixture]
[Parallelizable(ParallelScope.All)]
public class MyClassTests {
[Test]
public void MyParallelTest() {
using(var scope = new TestScope()) {
scope.Sut.DoSomething();
scope.Repository.Received(1).Save();
}
}
private sealed class TestScope : IDisposable {
public IRepository Repository{get;}
public MyClass Sut {get;}
public TestScope() {
Repository = Substitute.For<IRepository>();
Sut = new MyClass(Repository);
}
public void Dispose() {
//clean-up code goes here
Repository?.Dispose()
}
}
}
You should take precautions to ensure that while running in parallel, your tests do not interfere with each other.
As the article states:
How to safely run tests in parallel
To allow tests to run in parallel
without them interfering with each other, I have been applying the
following pattern for a while:
Create a nested private TestScope class that implements IDisposable.
All initialization or startup code that would go into the SetUp method
goes into the constructor of the TestScope class.
Any clean-up or
teardown code that would go into the TearDown method goes into the
Dispose method All tests run inside a using block that handles the
creation and disposal of the TestScope.
[TestFixture]
[Parallelizable(ParallelScope.All)]
public class MyClassTests {
[Test]
public void MyParallelTest() {
using(var scope = new TestScope()) {
scope.Sut.DoSomething();
scope.Repository.Received(1).Save();
}
}
private sealed class TestScope : IDisposable {
public IRepository Repository{get;}
public MyClass Sut {get;}
public TestScope() {
Repository = Substitute.For<IRepository>();
Sut = new MyClass(Repository);
}
public void Dispose() {
//clean-up code goes here
Repository?.Dispose()
}
}
}
The article provides more valuable advice. I suggest reading it, and thanking the author.
Parallel test run depends on arguments of test runner, if you are using xUnit console test runner there is a -parallel argument or MSBuild Options, see: https://xunit.net/docs/running-tests-in-parallel. But in any case you have to split your long running tests on separate Test Classes.
It is harder to guarantee order of test running you could use TestCollection (however according to quide collection running sequentially). You could rename your long running test to place them at the end of list, i.e TestClass2 will be executed after TestClass1. You also could use category attribute parameter to separate tests and run them via 2 commands from dotnet test --filter=TestCategory=LongTests (one for long and another for others), see https://learn.microsoft.com/ru-ru/dotnet/core/testing/selective-unit-tests?pivots=mstest
I'm using NUnit 2.X library but want to use NUnit 3.X now. I have some problems about migration from 2.X to 3.X. First i have a setup fixture class. Here is the 2.X version;
using System;
using System.IO;
using System.Reflection;
using HalisEnerji.QuantSignal.Logging;
using NUnit.Framework;
namespace HalisEnerji.QuantSignal.Tests
{
[SetUpFixture]
public class Initialize
{
[SetUp]
public void SetLogHandler()
{
Log.LogHandler = new ConsoleLogHandler();
}
}
}
First problem is fixed via change "Setup" attribute with "OneTimeSetUp" attribute. Second problem fixed via add some codes for set test directory. Because i'm using Re-Sharper test engine. Here is final shape of setup fixture;
using System;
using System.IO;
using System.Reflection;
using HalisEnerji.QuantSignal.Logging;
using NUnit.Framework;
namespace HalisEnerji.QuantSignal.Tests
{
[SetUpFixture]
public class Initialize
{
[OneTimeSetUp]
public void SetLogHandler()
{
Log.LogHandler = new ConsoleLogHandler();
var assembly = Assembly.GetExecutingAssembly();
var localPath = new Uri(assembly.CodeBase).LocalPath;
var direcotyName = Path.GetDirectoryName(localPath);
if (direcotyName != null)
{
Environment.CurrentDirectory = direcotyName;
}
}
}
}
Well after solve setup fixture problem, my real problems begins with use TestCaseSource/TestCaseData. Here is sample 2.X version;
[Theory]
[TestCaseSource("CreateSymbolTestCaseData")]
public void CreateSymbol(string ticker, Symbol expected)
{
Assert.AreEqual(Symbol.Create(ticker), expected);
}
private TestCaseData[] CreateSymbolTestCaseData()
{
return new []
{
new TestCaseData("SPY", new Symbol(Security.GenerateEquity("SPY"), "SPY")),
new TestCaseData("EURUSD", new Symbol(Security.GenerateForex("EURUSD"), "EURUSD"))
};
}
2.X version creating exception and my tests are fail. Shortly exception telling that TestCaseData provider method must be static. Well, after mark method with static identifier test working correctly but this time my other test failing (before use static identifier it's not failing). Why my other test failing? Because it's reading a file from test directory and somehow test working before setup up fixture codes run and change test directory!
Before use static identifier first SetUpFixture codes run and then tests code run. After use static identifier order changing my test that read file from test directory (which is Re-Sharper's temporary directory and not contain necessary file) run first after that SetUpFixture codes run. Any idea how all my tests to be successful?
UPDATE:
Explain some units;
I have Initialize.cs (part of my test assembly) which is responsible setup CurrentDirectory.
I have Config.cs (part of my project infrastructure assembly) which is my project configuration file and it has public static readonly Setttings property which is reading configuration file from CurrentDirectory.
I have ConfigTests.cs (part of my test assembly) which is contain some test methods for read/write Settings property.
When i debug tests;
Before use any static TestCaseSource, they are working below order;
A. Initialize.cs => Setup method
B. Config.cs => static Settings property getter method
C. ConfigTests.cs => first test method
So initialize working first others working later all tests successfully passing from test.
After use static TestCaseSource for inside other test file lets say OrdersTests.cs (excluded from project for first scenario after that included again), somehow working order is changing like below;
A. Config.cs => static Settings property getter method
B. OrdersTests.cs => static TestCaseSource method (not test method)
C. Initialize.cs => Setup method
D. ConfigTests.cs => first test method
E. OrdersTests.cs => first test method
So, my ConfigTests.cs tests failing because Initialize.cs working after Config.cs. I hope with this update my problem is more clear.
Is this problem related NUnit or Resharper or V.Studio? I don't know and all i know is my successfully passing tests are failing now!
UPDATE 2:
Chris,
Yes you are right. I explore project in detail and i saw the problem is related that my project's some classes accessing to static Config class and it's static Settings property (before run test setup fixture method and even before static test case source method!). You talk about order of process of test methods; NUnit doing tests like your said, not like i said. But when i try to use your solution (set current directory before test case source) it's not working. Because of that i solve my problem in another way. I'm not happy but at least my test methods working now. Could you please tell me what are the technical reasons that run static test case methods before initialize/setup method? Is this because of NUnit or because of infrastructure of .Net Framework? I'm not fanatic about NUnit and/or TDD. I don't have deep knowledge about these concepts but it does not make sense to me: run any method before setup method.
Thanks for your interest.
Because it's reading a file from test directory and somehow test working before setup up fixture codes run and change test directory!
How are you reading this file? You should use TestContext.CurrentContext.TestDirectory to get the test directory in NUnit 3, rather than relying on the location of the current directory. See the Breaking Changes page for details.
Edit: I also see you've tagged this ReSharper 7.1. You should be aware that this version of resharper does not support NUnit 3 - the first version that did is ReSharper 10. Your tests will appear to run correctly - however you may experience weird side effects, and this may break in any future version of NUnit.
Response to update:
Take a look at NUnit 3's Breaking Changes page. There are two relevant breaking changes between NUnit 2 and 3.
TestCaseSource's must now be static.
The CurrentDirectory is no longer set to Environment.CurrentDirectory by default.
The first you've solved easily enough. The second, is what's now causing you issues.
NUnit 3 runs it's methods in this order:
Evalute TestCaseSource methods (OrdersTests.cs)
Run SetUpFixture (Initialize.cs)
Run Test (ConfigTests/OrdersTests)
I'm not sure what Config.cs is being called before your TestCaseSource method - are you sure of that order? Does anything in CreateSymbolTestCaseData() call anything in Config.cs You could try rewriting your TestCaseSource as such:
private TestCaseData[] CreateSymbolTestCaseData()
{
Environment.CurrentDirectory = "c:\RequiredDirectory";
return new []
{
new TestCaseData("SPY", new Symbol(Security.GenerateEquity("SPY"), "SPY")),
new TestCaseData("EURUSD", new Symbol(Security.GenerateForex("EURUSD"), "EURUSD"))
};
}
.Net 4.6.2 has a known breaking change. Path.GetDirectoryName no longer works with URI's. However for existing code which already uses this method, it breaks.
There is a workaround provided by Microsoft. We can add following config entry in runtime <AppContextSwitchOverrides value="Switch.System.IO.UseLegacyPathHandling=true"/>
UseLegacyPathHandling runtime switch supposed to allow existing code to work. It does work when I am running the application normally. However if same code block is executed in a Unit Test, it fails.
e.g.
class Program
{
static void Main(string[] args)
{
System.IO.Path.GetDirectoryName("file://localhost/etc/fstab");
}
}
[Microsoft.VisualStudio.TestTools.UnitTesting.TestClass]
public class TestClass1
{
[TestMethod]
public void TestUrl()
{
System.IO.Path.GetDirectoryName("file://localhost/etc/fstab");
}
}
In above code if I run the program directly then System.IO.Path.GetDirectoryName("file://localhost/etc/fstab"); statement runs fine. However if I run the unit test case then it fails. Why is it behaving like this?
I have some old MbUnit code which looks like this:
public class MyFixture {
[SetUp]
public void SetUp() {
// Add data to database
}
[Test, Rollback]
public void DoTest() {
// Tests with the data
}
}
My new NUnit Rollback attribute looks a bit like this:
public class RollbackAttribute : TestActionAttribute
{
public override void BeforeTest(TestDetails testDetails)
{
// Begin transaction
}
public override void AfterTest(TestDetails testDetails)
{
// Abort transaction
}
}
The Rollback should roll back the new data added in the SetUp method as well as any modifications during the test itself. Unfortunately, it seems that NUnit's BeforeTest runs after the fixture's SetUp method, so the data added during SetUp is not rolled back.
Is there a way to run BeforeTest before SetUp?
One option would be a base class, and replace the existing Rollback attributes with additional code in SetUp and TearDown, however some of my tests require running outside a transaction (they create multiple transactions themselves during the test run), so adding transactions around all test cases would require a bit of care. I'd rather find a solution which can re-use the existing Rollback attributes.
Is there a way to run BeforeTest before SetUp?
I don't think so, see e.g. this related discussion on google groups. Issue being discussed there is very similar, as you can see, code in SetUp method would run even prior to BeforeTest method used on test fixture level (you have it on test level).
Workaround from my point of view would be to remove the SetUpAttribute from the SetUp method and call the SetUp method explicitly at the beginning of the each test, i.e.:
public class MyFixture
{
public void SetUp()
{
// Add data to database
}
[Test, Rollback]
public void DoTest()
{
SetUp();
// Tests with the data
}
}
Your question also reminded me of question that marc_s raised in this SO thread. Question is unrelated to your problem, but he used the same construct as I am proposing above so it is perhaps not that bad idea.
EDIT:
Here is an opened issue on NUnit's github. But still, requested order there is:
BeforeTest (BaseFixture)
BaseSetUp BeforeTest (Fixture)
SetUp
BeforeTest (Test)
Test AfterTest (Test)
TearDown AfterTest (Fixture)
BaseTearDown AfterTest (BaseFixture)
So not exactly what you desire, "BeforeTest (Test)" would be executed after SetUp.