.Net 4.6.2 has a known breaking change. Path.GetDirectoryName no longer works with URI's. However for existing code which already uses this method, it breaks.
There is a workaround provided by Microsoft. We can add following config entry in runtime <AppContextSwitchOverrides value="Switch.System.IO.UseLegacyPathHandling=true"/>
UseLegacyPathHandling runtime switch supposed to allow existing code to work. It does work when I am running the application normally. However if same code block is executed in a Unit Test, it fails.
e.g.
class Program
{
static void Main(string[] args)
{
System.IO.Path.GetDirectoryName("file://localhost/etc/fstab");
}
}
[Microsoft.VisualStudio.TestTools.UnitTesting.TestClass]
public class TestClass1
{
[TestMethod]
public void TestUrl()
{
System.IO.Path.GetDirectoryName("file://localhost/etc/fstab");
}
}
In above code if I run the program directly then System.IO.Path.GetDirectoryName("file://localhost/etc/fstab"); statement runs fine. However if I run the unit test case then it fails. Why is it behaving like this?
Related
I'm running a set of integration tests and while most of them finish within reasonable timeline, there're two tests that are waiting for specific conditions (financial markets conditions to be precise) and they can last for 2-3 hours. So ideally I'd like to achieve two things:
Start those two tests after other tests are finished
Run them in parallel
Is there a way to achieve that in NUnit/XUnit (or other test runner)?
Start those two tests after other tests are finished
You could keep those two tests in a separate nunit test project, allowing you to run all the other tests separately.
For running tests in parallel, this blog has a nice article:
https://blog.sanderaernouts.com/running-unit-tests-in-parallel-with-nunit
Mark your test fixtures with the Parallelizable attribute and set the parallel scope to ParallelScope.All.
Create a private class called TestScope and implement IDisposable.
Put all startup and clean-up logic inside the TestScope constructor and .Dispose() method respectively.
Wrap your test code in a using (var scope = new TestScope) { ... } block
[TestFixture]
[Parallelizable(ParallelScope.All)]
public class MyClassTests {
[Test]
public void MyParallelTest() {
using(var scope = new TestScope()) {
scope.Sut.DoSomething();
scope.Repository.Received(1).Save();
}
}
private sealed class TestScope : IDisposable {
public IRepository Repository{get;}
public MyClass Sut {get;}
public TestScope() {
Repository = Substitute.For<IRepository>();
Sut = new MyClass(Repository);
}
public void Dispose() {
//clean-up code goes here
Repository?.Dispose()
}
}
}
You should take precautions to ensure that while running in parallel, your tests do not interfere with each other.
As the article states:
How to safely run tests in parallel
To allow tests to run in parallel
without them interfering with each other, I have been applying the
following pattern for a while:
Create a nested private TestScope class that implements IDisposable.
All initialization or startup code that would go into the SetUp method
goes into the constructor of the TestScope class.
Any clean-up or
teardown code that would go into the TearDown method goes into the
Dispose method All tests run inside a using block that handles the
creation and disposal of the TestScope.
[TestFixture]
[Parallelizable(ParallelScope.All)]
public class MyClassTests {
[Test]
public void MyParallelTest() {
using(var scope = new TestScope()) {
scope.Sut.DoSomething();
scope.Repository.Received(1).Save();
}
}
private sealed class TestScope : IDisposable {
public IRepository Repository{get;}
public MyClass Sut {get;}
public TestScope() {
Repository = Substitute.For<IRepository>();
Sut = new MyClass(Repository);
}
public void Dispose() {
//clean-up code goes here
Repository?.Dispose()
}
}
}
The article provides more valuable advice. I suggest reading it, and thanking the author.
Parallel test run depends on arguments of test runner, if you are using xUnit console test runner there is a -parallel argument or MSBuild Options, see: https://xunit.net/docs/running-tests-in-parallel. But in any case you have to split your long running tests on separate Test Classes.
It is harder to guarantee order of test running you could use TestCollection (however according to quide collection running sequentially). You could rename your long running test to place them at the end of list, i.e TestClass2 will be executed after TestClass1. You also could use category attribute parameter to separate tests and run them via 2 commands from dotnet test --filter=TestCategory=LongTests (one for long and another for others), see https://learn.microsoft.com/ru-ru/dotnet/core/testing/selective-unit-tests?pivots=mstest
I'm using NUnit 2.X library but want to use NUnit 3.X now. I have some problems about migration from 2.X to 3.X. First i have a setup fixture class. Here is the 2.X version;
using System;
using System.IO;
using System.Reflection;
using HalisEnerji.QuantSignal.Logging;
using NUnit.Framework;
namespace HalisEnerji.QuantSignal.Tests
{
[SetUpFixture]
public class Initialize
{
[SetUp]
public void SetLogHandler()
{
Log.LogHandler = new ConsoleLogHandler();
}
}
}
First problem is fixed via change "Setup" attribute with "OneTimeSetUp" attribute. Second problem fixed via add some codes for set test directory. Because i'm using Re-Sharper test engine. Here is final shape of setup fixture;
using System;
using System.IO;
using System.Reflection;
using HalisEnerji.QuantSignal.Logging;
using NUnit.Framework;
namespace HalisEnerji.QuantSignal.Tests
{
[SetUpFixture]
public class Initialize
{
[OneTimeSetUp]
public void SetLogHandler()
{
Log.LogHandler = new ConsoleLogHandler();
var assembly = Assembly.GetExecutingAssembly();
var localPath = new Uri(assembly.CodeBase).LocalPath;
var direcotyName = Path.GetDirectoryName(localPath);
if (direcotyName != null)
{
Environment.CurrentDirectory = direcotyName;
}
}
}
}
Well after solve setup fixture problem, my real problems begins with use TestCaseSource/TestCaseData. Here is sample 2.X version;
[Theory]
[TestCaseSource("CreateSymbolTestCaseData")]
public void CreateSymbol(string ticker, Symbol expected)
{
Assert.AreEqual(Symbol.Create(ticker), expected);
}
private TestCaseData[] CreateSymbolTestCaseData()
{
return new []
{
new TestCaseData("SPY", new Symbol(Security.GenerateEquity("SPY"), "SPY")),
new TestCaseData("EURUSD", new Symbol(Security.GenerateForex("EURUSD"), "EURUSD"))
};
}
2.X version creating exception and my tests are fail. Shortly exception telling that TestCaseData provider method must be static. Well, after mark method with static identifier test working correctly but this time my other test failing (before use static identifier it's not failing). Why my other test failing? Because it's reading a file from test directory and somehow test working before setup up fixture codes run and change test directory!
Before use static identifier first SetUpFixture codes run and then tests code run. After use static identifier order changing my test that read file from test directory (which is Re-Sharper's temporary directory and not contain necessary file) run first after that SetUpFixture codes run. Any idea how all my tests to be successful?
UPDATE:
Explain some units;
I have Initialize.cs (part of my test assembly) which is responsible setup CurrentDirectory.
I have Config.cs (part of my project infrastructure assembly) which is my project configuration file and it has public static readonly Setttings property which is reading configuration file from CurrentDirectory.
I have ConfigTests.cs (part of my test assembly) which is contain some test methods for read/write Settings property.
When i debug tests;
Before use any static TestCaseSource, they are working below order;
A. Initialize.cs => Setup method
B. Config.cs => static Settings property getter method
C. ConfigTests.cs => first test method
So initialize working first others working later all tests successfully passing from test.
After use static TestCaseSource for inside other test file lets say OrdersTests.cs (excluded from project for first scenario after that included again), somehow working order is changing like below;
A. Config.cs => static Settings property getter method
B. OrdersTests.cs => static TestCaseSource method (not test method)
C. Initialize.cs => Setup method
D. ConfigTests.cs => first test method
E. OrdersTests.cs => first test method
So, my ConfigTests.cs tests failing because Initialize.cs working after Config.cs. I hope with this update my problem is more clear.
Is this problem related NUnit or Resharper or V.Studio? I don't know and all i know is my successfully passing tests are failing now!
UPDATE 2:
Chris,
Yes you are right. I explore project in detail and i saw the problem is related that my project's some classes accessing to static Config class and it's static Settings property (before run test setup fixture method and even before static test case source method!). You talk about order of process of test methods; NUnit doing tests like your said, not like i said. But when i try to use your solution (set current directory before test case source) it's not working. Because of that i solve my problem in another way. I'm not happy but at least my test methods working now. Could you please tell me what are the technical reasons that run static test case methods before initialize/setup method? Is this because of NUnit or because of infrastructure of .Net Framework? I'm not fanatic about NUnit and/or TDD. I don't have deep knowledge about these concepts but it does not make sense to me: run any method before setup method.
Thanks for your interest.
Because it's reading a file from test directory and somehow test working before setup up fixture codes run and change test directory!
How are you reading this file? You should use TestContext.CurrentContext.TestDirectory to get the test directory in NUnit 3, rather than relying on the location of the current directory. See the Breaking Changes page for details.
Edit: I also see you've tagged this ReSharper 7.1. You should be aware that this version of resharper does not support NUnit 3 - the first version that did is ReSharper 10. Your tests will appear to run correctly - however you may experience weird side effects, and this may break in any future version of NUnit.
Response to update:
Take a look at NUnit 3's Breaking Changes page. There are two relevant breaking changes between NUnit 2 and 3.
TestCaseSource's must now be static.
The CurrentDirectory is no longer set to Environment.CurrentDirectory by default.
The first you've solved easily enough. The second, is what's now causing you issues.
NUnit 3 runs it's methods in this order:
Evalute TestCaseSource methods (OrdersTests.cs)
Run SetUpFixture (Initialize.cs)
Run Test (ConfigTests/OrdersTests)
I'm not sure what Config.cs is being called before your TestCaseSource method - are you sure of that order? Does anything in CreateSymbolTestCaseData() call anything in Config.cs You could try rewriting your TestCaseSource as such:
private TestCaseData[] CreateSymbolTestCaseData()
{
Environment.CurrentDirectory = "c:\RequiredDirectory";
return new []
{
new TestCaseData("SPY", new Symbol(Security.GenerateEquity("SPY"), "SPY")),
new TestCaseData("EURUSD", new Symbol(Security.GenerateForex("EURUSD"), "EURUSD"))
};
}
I have my unit tests written in Nunit 2.6 but planning to upgrade to Nunit 3.6.1 , however I noticed a weird problem with Nunit 3.6.1 (or may be I did not understood it correctly). The problem is around OneTimeSetUp().
In Nunit 2.6.3, I had SetUpFixtureAttribute [SetUpFixture] and inside that SetUpAttribute [SetUp] and it worked as expected for me, the flow was
SetUpFixture.Setup
TestFixture.Setup
TestFixture.Test
TestFixture.TearDown
TestFixture.Setup
TestFixture.Test
TestFixture.TearDown
SetUpFixture.TearDown
When I upgraded to Nunit 3, I replaced the SetUp() inside SetUpFixture with OneTimeSetUp, and after running my code I got following flow
TestFixture.Setup
TestFixture.Test
TestFixture.TearDown
SetUpFixture.OneTimeSetUp
SetUpFixture.OneTimeTearDown
Following is the sample code which I tried on my machine and also the command line output
[SetUpFixture]
public class TestBase
{
[OneTimeSetUp]
//[SetUp]
public static void MyTestSetup()
{
Console.WriteLine(" ---------- Calling OneTimeSetUp ----------");
}
}
[TestFixture]
class TestClass : TestBase
{
[Test]
public void test()
{
Console.WriteLine("\n ....I'm inside TestClass.test() ....");
}
}
Console Output
=> TestSample.TestClass.test
....I'm inside TestClass.test() ....
=> TestSample.TestClass
---------- Calling OneTimeSetUp ----------
=> TestSpecflow.TestBase
---------- Calling OneTimeSetUp ----------
Can someone please suggest what am i missing here ?
I'm running the test via nunit-console
The issue is that the output is misleading and not in the order the code is executed. Because NUnit 3 supports parallel execution, it captures output and displays it on the console when that level of test execution is completed.
In your case, the fixture setup wraps the tests, so it finishes executing after the tests and outputs the captured text afterward.
If you debug your tests, or switch your Console.WriteLine calls to TestContext.Progress.WriteLine which outputs immediately, you will see that the code is executed in the order you expect.
If it is not in the order you expect, look at the namespaces. Remember that a [SetupFixture] is used to setup at the namespace level. If your tests are in a different namespace, they may be called in a different order. If you want a setup for all of your tests, put the class in your top level namespace, or if you have multiple namespaces, in no namespace.
Here is some test code,
namespace NUnitFixtureSetup
{
[SetUpFixture]
public class SetupClass
{
[OneTimeSetUp]
public void MyTestSetup()
{
TestContext.Progress.WriteLine("One time setup");
}
}
[TestFixture]
public class TestClass
{
[Test]
public void TestMethod()
{
TestContext.Progress.WriteLine("Test Method");
}
}
}
And here is the output from running with nunit3-console.exe
=> NUnitFixtureSetup.SetupClass
One time setup
=> NUnitFixtureSetup.TestClass.TestMethod
Test Method
Rob's answer is the fundamental reason you have a problem, but there is an additional one, which is present in your code although not in Rob's.
In your code, you are using TestBase twice: as a SetUpFixture and as the base class for your TestFixture.
That means the OneTimeSetUp method will be used twice... Once before all the fixtures in the namespace and once before any test fixture that inherits from it. Using a SetUpFixture in this way defeats it's purpose, which is to have some code that runs only once before all the fixtures in a namespace.
Use separate classes as a base class (if you need one) and as a setup fixture (if you need one of those).
I have some old MbUnit code which looks like this:
public class MyFixture {
[SetUp]
public void SetUp() {
// Add data to database
}
[Test, Rollback]
public void DoTest() {
// Tests with the data
}
}
My new NUnit Rollback attribute looks a bit like this:
public class RollbackAttribute : TestActionAttribute
{
public override void BeforeTest(TestDetails testDetails)
{
// Begin transaction
}
public override void AfterTest(TestDetails testDetails)
{
// Abort transaction
}
}
The Rollback should roll back the new data added in the SetUp method as well as any modifications during the test itself. Unfortunately, it seems that NUnit's BeforeTest runs after the fixture's SetUp method, so the data added during SetUp is not rolled back.
Is there a way to run BeforeTest before SetUp?
One option would be a base class, and replace the existing Rollback attributes with additional code in SetUp and TearDown, however some of my tests require running outside a transaction (they create multiple transactions themselves during the test run), so adding transactions around all test cases would require a bit of care. I'd rather find a solution which can re-use the existing Rollback attributes.
Is there a way to run BeforeTest before SetUp?
I don't think so, see e.g. this related discussion on google groups. Issue being discussed there is very similar, as you can see, code in SetUp method would run even prior to BeforeTest method used on test fixture level (you have it on test level).
Workaround from my point of view would be to remove the SetUpAttribute from the SetUp method and call the SetUp method explicitly at the beginning of the each test, i.e.:
public class MyFixture
{
public void SetUp()
{
// Add data to database
}
[Test, Rollback]
public void DoTest()
{
SetUp();
// Tests with the data
}
}
Your question also reminded me of question that marc_s raised in this SO thread. Question is unrelated to your problem, but he used the same construct as I am proposing above so it is perhaps not that bad idea.
EDIT:
Here is an opened issue on NUnit's github. But still, requested order there is:
BeforeTest (BaseFixture)
BaseSetUp BeforeTest (Fixture)
SetUp
BeforeTest (Test)
Test AfterTest (Test)
TearDown AfterTest (Fixture)
BaseTearDown AfterTest (BaseFixture)
So not exactly what you desire, "BeforeTest (Test)" would be executed after SetUp.
I'm new to the test automation scene so forgive me if this is a stupid question but google has failed me this time. Or at least anything I've read has just confused me further.
I'm using JUnit 4 and Selenium Webdriver within Eclipse. I have several tests that I need to run as a suite and also individually. At the moment these tests run fine when run on their own. At the start of the test an input box is presented to the tester/user asking first what server they wish to test on (this is a string variable which becomes part of a URL) and what browser they wish to test against. At the moment when running the tests in a suite the user is asked this at the beginning of each test, because obviously this is coded into each of their #Before methods.
How do I take in these values once, and pass them to each of the test methods?
So if server = "server1" and browser = "firefox" then firefox is the browser I want selenium to use and the URL I want it to open is http://server1.blah.com/ for all of the following test methods. The reason I've been using seperate #Before methods is because the required URL is slightly different for each test method. i.e each method tests a different page, such as server1.blah.com/something and server1.blah.com/somethingElse
The tests run fine, I just don't want to keep inputting the values because the number of test methods will eventually be quiet large.
I could also convert my tests to testNG if there is an easier way of doing this in testNG. I thought the #BeforeSuite annotation might work but now I'm not sure.
Any suggestions and criticism (the constructive kind) are much appreciated
You can adapt the solution for setting a global variable for a suite in this answer to JUnit 4 Test invocation.
Basically, you extend Suite to create MySuite. This creates a static variable/method which is accessible from your tests. Then, in your tests, you check the value of this variable. If it's set, you use the value. If not, then you get it. This allows you to run a single test and a suite of tests, but you'll only ask the user once.
So, your suite will look like:
public class MySuite extends Suite {
public static String url;
/**
* Called reflectively on classes annotated with <code>#RunWith(Suite.class)</code>
*
* #param klass the root class
* #param builder builds runners for classes in the suite
* #throws InitializationError
*/
public MySuite(Class<?> klass, RunnerBuilder builder) throws InitializationError {
this(builder, klass, getAnnotatedClasses(klass));
// put your global setup here
MySuite.url = getUrlFromUser();
}
}
This would be used in your Suite like so:
#RunWith(MySuite.class)
#SuiteClasses({FooTest.class, BarTest.class, BazTest.class});
Then, in your test classes, you can either do something in the #Before/#After, or better look at TestRule, or if you want Before and After behaviour, look at ExternalResource. ExternalResource looks like this:
public static class FooTest {
private String url;
#Rule
public ExternalResource resource= new ExternalResource() {
#Override
protected void before() throws Throwable {
url = (MySuite.url != null) ? MySuite.url : getUrlFromUser();
};
#Override
protected void after() {
// if necessary
};
};
#Test
public void testFoo() {
// something which uses resource.url
}
}
You can of course externalize the ExternalResource class, and use it from multiple Test Cases.
I think the main functionality of TestNG that will be useful here is not just #BeforeSuite but #DataProviders, which make it trivial to run the same test with a different set of values (and won't require you to use statics, which always become a liability down the road).
You might also be interested in TestNG's scripting support, which makes it trivial to ask the user for some input before the tests start, here is an example of what you can do with BeanShell.
It might make sense to group tests so that the test suite will have the same #Before method code, so you have a test suite for each separate.
Another option might be to use the same base url for each test but navigate to the specific page by getting selenium to click through to where you want to carry out the test.
If using #RunWith(Suite.class), you can add static methods with #BeforeClass (and #AfterClass), which will run before (and after) the entire Suite you define. See this question.
This of course won't help if you are referring to the entire set of classes found dynamically, and are not using Suite runner.