I've got an odd question for which Google has proven barren:
I've got a project in .net with ~20 classes that all have tests in them. One of the classes has common test setup code, although a few of the classes have their own TestFixtureSetup that looks exactly like the common class (not my architecture choice - this predates my employment). I have my own test class for which I have some different code that runs prior to running a few particular tests within the class.
Some more info that's relevant: The custom setup code that I have enables data to be available for a few combinatorial tests I have in my own test class. As the value source for the combinatorial params, the List that is returned first initializes some data.
Alright, here's the question: When I try to run a test in ANOTHER test class, it's "building" the tests from every other class. In my case, it's building the combinatorial test that I have - and thus, triggering the custom setup method that I have.
How do I prevent NUnit from building tests in other classes? As in, I run a test in one class, all I'd like NUnit to do is build tests from that class ONLY.
I tried to remove any NDA-no-no language, but here's the combinatorial I have:
[Test, Combinatorial, Category("Regressive")]
public void Test05_CombiTestExample(
[ValueSource("ListA")] User user,
[ValueSource("ListB")] KeyValuePair<string, string> searchKvp,
[ValueSource("ListC")] string scope)
{
And here's one of the lists that is being reference:
public IEnumerable<KeyValuePair<string, string>> ListB
{
get
{
InitCustomData();
if ([Redacted] != null)
{
return new Dictionary<string, string>()
{
[Redacted]
};
}
return null;
}
}
The line in question is "InitCustomData();" which, because my combinatorial is being built prior to running any setup or anything, is being executed anyway. I want this to stay here - I just don't want NUnit to start building test cases from any other class besides the one it's currently running a test in.
Related
Im having 4 test classes inside one project (Lets call them Class A, Class B, Class C, Class D)
Each of these 3 classes have two [TestFixture("string")], which makes it to 8 tests in total.
All classes are having the [Parallelizable] parameter.
When i start the test all at once by clicking inside the Test Explorer on the name of the project and "Run", then it will start all 8 tests at the same time.
The problem here is, that it consumes a lot RAM and the tests fail because it takes too long to load and i get a timeout error (Im doing automation tests with selenium in chrome)
Now i want to define a order.
For example:
Class A and Class B should start parallel
Class C and Class D should start parallel when Class A and Class B is done
Is it possible?
I tried the parameter [Order(1)] for Class A and Class B and Order(2)] for Class C and Class D
But when i run the tests, all 8 tests start to load.
Example from my code:
[TestFixture("normalUser")]
[TestFixture("adminUser")]
[Parallelizable]
public class ImportTest
{
private IWebDriver webDriver;
private const int waitTimer = 60;
public WebDriverWait w;
public string userRole;
// Constructor
public ImportTest(string userRole)
{
this.userRole = userRole;
Console.WriteLine(userRole);
}
////-----------------------------
[SetUp]
{
}
//-------------------------------
[Test]
public void Test1()
{
Do Test
}
[Test]
public void Test2()
{
Do Test
}
//--------------------------
[TearDown]
public void CloseBrowser()
{
webDriver.Quit();
}
}
First, I'll describe what's happening...
The OrderAttribute was created in NUnit V2, before parallel tests existed. It defines the order in which tests are started. Since there was no parallelism at the time, one test had to finish before the next one started.
When parallel execution was introduced in NUnit 3, Order was not exactly broken, because it continued to start tests in the specified order. But many users perceptions were "broken", because they thought that one test would not start until the prior one finished.
Order could, of course, be changed to work like that. However, at this point, that would be a breaking change for some people, so you most likely won't see it happen until there's an NUnit 4.
So... what can you do as a workaround? I can see three options...
The simplest approach would be to make each fixture [NonParallelizable]. Then they would all run separately. You should try that first and see if the performance is acceptable to you. If you want the tests within each fixture to run in parallel, you could use [Parallelizable(ParallelScope.Children)] instead but that might break things if the tests change the state of the fixture or of any common references found in the fixture.
Alternatively, you could pick only some fixtures to mark as [NonParallelizable]. In that case, I'd do it for the ones that consume a lot of memory.
For the most effort required, you could implement ordering yourself for these classes. I'd do that by creating some sort of shared token... e.g. a lock... which each class had to acquire on startup. I'd grab the lock in the OneTimeSetUp for a fixture and release it in the onetime teardown. The locking code should be before any setup code, which acquires resources and should be released after your teardown releases those resources.
I made option 3 rather sketchy because (a) I don't know precisely how your application works and (b) I presume that you won't do it unless it's absolutely necessary.
Final advice: don't make any assumptions about the performance impact of any of these options, even the first. Measure first!
In my existing [OneTimeSetup] method, I want to check some preconditions before running any test. But I can't do so as the object which I'll be needing to check preconditions is initialized in Base class [Setup] method. I can't initialize this earlier due to some project limitations.
So, Is there any way where I can execute some code after Base [Setup] method (to check some preconditions) and before any suite execution? I want to execute this once per suite.
[SetUpFixture]
Class GlobalSetup
{
[OneTimeSetUp]
public void OneTimeSetUp(){
setup();
CheckIfDataIsPresent(); // I can't do this here as this code needs Obj O to be initialized. which will be initialized in Base class's [Setup] methed
}
}
Public class Base
{
[Setup]
public void setUp()
{
//some code where we initialize obj O;
}
}
[TestFixture]
public class Test : Base
{
// tests to be executed
}
You already did a nice job of explaining why what you want to do won't work, so I don't have to. :-)
The problem is that each your tests needs a fresh instance of that object, so you properly create it in a [SetUp] method. You would like to ensure that it's possible to create such an object once before you run any tests.
I can only give you a non-specific answer, since you haven't given a lot of info in your example code. If you update your question, I may be able to update my answer. Here goes...
Both your tests and the check you want to perform require an instance of object o. So one approach would be to initialize o one more time in the OneTimeSetup, perform the check and then throw it away. Since you are initializing o in every test, I assume it's not expensive to do so. Say you have 100 tests. You are setting up o 100 times. So make it 101 and be done!
Alternatively, determine what is required for o to be initialized successfully and check that. For example, if it needs a file to be present, check that the file is present. If the file has to have 100 records in some format, check that it's so. Perhaps you might give us more detail about what those prerequisites are.
Finally, you might reconsider whether you really need a new instance per test. Since you suggest you would be willing to make a check once per fixture (I assume that's what you mean by suite) then perhaps you really only need one instance per fixture rather than a new one for each test.
I'm using NUnit 2.X library but want to use NUnit 3.X now. I have some problems about migration from 2.X to 3.X. First i have a setup fixture class. Here is the 2.X version;
using System;
using System.IO;
using System.Reflection;
using HalisEnerji.QuantSignal.Logging;
using NUnit.Framework;
namespace HalisEnerji.QuantSignal.Tests
{
[SetUpFixture]
public class Initialize
{
[SetUp]
public void SetLogHandler()
{
Log.LogHandler = new ConsoleLogHandler();
}
}
}
First problem is fixed via change "Setup" attribute with "OneTimeSetUp" attribute. Second problem fixed via add some codes for set test directory. Because i'm using Re-Sharper test engine. Here is final shape of setup fixture;
using System;
using System.IO;
using System.Reflection;
using HalisEnerji.QuantSignal.Logging;
using NUnit.Framework;
namespace HalisEnerji.QuantSignal.Tests
{
[SetUpFixture]
public class Initialize
{
[OneTimeSetUp]
public void SetLogHandler()
{
Log.LogHandler = new ConsoleLogHandler();
var assembly = Assembly.GetExecutingAssembly();
var localPath = new Uri(assembly.CodeBase).LocalPath;
var direcotyName = Path.GetDirectoryName(localPath);
if (direcotyName != null)
{
Environment.CurrentDirectory = direcotyName;
}
}
}
}
Well after solve setup fixture problem, my real problems begins with use TestCaseSource/TestCaseData. Here is sample 2.X version;
[Theory]
[TestCaseSource("CreateSymbolTestCaseData")]
public void CreateSymbol(string ticker, Symbol expected)
{
Assert.AreEqual(Symbol.Create(ticker), expected);
}
private TestCaseData[] CreateSymbolTestCaseData()
{
return new []
{
new TestCaseData("SPY", new Symbol(Security.GenerateEquity("SPY"), "SPY")),
new TestCaseData("EURUSD", new Symbol(Security.GenerateForex("EURUSD"), "EURUSD"))
};
}
2.X version creating exception and my tests are fail. Shortly exception telling that TestCaseData provider method must be static. Well, after mark method with static identifier test working correctly but this time my other test failing (before use static identifier it's not failing). Why my other test failing? Because it's reading a file from test directory and somehow test working before setup up fixture codes run and change test directory!
Before use static identifier first SetUpFixture codes run and then tests code run. After use static identifier order changing my test that read file from test directory (which is Re-Sharper's temporary directory and not contain necessary file) run first after that SetUpFixture codes run. Any idea how all my tests to be successful?
UPDATE:
Explain some units;
I have Initialize.cs (part of my test assembly) which is responsible setup CurrentDirectory.
I have Config.cs (part of my project infrastructure assembly) which is my project configuration file and it has public static readonly Setttings property which is reading configuration file from CurrentDirectory.
I have ConfigTests.cs (part of my test assembly) which is contain some test methods for read/write Settings property.
When i debug tests;
Before use any static TestCaseSource, they are working below order;
A. Initialize.cs => Setup method
B. Config.cs => static Settings property getter method
C. ConfigTests.cs => first test method
So initialize working first others working later all tests successfully passing from test.
After use static TestCaseSource for inside other test file lets say OrdersTests.cs (excluded from project for first scenario after that included again), somehow working order is changing like below;
A. Config.cs => static Settings property getter method
B. OrdersTests.cs => static TestCaseSource method (not test method)
C. Initialize.cs => Setup method
D. ConfigTests.cs => first test method
E. OrdersTests.cs => first test method
So, my ConfigTests.cs tests failing because Initialize.cs working after Config.cs. I hope with this update my problem is more clear.
Is this problem related NUnit or Resharper or V.Studio? I don't know and all i know is my successfully passing tests are failing now!
UPDATE 2:
Chris,
Yes you are right. I explore project in detail and i saw the problem is related that my project's some classes accessing to static Config class and it's static Settings property (before run test setup fixture method and even before static test case source method!). You talk about order of process of test methods; NUnit doing tests like your said, not like i said. But when i try to use your solution (set current directory before test case source) it's not working. Because of that i solve my problem in another way. I'm not happy but at least my test methods working now. Could you please tell me what are the technical reasons that run static test case methods before initialize/setup method? Is this because of NUnit or because of infrastructure of .Net Framework? I'm not fanatic about NUnit and/or TDD. I don't have deep knowledge about these concepts but it does not make sense to me: run any method before setup method.
Thanks for your interest.
Because it's reading a file from test directory and somehow test working before setup up fixture codes run and change test directory!
How are you reading this file? You should use TestContext.CurrentContext.TestDirectory to get the test directory in NUnit 3, rather than relying on the location of the current directory. See the Breaking Changes page for details.
Edit: I also see you've tagged this ReSharper 7.1. You should be aware that this version of resharper does not support NUnit 3 - the first version that did is ReSharper 10. Your tests will appear to run correctly - however you may experience weird side effects, and this may break in any future version of NUnit.
Response to update:
Take a look at NUnit 3's Breaking Changes page. There are two relevant breaking changes between NUnit 2 and 3.
TestCaseSource's must now be static.
The CurrentDirectory is no longer set to Environment.CurrentDirectory by default.
The first you've solved easily enough. The second, is what's now causing you issues.
NUnit 3 runs it's methods in this order:
Evalute TestCaseSource methods (OrdersTests.cs)
Run SetUpFixture (Initialize.cs)
Run Test (ConfigTests/OrdersTests)
I'm not sure what Config.cs is being called before your TestCaseSource method - are you sure of that order? Does anything in CreateSymbolTestCaseData() call anything in Config.cs You could try rewriting your TestCaseSource as such:
private TestCaseData[] CreateSymbolTestCaseData()
{
Environment.CurrentDirectory = "c:\RequiredDirectory";
return new []
{
new TestCaseData("SPY", new Symbol(Security.GenerateEquity("SPY"), "SPY")),
new TestCaseData("EURUSD", new Symbol(Security.GenerateForex("EURUSD"), "EURUSD"))
};
}
I'm new to the test automation scene so forgive me if this is a stupid question but google has failed me this time. Or at least anything I've read has just confused me further.
I'm using JUnit 4 and Selenium Webdriver within Eclipse. I have several tests that I need to run as a suite and also individually. At the moment these tests run fine when run on their own. At the start of the test an input box is presented to the tester/user asking first what server they wish to test on (this is a string variable which becomes part of a URL) and what browser they wish to test against. At the moment when running the tests in a suite the user is asked this at the beginning of each test, because obviously this is coded into each of their #Before methods.
How do I take in these values once, and pass them to each of the test methods?
So if server = "server1" and browser = "firefox" then firefox is the browser I want selenium to use and the URL I want it to open is http://server1.blah.com/ for all of the following test methods. The reason I've been using seperate #Before methods is because the required URL is slightly different for each test method. i.e each method tests a different page, such as server1.blah.com/something and server1.blah.com/somethingElse
The tests run fine, I just don't want to keep inputting the values because the number of test methods will eventually be quiet large.
I could also convert my tests to testNG if there is an easier way of doing this in testNG. I thought the #BeforeSuite annotation might work but now I'm not sure.
Any suggestions and criticism (the constructive kind) are much appreciated
You can adapt the solution for setting a global variable for a suite in this answer to JUnit 4 Test invocation.
Basically, you extend Suite to create MySuite. This creates a static variable/method which is accessible from your tests. Then, in your tests, you check the value of this variable. If it's set, you use the value. If not, then you get it. This allows you to run a single test and a suite of tests, but you'll only ask the user once.
So, your suite will look like:
public class MySuite extends Suite {
public static String url;
/**
* Called reflectively on classes annotated with <code>#RunWith(Suite.class)</code>
*
* #param klass the root class
* #param builder builds runners for classes in the suite
* #throws InitializationError
*/
public MySuite(Class<?> klass, RunnerBuilder builder) throws InitializationError {
this(builder, klass, getAnnotatedClasses(klass));
// put your global setup here
MySuite.url = getUrlFromUser();
}
}
This would be used in your Suite like so:
#RunWith(MySuite.class)
#SuiteClasses({FooTest.class, BarTest.class, BazTest.class});
Then, in your test classes, you can either do something in the #Before/#After, or better look at TestRule, or if you want Before and After behaviour, look at ExternalResource. ExternalResource looks like this:
public static class FooTest {
private String url;
#Rule
public ExternalResource resource= new ExternalResource() {
#Override
protected void before() throws Throwable {
url = (MySuite.url != null) ? MySuite.url : getUrlFromUser();
};
#Override
protected void after() {
// if necessary
};
};
#Test
public void testFoo() {
// something which uses resource.url
}
}
You can of course externalize the ExternalResource class, and use it from multiple Test Cases.
I think the main functionality of TestNG that will be useful here is not just #BeforeSuite but #DataProviders, which make it trivial to run the same test with a different set of values (and won't require you to use statics, which always become a liability down the road).
You might also be interested in TestNG's scripting support, which makes it trivial to ask the user for some input before the tests start, here is an example of what you can do with BeanShell.
It might make sense to group tests so that the test suite will have the same #Before method code, so you have a test suite for each separate.
Another option might be to use the same base url for each test but navigate to the specific page by getting selenium to click through to where you want to carry out the test.
If using #RunWith(Suite.class), you can add static methods with #BeforeClass (and #AfterClass), which will run before (and after) the entire Suite you define. See this question.
This of course won't help if you are referring to the entire set of classes found dynamically, and are not using Suite runner.
I'm testing some code that uses StructureMap for Inversion of Control and problems have come up when I use different concrete classes for the same interface.
For example:
[Test]
public void Test1()
{
ObjectFactory.Inject<IFoo>(new TestFoo());
...
}
[Test]
public void Test2()
{
ObjectFactory.Initialize(
x => x.ForRequestedType<IFoo>().TheDefaultIsConcreteType<RealFoo>()
);
// ObjectFactory.Inject<IFoo>(new RealFoo()) doesn't work either.
...
}
Test2 works fine if it runs by itself, using a RealFoo. But if Test1 runs first, Test2 ends up using a TestFoo instead of RealFoo. Aren't NUnit tests supposed to be isolated? How can I reset StructureMap?
Oddly enough, Test2 fails if I don't include the Initialize expression. But if I do include it, it gets ignored...
If you must use ObjectFactory in your tests, in your SetUp or TearDown, make a call to ObjectFactory.ResetAll().
Even better, try to migrate your code away from depending on ObjectFactory. Any class that needs to pull stuff out of the container (other than the startup method) can take in an IContainer, which will automatically be populated by StructureMap (assuming the class itself is retrieved from the container). You can reference the IContainer wrapped by ObjectFactory through its Container property. You can also avoid using ObjectFactory completely and just create an instance of a Container that you manage yourself (it can be configured in the same way as ObjectFactory).
Yes, NUnit tests are supposed to be isolated and it is your responsibility to make sure they are isolated. The solution would be to reset ObjectFactory in the TearDown method of your test fixture. You can use ObjectFactory.EjectAllInstancesOf() for example.
Of course it doesn't reset between tests. ObjectFactory is a static wrapper around an InstanceManager; it is static through an AppDomain and as tests run in the same AppDomain this is why it is not reset. You need to TearDown the ObjectFactory between tests or configure a new Container for each test (i.e., get away from using the static ObjectFactory).
Incidentally, this is the main reason for avoiding global state and singletons: they are not friendly to testing.
From the Google guide to Writing Testable Code:
Global State: Global state is bad from theoretical, maintainability, and understandability point of view, but is tolerable at run-time as long as you have one instance of your application. However, each test is a small instantiation of your application in contrast to one instance of application in production. The global state persists from one test to the next and creates mass confusion. Tests run in isolation but not together. Worse yet, tests fail together but problems can not be reproduced in isolation. Order of the tests matters. The APIs are not clear about the order of initialization and object instantiation, and so on. I hope that by now most developers agree that global state should be treated like GOTO.