I have three classes all containing multiple tests. Each class has the annotation of TestFixture. I need each class to run the tests in order. (e.g. TextFixture1 runs all of its tests, then TestFixture2 runs all of its tests, and finally TestFixture3 runs all of its tests.) How can I accomplish this?
Use the OrderAttribute on each fixture, specifying the order in which you want the fixtures to run. Use the same attribute on each test method, specifying the order the test should run within the fixture.
See OrderAttribute in the docs for more info.
Related
I have a testng.xml file and i have three classes and all three have tests with priority from 1 to 4(#Test(priority=1 to 4)). On running it the tests with priority 1 from different classes run first and affect the flow of my test.What kind of testng annotation can I use to avoid this?
#Test(priority=1) given in multiple classes
use preserve order
<test name="Test" preserve-order="true">
So that all classes specified in testng.xml will run in specified order and then in each class methods will run as per priority.
In testng, the order of the classes mentioned in the testng.xml doesn't matter if you have the priority set in the tests inside those classes. The tests will run according to the priorities (priority=1 tests will run first and then priority=2 and further on).
To solve the above issue, you need to remove the priorities from the tests inside the classes and then put the classes in the testng in the order you want to execute those classes and put <preserve-order="true"> in the testng xml.
If you want to run the tests inside a class also in a particular order, then you can use dependsOnMethods between the tests mentioned inside the class like:
#Test(dependsOnMethods = {"parentTest"})
public void childTest() {
}
#Test
public void parentTest() {
}
In the above case, when the parentTest() pass only then the childTest() will execute.
I have a class which extends org.scalatest.junit.JUnitSuite. This class has a couple of tests. I do not want these tests to run in parallel.
I know how simple it is with Specs2 (extend the class with Specification and add a single line sequential inside the class) as shown here: How to run specifications sequentially.
I do not want to alter the Build file by setting:
parallelExecution in Test := false
nor I want to use tags to run specific test files sequentially.
All I want is a way to make sure that all tests inside my class run sequentially. Is this possible with ScalaTest ? Any sample test/template is appreciated.
A quick google search pointed me to this: http://doc.scalatest.org/2.0/index.html#org.scalatest.Sequential
Just for the couple of tests I have, I think it is a total overkill to create StepSuites. I am not completely sure if that's the way to go about with my case!
The doc for org.scalatest.ParallelTestExecution says
ScalaTest's normal approach for running suites of tests in parallel is to run different suites in parallel, but the tests of any one suite sequentially.
So it looks like you don't have to do anything to get what you want, if your tests are in a single suite.
Can I force each nunit test method to run on a separate process ?
I need to do this because calling some of the methods-under-test may have side-effects. So, I need to make sure that each unit test run in complete isolation from the other unit tests.
You can make use of the "/process" option to specify - single or separate or multiple options. Here is the reference to the NUnit documentation for version 2.5: http://www.nunit.org/index.php?p=consoleCommandLine&r=2.5. Look at the Controlling the Use of Processes section.
We use Spec trait for our tests in ScalaTest. when we run the entire suite, it does not always run in the same order. Most answers in google suggest defining a Suite and specifying all the test names. But this requires us to add the test name every time we add a new test.
Is it possible to use the DiscoverySuite itself and define the test execution order? Like run the tests in alphabetical order. I looked at extending the DiscoverySuite but DiscoverySuite seems to be private to scalatest.
---More info----
By ordering i mean, If there are tests A, B, C.
class A extends Spec {..}
class B extends Spec {..}
class C extends Spec {..}
Then i want the tests to run in order (A, B, C). But what happens now is, it run in a different order everytime.
DiscoverySuite is private to ScalaTest, yes. The execution order of tests in a Spec (now called FunSpec, by the way) is defined to be the order of appearance in the source file. To define the order of the test classes themselves, you will need to define a nestedSuites method and run that wrapper Suite instead of using Discovery. You can go back to using discovery once you no longer need an order. I'll look at adding a defined order to DiscoverySuite in the next ScalaTest release.
See http://doc.scalatest.org/1.0/org/scalatest/SequentialNestedSuiteExecution.html
This seems to provide the specific behavior you're looking for.
I have some NUnit tests which uses a TestCaseSource function. Unfortunately, the TestCaseSource function that I need takes a long time to initialize, because it scans a folder tree recursively to find all of the test images that would be passed into the test function. (Alternatively it could load from a file list XML every time it's run, but automatic discovery of new image files is still a requirement.)
Is it possible to specify an NUnit attribute together with TestCaseSource such that NUnit does not enumerate the test cases (does not call the TestCaseSource function) until either the user clicks on the node, or until the test suite is being run?
The need to get all test images stored in a folder is a project requirement because other people who do not have access to the test project will need to add new test images to the folder, without having to modify the test project's source code. They would then be able to view the test result.
Some dogmatic unit-testers may counter that I am using NUnit to do something it's not supposed to do. I have to admit that I have to meet a requirement, and NUnit is such a great tool with a great GUI that satisfies most of my requirements, such that I do not care about whether it is proper unit testing or not.
Additional info (from NUnit documentation)
Note on Object Construction
NUnit locates the test cases at the
time the tests are loaded, creates
instances of each class with
non-static sources and builds a list
of tests to be executed. Each source
object is only created once at this
time and is destroyed after all tests
are loaded.
If the data source is in the test
fixture itself, the object is created
using the appropriate constructor for
the fixture parameters provided on the
TestFixtureAttribute or the default
constructor if no parameters were
specified. Since this object is
destroyed before the tests are run, no
communication is possible between
these two phases - or between
different runs - except through the
parameters themselves.
It seems the purpose of loading the test cases up front is to avoid having communications (or side-effects) between TestCaseSource and the execution of the tests. Is this true? Is this the only reason to require test cases to be loaded up front?
Note:
A modification of NUnit was needed, as documented in http://blog.sponholtz.com/2012/02/late-binded-parameterized-tests-in.html
There are plans to introduce this option to later versions of NUnit.
I don't know of a way to delay-load test names in the GUI. My recommendation would be to move those tests to a separate assembly. That way, you can quickly run all of your other tests, and load the slower exhaustive tests only when needed.