I am using junit with eclipse to write function tests.
When running an individual test it runs in the order that I have them set within the class.
Eg.
testCreateUser
testJoinUserToRoom
testVerify
testDeleteUser
However when I run this test as part of a suite, (so in a package) the order is random.
It will for example do the verify, then delete user then joinuserToRoom then Createuser.
My tests within the suite are not dependent on each other. However each individual test within a test is dependent on them being run in the correct order.
Is there any way I can achieve this?
Thanks.
You can't guarantee the order of execution of test methods in JUnit.
The order of execution of test classes within a suite is guaranteed (if you're using Suite), but the order of execution if the test classes are found by reflection isn't (for instance, if you're running a package in Eclipse, or a set of tests from maven or ant). This may be definable by ant or maven, but it isn't defined by JUnit.
In general, JUnit executes the test methods in the order in which they are defined in the source file, but not all JVMs guarantee this (particulary with JVM 7). If some of the methods are inherited from an abstract base test class, then this may not hold either. (This sounds like your case, but I can't tell from your description).
For more information on this, see my answer to Has JUnit4 begun supporting ordering of test? Is it intentional?.
So what can you do to fix your problem? There are two solutions.
In your original example, you've actually only got one test (verify), but you've got 4 methods, two setup (createUser, joinUserToRoom) and one teardown (deleteUser). So your first option is to better define your test cases, using a TestRule, in particular ExternalResource. ExternalResource allows you to define before/after behaviour for a test, similar to #Before/#After. However, the advantage of ExternalResource is that you can factor this out of your test.
So, you would create/delete the user in your external resource:
public class UsesExternalResource {
#Rule
public ExternalResource resource= new ExternalResource() {
#Override
protected void before() throws Throwable {
// create user
};
#Override
protected void after() {
// destroy user
};
};
#Test
public void testJoinUserToRoom() {
// join user to room
// verify all ok
}
}
For me, this is simpler and easier to understand, and you get independent tests, which is a good thing. This is what I would do, but you will need to refactor your tests a bit. You can also stack these Rules using RuleChain.
Your second option, if you really want to introduce dependencies between your test methods, is to look at TestNG, in which you can define dependencies from one test to another.
If they have a 'correct' order, then they are not multiple tests, but a single test that you have incorrectly annotated as being multiple independent tests.
Best practise would to rewrite them in approved junit style (setup - act - verify), supported by #Before or #BeforeClass methods that did any required common setup.
Quick workaround would be to have a single #Test-annotated method that called the other test methods in sequence. That becomes something like the preferred alternative if you are using Junit not to do strict unit testing, but something more like scenario-driven systems testing. It's not necessarily the best tool for such use, but it does work perfectly well in some cases.
Then, what you would have so far is have a single test:
#Test public void testUserNominalLifeCycle(...
which could then, if you are feeling virtuous, be supplemented by extra new tests like
#Test public void testUserWhoNeverJoinsARoom(...
Related
I have a testng.xml file and i have three classes and all three have tests with priority from 1 to 4(#Test(priority=1 to 4)). On running it the tests with priority 1 from different classes run first and affect the flow of my test.What kind of testng annotation can I use to avoid this?
#Test(priority=1) given in multiple classes
use preserve order
<test name="Test" preserve-order="true">
So that all classes specified in testng.xml will run in specified order and then in each class methods will run as per priority.
In testng, the order of the classes mentioned in the testng.xml doesn't matter if you have the priority set in the tests inside those classes. The tests will run according to the priorities (priority=1 tests will run first and then priority=2 and further on).
To solve the above issue, you need to remove the priorities from the tests inside the classes and then put the classes in the testng in the order you want to execute those classes and put <preserve-order="true"> in the testng xml.
If you want to run the tests inside a class also in a particular order, then you can use dependsOnMethods between the tests mentioned inside the class like:
#Test(dependsOnMethods = {"parentTest"})
public void childTest() {
}
#Test
public void parentTest() {
}
In the above case, when the parentTest() pass only then the childTest() will execute.
I am trying to find a way to create custom suites of NUnit tests to target our wide variety of environments. The closest thing I found was this http://nunit.org/docs/2.5.6/suite.html which is exactly what I am looking for. Tying to implement this though, the [Suite] annotation doesnt even seem to exist.. Was this taken away? Is there a better solution now?
The SuiteAttribute was eliminated in NUnit 3. It never got a lot of use as most people simply organize their tests by namespace, which provides the same grouping of tests that the SuiteAttribute used to do.
FUN FACT: "Automatic namespace suites" were once a new cool thing!
If you want the ability to group tests in different ways, across namespace boundaries, you can use categories to do it. It's not as easy of course.
An alternative, if you are using the command-line console runner, is to list the fixtures you want to run in a file and use the --testlist option.
Building off of Charlies post from above - The way I was able to set this up was using the --testlist option.
First create a testlist.txt file and store it somewhere in your solution. Structure the file in such a way so if you have a class like.
namespace NamespaceA
{
class TestGroup
{
[Test]
public void TestOne()
{
}
[Test]
public void TestTwo()
{
}
}
}
the files contents would look like this..
NamespaceA.TestGroup.TestOne
or for both..
NamespaceA.TestGroup
Then just your standard consule runner command
"nunit-console.exe" "path/to/.dll" --testlist="path/to/testlist.txt"
:D)
I have written a couple of tests, very identical but different in only one way. One test has an international address and the other has domestic address: DomesticAddress.scala & InternationalAddress.scala
DomesticAddress.scala extends another class ShipMethods.scala that has a #Test method. Here I am validating that DomesticAddress.scala has valid ship methods present. InternationalAddress.scala does not extend ShipMethods.scala.
Both test classes (DomesticAddress.scala and InternationalAddress.scala) have different users; the only similarity is that the address is stored in a val named 'address'.
When I run these two tests some times (and only some times) the test fails for DomesticAddress.scala because I see an international address in there.
Is it possible that there is a race condition happening in this scenario? My testng xml is preserving the order of tests, so this is more confusing that a race condition could occur especially since I am not sharing any resources among the tests.
Why does your ShipMethods have a #Test annotation on it? Is it a test class? If it's not a test class (as opposed to a class which you want to test), then it shouldn't have a #Test annotation on it.
Your classes under test shouldn't have test methods inside them.
We use Spec trait for our tests in ScalaTest. when we run the entire suite, it does not always run in the same order. Most answers in google suggest defining a Suite and specifying all the test names. But this requires us to add the test name every time we add a new test.
Is it possible to use the DiscoverySuite itself and define the test execution order? Like run the tests in alphabetical order. I looked at extending the DiscoverySuite but DiscoverySuite seems to be private to scalatest.
---More info----
By ordering i mean, If there are tests A, B, C.
class A extends Spec {..}
class B extends Spec {..}
class C extends Spec {..}
Then i want the tests to run in order (A, B, C). But what happens now is, it run in a different order everytime.
DiscoverySuite is private to ScalaTest, yes. The execution order of tests in a Spec (now called FunSpec, by the way) is defined to be the order of appearance in the source file. To define the order of the test classes themselves, you will need to define a nestedSuites method and run that wrapper Suite instead of using Discovery. You can go back to using discovery once you no longer need an order. I'll look at adding a defined order to DiscoverySuite in the next ScalaTest release.
See http://doc.scalatest.org/1.0/org/scalatest/SequentialNestedSuiteExecution.html
This seems to provide the specific behavior you're looking for.
More than once the question has been asked on SO. But the only answers that are given read "you should not need to order your unit tests, it is bad because" or "you can avoid that if..."
I already know it is bad, why it is bad, and techniques to avoid it. But that is not what I want to know. I'd like to know if it is possible to order the execution of NUnit tests, other than an alphabetical order. To be blunt: I actually want state to propogate from one test to the next. Trust me that I have a clever reason for this, that defies the usual philosophy.
MSTest has the "ordered test" capability, which is very useful in certain cases. I'd like to have that ability in NUnit. Can it be done?
Update for NUnit 3.2.0 - now it support OrderAttribute.
The OrderAttribute may be placed on a test method to specify the order in which tests are run.
Example:
public class MyFixture
{
[Test, Order(1)]
public void TestA() { ... }
[Test, Order(2)]
public void TestB() { ... }
[Test]
public void TestC() { ... }
}
https://github.com/nunit/docs/wiki/Order-Attribute
The work-around (hack) is to alphabetize your test case names. See this thread:
https://bugs.launchpad.net/nunit-3.0/+bug/740539
Relying on alphabetical order is a workaround that you can use but it is not documented and supported beyond the visual order of the display. In theory it could change at any time. In practice it won't change until NUnit 3.0, so you're pretty safe using it as a workaround
This quote is from Charlie Poole, the main dev on NUnit.
It also seems they have a scheme cooking to support ordered tests in NUnit 3, though how they will do so is still under discussion.
Just an update for NUnit 2.5.1. According to documentation there are cases that even alphabetical order is not supported.
NUnit TestCaseAttribute
Order of Execution
In NUnit 2.5, individual test cases are sorted alphabetically and
executed in that order. With NUnit 2.5.1, the individual cases are not
sorted, but are executed in the order in which NUnit discovers them.
This order does not follow the lexical order of the attributes and
will often vary between different compilers or different versions of
the CLR.
As a result, when TestCaseAttribute appears multiple times on a method
or when other data-providing attributes are used in combination with
TestCaseAttribute, the order of the test cases is undefined.
Try to use NameParameters argument to pass the TestName with a string you wish, in order the TestCase() to be ordered by TestName.
[TestCase(..., TestName = "1stTest")]
[TestCase(..., TestName = "2ndTest")]
for Nuint you can use following code .
[TestMethod]
[Priority(2)]