For my project I want to run the exact same test cases twice, once locally and on a different VM in parallel in the cloud (Azure in my case).
I duplicated the TestCase and tagged one Category("Local") and the other Category("Cloud").
Running nunit3 from the console with --where="cat == Cloud" will thus run all TestCases of every test that has one or more TestCases tagged with Category("Cloud").
Is there a different way of only running selected TestCases by a commandline switch?
Simplified example:
[TestCase(TestName = "Canary, Run in cloud."), Category("Cloud")]
[TestCase(TestName = "Canary, Run locally."), Category("Local")]
public void Canary()
{
Assert.True(true);
}
Found a work-around.
Using --params:Cloud=true as command line argument and in the code
private bool ShallRunInCloud => TestContext.Parameters["Cloud"]?.ToLowerInvariant() == "true";
Related
I have 3 normal and 1 explicit test but when I run my test using the Test Explorer window I get this output under "Tests" in the Output window
========== Starting test run ==========
NUnit Adapter 4.2.0.0: Test execution started
Running all tests in xyz.dll
NUnit3TestExecutor discovered 1 of 1 NUnit test cases using Current Discovery mode, Explicit run
ExplicitMethod(03/10/2022 08:00:00,03/10/2022 16:00:00): OneTimeSetUp:
NUnit Adapter 4.2.0.0: Test execution complete
========== Test run finished: 1 Tests (0 Passed, 0 Failed, 0 Skipped) run in 462 ms ==========
*Note I'm using NUnit Adapter 4.2.1 instead of 4.2.0 so that's already weird
And this is how the Test Explorer window looks
Test Explorer window
*Note the full blue test is the Explicit test that get's skipped like it should
This is a problem because it does seem to only discover tests which are explicit.
The tests I want to run are of course not explicit, here is an example
[Test]
[TestCaseSource(nameof(TestNameData))]
public async Task<float> TestName(DateTime start, DateTime end, List<CalculateHoursObj>? list = default)
{
if (list == null) list = new List<CalculateHoursObj>();
return await EmployeeService.CalculateOverTimeHours(start, end, list);
}
public static IEnumerable TestNameData
{
get
{
yield return new TestCaseData(TenthMarch8_2022, TenthMarch16_2022).Returns(8.0f);
}
}
It fails to discover and/or run this test.
But curiously if I break all my non explicit tests by making the data non-static like this
public IEnumerable TestNameData
{
get
{
yield return new TestCaseData(TenthMarch8_2022, TenthMarch16_2022).Returns(8.0f);
}
}
It of course breaks the test and when I run all tests it does actually discover all tests
========== Starting test run ==========
NUnit Adapter 4.2.0.0: Test execution started
Running selected tests in xyz.dll
NUnit3TestExecutor discovered 2 of 4 NUnit test cases using Current Discovery mode, Non-Explicit run
NUnit Adapter 4.2.0.0: Test execution complete
========== Test run finished: 2 Tests (0 Passed, 2 Failed, 0 Skipped) run in 394 ms ==========
But even now it only runs 2 of the 3 broken non explicit tests and of course they all fail
I have looked up everything online for 1.5 hours and really can't find a solution.
Don't bother responding with "have you updated visual studio or the nugget packages"
This question was part of a bug the solution is to not use new DateTime() withing nunit tests or to have nunit test be discover with id's using the <UseParentFQNForParametrizedTests>true</UseParentFQNForParametrizedTests> <UseNUnitIdforTestCaseId>true</UseNUnitIdforTestCaseId> flags
This was a bug because it was not reported correctly in the output window and/or Error List window.
I've created an Issue on GitHub about this if anyone wants to read.
Now stop editing this post because it truly is answered and the Issue link which I included shows the valid, correct and readable answer by "OsirisTerje" whom explains it way better than I could.
I'm running into a problem like this. I'm currently using pytest to run test cases, and reducing execution time using xdist to run tests in parallel and publishing tests results to TestRail. The issue is when using xdist, pytest-testrail plugin creates Test-Run for each xdist workers and then publishes test cases like Untested.
I tried this hook pytest_terminal_summary to prevent pytest_sessionfinish plugin hook from being call multiple times.
I expect only one test run is created, but still multiple test runs are created.
I ran in to the same problem, but found a kind of workaround with duct tape.
I found that all results are collecting properly in test run, if we run the tests with --tr-run-id key.
If you are using jenkins jobs to automate processes, you can do following:
1) create testrun using testrail API
2) get ID of this test run
3) run the tests with --tr-run-id=$TEST_RUN_ID
I used these docs:
http://docs.gurock.com/testrail-api2/bindings-python
http://docs.gurock.com/testrail-api2/reference-runs
from testrail import *
import sys
client = APIClient('URL')
client.user = 'login'
client.password = 'password'
result = client.send_post('add_run/1', {"name": sys.argv[1], "assignedto_id": 1}).get("id")
print(result)
then in jenkins shell
RUN_ID=`python3 testrail_run.py $BUILD_TAG`
and then
python3 -m pytest -n 3 --testrail --tr-run-id=$RUN_ID --tr-config=testrail.cfg ...
NUnit 3 has "--where" parameter in console that allows us to select different tests to run. It can include different namespaces or test categories.
I want (but don't know how) to include some namespaces to run tests. I have specific examples and I ask you for help.
Let's assume we have the next namespaces with tests:
Project.MainSuite (includes 1 tests)
Project.MainSuite.Category1 (has 2 tests)
Project.MainSuite.Category1.TestSuite1 (has 3 tests)
How to run the next tests using --where parameter:
Tests only from Project.MainSuite.Category1 (2 tests should be run)
Tests from Project.MainSuite.Category1 and Project.MainSuite.Category1.TestSuite1 together (5 tests should be run)
All test from Project.MainSuite including sub-namespaces (6 tests should be run)
Thanks in advance for your help.
I recently ran into a similar issue and wanted to get a solid answer for this.
The short answer to your question is that you cannot do what you are asking without being more explicit.
When you run your tests with a where clause of --where "test == Project.MainSuite" (the highest namespace in your project), it will run all of the tests in that namespace and all sub-namespaces.
If you run your tests with a where clause of --where "test == Project.MainSuite.Category1.TestSuite1" (the lowest sub namespace in Project.MainSuite), it will only run all of the tests inside of that namespace.
You can do a few things to get what you are trying to accomplish.
1. Tests only from Project.MainSuite.Category1
--where "class == Project.MainSuite.Category1.ClassWithTests"
Just be explicit about the classes that are inside of this namespace.
Or if you are worried about adding more tests inside of this namespace in the future and don't want to update the script to run the tests. You can add Category attributes to the Suites/Tests inside this namespace and run them based off that Category.
--where "cat == TestsInCategory1Namespace"
2. Tests from Project.MainSuite.Category1 and Project.MainSuite.Category1.TestSuite1 together
Similarly for this scenario, you can combine the category and the class clause together.
--where "cat == TestsInCategory1Namespace and class == Project.MainSuite.Category1.TestSuite1"
3. All test from Project.MainSuite including sub-namespaces
--where test == Project.MainSuite
Does this help? Test Selection Language
this should work for categorys --where "cat == SmokeTests" --noresult
this for namespace: --where test == "My.Namespace" and cat == Urgent
I am trying to run the jmeter test-suites in eclipse.
In my test-suite I am using a BeanShellAssertion to count the number of rows in a csv file.
I have a custom jmeter function to do so.
The script of the BeanShellAssertion is :
String str = "${__CustomFunction("Path to the CSV file")}";
int i = Integer.parseInt(str);
if(i ==0)
{
Failure = true;
FailureMessage = "Failed!";
}
return i;
This test-suite works fine when I run it using the jmeter on my local machine.
Only when I try to run it with eclipse, (using the jmeter maven plugin) I see the following error:
jmeter.util.BeanShellInterpreter: Error invoking bsh method:
eval Sourced file: inline evaluation of: `` String str =
"${__CustomFunction("FilePath")}"; int i = Integ . . . '' : Typed
variable declaration : Method Invocation Integer.parseInt
I am wondering if there's some other way to invoke the jmeter functions when executing it using eclipse cause I am sure that the function is correct as I mentioned before that it works fine when the test suite is run using the jmeter on my local machine.
Any help would be appreciated.
Thanks.
Are you sure your custom function jar is visible for the Maven Plugin ?
As when you run it from JMeter, it works , I suppose you have a jar in lib/ext.
So you need to make this jar available to the jmeter maven plugin.
I tried to follow the PHPUnit manual on how to setup a testsuite with a custom test execution order. I now realized that i only need these lines and some includes to get the suite working:
public static function suite()
{
$suite = new PHPUnit_Framework_TestSuite('Package');
return $suite;
}
But when i use the above lines the test execution order is defined by the sort order of my includes. And when i try to change it via suite() as followes the tests are executed twice, first in the sort order as defined by suite() and after that in the sort order of the includes:
public static function suite()
{
$suite = new PHPUnit_Framework_TestSuite('Package');
$suite->addTestSuite('Package_Class1Test');
$suite->addTestSuite('Package_Class2Test');
$suite->addTestSuite('Package_Class3Test');
return $suite;
}
Includes are done by:
require_once 'Package/Class3Test.php';
require_once 'Package/Class2Test.php';
require_once 'Package/Class1Test.php';
Result (test execution order):
1) Class1Test
2) Class2Test
3) Class3Test
4) Class3Test
5) Class2Test
6) Class1Test
I am using Netbeans 7.0beta to run the PHP 5.3.5 / PHPUnit 3.5.11 on windows 7.
I read the phpunit manual (http://www.phpunit.de/manual/3.5/en/organizing-tests.html) but i have no clue what i am doing wrong...
Please help.
PS: This problem can be solved by autoloading the classes.
PHPUnit 3.5 was released five years ago and has not been supported for four years. But even five years ago using PHPUnit_Framework_TestSuite objects to compose a test suite was no longer considered a good practice.
Please read the Getting Started article on the PHPUnit website as well as the chapter on organizing tests in the PHPUnit manual to learn how to properly organize your test suite.
Are you calling phpunit with the right parameters?
I have this setup, which works fine with suites.
/tests/
/tests/allTests.php
/tests/lib/
/tests/lib/libAllTests.php
/tests/lib/baseTest.php
/tests/lib/coreTest.php
/tests/lib/...
allTests.php:
require_once 'lib/libAllTests.php';
class AllTests
{
public static function suite()
{
$suite = new PHPUnit_Framework_TestSuite('Project');
$suite->addTestSuite('LibAllTests');
return $suite;
}
}
libAllTests.php:
require_once 'baseTest.php';
require_once 'coreTest.php';
class LibAllTests
{
public static function suite()
{
$suite = new PHPUnit_Framework_TestSuite('Lib');
$suite->addTestSuite('CoreTest');
$suite->addTestSuite('BaseTest');
return $suite;
}
}
From a command prompt I can call:
phpunit /tests/allTests.php: Runs all tests
phpunit /tests/lib/libAllTests.php: Runs all lib tests
phpunit /tests/lib/baseTest.php: Runs all base tests
phpunit /tests/*: Runs all tests
And in all four scenarios, the core tests are run before base tests, and no tests are repeated twice.
I'm using phpUnit 3.5.7.