I have the following
[assembly: LevelOfParallelism(10)]
[Parallelizable(ParallelScope.Self)]
public class MessageHandlerTests
{
[Test]
public async Task WhenCallingHandle_ShouldInvokeConsumer(
[Values(1, 25)] int messageCount,
[Values(5, 12)] int processingTimeSeconds,
[Values(SendMode.AzureServiceBus, SendMode.BrokeredMessageSender)] SendMode sendMode,
[Values(ConsumerAction.None, ConsumerAction.Publish, ConsumerAction.Reply, ConsumerAction.Send)] ConsumerAction consumerAction)
{
...
}
}
It's a semi long running test (about 30 seconds) and I have logging throughout the test and as far as I can tell it's not running in parallel, either via Test Explorer or via nunit3-console.exe
Any ideas on what I'm doing wrong?
Resolved this by setting
[Parallelizable(ParallelScope.All)]
Issue seems to be an unexpected behavior when using combinatorial tests. Note this is a newly added enum value.
Related
I have 3 normal and 1 explicit test but when I run my test using the Test Explorer window I get this output under "Tests" in the Output window
========== Starting test run ==========
NUnit Adapter 4.2.0.0: Test execution started
Running all tests in xyz.dll
NUnit3TestExecutor discovered 1 of 1 NUnit test cases using Current Discovery mode, Explicit run
ExplicitMethod(03/10/2022 08:00:00,03/10/2022 16:00:00): OneTimeSetUp:
NUnit Adapter 4.2.0.0: Test execution complete
========== Test run finished: 1 Tests (0 Passed, 0 Failed, 0 Skipped) run in 462 ms ==========
*Note I'm using NUnit Adapter 4.2.1 instead of 4.2.0 so that's already weird
And this is how the Test Explorer window looks
Test Explorer window
*Note the full blue test is the Explicit test that get's skipped like it should
This is a problem because it does seem to only discover tests which are explicit.
The tests I want to run are of course not explicit, here is an example
[Test]
[TestCaseSource(nameof(TestNameData))]
public async Task<float> TestName(DateTime start, DateTime end, List<CalculateHoursObj>? list = default)
{
if (list == null) list = new List<CalculateHoursObj>();
return await EmployeeService.CalculateOverTimeHours(start, end, list);
}
public static IEnumerable TestNameData
{
get
{
yield return new TestCaseData(TenthMarch8_2022, TenthMarch16_2022).Returns(8.0f);
}
}
It fails to discover and/or run this test.
But curiously if I break all my non explicit tests by making the data non-static like this
public IEnumerable TestNameData
{
get
{
yield return new TestCaseData(TenthMarch8_2022, TenthMarch16_2022).Returns(8.0f);
}
}
It of course breaks the test and when I run all tests it does actually discover all tests
========== Starting test run ==========
NUnit Adapter 4.2.0.0: Test execution started
Running selected tests in xyz.dll
NUnit3TestExecutor discovered 2 of 4 NUnit test cases using Current Discovery mode, Non-Explicit run
NUnit Adapter 4.2.0.0: Test execution complete
========== Test run finished: 2 Tests (0 Passed, 2 Failed, 0 Skipped) run in 394 ms ==========
But even now it only runs 2 of the 3 broken non explicit tests and of course they all fail
I have looked up everything online for 1.5 hours and really can't find a solution.
Don't bother responding with "have you updated visual studio or the nugget packages"
This question was part of a bug the solution is to not use new DateTime() withing nunit tests or to have nunit test be discover with id's using the <UseParentFQNForParametrizedTests>true</UseParentFQNForParametrizedTests> <UseNUnitIdforTestCaseId>true</UseNUnitIdforTestCaseId> flags
This was a bug because it was not reported correctly in the output window and/or Error List window.
I've created an Issue on GitHub about this if anyone wants to read.
Now stop editing this post because it truly is answered and the Issue link which I included shows the valid, correct and readable answer by "OsirisTerje" whom explains it way better than I could.
For my project I want to run the exact same test cases twice, once locally and on a different VM in parallel in the cloud (Azure in my case).
I duplicated the TestCase and tagged one Category("Local") and the other Category("Cloud").
Running nunit3 from the console with --where="cat == Cloud" will thus run all TestCases of every test that has one or more TestCases tagged with Category("Cloud").
Is there a different way of only running selected TestCases by a commandline switch?
Simplified example:
[TestCase(TestName = "Canary, Run in cloud."), Category("Cloud")]
[TestCase(TestName = "Canary, Run locally."), Category("Local")]
public void Canary()
{
Assert.True(true);
}
Found a work-around.
Using --params:Cloud=true as command line argument and in the code
private bool ShallRunInCloud => TestContext.Parameters["Cloud"]?.ToLowerInvariant() == "true";
I am running 5 API test scenarios using karate. When I run the test in non-parallel mode using #RunWith(Karate.class) then in xml generated by surefire-reports, all scenarios are reported individually as .
<testcase classname="[healthCheck]" name="[1:3] Check health check API returns status code 200" time="2.846"/>
<testcase classname="[healthCheckHeader]" name="[1:6] Check health check API returns status code 200" time="0.285"/>
<testcase classname="[userLogin]" name="[1:3] Check User Login API returns status code 200" time="0.108"/>
<testcase classname="[requestChaining]" name="[1:7] chain request demo" time="0.521"/>
<testcase classname="[viewRequests]" name="[1:10] Check View Requests API returns status code 200" time="0.278"/>
However, when I use karate parallel runner, then each scenario is not reported individually.
<testcase classname="demoTest.AutomationSuiteParallelCucRunner" name="testParallel" time="10.917"/>
I want to have similar report for parallel runner as generated when tests executed in non-parallel mode.
Here is the code for running tests in non-parallel mode:
#RunWith(Karate.class)
public class AutomationSuiteTest {
}
Here is the code for running tests in parallel mode:
#CucumberOptions(tags = {"~#ignore"})
public class AutomationSuiteParallelCucRunner {
#Test
public void testParallel() {
String karateOutputPath = "target/surefire-reports";
KarateStats stats = CucumberRunner.parallel(getClass(), 3, karateOutputPath);
assertTrue("SCENARIO FAILURES!!", stats.getFailCount() == 0);
}
}
Having struggled with this myself, the solution was surprisingly simple, though not really covered in the karate docs (as of this time).
The point is: when using the parallel runner in this manner, there is from JUnit's perspective just one unit test - the single method annotated with #Test. So this single test is all that's listed in target/surefire-reports. Karate doesn't seem to hook into the JUnit execution or so, so that's it.
But - and that was the missing piece for me - the karate runner itself can generate JUnit XML files. You just have to call .outputJunitXml(true) when creating it. This causes it to generate JUnit XML files in the karate report dir along with the HTML report and (optionally) cucumber JSON files:
import com.intuit.karate.Results;
import com.intuit.karate.Runner;
public class KarateTestRunner {
#Test
public void runParallel() {
final Results results = Runner.path("classpath:api/karate")
.outputCucumberJson(true)
.outputJunitXml(true) # <----
.parallel(8);
assertEquals(0, results.getFailCount());
}
}
UPDATE:
AutoFixture team released a fix for this in version 3.51.
Simply extend the AutoDataAttribute doing so:
public class AutoDataFixedNameAttribute : AutoDataAttribute
{
public AutoDataFixedNameAttribute()
{
this.TestMethodBuilder = new FixedNameTestMethodBuilder();
}
}
Then use this new attribute instead of the built-in AutoData in your NUnit tests.
Starting from v4, this behavior is the default one.
Previous post
I'm trying to use AutoFixture with NUnit and Moq, using the following AutoMoqDataAttribute :
public class AutoMoqDataAttribute : AutoDataAttribute
{
public AutoMoqDataAttribute()
: base(new Fixture().Customize(new AutoMoqCustomization()))
{
}
}
But when I run this test :
[Test, AutoMoqData]
public void Test(Mock<IUser> user)
{
// do stuff with user
}
The test never runs. AutomMoqData is hit correctly, but the code inside the test is never executed and everything ends without any warning with the following message :
Test adapter sent back a result for an unknown test case. Ignoring result for 'Test(Mock<Sandbox.IUser>)'
The test also doesn't appear in the test runner list.
But if I remove the parameter :
[Test, AutoMoqData]
public void Test()
{
// do stuff without user
}
Everything runs fine, but this is less useful without the parameters passed :)
Am I missing something here ?
Here is the list of the Nuget packages versions :
<package id="AutoFixture" version="3.50.2" targetFramework="net452" />
<package id="AutoFixture.AutoMoq" version="3.50.2" targetFramework="net452" />
<package id="AutoFixture.NUnit3" version="3.50.2" targetFramework="net452" />
<package id="Moq" version="4.5.3" targetFramework="net452" />
<package id="NUnit" version="3.7.1" targetFramework="net452" />
EDIT:
Following #MarkSeemann's advice, I filed an issue on Github.
Visual Studio Test Runner repro
This looks like an issue with the NUnit Visual Studio test adapter. I can reproduce the issue when I also add the NUnit3TestAdapter package to my repro solution.
I'm also assuming that the test class has the [TestFixture] attribute, so that the entire repro class looks like this:
[TestFixture]
public class Tests
{
[Test, AutoMoqData]
public void Test(Mock<IUser> user)
{
Assert.NotNull(user);
}
}
When I attempt to run all tests using Visual Studio 2015's test runner, the test never runs, and this is the output to test output window:
------ Run test started ------
NUnit Adapter 3.7.0.0: Test execution started
Running all tests in C:\Users\mark\Documents\Stack Overflow\44564377\44564377\bin\Debug\Ploeh.StackOverflow.Q44564377.dll
NUnit3TestExecutor converted 1 of 1 NUnit test cases
NUnit Adapter 3.7.0.0: Test execution complete
Test adapter sent back a result for an unknown test case. Ignoring result for 'Test(Mock<Ploeh.StackOverflow.Q44564377.IUser:8e33>)'.
========== Run test finished: 0 run (0:00:01,1763498) ==========
TestDriven.Net
If, on the other hand, I try to run it with TestDriven.Net, it runs just fine:
------ Test started: Assembly: Ploeh.StackOverflow.Q44564377.dll ------
1 passed, 0 failed, 0 skipped, took 0,79 seconds (NUnit 3.7.1).
TestDriven.Net is sometimes extraordinarily tolerant of small errors in the test code, so this may not be that telling in itself.
NUnit 3 console runner
Since TestDriven.Net may be too liberal in what it accepts, a better test would be to try with the official NUnit 3 console runner:
$ packages/NUnit.ConsoleRunner.3.6.1/tools/nunit3-console.exe 44564377/bin/Debug/Ploeh.StackOverflow.Q44564377.dll
NUnit Console Runner 3.6.1
Copyright (C) 2017 Charlie Poole
Runtime Environment
OS Version: Microsoft Windows NT 10.0.15063.0
CLR Version: 4.0.30319.42000
Test Files
44564377/bin/Debug/Ploeh.StackOverflow.Q44564377.dll
Run Settings
DisposeRunners: True
WorkDirectory: C:\Users\mark\Documents\Stack Overflow\44564377
ImageRuntimeVersion: 4.0.30319
ImageTargetFrameworkName: .NETFramework,Version=v4.6.1
ImageRequiresX86: False
ImageRequiresDefaultAppDomainAssemblyResolver: False
NumberOfTestWorkers: 4
Test Run Summary
Overall result: Passed
Test Count: 1, Passed: 1, Failed: 0, Warnings: 0, Inconclusive: 0, Skipped: 0
Start time: 2017-06-15 11:09:21Z
End time: 2017-06-15 11:09:22Z
Duration: 0.933 seconds
Results (nunit3) saved as TestResult.xml
This, too, successfully executes the test.
Interim conclusion
Since both the official console runner and TestDriven.Net successfully executes the test, I'd tentatively conclude that this looks like a defect in the NUnit3TestAdapter package. May I suggest filing an issue for it?
I tried to follow the PHPUnit manual on how to setup a testsuite with a custom test execution order. I now realized that i only need these lines and some includes to get the suite working:
public static function suite()
{
$suite = new PHPUnit_Framework_TestSuite('Package');
return $suite;
}
But when i use the above lines the test execution order is defined by the sort order of my includes. And when i try to change it via suite() as followes the tests are executed twice, first in the sort order as defined by suite() and after that in the sort order of the includes:
public static function suite()
{
$suite = new PHPUnit_Framework_TestSuite('Package');
$suite->addTestSuite('Package_Class1Test');
$suite->addTestSuite('Package_Class2Test');
$suite->addTestSuite('Package_Class3Test');
return $suite;
}
Includes are done by:
require_once 'Package/Class3Test.php';
require_once 'Package/Class2Test.php';
require_once 'Package/Class1Test.php';
Result (test execution order):
1) Class1Test
2) Class2Test
3) Class3Test
4) Class3Test
5) Class2Test
6) Class1Test
I am using Netbeans 7.0beta to run the PHP 5.3.5 / PHPUnit 3.5.11 on windows 7.
I read the phpunit manual (http://www.phpunit.de/manual/3.5/en/organizing-tests.html) but i have no clue what i am doing wrong...
Please help.
PS: This problem can be solved by autoloading the classes.
PHPUnit 3.5 was released five years ago and has not been supported for four years. But even five years ago using PHPUnit_Framework_TestSuite objects to compose a test suite was no longer considered a good practice.
Please read the Getting Started article on the PHPUnit website as well as the chapter on organizing tests in the PHPUnit manual to learn how to properly organize your test suite.
Are you calling phpunit with the right parameters?
I have this setup, which works fine with suites.
/tests/
/tests/allTests.php
/tests/lib/
/tests/lib/libAllTests.php
/tests/lib/baseTest.php
/tests/lib/coreTest.php
/tests/lib/...
allTests.php:
require_once 'lib/libAllTests.php';
class AllTests
{
public static function suite()
{
$suite = new PHPUnit_Framework_TestSuite('Project');
$suite->addTestSuite('LibAllTests');
return $suite;
}
}
libAllTests.php:
require_once 'baseTest.php';
require_once 'coreTest.php';
class LibAllTests
{
public static function suite()
{
$suite = new PHPUnit_Framework_TestSuite('Lib');
$suite->addTestSuite('CoreTest');
$suite->addTestSuite('BaseTest');
return $suite;
}
}
From a command prompt I can call:
phpunit /tests/allTests.php: Runs all tests
phpunit /tests/lib/libAllTests.php: Runs all lib tests
phpunit /tests/lib/baseTest.php: Runs all base tests
phpunit /tests/*: Runs all tests
And in all four scenarios, the core tests are run before base tests, and no tests are repeated twice.
I'm using phpUnit 3.5.7.