Junit xml report don't report each test case when 'Karate' parallel runner is used - junit4

I am running 5 API test scenarios using karate. When I run the test in non-parallel mode using #RunWith(Karate.class) then in xml generated by surefire-reports, all scenarios are reported individually as .
<testcase classname="[healthCheck]" name="[1:3] Check health check API returns status code 200" time="2.846"/>
<testcase classname="[healthCheckHeader]" name="[1:6] Check health check API returns status code 200" time="0.285"/>
<testcase classname="[userLogin]" name="[1:3] Check User Login API returns status code 200" time="0.108"/>
<testcase classname="[requestChaining]" name="[1:7] chain request demo" time="0.521"/>
<testcase classname="[viewRequests]" name="[1:10] Check View Requests API returns status code 200" time="0.278"/>
However, when I use karate parallel runner, then each scenario is not reported individually.
<testcase classname="demoTest.AutomationSuiteParallelCucRunner" name="testParallel" time="10.917"/>
I want to have similar report for parallel runner as generated when tests executed in non-parallel mode.
Here is the code for running tests in non-parallel mode:
#RunWith(Karate.class)
public class AutomationSuiteTest {
}
Here is the code for running tests in parallel mode:
#CucumberOptions(tags = {"~#ignore"})
public class AutomationSuiteParallelCucRunner {
#Test
public void testParallel() {
String karateOutputPath = "target/surefire-reports";
KarateStats stats = CucumberRunner.parallel(getClass(), 3, karateOutputPath);
assertTrue("SCENARIO FAILURES!!", stats.getFailCount() == 0);
}
}

Having struggled with this myself, the solution was surprisingly simple, though not really covered in the karate docs (as of this time).
The point is: when using the parallel runner in this manner, there is from JUnit's perspective just one unit test - the single method annotated with #Test. So this single test is all that's listed in target/surefire-reports. Karate doesn't seem to hook into the JUnit execution or so, so that's it.
But - and that was the missing piece for me - the karate runner itself can generate JUnit XML files. You just have to call .outputJunitXml(true) when creating it. This causes it to generate JUnit XML files in the karate report dir along with the HTML report and (optionally) cucumber JSON files:
import com.intuit.karate.Results;
import com.intuit.karate.Runner;
public class KarateTestRunner {
#Test
public void runParallel() {
final Results results = Runner.path("classpath:api/karate")
.outputCucumberJson(true)
.outputJunitXml(true) # <----
.parallel(8);
assertEquals(0, results.getFailCount());
}
}

Related

Nunit won't discover working non explicit tests Visual studio 2022

I have 3 normal and 1 explicit test but when I run my test using the Test Explorer window I get this output under "Tests" in the Output window
========== Starting test run ==========
NUnit Adapter 4.2.0.0: Test execution started
Running all tests in xyz.dll
NUnit3TestExecutor discovered 1 of 1 NUnit test cases using Current Discovery mode, Explicit run
ExplicitMethod(03/10/2022 08:00:00,03/10/2022 16:00:00): OneTimeSetUp:
NUnit Adapter 4.2.0.0: Test execution complete
========== Test run finished: 1 Tests (0 Passed, 0 Failed, 0 Skipped) run in 462 ms ==========
*Note I'm using NUnit Adapter 4.2.1 instead of 4.2.0 so that's already weird
And this is how the Test Explorer window looks
Test Explorer window
*Note the full blue test is the Explicit test that get's skipped like it should
This is a problem because it does seem to only discover tests which are explicit.
The tests I want to run are of course not explicit, here is an example
[Test]
[TestCaseSource(nameof(TestNameData))]
public async Task<float> TestName(DateTime start, DateTime end, List<CalculateHoursObj>? list = default)
{
if (list == null) list = new List<CalculateHoursObj>();
return await EmployeeService.CalculateOverTimeHours(start, end, list);
}
public static IEnumerable TestNameData
{
get
{
yield return new TestCaseData(TenthMarch8_2022, TenthMarch16_2022).Returns(8.0f);
}
}
It fails to discover and/or run this test.
But curiously if I break all my non explicit tests by making the data non-static like this
public IEnumerable TestNameData
{
get
{
yield return new TestCaseData(TenthMarch8_2022, TenthMarch16_2022).Returns(8.0f);
}
}
It of course breaks the test and when I run all tests it does actually discover all tests
========== Starting test run ==========
NUnit Adapter 4.2.0.0: Test execution started
Running selected tests in xyz.dll
NUnit3TestExecutor discovered 2 of 4 NUnit test cases using Current Discovery mode, Non-Explicit run
NUnit Adapter 4.2.0.0: Test execution complete
========== Test run finished: 2 Tests (0 Passed, 2 Failed, 0 Skipped) run in 394 ms ==========
But even now it only runs 2 of the 3 broken non explicit tests and of course they all fail
I have looked up everything online for 1.5 hours and really can't find a solution.
Don't bother responding with "have you updated visual studio or the nugget packages"
This question was part of a bug the solution is to not use new DateTime() withing nunit tests or to have nunit test be discover with id's using the <UseParentFQNForParametrizedTests>true</UseParentFQNForParametrizedTests> <UseNUnitIdforTestCaseId>true</UseNUnitIdforTestCaseId> flags
This was a bug because it was not reported correctly in the output window and/or Error List window.
I've created an Issue on GitHub about this if anyone wants to read.
Now stop editing this post because it truly is answered and the Issue link which I included shows the valid, correct and readable answer by "OsirisTerje" whom explains it way better than I could.

How to run a sub-set of TestCases using --where for nunit

For my project I want to run the exact same test cases twice, once locally and on a different VM in parallel in the cloud (Azure in my case).
I duplicated the TestCase and tagged one Category("Local") and the other Category("Cloud").
Running nunit3 from the console with --where="cat == Cloud" will thus run all TestCases of every test that has one or more TestCases tagged with Category("Cloud").
Is there a different way of only running selected TestCases by a commandline switch?
Simplified example:
[TestCase(TestName = "Canary, Run in cloud."), Category("Cloud")]
[TestCase(TestName = "Canary, Run locally."), Category("Local")]
public void Canary()
{
Assert.True(true);
}
Found a work-around.
Using --params:Cloud=true as command line argument and in the code
private bool ShallRunInCloud => TestContext.Parameters["Cloud"]?.ToLowerInvariant() == "true";

NUnit3 Tests Don't seem to be running in parallel

I have the following
[assembly: LevelOfParallelism(10)]
[Parallelizable(ParallelScope.Self)]
public class MessageHandlerTests
{
[Test]
public async Task WhenCallingHandle_ShouldInvokeConsumer(
[Values(1, 25)] int messageCount,
[Values(5, 12)] int processingTimeSeconds,
[Values(SendMode.AzureServiceBus, SendMode.BrokeredMessageSender)] SendMode sendMode,
[Values(ConsumerAction.None, ConsumerAction.Publish, ConsumerAction.Reply, ConsumerAction.Send)] ConsumerAction consumerAction)
{
...
}
}
It's a semi long running test (about 30 seconds) and I have logging throughout the test and as far as I can tell it's not running in parallel, either via Test Explorer or via nunit3-console.exe
Any ideas on what I'm doing wrong?
Resolved this by setting
[Parallelizable(ParallelScope.All)]
Issue seems to be an unexpected behavior when using combinatorial tests. Note this is a newly added enum value.

How to execute JMeter test case from Java code

How do I run a JMeter test case from Java code?
I have followed the example Here from Blazemeter.com
My code is as follows:
public class BasicSampler {
public static void main(String[] argv) throws Exception {
// JMeter Engine
StandardJMeterEngine jmeter = new StandardJMeterEngine();
// Initialize Properties, logging, locale, etc.
JMeterUtils.loadJMeterProperties("/home/stone/Workbench/automated-testing/apache-jmeter-2.11/bin/jmeter.properties");
JMeterUtils.setJMeterHome("/home/stone/Workbench/automated-testing/apache-jmeter-2.11");
JMeterUtils.initLogging();// you can comment this line out to see extra log messages of i.e. DEBUG level
JMeterUtils.initLocale();
// Initialize JMeter SaveService
SaveService.loadProperties();
// Load existing .jmx Test Plan
FileInputStream in = new FileInputStream("/home/stone/Workbench/automated-testing/apache-jmeter-2.11/bin/examples/CSVSample.jmx");
HashTree testPlanTree = SaveService.loadTree(in);
in.close();
// Run JMeter Test
jmeter.configure(testPlanTree);
jmeter.run();
}
}
but I keep getting the following messages in the console and my test never executes.
INFO 2014-09-23 12:04:40.492 [jmeter.e] (): Listeners will be started after enabling running version
INFO 2014-09-23 12:04:40.511 [jmeter.e] (): To revert to the earlier behaviour, define jmeterengine.startlistenerslater=false
I have also tried uncommented jmeterengine.startlistenerslater=false from jmeter.properties file
How do you know that your "test never executes"?
What is in jmeter.log file (it should be in the root of your project). Or alternatively comment JMeterUtils.initLogging() line to see the full output in STDOUT
Have you changed relative path CSVSample_user.csv in "Get user details" CSV Data Set Config as it may resolve into a different location as it recommended in Using CSV DATA SET CONFIG
Is CSVSample.jtl file generated anywhere (again it should be in the root of your project by default)? What is in it?
The code looks good and I'm pretty sure that the problem is with the path to CSVSample_user.csv file and you have something like java.io.FileNotFoundException in your log. Please double check that CSVSample.jmx file contains valid full path to CSVSample_user.csv.
UPDATE TO ANSWER QUESTIONS IN COMMENTS
jmeter.log file should be under your Eclipse workspace folder by default
Looking into CSVSample.jmx there is a View Resulst in Table listener which is configured to store results under ~/CSVSample.jtl
If you want to see summarizer messages and "classic" .jtl reporting add next few lines before jmeter.configure(testPlanTree); stanza
Summariser summer = null;
String summariserName = JMeterUtils.getPropDefault("summariser.name", "summary");
if (summariserName.length() > 0) {
summer = new Summariser(summariserName);
}
String logFile = "/path/to/jtl/results/file.jtl";
ResultCollector logger = new ResultCollector(summer);
logger.setFilename(logFile);
testPlanTree.add(testPlanTree.getArray()[0], logger);
Try using library - https://github.com/abstracta/jmeter-java-dsl.
It supports implementing JMeter test as java code.
Below example shows how to implement and execute test for REST API. Same approach could be applied to other type of tests as well.
#Test
public void testPerformance() throws IOException {
TestPlanStats stats = testPlan(
threadGroup(2, 10,
httpSampler("http://my.service")
.post("{\"name\": \"test\"}", Type.APPLICATION_JSON)
),
//this is just to log details of each request stats
jtlWriter("test" + Instant.now().toString().replace(":", "-") + ".jtl")
).run();
assertThat(stats.overall().elapsedTimePercentile99()).isLessThan(Duration.ofSeconds(5));
}

How to stop PHPUnit from adding included/required files as part of the testsuite?

I tried to follow the PHPUnit manual on how to setup a testsuite with a custom test execution order. I now realized that i only need these lines and some includes to get the suite working:
public static function suite()
{
$suite = new PHPUnit_Framework_TestSuite('Package');
return $suite;
}
But when i use the above lines the test execution order is defined by the sort order of my includes. And when i try to change it via suite() as followes the tests are executed twice, first in the sort order as defined by suite() and after that in the sort order of the includes:
public static function suite()
{
$suite = new PHPUnit_Framework_TestSuite('Package');
$suite->addTestSuite('Package_Class1Test');
$suite->addTestSuite('Package_Class2Test');
$suite->addTestSuite('Package_Class3Test');
return $suite;
}
Includes are done by:
require_once 'Package/Class3Test.php';
require_once 'Package/Class2Test.php';
require_once 'Package/Class1Test.php';
Result (test execution order):
1) Class1Test
2) Class2Test
3) Class3Test
4) Class3Test
5) Class2Test
6) Class1Test
I am using Netbeans 7.0beta to run the PHP 5.3.5 / PHPUnit 3.5.11 on windows 7.
I read the phpunit manual (http://www.phpunit.de/manual/3.5/en/organizing-tests.html) but i have no clue what i am doing wrong...
Please help.
PS: This problem can be solved by autoloading the classes.
PHPUnit 3.5 was released five years ago and has not been supported for four years. But even five years ago using PHPUnit_Framework_TestSuite objects to compose a test suite was no longer considered a good practice.
Please read the Getting Started article on the PHPUnit website as well as the chapter on organizing tests in the PHPUnit manual to learn how to properly organize your test suite.
Are you calling phpunit with the right parameters?
I have this setup, which works fine with suites.
/tests/
/tests/allTests.php
/tests/lib/
/tests/lib/libAllTests.php
/tests/lib/baseTest.php
/tests/lib/coreTest.php
/tests/lib/...
allTests.php:
require_once 'lib/libAllTests.php';
class AllTests
{
public static function suite()
{
$suite = new PHPUnit_Framework_TestSuite('Project');
$suite->addTestSuite('LibAllTests');
return $suite;
}
}
libAllTests.php:
require_once 'baseTest.php';
require_once 'coreTest.php';
class LibAllTests
{
public static function suite()
{
$suite = new PHPUnit_Framework_TestSuite('Lib');
$suite->addTestSuite('CoreTest');
$suite->addTestSuite('BaseTest');
return $suite;
}
}
From a command prompt I can call:
phpunit /tests/allTests.php: Runs all tests
phpunit /tests/lib/libAllTests.php: Runs all lib tests
phpunit /tests/lib/baseTest.php: Runs all base tests
phpunit /tests/*: Runs all tests
And in all four scenarios, the core tests are run before base tests, and no tests are repeated twice.
I'm using phpUnit 3.5.7.