I have a smoke suite having xx test Cases. I have applied the Meta filter on only smoke level test cases. But when I tried with *.story as argument for eclipse, its hangup the execution while excluding the test cases as per the Meta Filter.
ENVIRONMENT: variable:- STORY_META_FILTER
Value:- +smoke
Story file Structure:
Scenario : test_xyz
Meta : #smoke
Given TEST1
When TEST2
Then TEST3
Can Anyone know the correct way to implement it in Eclipse? What will be the argument for Eclipse?
Set environment Variable with Meta_Filter and having value as +smoke
and run the story file with *.story
Related
I am trying to run performance test on scenario tagged as perf from the below feature file-
#tag1 #tag2 #tag3
**background:**
user login
#tag4 #perf
**scenario1:**
#tag4
**scenario2:**
Below is my .scala file setup-
class PerfTest extends Simulation {
val protocol = karateProtocol()
val getTags = scenario("Name goes here").exec(karateFeature("classpath:filepath"))
setUp(
getTags.inject(
atOnceUsers(1)
).protocols(protocol)
)
I have tried passing the tags from command line and as well as passing the tag as argument in exec method in scala setup.
Terminal command-
mvn clean test-compile gatling:test "-Dkarate.env={env}" "-Dkarate.options= --tags #perf"
.scala update:- I have also tried passing the tag as an argument in the karate execute.
val getTags = scenario("Name goes here").exec(karateFeature("classpath:filepath", "#perf"))
Both scenarios are being executed with either approach. Any pointers how i can force only the test with tag perf to run?
I wanted to share the finding here. I realized it is working fine when i am passing the tag info in .scala file.
My scenario with perf tag was a combination of GET and POST call as i needed some data from GET call to pass in POST call. That's why i was seeing both calls when running performance test.
I did not find any reference in karate gatling documentation for passing tags in terminal execution command. So i am assuming that might not be a valid case.
I am attempting to run my SpecFlow/xUnit tests on the command line, as described here:
http://gasparnagy.com/2016/02/running-specflow-scenarios-in-parallel-with-xunit-v2/
If I type this:
.\packages\xunit.runner.console.2.3.1\tools\net452\xunit.console.exe --help
One of the flags described is this:
-trait "name=value" : only run tests with matching name/value traits
: if specified more than once, acts as an OR operation
I have a SpecFlow scenario with the trait #justthisone which I would like to run on its own. The Visual Studio test explorer lists this as having the trait Category [justthisone] I have tried this:
.\packages\xunit.runner.console.2.3.1\tools\net452\xunit.console.exe .\MyProj.Tests\bin\Debug\MyProj.Tests.dll -trait "name=justthisone"
But I get this output:
=== TEST EXECUTION SUMMARY ===
Order.UserInterface.Tests.dll Total: 0
How should I write the -trait flag/option to tell xUnit which tests I want to run?
Turns out I just had to specify the correct file name:
.\packages\xunit.runner.console.2.3.1\tools\net452\xunit.console.exe .\MyProj.Tests\bin\Debug\MyProj.Tests.dll -trait "**Category**=justthisone"
As shown here:
https://github.com/techtalk/SpecFlow/issues/938
I want to run feature files in a desired order or sequence, for example:
tags:"`#ProtractorScenario` or #CucumberScenario"
But cucumber scenario is getting executed first. Can someone guide me on this?
Note: Cucumber is executing scenario based on alphabetical order of feature file in folder
Also, in cases with more than 50+ feature files, what would be the best way to define sequencing of cucumber feature files?
In order to have reliable tests, your tests should be independent and not rely on the order they are run in. The reason being that your test shouldn't depend on the system being in a certain state, as this will lead to flaky tests. Each of your tests should set up the expected state (and teardown!), so they can be run independently.
Below is how protractor executes cucumber feature files:
Protractor finds out all feature files specified in specs, save the absolute file path into an array, let's call it feature_list.
Protractor starts a session (start a browser instance)
Protractor generates a Cucumber CLI as below, and execute the CLI to hand over the running control cucumber:
./node_modules/bin/cucumber --require xxx --format xxx feature1,feature2,....featureN
feature1,feature2,....featureN calculated by feature_list.join(',')
From above, we can learn the only opportunity to change the order
is given an order-done feature_list to protractor specs.
Note: every member of the feature_list should be absolute/relative
path of single feature file. folder and wildcard are not recommended to appear in the path.
You can get a solution code from my github: spec.filter.js, which implements:
filter feature file by cucumberOpts.tags
order filter result of above step 1 by priority
Guide to use spec.filter.js:
// protractor conf file
const specFilter = require('./spec.filter.js');
var config = {
seleniumAddress: 'xxxxx',
capabilities:'xxxx',
framework: 'custom',
frameworkPath: require.resolve('protractor-cucumber-framework'),
ignoreUncaughtExceptions: true,
specs: [
'./aa/**/*.feature',
'./bb/**/*.feature'
],
cucumberOpts: {
require: [
'xxx'
],
priorities: {
// feature has tag #SPGC-21542 or #SPGC-21944 or #SPGC-21946
// will has priority 1
'1': ['#SPGC-21542 or #SPGC-21944', '#SPGC-21946'],
// feature has tag #SPGC-22055 will has priority 2,
// feature has heighest priority will put ahead at
// the `specs` list and get executed firstly.
'2': ['#SPGC-22055']
}
tags: ""
}
....
};
exports.config = specFilter(config);
How can I execute just a single test case through the NUnit ConsoleRunner using the --test=NAMES argument.
If I right click on a test case in Visual Studio and 'Copy' this is the information regarding the test.
Test Name: MakePaymentToAccountViaPayPal
Test FullName: POMAuctivaTest.TestSuite.ExistingUserTestSuite("firefox").MakePaymentToAccountViaPayPal
Test Source: c:\git\POMAuctivaTest\POMAuctivaTest.TestSuite\ExistingUserTestSuite.cs : line 263
Test Outcome: Not Run
Test Duration: 0:00:00
Here are some of the different ways I tried to run the tests.
I have tried with both the Test Name and the Test FullName and both attempts don't actually run the test cases. I have a feeling the issue is with the string being passed as a parameter but I am not sure.
Have just tried this, looks like you just need to escape the quotation marks. So use:
--test=POMAuctivaTest.TestSuite.ExistingUserTestSuite(\"firefox\").MakePaymentToAccountViaPayPal
I want to get the test name and test result during runtime.
I have setup and tearDown methods in my script. In setup, I need to get the test name, and in tearDown I need to get the test result and test execution time.
Is there a way I can do this?
You can, using a hook.
I have these files in my test directory:
./rest/
├── conftest.py
├── __init__.py
└── test_rest_author.py
In test_rest_author.py I have three functions, startup, teardown and test_tc15, but I only want to show the result and name for test_tc15.
Create a conftest.py file if you don't have one yet and add this:
import pytest
from _pytest.runner import runtestprotocol
def pytest_runtest_protocol(item, nextitem):
reports = runtestprotocol(item, nextitem=nextitem)
for report in reports:
if report.when == 'call':
print '\n%s --- %s' % (item.name, report.outcome)
return True
The hook pytest_runtest_protocol implements the runtest_setup/call/teardown protocol for the given test item, including capturing exceptions and calling reporting hooks. It is called when any test finishes (like startup or teardown or your test).
If you run your script you can see the result and name of the test:
$ py.test ./rest/test_rest_author.py
====== test session starts ======
/test_rest_author.py::TestREST::test_tc15 PASSED
test_tc15 --- passed
======== 1 passed in 1.47 seconds =======
See also the docs on pytest hooks and conftest.py.
unittest.TestCase.id() this will return the complete Details including class name , method name .
From this we can extract test method name.
Getting the results during can be achieved by checking if there any exceptions in executing the test.
If the test fails then there wil be an exception if sys.exc_info() returns None then test is pass else test will be fail.
Using pytest_runtest_protocol as suggested with fixture marker solved my problem. In my case it was enough just to use reports = runtestprotocol(item, nextitem=nextitem) within my pytest html fixture. So to finalize the item element contains the information you need.
Many Thanks.