I am attempting to run my SpecFlow/xUnit tests on the command line, as described here:
http://gasparnagy.com/2016/02/running-specflow-scenarios-in-parallel-with-xunit-v2/
If I type this:
.\packages\xunit.runner.console.2.3.1\tools\net452\xunit.console.exe --help
One of the flags described is this:
-trait "name=value" : only run tests with matching name/value traits
: if specified more than once, acts as an OR operation
I have a SpecFlow scenario with the trait #justthisone which I would like to run on its own. The Visual Studio test explorer lists this as having the trait Category [justthisone] I have tried this:
.\packages\xunit.runner.console.2.3.1\tools\net452\xunit.console.exe .\MyProj.Tests\bin\Debug\MyProj.Tests.dll -trait "name=justthisone"
But I get this output:
=== TEST EXECUTION SUMMARY ===
Order.UserInterface.Tests.dll Total: 0
How should I write the -trait flag/option to tell xUnit which tests I want to run?
Turns out I just had to specify the correct file name:
.\packages\xunit.runner.console.2.3.1\tools\net452\xunit.console.exe .\MyProj.Tests\bin\Debug\MyProj.Tests.dll -trait "**Category**=justthisone"
As shown here:
https://github.com/techtalk/SpecFlow/issues/938
Related
I am trying to run performance test on scenario tagged as perf from the below feature file-
#tag1 #tag2 #tag3
**background:**
user login
#tag4 #perf
**scenario1:**
#tag4
**scenario2:**
Below is my .scala file setup-
class PerfTest extends Simulation {
val protocol = karateProtocol()
val getTags = scenario("Name goes here").exec(karateFeature("classpath:filepath"))
setUp(
getTags.inject(
atOnceUsers(1)
).protocols(protocol)
)
I have tried passing the tags from command line and as well as passing the tag as argument in exec method in scala setup.
Terminal command-
mvn clean test-compile gatling:test "-Dkarate.env={env}" "-Dkarate.options= --tags #perf"
.scala update:- I have also tried passing the tag as an argument in the karate execute.
val getTags = scenario("Name goes here").exec(karateFeature("classpath:filepath", "#perf"))
Both scenarios are being executed with either approach. Any pointers how i can force only the test with tag perf to run?
I wanted to share the finding here. I realized it is working fine when i am passing the tag info in .scala file.
My scenario with perf tag was a combination of GET and POST call as i needed some data from GET call to pass in POST call. That's why i was seeing both calls when running performance test.
I did not find any reference in karate gatling documentation for passing tags in terminal execution command. So i am assuming that might not be a valid case.
How can I execute just a single test case through the NUnit ConsoleRunner using the --test=NAMES argument.
If I right click on a test case in Visual Studio and 'Copy' this is the information regarding the test.
Test Name: MakePaymentToAccountViaPayPal
Test FullName: POMAuctivaTest.TestSuite.ExistingUserTestSuite("firefox").MakePaymentToAccountViaPayPal
Test Source: c:\git\POMAuctivaTest\POMAuctivaTest.TestSuite\ExistingUserTestSuite.cs : line 263
Test Outcome: Not Run
Test Duration: 0:00:00
Here are some of the different ways I tried to run the tests.
I have tried with both the Test Name and the Test FullName and both attempts don't actually run the test cases. I have a feeling the issue is with the string being passed as a parameter but I am not sure.
Have just tried this, looks like you just need to escape the quotation marks. So use:
--test=POMAuctivaTest.TestSuite.ExistingUserTestSuite(\"firefox\").MakePaymentToAccountViaPayPal
I have a smoke suite having xx test Cases. I have applied the Meta filter on only smoke level test cases. But when I tried with *.story as argument for eclipse, its hangup the execution while excluding the test cases as per the Meta Filter.
ENVIRONMENT: variable:- STORY_META_FILTER
Value:- +smoke
Story file Structure:
Scenario : test_xyz
Meta : #smoke
Given TEST1
When TEST2
Then TEST3
Can Anyone know the correct way to implement it in Eclipse? What will be the argument for Eclipse?
Set environment Variable with Meta_Filter and having value as +smoke
and run the story file with *.story
I'm using CTest and want to pass command-line arguments to the underlying tests at runtime. I know there are ways to hard code command-line arguments into the CMake/CTest script, but I want to specify the command-line arguments at runtime and have those arguments passed through CTest to the underlying test.
Is this even possible?
I've figured out a way to do it (using the Fundamental theorem of software engineering). It's not as simple as I'd like, but here it is.
First, create a file ${CMAKE_SOURCE_DIR}/cmake/RunTests.cmake with the content
if(NOT DEFINED ENV{TESTS_ARGUMENTS})
set(ENV{TESTS_ARGUMENTS} "--default-arguments")
endif()
execute_process(COMMAND ${TEST_EXECUTABLE} $ENV{TESTS_ARGUMENTS} RESULT_VARIABLE result)
if(NOT "${result}" STREQUAL "0")
message(FATAL_ERROR "Test failed with return value '${result}'")
endif()
Then, when you add the test, use
add_test(
NAME MyTest
COMMAND ${CMAKE_COMMAND} -DTEST_EXECUTABLE=$<TARGET_FILE:MyTest> -P ${CMAKE_SOURCE_DIR}/cmake/RunTests.cmake
)
Finally, you can run the test with custom arguments using
cmake -E env TESTS_ARGUMENTS="--custom-arguments" ctest
Note that if you use bash, you can simplify this to
TESTS_ARGUMENTS="--custom-arguments" ctest
There are some problems with this approach, e.g. it ignores the WILL_FAIL property of the tests. Of course I wish it could be as simple as calling ctest -- --custom-arguments, but, as the Stones said, You can't always get what you want.
I'm not sure I fully understand what you want, but I still can give you a way to pass arguments to tests in CTest, at runtime.
I'll give you an example, with CTK (the Common Toolkit, https://github.com/commontk/CTK):
In the build dir (ex: CTK-build/CTK-build, it's a superbuild), if I run: ('-V' for Verbose, and '-N' for View Mode only)
ctest -R ctkVTKDataSetArrayComboBoxTest1 -V -N
I get:
UpdateCTestConfiguration from : /CTK-build/CTK-build/DartConfiguration.tcl
Parse Config file:/CTK-build/CTK-build/DartConfiguration.tcl
Add coverage exclude regular expressions.
Add coverage exclude: /CMakeFiles/CMakeTmp/
Add coverage exclude: .*/moc_.*
Add coverage exclude: .*/ui_.*
Add coverage exclude: .*/Testing/.*
Add coverage exclude: .*/CMakeExternals/.*
Add coverage exclude: ./ctkPixmapIconEngine.*
Add coverage exclude: ./ctkIconEngine.*
UpdateCTestConfiguration from :/CTK-build/CTK-build/DartConfiguration.tcl
Parse Config file:/CTK-build/CTK-build/DartConfiguration.tcl
Test project /CTK-build/CTK-build
Constructing a list of tests
Done constructing a list of tests
178: Test command: /CTK-build/CTK-build/bin/CTKVisualizationVTKWidgetsCppTests "ctkVTKDataSetArrayComboBoxTest1"
Labels: CTKVisualizationVTKWidgets
Test #178: ctkVTKDataSetArrayComboBoxTest1
Total Tests: 1
You can copy-paste the "Test command" in your terminal:
/CTK-build/CTK-build/bin/CTKVisualizationVTKWidgetsCppTests "ctkVTKDataSetArrayComboBoxTest1"
And add the arguments, for example "-I" for interactive testing:
/CTK-build/CTK-build/bin/CTKVisualizationVTKWidgetsCppTests "ctkVTKDataSetArrayComboBoxTest1" "-I"
Tell me if it helps.
matthieu's answer gave me the clue to get it to work for me.
For my code I did the following:
Type the command ctest -V -R TestMembraneCellCrypt -N to get the output:
...
488: Test command: path/to/ctest/executable/TestMembraneCellCrypt
Labels: Continuous_project_ChasteMembrane
Test #488: TestMembraneCellCrypt
...
Then I copied the Test command and provided the arguments there:
path/to/ctest/executable/TestMembraneCellCrypt -e 2 -em 5 -ct 10
I'll note that the package I'm using (Chaste), is pretty large so there might be things going on that I don't know about.
I know that a single test can be ran by running, in sbt,
testOnly *class -- -n Tag
Is there a way of telling sbt/scalatest to run a single test without tags? For example:
testOnly *class -- -X 2
it would mean "run the second test in the class. Whatever it is". We have a bunch of tests and no one bothered to tag them, so is there a way to run a single test without it having a tag?
This is now supported (since ScalaTest 2.1.3) within interactive mode:
testOnly *MySuite -- -z foo
to run only the tests whose name includes the substring "foo".
For exact match rather than substring, use -t instead of -z.
If you run it from the command line, it should be as single argument to sbt:
sbt 'testOnly *MySuite -- -z foo'
I wanted to add a concrete example to accompany the other answers
You need to specify the name of the class that you want to test, so if you have the following project (this is a Play project):
You can test just the Login tests by running the following command from the SBT console:
test:testOnly *LoginServiceSpec
If you are running the command from outside the SBT console, you would do the following:
sbt "test:testOnly *LoginServiceSpec"
I don't see a way to run a single untagged test within a test class but I am providing my workflow since it seems to be useful for anyone who runs into this question.
From within a sbt session:
test:testOnly *YourTestClass
(The asterisk is a wildcard, you could specify the full path com.example.specs.YourTestClass.)
All tests within that test class will be executed. Presumably you're most concerned with failing tests, so correct any failing implementations and then run:
test:testQuick
... which will only execute tests that failed. (Repeating the most recently executed test:testOnly command will be the same as test:testQuick in this case, but if you break up your test methods into appropriate test classes you can use a wildcard to make test:testQuick a more efficient way to re-run failing tests.)
Note that the nomenclature for test in ScalaTest is a test class, not a specific test method, so all untagged methods are executed.
If you have too many test methods in a test class break them up into separate classes or tag them appropriately. (This could be a signal that the class under test is in violation of single responsibility principle and could use a refactoring.)
Just to simplify the example of Tyler.
test:-prefix is not needed.
So according to his example:
In the sbt-console:
testOnly *LoginServiceSpec
And in the terminal:
sbt "testOnly *LoginServiceSpec"
Here's the Scalatest page on using the runner and the extended discussion on the -t and -z options.
This post shows what commands work for a test file that uses FunSpec.
Here's the test file:
package com.github.mrpowers.scalatest.example
import org.scalatest.FunSpec
class CardiBSpec extends FunSpec {
describe("realName") {
it("returns her birth name") {
assert(CardiB.realName() === "Belcalis Almanzar")
}
}
describe("iLike") {
it("works with a single argument") {
assert(CardiB.iLike("dollars") === "I like dollars")
}
it("works with multiple arguments") {
assert(CardiB.iLike("dollars", "diamonds") === "I like dollars, diamonds")
}
it("throws an error if an integer argument is supplied") {
assertThrows[java.lang.IllegalArgumentException]{
CardiB.iLike()
}
}
it("does not compile with integer arguments") {
assertDoesNotCompile("""CardiB.iLike(1, 2, 3)""")
}
}
}
This command runs the four tests in the iLike describe block (from the SBT command line):
testOnly *CardiBSpec -- -z iLike
You can also use quotation marks, so this will also work:
testOnly *CardiBSpec -- -z "iLike"
This will run a single test:
testOnly *CardiBSpec -- -z "works with multiple arguments"
This will run the two tests that start with "works with":
testOnly *CardiBSpec -- -z "works with"
I can't get the -t option to run any tests in the CardiBSpec file. This command doesn't run any tests:
testOnly *CardiBSpec -- -t "works with multiple arguments"
Looks like the -t option works when tests aren't nested in describe blocks. Let's take a look at another test file:
class CalculatorSpec extends FunSpec {
it("adds two numbers") {
assert(Calculator.addNumbers(3, 4) === 7)
}
}
-t can be used to run the single test:
testOnly *CalculatorSpec -- -t "adds two numbers"
-z can also be used to run the single test:
testOnly *CalculatorSpec -- -z "adds two numbers"
See this repo if you'd like to run these examples. You can find more info on running tests here.