I'm using CTest and want to pass command-line arguments to the underlying tests at runtime. I know there are ways to hard code command-line arguments into the CMake/CTest script, but I want to specify the command-line arguments at runtime and have those arguments passed through CTest to the underlying test.
Is this even possible?
I've figured out a way to do it (using the Fundamental theorem of software engineering). It's not as simple as I'd like, but here it is.
First, create a file ${CMAKE_SOURCE_DIR}/cmake/RunTests.cmake with the content
if(NOT DEFINED ENV{TESTS_ARGUMENTS})
set(ENV{TESTS_ARGUMENTS} "--default-arguments")
endif()
execute_process(COMMAND ${TEST_EXECUTABLE} $ENV{TESTS_ARGUMENTS} RESULT_VARIABLE result)
if(NOT "${result}" STREQUAL "0")
message(FATAL_ERROR "Test failed with return value '${result}'")
endif()
Then, when you add the test, use
add_test(
NAME MyTest
COMMAND ${CMAKE_COMMAND} -DTEST_EXECUTABLE=$<TARGET_FILE:MyTest> -P ${CMAKE_SOURCE_DIR}/cmake/RunTests.cmake
)
Finally, you can run the test with custom arguments using
cmake -E env TESTS_ARGUMENTS="--custom-arguments" ctest
Note that if you use bash, you can simplify this to
TESTS_ARGUMENTS="--custom-arguments" ctest
There are some problems with this approach, e.g. it ignores the WILL_FAIL property of the tests. Of course I wish it could be as simple as calling ctest -- --custom-arguments, but, as the Stones said, You can't always get what you want.
I'm not sure I fully understand what you want, but I still can give you a way to pass arguments to tests in CTest, at runtime.
I'll give you an example, with CTK (the Common Toolkit, https://github.com/commontk/CTK):
In the build dir (ex: CTK-build/CTK-build, it's a superbuild), if I run: ('-V' for Verbose, and '-N' for View Mode only)
ctest -R ctkVTKDataSetArrayComboBoxTest1 -V -N
I get:
UpdateCTestConfiguration from : /CTK-build/CTK-build/DartConfiguration.tcl
Parse Config file:/CTK-build/CTK-build/DartConfiguration.tcl
Add coverage exclude regular expressions.
Add coverage exclude: /CMakeFiles/CMakeTmp/
Add coverage exclude: .*/moc_.*
Add coverage exclude: .*/ui_.*
Add coverage exclude: .*/Testing/.*
Add coverage exclude: .*/CMakeExternals/.*
Add coverage exclude: ./ctkPixmapIconEngine.*
Add coverage exclude: ./ctkIconEngine.*
UpdateCTestConfiguration from :/CTK-build/CTK-build/DartConfiguration.tcl
Parse Config file:/CTK-build/CTK-build/DartConfiguration.tcl
Test project /CTK-build/CTK-build
Constructing a list of tests
Done constructing a list of tests
178: Test command: /CTK-build/CTK-build/bin/CTKVisualizationVTKWidgetsCppTests "ctkVTKDataSetArrayComboBoxTest1"
Labels: CTKVisualizationVTKWidgets
Test #178: ctkVTKDataSetArrayComboBoxTest1
Total Tests: 1
You can copy-paste the "Test command" in your terminal:
/CTK-build/CTK-build/bin/CTKVisualizationVTKWidgetsCppTests "ctkVTKDataSetArrayComboBoxTest1"
And add the arguments, for example "-I" for interactive testing:
/CTK-build/CTK-build/bin/CTKVisualizationVTKWidgetsCppTests "ctkVTKDataSetArrayComboBoxTest1" "-I"
Tell me if it helps.
matthieu's answer gave me the clue to get it to work for me.
For my code I did the following:
Type the command ctest -V -R TestMembraneCellCrypt -N to get the output:
...
488: Test command: path/to/ctest/executable/TestMembraneCellCrypt
Labels: Continuous_project_ChasteMembrane
Test #488: TestMembraneCellCrypt
...
Then I copied the Test command and provided the arguments there:
path/to/ctest/executable/TestMembraneCellCrypt -e 2 -em 5 -ct 10
I'll note that the package I'm using (Chaste), is pretty large so there might be things going on that I don't know about.
Related
My goal is to use bcftools to check that the reference alleles in my dataset (vcf file) match with a reference genome (fasta file) using the fixref plugin.
Working on command line, I first set the following environment:
export BCFTOOLS_PLUGINS=/path/to/bcftools/plugins
The following code is recommended for test datasets with mismatches:
bcftools +fixref test.bcf -Ob -o output.bcf -- -f ref.fa -m top
When I run this code using my own files (please note that my data is .vcf, not .bcf) I get the following error:
[main] Unrecognized command
If I simply enter:
bcftools
I get a list of the only 5 commands (view, index, cat, ld, ldpair) that I can use. So although I've set the environment, does it somehow need to be activated? Do I need to run my command through a bash script?
bcftools
was pointing to a deprecated version of bcftools (0.1.19) in ../bin/, while
BCFTOOLS_PLUGINS=/path/to/bcftools/plugins
was pointing to the plugins for bcftools version 1.10.2 outside /bin/
Replacing ../bin/bcftools (0.1.19 with 1.10.2) was the fix.
Does anyone know how to provide command-line argument for swift test ?
I have tried :
swift test "myDBName"
But I got unexpected argument error.
List of possible arguments are :
OVERVIEW: Build and run tests
USAGE: swift test [options]
OPTIONS:
--build-path Specify build/cache directory [default: ./.build]
--chdir, -C Change working directory before any other operation
--color Specify color mode (auto|always|never) [default: auto]
--configuration, -c Build with configuration (debug|release) [default: debug]
--enable-prefetching Enable prefetching in resolver
--list-tests, -l Lists test methods in specifier format
--parallel Run the tests in parallel.
--skip-build Skip building the test target
--specifier, -s Run a specific test class or method, Format: <test-module>.<test-case> or <test-module>.<test-case>/<test>
--verbose, -v Increase verbosity of informational output
-Xcc Pass flag through to all C compiler invocations
-Xlinker Pass flag through to all linker invocations
-Xswiftc Pass flag through to all Swift compiler invocations
--help Display available options
Is there any other way to pass args ? ( Environment variable, etc ? )
You can, indeed, use the environment to do it.
A test with the following contents:
final class HelloTests: XCTestCase {
func testExample() {
XCTAssertEqual(String(cString: getenv("SOMETHING")), "else")
}
static var allTests = [
("testExample", testExample),
]
}
Will succeed, using the swift command line:
SOMETHING=else swift test
(This is not explicitly mentioned in OP's question, but swift test refers to the test tool of the Swift Package Manager; likewise does the answer below)
Swift Package Manager Test Tool does not support passing or arguments
If we compare the --help for the Test (swift test) and the Run (swift run) tools, you may note that the USAGE grammar for the former does not allow passing arguments to the binary that the Test Tool wraps, something that is a feature for the latter:
$ swift test --help
OVERVIEW: Build and run tests
USAGE: swift test [options]
...
$ swift run --help
OVERVIEW: Build and run an executable product
USAGE: swift run [options] [executable [arguments ...]]
...
We may verify this if we visit the source code, where the ArgumentParser for the SwiftRunTool binds the command line arguments, whereas the ArgumentParser for SwiftTestTool does not.
If you believe this could be a useful feature, please have a look at the Support page of the Swift Package Manager at GitHub for instructions on how to propose new features / bug fixes.
Using conditional compilation flags
For completeness (in addition to making use of the environment), conditional compilation flags can be used, and possibly suffice, depending on the use case.
This is not providing actual values when launching the tests over the command line, but you may make use of conditional compilation flags (true/false) to the Swift compiler swiftc using the -D flag.
E.g., the following test would pass:
import XCTest
#if USE_LOCALHOST
let server_ip = "127.0.0.1"
#else
let server_ip = "1.2.3.4"
#endif
final class HelloTests: XCTestCase {
func testExample() {
XCTAssertEqual(server_ip, "127.0.0.1")
}
static var allTests = [
("testExample", testExample),
]
}
if the test suite is launched using:
$ swift test -Xswiftc -DUSE_LOCALHOST
Whereas the test would fail if launching using:
$ swift test
I'm trying to debug a python codebase that uses tox for unit tests. One of the failing tests is proving difficult due to figure out, and I'd like to use pudb to step through the code.
At first thought, one would think to just pip install pudb then in the unit test code add in import pudb and pudb.settrace(). But that results in a ModuleNotFoundError:
> import pudb
>E ModuleNotFoundError: No module named 'pudb'
>tests/mytest.py:130: ModuleNotFoundError
> ERROR: InvocationError for command '/Users/me/myproject/.tox/py3/bin/pytest tests' (exited with code 1)
Noticing the .tox project folder leads one to realize there's a site-packages folder within tox, which makes sense since the point of tox is to manage testing under different virtualenv scenarios. This also means there's a tox.ini configuration file, with a deps section that may look like this:
[tox]
envlist = lint, py3
[testenv]
deps =
pytest
commands = pytest tests
adding pudb to the deps list should solve the ModuleNotFoundError, but leads to another error:
self = <_pytest.capture.DontReadFromInput object at 0x103bd2b00>
def fileno(self):
> raise UnsupportedOperation("redirected stdin is pseudofile, "
"has no fileno()")
E io.UnsupportedOperation: redirected stdin is pseudofile, has no fileno()
.tox/py3/lib/python3.6/site-packages/_pytest/capture.py:583: UnsupportedOperation
So, I'm stuck at this point. Is it not possible to use pudb instead of pdb within Tox?
There's a package called pytest-pudb which overrides the pudb entry points within an automated test environment like tox to successfully jump into the debugger.
To use it, just make your tox.ini file have both the pudb and pytest-pudb entries in its testenv dependencies, similar to this:
[tox]
envlist = lint, py3
[testenv]
deps =
pytest
pudb
pytest-pudb
commands = pytest tests
Using the original PDB (not PUDB) could work too. At least it works on Django and Nose testers. Without changing tox.ini, simply add a pdb breakpoint wherever you need, with:
import pdb; pdb.set_trace()
Then, when it get to that breakpoint, you can use the regular PDB commands:
w - print stacktrace
s - step into
n - step over
c - continue
p - print an argument value
a - print arguments of current function
I am generating the client scala code for an API using the Swagger Edtior. I pasted the json then did a Generate Client/Scala. It gives me a default root package of
io.swagger.client
I can't see any obvious way of specifying something different. Can this be done?
Step (1): Create a file config.json and add following lines and define package names:
{
"modelPackage" : "com.xyz.model",
"apiPackage" : "com.xyz.api"
}
Step (2): Now, pass the above file name along with codegen command with -c option:
$ java -jar swagger-codegen-cli.jar generate -i path/swagger.json -l java -o Code -c path/config.json
Now, it will generate your java packages like com.xyz… instead of default one io.swagger.client…
Run the following command to get information about the supported configuration options
java -jar swagger-codegen-cli.jar config-help -l scala
This will give you information about supported by this generator (Scala in this example):
CONFIG OPTIONS
sortParamsByRequiredFlag
Sort method arguments to place required parameters before optional parameters. (Default: true)
ensureUniqueParams
Whether to ensure parameter names are unique in an operation (rename parameters that are not). (Default: true)
modelPackage
package for generated models
apiPackage
package for generated api classes
Next, define a config.json file with the above parameters:
{
"modelPackage": "your package name",
"apiPackage": "your package name"
}
And supply config.json as input to swagger-codegen using the -c flag.
I know that a single test can be ran by running, in sbt,
testOnly *class -- -n Tag
Is there a way of telling sbt/scalatest to run a single test without tags? For example:
testOnly *class -- -X 2
it would mean "run the second test in the class. Whatever it is". We have a bunch of tests and no one bothered to tag them, so is there a way to run a single test without it having a tag?
This is now supported (since ScalaTest 2.1.3) within interactive mode:
testOnly *MySuite -- -z foo
to run only the tests whose name includes the substring "foo".
For exact match rather than substring, use -t instead of -z.
If you run it from the command line, it should be as single argument to sbt:
sbt 'testOnly *MySuite -- -z foo'
I wanted to add a concrete example to accompany the other answers
You need to specify the name of the class that you want to test, so if you have the following project (this is a Play project):
You can test just the Login tests by running the following command from the SBT console:
test:testOnly *LoginServiceSpec
If you are running the command from outside the SBT console, you would do the following:
sbt "test:testOnly *LoginServiceSpec"
I don't see a way to run a single untagged test within a test class but I am providing my workflow since it seems to be useful for anyone who runs into this question.
From within a sbt session:
test:testOnly *YourTestClass
(The asterisk is a wildcard, you could specify the full path com.example.specs.YourTestClass.)
All tests within that test class will be executed. Presumably you're most concerned with failing tests, so correct any failing implementations and then run:
test:testQuick
... which will only execute tests that failed. (Repeating the most recently executed test:testOnly command will be the same as test:testQuick in this case, but if you break up your test methods into appropriate test classes you can use a wildcard to make test:testQuick a more efficient way to re-run failing tests.)
Note that the nomenclature for test in ScalaTest is a test class, not a specific test method, so all untagged methods are executed.
If you have too many test methods in a test class break them up into separate classes or tag them appropriately. (This could be a signal that the class under test is in violation of single responsibility principle and could use a refactoring.)
Just to simplify the example of Tyler.
test:-prefix is not needed.
So according to his example:
In the sbt-console:
testOnly *LoginServiceSpec
And in the terminal:
sbt "testOnly *LoginServiceSpec"
Here's the Scalatest page on using the runner and the extended discussion on the -t and -z options.
This post shows what commands work for a test file that uses FunSpec.
Here's the test file:
package com.github.mrpowers.scalatest.example
import org.scalatest.FunSpec
class CardiBSpec extends FunSpec {
describe("realName") {
it("returns her birth name") {
assert(CardiB.realName() === "Belcalis Almanzar")
}
}
describe("iLike") {
it("works with a single argument") {
assert(CardiB.iLike("dollars") === "I like dollars")
}
it("works with multiple arguments") {
assert(CardiB.iLike("dollars", "diamonds") === "I like dollars, diamonds")
}
it("throws an error if an integer argument is supplied") {
assertThrows[java.lang.IllegalArgumentException]{
CardiB.iLike()
}
}
it("does not compile with integer arguments") {
assertDoesNotCompile("""CardiB.iLike(1, 2, 3)""")
}
}
}
This command runs the four tests in the iLike describe block (from the SBT command line):
testOnly *CardiBSpec -- -z iLike
You can also use quotation marks, so this will also work:
testOnly *CardiBSpec -- -z "iLike"
This will run a single test:
testOnly *CardiBSpec -- -z "works with multiple arguments"
This will run the two tests that start with "works with":
testOnly *CardiBSpec -- -z "works with"
I can't get the -t option to run any tests in the CardiBSpec file. This command doesn't run any tests:
testOnly *CardiBSpec -- -t "works with multiple arguments"
Looks like the -t option works when tests aren't nested in describe blocks. Let's take a look at another test file:
class CalculatorSpec extends FunSpec {
it("adds two numbers") {
assert(Calculator.addNumbers(3, 4) === 7)
}
}
-t can be used to run the single test:
testOnly *CalculatorSpec -- -t "adds two numbers"
-z can also be used to run the single test:
testOnly *CalculatorSpec -- -z "adds two numbers"
See this repo if you'd like to run these examples. You can find more info on running tests here.