If I write a parameterized NUnit test, using something like [TestCaseSource] or [ValueSource], NUnit will pass the parameters directly to my test method. But is there any other way to access those parameters, e.g. from SetUp, or from a helper method (without having to explicitly pass the parameter value to that helper method)?
For example, suppose I have three different scenarios (maybe it's "rising rates", "falling rates", and "constant rates"). I'm writing tests for a particular calculation, and some tests will have the same behavior in all three scenarios; others in two of the three (and I'll write a second test for the other scenario); others will have a separate test for each scenario. Parameterized tests seem like a good way to model this; I can write a strategy object for each scenario, and parameterize the tests based on which scenarios each test should apply to.
I can do something like this:
public IEnumerable<RateStrategy> AllScenarios {
get {
yield return new RisingRatesStrategy();
yield return new FallingRatesStrategy();
yield return new ConstantRatesStrategy();
}
}
[TestCaseSource("AllScenarios")]
public void SomethingThatIsTheSameInAllScenarios(RateStrategy scenario) {
InitializeScenario(scenario);
... arrange ...
... act ...
... assert ...
}
The downside to this is that I need to remember to call InitializeScenario in every test. This is easy to mess up, and it also makes the tests harder to read -- in addition to the attribute that says exactly which scenarios this test applies to, I also need an extra line of code cluttering up my test, saying that oh yeah, there are scenarios.
Is there some other way I could access the test parameters? Is there a static property, similar to those on TestContext, that would let me access the test's parameters from, say, my SetUp method, so I could make my tests more declarative (convention-based) and less repetitive?
(TestContext looked promising, but it only tells me the test's name and whether it passed or failed. The test's parameters are sort of there, but only as part of a display string, not as actual objects; I can't grab the strategy object and start calling methods on it.)
Related
I have some vanilla Scala code that looks like this:
addCooker(new Cooker(getOvenState(), cookingTime, CookieNames.GingerSnaps) {
override def cook(customer: Customer, priority: Priority): Boolean = {
// Use `customer` and `priority` to cook a cookie and return true if successful.
???
}
})
I.e. I create a callback-like Cooker object that's passed to an addCooker method. CookieCutter takes some values (cookingTime etc.) that are available when calling addCooker (these are passed to its ctor) and it takes some values (customer etc.) that will only be available at some later point in time (these will be passed as arguments to its cook method).
I'd like to create a DSL where I can write this as:
addCooker(getOvenState(), cookingTime, CookieNames.GingerSnaps) {
// Somehow make a `customer` and `priority` value (that are not available at
// the time `addCooker` happens) visible to the code within this block.
}
I could declare addCooker as a method like so:
def addCooker(overState: OvenState, cookingTime: Duration, name: CookieName)(
block: () => Boolean
): Unit = ???
But I don't see a way to make cookingTime etc. available such that they can be used within the lambda passed as block.
The best I can do results in something like this:
addCooker(getOvenState(), cookingTime, CookieNames.GingerSnaps) { (customer, priority) =>
true
}
Normally, this would be good enough for me but in this situation, hundreds of such blocks will be written (and there'll be lots of different but similar constructs) and a DSL where many of the values are just there rather than one needing to always declare them as parameters would be ideal.
I guess one way is to make customer etc. protected var variables of the class where the addCooker calls are being made, i.e. they'd be visible not just to my { ... } block but also to the logic that calls addCooker (but without yet being set to anything meaningful).
PS are there any good guides to the kinds of non-obvious tricks that you need to use to create DSLs? I found lots of guides that didn't go very deep (focusing on little more than using implicits to do type conversions or do fun things with operators). The only substantial thing I found was DSLs in Action but it was written in 2010 and uses Scala 2.8 - I imagine the thinking on many things related to implicits and the like has changed noticeably since then.
If the above snippets are unclear, you can them (with supporting stubs such that things will compile) here:
https://gist.github.com/george-hawkins/a9db64f05e14ea7d191bc4cf85dd64f6
I have a parameterised test that unit tests a certain logic. There are several test cases captured by the NUnit TestCaseAttribute.
Now I wish to utilize exactly the same parameters to test a slightly different logic.
I realize that I can deliver the parameters through a different attribute - TestCaseSourceAttribute and use the same source for multiple unit tests.
But I wonder if one can both use TestCaseAttribute (which I find more convenient in this particular test) and reuse the parameters for another test?
My solution involves reflection:
[TestCase(Impl.SqlErrorCode.PartiallyDocumentedColumn, 1978.14, "MyTable", ChangeTypeCode.AddTable, "dbo.MyAuxTable:MyTableId")]
[TestCase(Impl.SqlErrorCode.UndocumentedColumn, 1978.15, "MyAuxTable", ChangeTypeCode.AddTable, "dbo.MyAuxTable:MyAuxTableId")]
[TestCase(Impl.SqlErrorCode.UndocumentedColumn, 1978.16, "MyTable", ChangeTypeCode.AddTable, "dbo.MyTable:MyAuxTableId")]
[TestCase(Impl.SqlErrorCode.NonExistingColumnInComments, 1969.19, "MyTable", ChangeTypeCode.None, "dbo.MyTable:Remarks")]
public async Task AddTableWithBadComments(Impl.SqlErrorCode expectedSqlErrorCode, decimal step, string tableName, int sqlErrorState, string expectedObjectName)
{
// ...
}
private static IEnumerable GetParametersOfAnotherTest(string testName)
{
var testCaseAttrs = typeof(IntegrationTests).GetMethod(testName).GetCustomAttributes<TestCaseAttribute>();
return testCaseAttrs.Select(a => a.Arguments);
}
[TestCaseSource(nameof(GetParametersOfAnotherTest), new object[] { nameof(AddTableWithBadComments) })]
public async Task AddTableWithBadCommentsNoVerify(Impl.SqlErrorCode expectedSqlErrorCode, double _step, string tableName, int sqlErrorState, string expectedObjectName)
{
// A different logic, but with the same parameters.
}
It has some problems though.
So, my question is this - is there an NUnit way to run a test method Y with the parameters of the test method X, where the latter uses TestCaseAttribute to provide the parameters?
I use nunit 3.7.1
The actual answer is quite short. The NUnit way to reuse parameters is TestCaseSourceAttribute. :-)
I thought I would explain why your solution doesn't work.
In NUnit 3+, attributes like TestCase and TestCaseSource are not just containers of data. They implement interfaces, which NUnit calls in order to have the attributes operate on a particular test.
Your code is treating TestCaseAttribute as if it were no more than a data store for arguments. But the attribute actually does some things and some of them are different from what TestCaseSourceAttribute does.
From your code, I can see you figured part of that out yourself. Your first method relies on the attribute converting double to decimal, while your second takes the argument as a double. That difference is of course due to the fact that you can't have a decimal argument to an attribute.
Unfortunately, for a full solution, you would have to duplicate or make allowances for other differences between the two attributes, which are all due to the restrictions C# places on attribute arguments. IMO, it's not worth it. It's trivial to create a static array of TestCaseData items and use them for both methods. If you make your approach work (which is possible) it's only advantage will be in its cleverness. :-)
I'm new to BDD, even to whole testing world.
I'm trying to take BDD practices when writing a simple linear algebra library in swift. So there would be many value object types like Matrix, Vector etc. When writing code, I suppose I still need to stick to the TDD principle (am I right?):
Not writing any single line of code without a failing test
To implement a value object type, I need to make it conform to Equatable protocol and implement its == operator. This is adding code, so I need a failing test. How to write spec for this kinda scenarios ?
One may suggest some approach like:
describe("Matrix") {
it("should be value object") {
let aMatrix = Matrix<Double>(rows: 3, cols:2)
let sameMatrix = Matrix<Double>(rows: 3, cols:2)
expect(sameMatrix) == aMatrix
let differentMatrix = Matrix<Double>(rows: 4, cols: 2)
expect(differentMatrix) != aMatrix
}
}
This would be an ugly boilerplate for two reasons:
There may be plenty of value object types and I need to repeat it for all of them
There may be plenty of cases that would cause two objects being not equal. Taking the spec above for example, an implementation of == like return lhs.rows == rhs.rows would pass the test. In order to reveal this "bug", I need to add another expectation like expect(matrixWithDifferentColmunCount) != aMatrix. And again, this kinda repetition happens for all value object types.
So, how should I test this "isEqual" ( or operator== ) method elegantly ? or shouldn't I test it at all ?
I'm using swift and Quick for testing framework. Quick provides a mechanism called SharedExample to reduce boilerplates. But since swift is a static typing language and Quick's shared example doesn't support generics, I can't directly use a shared example to test value objects.
I came up with a workaround but don't consider it as an elegant one.
Elegance is the enemy of test suites. Test suites are boring. Test suites are repetitive. Test suites are not "DRY." The more clever you make your test suites, the more you try to avoid boilerplate, the more you are testing your test infrastructure instead of your code.
The rule of BDD is to write the test before you write the code. It's a small amount of test code because it's testing a small amount of live code; you just write it.
Yes, that can be taken too far, and I'm not saying you never use a helper function. But when you find yourself refactoring your test suite, you need to ask yourself what the test suite was for.
As a side note, your test doesn't test what it says it does. You're testing that identical constructors create Equal objects and non-identical constructors create non-Equal objects (which in principle is two tests). This doesn't test at all that it's a value object (though it's a perfectly fine thing to test). This isn't a huge deal. Don't let testing philosophy get in the way of useful testing; but it's good to have your titles match your test intent.
Im working with scalatest and scalacheck, alsso working with FeatureSpec.
I have a generator class that generate object for me that looks something like this:
object InvoiceGen {
def myObj = for {
country <- Gen.oneOf(Seq("France", "Germany", "United Kingdom", "Austria"))
type <- Gen.oneOf(Seq("Communication", "Restaurants", "Parking"))
amount <- Gen.choose(100, 4999)
number <- Gen.choose(1, 10000)
valid <- Arbitrary.arbitrary[Boolean]
} yield SomeObject(country, type, "1/1/2014", amount,number.toString, 35, "something", documentTypeValid, valid, "")
Now, I have the testing class which works with FeatureSpec and everything that I need to run the tests.
In this class I have scenarios, and in each scenario I want to generate a different object.
The thing is from what I understand is that to generate object is better to use forAll func, but for all will not sure to bring you an object so you can add minSuccessful(1) to make sure you get at list 1 obj....
I did it like this and it works:
scenario("some scenario") {
forAll(MyGen.myObj, minSuccessful(1)) { someObject =>
Given("A connection to the system")
loginActions shouldBe 'Connected
When("something")
//blabla
Then("something should happened")
//blabla
}
}
but im not sure exactly what it means.
What I want is to generate an invoice each scenario and do some actions on it...
im not sure why i care if the generation work or didnt work...i just want a generated object to work with.
TL;DR: To get one object, and only one, use myObj.sample.get. Unless your generator is doing something fancy that's perfectly safe and won't blow up.
I presume that your intention is to run some kind of integration/acceptance test with some randomly generated domain object—in other words (ab-)use scalacheck as a simple data generator—and you hope that minSuccessful(1) would ensure that the test only runs once.
Be aware that this is not the case!. scalacheck will run your test multiple times if it fails, to try and shrink the input data to a minimal counterexample.
If you'd like to ensure that your test runs only once you must use sample.
However, if running the test multiple times is fine, prefer minSuccessful(1) to "succeed fast" but still profit from minimized counterexamples in case the test fails.
Gen.sample returns an option because generators can fail:
ScalaCheck generators can fail, for instance if you're adding a filter (listingGen.suchThat(...)), and that failure is modeled with the Option type.
But:
[…] if you're sure that your generator never will fail, you can simply call Option.get like you do in your example above. Or you can use Option.getOrElse to replace None with a default value.
Generally if your generator is simple, i.e. does not use generators that could fail and does not use any filters on its own, it's perfectly safe to just call .get on the option returned by .sample. I've been doing that in the past and never had problems with it. If your generators frequently return None from .sample they'd likely make scalacheck fail to successfully generate values as well.
If all that you want is a single object use Gen.sample.get.
minSuccessful has a very different meaning: It's the minimal number of successful tests that scalacheck runs—which by no means implies
that scalacheck takes only a single value out of the generator, or
that the test runs only once.
With minSuccessful(1) scalacheck wants one successful test. It'll take samples out of the generator until the test runs at least once—i.e. if you filter the generated values with whenever in your test body scalacheck will take samples as long as whenever discards them.
If the test passes scalacheck is happy and won't run the test a second time.
However if the test fails scalacheck will try and produce a minimal example to fail the test. It'll shrink the input data and run the test as long as it fails and then provides you with the minimized counter example rather than the actual input that triggered the initial failure.
That's an important property of property testing as it helps you to discover bugs: The original data is frequently too large to lend itself for debugging. Minimizing it helps you discover the piece of input data that actually triggers the failure, i.e. corner cases like empty strings that you didn't think of.
I think the way you want to use Scalacheck (generate only one object and execute the test for it) defeats the purpose of property-based testing. Let me explain a bit in detail:
In classical unit-testing, you would generate your system under test, be it an object or a system of dependent objects, with some fixed data. This could e.g. be strings like "foo" and "bar" or, if you needed a name, you would use something like "John Doe". For integers and other data, you can also randomly choose some values.
The main advantage is that these are "plain" values—you can directly see them in the code and correlate them with the output of a failed test. The big disadvantage is that the tests will only ever run with the values you specified, which in turn means that your code is also only tested with these values.
In contrast, property-based testing allows you to just describe how the data should look like (e.g. "a positive integer", "a string of maximum 20 characters"). The testing framework will then—with the help of generators—generate a number of matching objects and execute the test for all of them. This way, you can be more sure that your code will actually be correct for different inputs, which after all is the purpose of testing: to check if your code does what it should for the possible inputs.
I never really worked with Scalacheck, but a colleague explained it to me that it also tries to cover edge-cases, e.g. putting in a 0 and MAX_INT for a positive integer, or an empty string for the aforementioned string with max. 20 characters.
So, to sum it up: Running a property-based test only once for one generic object is the wrong thing to do. Instead, once you have the generator infrastructure in place, embrace the advantage you then have and let your code be checked a lot more times!
I'm trying to test-drive some Scala code using Specs2 and Mockito. I'm relatively new to all three, and having difficulty with the mocked methods returning null.
In the following (transcribed with some name changes)
"My Component's process(File)" should {
"pass file to Parser" in new modules {
val file = mock[File]
myComponent.process(file)
there was one(mockParser).parse(file)
}
"pass parse result to Translator" in new modules {
val file = mock[File]
val myType1 = mock[MyType1]
mockParser.parse(file) returns (Some(myType1))
myComponent.process(file)
there was one(mockTranslator).translate(myType1)
}
}
The "pass file to Parser" works until I add the translator call in the SUT, and then dies because the mockParser.parse method has returned a null, which the translator code can't take.
Similarly, the "pass parse result to Translator" passes until I try to use the translation result in the SUT.
The real code for both of these methods can never return null, but I don't know how to tell Mockito to make the expectations return usable results.
I can of course work around this by putting null checks in the SUT, but I'd rather not, as I'm making sure to never return nulls and instead using Option, None and Some.
Pointers to a good Scala/Specs2/Mockito tutorial would be wonderful, as would a simple example of how to change a line like
there was one(mockParser).parse(file)
to make it return something that allows continued execution in the SUT when it doesn't deal with nulls.
Flailing about trying to figure this out, I have tried changing that line to
there was one(mockParser).parse(file) returns myResult
with a value for myResult that is of the type I want returned. That gave me a compile error as it expects to find a MatchResult there rather than my return type.
If it matters, I'm using Scala 2.9.0.
If you don't have seen it, you can look the mock expectation page of the specs2 documentation.
In your code, the stub should be mockParser.parse(file) returns myResult
Edited after Don's edit:
There was a misunderstanding. The way you do it in your second example is the good one and you should do exactly the same in the first test:
val file = mock[File]
val myType1 = mock[MyType1]
mockParser.parse(file) returns (Some(myType1))
myComponent.process(file)
there was one(mockParser).parse(file)
The idea of unit testing with mock is always the same: explain how your mocks work (stubbing), execute, verify.
That should answer the question, now a personal advice:
Most of the time, except if you want to verify some algorithmic behavior (stop on first success, process a list in reverse order) you should not test expectation in your unit tests.
In your example, the process method should "translate things", thus your unit tests should focus on it: mock your parsers and translators, stub them and only check the result of the whole process. It's less fine grain but the goal of a unit test is not to check every step of a method. If you want to change the implementation, you should not have to modify a bunch of unit tests that verify each line of the method.
I have managed to solve this, though there may be a better solution, so I'm going to post my own answer, but not accept it immediately.
What I needed to do was supply a sensible default return value for the mock, in the form of an org.mockito.stubbing.Answer<T> with T being the return type.
I was able to do this with the following mock setup:
val defaultParseResult = new Answer[Option[MyType1]] {
def answer(p1: InvocationOnMock): Option[MyType1] = None
}
val mockParser = org.mockito.Mockito.mock(implicitly[ClassManifest[Parser]].erasure,
defaultParseResult).asInstanceOf[Parser]
after a bit of browsing of the source for the org.specs2.mock.Mockito trait and things it calls.
And now, instead of returning null, the parse returns None when not stubbed (including when it's expected as in the first test), which allows the test to pass with this value being used in the code under test.
I will likely make a test support method hiding the mess in the mockParser assignment, and letting me do the same for various return types, as I'm going to need the same capability with several return types just in this set of tests.
I couldn't locate support for a shorter way of doing this in org.specs2.mock.Mockito, but perhaps this will inspire Eric to add such. Nice to have the author in the conversation...
Edit
On further perusal of source, it occurred to me that I should be able to just call the method
def mock[T, A](implicit m: ClassManifest[T], a: org.mockito.stubbing.Answer[A]): T = org.mockito.Mockito.mock(implicitly[ClassManifest[T]].erasure, a).asInstanceOf[T]
defined in org.specs2.mock.MockitoMocker, which was in fact the inspiration for my solution above. But I can't figure out the call. mock is rather overloaded, and all my attempts seem to end up invoking a different version and not liking my parameters.
So it looks like Eric has already included support for this, but I don't understand how to get to it.
Update
I have defined a trait containing the following:
def mock[T, A](implicit m: ClassManifest[T], default: A): T = {
org.mockito.Mockito.mock(
implicitly[ClassManifest[T]].erasure,
new Answer[A] {
def answer(p1: InvocationOnMock): A = default
}).asInstanceOf[T]
}
and now by using that trait I can setup my mock as
implicit val defaultParseResult = None
val mockParser = mock[Parser,Option[MyType1]]
I don't after all need more usages of this in this particular test, as supplying a usable value for this makes all my tests work without null checks in the code under test. But it might be needed in other tests.
I'd still be interested in how to handle this issue without adding this trait.
Without the full it's difficult to say but can you please check that the method you're trying to mock is not a final method? Because in that case Mockito won't be able to mock it and will return null.
Another piece of advice, when something doesn't work, is to rewrite the code with Mockito in a standard JUnit test. Then, if it fails, your question might be best answered by someone on the Mockito mailing list.