I'm trying to understand what is the correct way to use GivenWhenThen with ScalaCheck generators. If I put the GivenWhenThen statements inside the forAll loop, the test report will print out each GivenWhenThen statement as many times as there are trials of that test. However, if I put them outside (i.e. all of them are immediately before the forAll), then the code loses the advantage that GivenWhenThen provides, namely of having easily readable descriptions of expected behavior immediately next to where that behavior is tested.
Is there a better way to do this?
Related
I'm just starting out using ZIO in Scala. I've written some Scalacheck-style tests using ZIO's Gen type, and they appear to work, but I'd like to manually test the generators in the REPL to ensure that they're actually producing the data I expect them to.
The problem: everything in ZIO is wrapped in the ZIO monad, and I need to pass the right data into this monad to unwrap it and view the results. And there's no documentation explaining how to do this in the REPL.
I think I understand how to do it for a basic program with no environment dependencies: call zio.Runtime.default.unsafeRun. But the Gen objects expect an environment of type Random with Sized, and I don't know how to produce an instance of this.
Given a Gen[Random with Sized, T], what is the quickest way to execute it on the REPL and get a List[T] of generated values?
I think I've found a partial solution, but I'm not completely satisfied with it.
For just printing some samples from a Gen on the REPL, this works:
zio.Runtime.default.unsafeRun(
yourGenerator
.runCollectN(50)
.provideLayer(zio.random.Random.live +!+ zio.test.Sized.live(100))
) foreach println
But I don't think this is how it's supposed to be done. provideLayer doesn't typecheck unless I provide both Random and Sized, even though Random should be part of zio.Runtime.default. I think there's something about ZLayer that I still don't understand.
I am writing unit tests for a class(class A) that takes another class(class B) as its parameter which in turn takes another class(class C) as its parameter. I need to mock class B in order to test class A. But, mock doesn't take class parameters.
So, I created a trait(trait BsTrait) and my class B extends BsTrait. According to this answer - (ScalaMock. Mock a class that takes arguments).
Class I want to test - class A(b: BsTrait){}
class B - class B(c: C){}
class C - class C{}
trait of class B - trait BsTrait{}
My Unit Test -
val mockFactory = mockFunction[C, B]
val mockClient = mock[BsTrait]
mockFactory.expects(new C).returning(mockClient)```
Error: /path/to/file/Test.scala:63: type mismatch; found : com.example.BsTrait required: com.example.B
mockFactory.expects(new C).returning(mockClient)
I tried to add return type for mockFunction as well, but it threw the same error.
I know this is a not directly a solution, and maybe it actually is.
In my experience, ditch the mocking framework in 99% of the cases where you want it, and just write mock implementations of traits (or interfaces in other languages). Yes it will feel like boilerplate, but:
Often it turns out you will write less code - the mock code is short to begin with, but almost always starts having all sorts of work arounds to allow all sorts of calls, sometimes even calls your test does not care about, but has to be there. That you will end up having in a lot of the tests, just for the sake of the test
Once you in your framework mock a function/method with some testing behaviour people often don't discover when it isn't needed for the test any more, and forget to delete it. Auch dead code that nothing warns us is dead
Code you control is easier to extend or modify to your needs. Your own problem as a good example. Had your mock just been a class you wouldn't need to ask a question. Now your mock is hidden i all sorts of reflections, perhaps byte code manipulations (I don't know if ScalaMock does this, but a lot of frameworks do), and most likely other nasty things, that is impossible to debug.
Fameworked mocks are often hard and unwieldy to maintain. If you add a method to your trait you can end needing to update a lot of tests, even though it should not affect them. Just because those tests in some hidden way use them
You don't fall into the mocking framework pitfall with testing how you implemented your function/method rather than what it does. (E.g. is it important if you rest client specifically calls the method named get, or is it important that it performs a get-request, no matter if it calls exchange, get or something else?)
So even though it feels like overkill, heavily consider doing it anyway. You Will most likely write less code in the long run. The code you write is more clear and readable, if a function or method is no longer used you remove it from your trait and get a compiler warning on your mock. Your question would automatically have been solved, because you'd just extend your mock's behavior just enough to get it going. And your are more likely to actually write good tests.
Lastly, if you follow this suggestion, just one thing: keep your mocks independent. Do not let B take a parameter C in your mock. Make your B work with lists, maps, or whatever you need. The point is to test A, not B and C :)
I am going through the coursera functional programming and have an assignment where the scalatest is written using FunSuite and Checkers.
This test framework is new to me but I have some basic idea of using assertion, as I have developed pigunit for an user defined function using assert.
As google didn't give me clear usage of Checkers and how it is different from assert, could anyone clarify where Checkers can be used and why not assert be used.
Thanks
As you know, an assertion is a way of testing that a certain condition holds. These are pretty simple in ScalaTest, as you only need to use assert. For example:
assert(List(1, 2, 3).length == 3)
"Checkers," or, as they are more often called, properties, are a bit different. They are a way to assert that a condition holds for all possible inputs instead of for a single case. For example, here is a property that tests that a list always has a nonnegative length:
check((ls: List[Int]) => ls.length >= 0)
At this point, ScalaTest defers to ScalaCheck to do the heavy lifting. ScalaCheck generates random values for ls in an effort to find one that fails the test. This concept is called property-based testing. You can read more about how to use it in ScalaTest here.
Im working with scalatest and scalacheck, alsso working with FeatureSpec.
I have a generator class that generate object for me that looks something like this:
object InvoiceGen {
def myObj = for {
country <- Gen.oneOf(Seq("France", "Germany", "United Kingdom", "Austria"))
type <- Gen.oneOf(Seq("Communication", "Restaurants", "Parking"))
amount <- Gen.choose(100, 4999)
number <- Gen.choose(1, 10000)
valid <- Arbitrary.arbitrary[Boolean]
} yield SomeObject(country, type, "1/1/2014", amount,number.toString, 35, "something", documentTypeValid, valid, "")
Now, I have the testing class which works with FeatureSpec and everything that I need to run the tests.
In this class I have scenarios, and in each scenario I want to generate a different object.
The thing is from what I understand is that to generate object is better to use forAll func, but for all will not sure to bring you an object so you can add minSuccessful(1) to make sure you get at list 1 obj....
I did it like this and it works:
scenario("some scenario") {
forAll(MyGen.myObj, minSuccessful(1)) { someObject =>
Given("A connection to the system")
loginActions shouldBe 'Connected
When("something")
//blabla
Then("something should happened")
//blabla
}
}
but im not sure exactly what it means.
What I want is to generate an invoice each scenario and do some actions on it...
im not sure why i care if the generation work or didnt work...i just want a generated object to work with.
TL;DR: To get one object, and only one, use myObj.sample.get. Unless your generator is doing something fancy that's perfectly safe and won't blow up.
I presume that your intention is to run some kind of integration/acceptance test with some randomly generated domain object—in other words (ab-)use scalacheck as a simple data generator—and you hope that minSuccessful(1) would ensure that the test only runs once.
Be aware that this is not the case!. scalacheck will run your test multiple times if it fails, to try and shrink the input data to a minimal counterexample.
If you'd like to ensure that your test runs only once you must use sample.
However, if running the test multiple times is fine, prefer minSuccessful(1) to "succeed fast" but still profit from minimized counterexamples in case the test fails.
Gen.sample returns an option because generators can fail:
ScalaCheck generators can fail, for instance if you're adding a filter (listingGen.suchThat(...)), and that failure is modeled with the Option type.
But:
[…] if you're sure that your generator never will fail, you can simply call Option.get like you do in your example above. Or you can use Option.getOrElse to replace None with a default value.
Generally if your generator is simple, i.e. does not use generators that could fail and does not use any filters on its own, it's perfectly safe to just call .get on the option returned by .sample. I've been doing that in the past and never had problems with it. If your generators frequently return None from .sample they'd likely make scalacheck fail to successfully generate values as well.
If all that you want is a single object use Gen.sample.get.
minSuccessful has a very different meaning: It's the minimal number of successful tests that scalacheck runs—which by no means implies
that scalacheck takes only a single value out of the generator, or
that the test runs only once.
With minSuccessful(1) scalacheck wants one successful test. It'll take samples out of the generator until the test runs at least once—i.e. if you filter the generated values with whenever in your test body scalacheck will take samples as long as whenever discards them.
If the test passes scalacheck is happy and won't run the test a second time.
However if the test fails scalacheck will try and produce a minimal example to fail the test. It'll shrink the input data and run the test as long as it fails and then provides you with the minimized counter example rather than the actual input that triggered the initial failure.
That's an important property of property testing as it helps you to discover bugs: The original data is frequently too large to lend itself for debugging. Minimizing it helps you discover the piece of input data that actually triggers the failure, i.e. corner cases like empty strings that you didn't think of.
I think the way you want to use Scalacheck (generate only one object and execute the test for it) defeats the purpose of property-based testing. Let me explain a bit in detail:
In classical unit-testing, you would generate your system under test, be it an object or a system of dependent objects, with some fixed data. This could e.g. be strings like "foo" and "bar" or, if you needed a name, you would use something like "John Doe". For integers and other data, you can also randomly choose some values.
The main advantage is that these are "plain" values—you can directly see them in the code and correlate them with the output of a failed test. The big disadvantage is that the tests will only ever run with the values you specified, which in turn means that your code is also only tested with these values.
In contrast, property-based testing allows you to just describe how the data should look like (e.g. "a positive integer", "a string of maximum 20 characters"). The testing framework will then—with the help of generators—generate a number of matching objects and execute the test for all of them. This way, you can be more sure that your code will actually be correct for different inputs, which after all is the purpose of testing: to check if your code does what it should for the possible inputs.
I never really worked with Scalacheck, but a colleague explained it to me that it also tries to cover edge-cases, e.g. putting in a 0 and MAX_INT for a positive integer, or an empty string for the aforementioned string with max. 20 characters.
So, to sum it up: Running a property-based test only once for one generic object is the wrong thing to do. Instead, once you have the generator infrastructure in place, embrace the advantage you then have and let your code be checked a lot more times!
What is the proposed way of performing tests on scala macros?
I realize that one needs two projects due to the necessity of separate compilation. This step, if necessary, is acceptable and mostly clear.
But how do you assert a macro expansion fails when it should? Without some special facility, the test case won't compile and therefore the whole test project won't compile.
I think this assert would require another macro of the form
errors(code: => _): List[CompileError]
which returns the compile errors of the inner macro. The same would be required for testing that warnings occur if they should and so on...
Are there some existing testing facilities for Scala macros?
You can use assertDoesNotCompile from ScalaTest. Here is an example of using this to test the Scala typechecker:
import org.scalatest.FunSuite
class TypeTest extends FunSuite {
test("String vals can't hold ints") {
assertDoesNotCompile("val x: String = 1")
}
}
You can pass a string containing an example of when your macro expansion should fail to assertDoesNotCompile. Note that that there is also assertCompiles, if you feel the need for that.
you could use twitter eval to check if the code copiles
https://github.com/twitter/util/blob/master/util-eval/src/main/scala/com/twitter/util/Eval.scala#L247