I want to take a BDD approach to unit testing in an iOS project, and I just realized that there may not be an existing library that provides test doubles of the test spy variety. Ideally, I'm looking for something similar to Mockito, Jasmine, or RR.
Before I go and spend a week of free time writing a test spy library, I thought I'd pose the question here on SO first.
So far I've looked at OCMock and Kiwi, but they both seem to be of the traditional high-specification-by-default mocking frameworks that require expectation assertions be set in the arrange phase, prior to the act phase. Obviously, this is hampering my vision of beautiful, DRY, nested specs (which I plan on authoring in either Kiwi or Cedar).
Just saw this.
Kiwi definitely does not do this now. You are right that the mocks in it are built for a 'standard' arrange prior to act phase.
Moving on, albeit at first glance, it seems that adding the basics for spy functionality would not require too much reengineering. Every message (barring some implementation important, reserved selectors) that gets to a mock goes through -[KWMock forwardInvocation:].
Essentially, the current -[KWMock forwardInvocation:] would need be modified to record/copy all invocations that pass through it, instead of what it does now. This would be the primitive functionality that would allow expectations to be verified later by querying the recorded invocations. Of course, coming up with a nice readable form for verification isn't trivial either.
The spy/mock would still need to know what class/protocol it is standing in for upfront. This is so it will be able to generate valid method signatures for selectors of messages sent to it that allows the runtime forwarding machinery to generate the actual NSInvocation that will be forwarded.
I am preoccupied with other things right now to get an implementation in there, but I'll be happy to answer more questions or merge any pull requests. HTH.
Related
I have not used libbpf in a while. Now, when I'm looking at the source code and examples, it looks to me that all API now is built around bpf_object while before it was based on program FD (at least on the user-facing level). I believe that fd is now hidden in bpf_object or such.
Of course it keeps backward compatibility and I still can use bpf_prog_load for example, however it looks like the preferred way of writing application code using libbpf is by bpf_object API?
Correct me if I'm wrong. Thanks!
Sounds mostly correct to me.
Low-Level Wrappers
If I remember correctly, the functions returning file descriptors in libbpf, mostly defined in tools/lib/bpf/bpf.c, have always been very low-level. This is the case for bpf_load_program() for example, which is no more than a wrapper around the bpf() system call for loading programs. Such functions are still available, but their use may be tedious for complex use cases.
bpf_prog_load()
Some more advanced functions have long been provided. bpf_prog_load(), that you mention, is one of them, but it returns an error code, not a file descriptor. It is still available as one option to load programs with the library.
bpf_object__*()
Although I don't think there are strict guidelines, I believe it is true that most example now use the bpf_object__*() function. One reason is that they provide a more consistent user experience, being organised around the manipulation of an object file to extract all the relevant bytecode and metadata, and then to load and attach the program. One other reason, I think, is that since this model has been favoured over the last releases, these functions have better support for recent eBPF features and the bpf_object__*() functions offer features that the older bpf_prog_load() workflow does not support.
Libbpf Evolves
At last, it's worth mentioning that libbpf's API is currently undergoing some review and will likely be reworked as part of a major v1.0 release. You may want to have a look at the work document linked in the announcement: Some bpf_object__ functions may be deprecated, and similarly there is currently a proposal to:
Deprecate bpf_prog_load() and bpf_prog_load_xattr() in favor of bpf_object__open_{mem, file}() and bpf_object__load() combo.
There is nothing certain yet regarding the v1.0 release, so I wouldn't worry too much about “deprecation” at the moment - I don't expect all functions to be removed just yet. But that's something you may want to consider when building your next applications.
This is clearly a practical question. Internet wisdom suggests logically organizing tests for maintainability. When it comes to ReST API Controllers one could include all integration-tests for all actions of the controller in a single file.
Assuming we have 4 CRUD actions per controller with an average of 6 tests per action we are bound to end up with at least 24 tests in one file. In industrial-grade web-servers I suspect that this number would baloon way further upwards.
The thing is that these actions even though they are part of one class (controller) they are complex and orthocanonical (different resources/artifacts/mockups needed to test each group etc).
I'm having a hard time coming to terms that all these tests should be placed in one file given the fact that they are testing almost entirely different things and can be placed in 4 separate files. Isn't this more aligned with the spirit of TDD afterall?
Is my intuition misplaced?
Nothing is stopping you from organizing your tests in any way that makes sense for your project.
Internet wisdom has nothing to do with this, do what works for you. You want one test file per crud method? Then do that.
Where your tests live is far less important than what you actually test and how.
Just one note of caution, I have seen people spend a long time and test completely the wrong things.
Let's instantiate controllers to call methods from them ( don't, this is a sign of really bad SOC ), let's mock who knows what.
You will achieve a much better result if you unit test what should be unit tested and that is functional stuff, classes and methods that do stuff. For an API this will mean business rules, data transformations, model transformations, that kind of stuff.
For the rest, I'd stick with integrations tests, call your endpoints like a normal user would and this means mostly integration testing. Use something like Postman to organize collections of tests.
You'll have a lot less to mock, your tests won't be affected much if you change your implementation details, you'll be able to do measurements on your endpoints and you'll actually test the real thing, going all the way to your storage and back which no amount of mocking will give you.
I'm a big believer in testing, but not a very good practitioner. I've done pretty well at getting coverage on my model objects and programming them in a TDD style. I'm actually enjoying it so much I'd love to extend this to my controller layer, particularly my UIViewController subclasses.
Unfortunately, many UIKit classes don't function in independent tests. However, I'm unhappy with the restriction of only running my dependent tests on the device. It's really important to me to run all unit tests before every build, and it seems to me like its possible and worthwhile to unit test (as opposed to other types of testing) controller code.
My question is simply this: How do I test UIViewControllers in such a way that the tests run before every build? I am aware of a couple of different solutions to this problem, but don't know a whole lot about the various benefits of each one.
In general if you are stuck with an environmental problem where unconditional testing seems to be made impossible, you can work around it if the benefits are great enough to warrant your jumping through some hoops.
The situation here is a special case of unwanted outside dependencies, which I generally solve in my unit tests by either #ifdefing out tiny bits of dependent code, or by implementing stub classes to fill the expected dependency roles enough that my code can be tested.
So in this specific case, you might create a new source file, linked only in the test bundle, called "UIKitStubClasses.m" … within it you could implement the bare necessities to simulate UIKit dependent classes such as UIViewController, so that your tests link and thoroughly exercise their own logic.
The important thing to remember is this is usually not necessarily all that much work. The tests will let you know what you need to implement in the stub, for example by issuing exceptions about unimplemented methods. You just add what you need to quiet the errors and test your code, and then your stub class is as sufficient for testing as any of the legitimate system framework classes would be.
Can you not use the tips/methods in Chris Hanson's blog here: http://chanson.livejournal.com/120263.html
It seems that you would probably need to run your app as the test harness, but it seems doable. Just because you are running the 'app' doesn't mean it necessarily has to do the normal thing your application does. Alternatively, you could create another application target that is just a simple dummy application that will load your test bundles and run them.
I agree it might not be perfect, but it sounds like it might work, and it seems like it could be automated.
So I was unhappy with this situation as it stood, and both Daniel and Ed's answers spurred me on to improve things further. I decided to take matters into my own hands.
What I ended up writing was a small Cocoa Touch application that in -applicationDidFinishLaunching scans the class hierarchy for any SenTestCase subclasses and runs them. Unlike GoogleToolboxForMac's unit testing stuff, I tried to use as much of the built in SenTesting machinery as possible and as a result it didn't really end up as that much code. I haven't done too much testing beyond verifying that UILabel allocates without crashing the test rig (otest does in fact crash if you try), but it should work.
I've put the source online on both Bitbucket and Github.
While apple's tools have improved over time, they still leave much to be desired. GHUnit is much better, has a real test runner, and makes is dead simple to set breakpoints, etc. You can even leverage all your existing SenTests.
Get it on github.
You can instantiate all your UIKit code with no worries. I use it to test UIViewControllers, assert outlets, and do behavior testing.
I have this grand idea to basically employ some brute force attack to test/verify that my web application doesn't crash.
Don't get me started on unit testing, and IoC stuff, this is something else entirely.
What I'm doing, and what I'm asking for help with is to create an intelligent exhaustive search, that explore parts of the program state.
What I have is a web page with things I can do, clicking is one thing, text input is another, some inputs like radio buttons and drop down lists are constrained to certain values. Pretty basic things. What I end up with a finite set of events and values and what I want to model is a progression of state. Maybe this is FSM optimization in a way, but the goal is to systematically go through arbitrary permutations of events and values and see what happens.
When a problem is found I want to try and provoke that error with as little effort as possible to be able to present a clear test case.
This relates to formal verification methods and I'm asking for help or insight from people with experience.
What you want to do sounds a little like model-checking, on the one hand, and automated test case generation on the other hand (in the latter category check out Concolic testing, a technique to avoid wasting time with unfeasible execution paths).
Model-checking would be the preferred method if you assume your web application is correct and want to prove that it is. But in the case of a warning, you may have to work to understand if the problem is real or not. Test case generation is oriented towards bug-finding: it does not prove that you app is correct, but if it finds a problem, it gives you an input vector to produce it so you don't need to wonder if the problem is real.
I am not aware of any existing tools for web apps, but that doesn't mean that they don't exist.
It sounds like you want a fuzzer. Peach is one such tool.
Exhaustive search can be non trivial task for limited resource (memory,space) ,but with many techniques the problem can be reduced ,like abstracting you code (ex: replacing database driver classes with stubs), an experience is presented in this paper: Abstract Model Checking of Web Applications Using Java PathFinder (Vinh Cuong Tran, Yoshinori Tanabe, Masami Hagiya, University of Tokyo).
If you look to a kind of formal verification of FSM like models, Java PathFinder has an extension to verify UML state charts written in Java+annotation (it depends on the Javapathfinder VM):
http://babelfish.arc.nasa.gov/trac/jpf/wiki/projects/jpf-statechart
Mock frameworks, e.g. EasyMock, make it easier to plugin dummy dependencies. Having said that, using them for ensuring how different methods on particular components are called (and in what order) seems bad to me. It exposes the behaviour to test class, which makes it harder to maintain production code. And I really don't see the benefit; mentally I feel like I've been chained to a heavy ball.
I much rather like to just test against interface, giving test data as input and asserting the result. Better yet, to use some testing tool that generates test data automatically for verifying given property. e.g. adding one element to a list, and removing it immediately yields the same list.
In our workplace, we use Hudson which gives testing coverage. Unfortunately it makes it easy to get blindly obsessed that everything is tested. I strongly feel that one shouldn't test everything if one wants to be productive also in maintenance mode. One good example would be controllers in web frameworks. As generally they should contain very little logic, testing with mock framework that controller calls such and such method in particular order is nonsensical in my honest opinion.
Dear SOers, what are your opinions on this?
I read 2 questions:
What is your opinion on testing that particular methods on components are called in a particular order?
I've fallen foul of this in the past. We use a lot more "stubbing" and a lot less "mocking" these days.
We try to write unit tests which test only one thing. When we do this it's normally possible to write a very simple test which stubs out
interactions with most other components. And we very rarely assert ordering. This helps to make the tests less brittle.
Tests which test only one thing are easier to understand and maintain.
Also, if you find yourself having to write lots of expectations for interactions with lots of components there could well be a problem in the code you're testing anyway. If it's difficult to maintain tests the code you're testing can often be refactored.
Should one be obsessed with test coverage?
When writing unit tests for a given class I'm pretty obsessed with test coverage. It makes it really easy to spot important bits of behaviour that I haven't tested. I can also make a judgement call about which bits I don't need to cover.
Overall unit test coverage stats? Not particularly interested so long as they're high.
100% unit test coverage for an entire system? Not interested at all.
I agree - I'm in favor of leaning heavily towards state verification rather than behavior verification (a loose interpretation of classical TDD while still using test doubles).
The book The Art of Unit Testing has plenty of good advice in these areas.
100% test coverage, GUI testing, testing getters/setters or other no-logic code, etc. seem unlikely to provide good ROI. TDD will provide high test coverage in any case. Test what might break.
It depends on how you model the domain(s) of your program.
If you model the domains in terms of data stored in data structures and methods that read data from one data structure and store derived data in another data structure (procedures or functions depending how procedural or functional your design is), then mock objects are not appropriate. So called "state-based" testing is what you want. The outcome you care about is that a procedure puts the right data in the right variables and what it calls to make that happen is just an implementation detail.
If you model the domains in terms of message-passing communication protocols by which objects collaborate, then the protocols are what you care about and what data the objects store to coordinate their behaviour in the protocols in which they play roles is just implementation detail. In that case, mock objects are the right tool for the job and state based testing ties the tests too closely to unimportant implementation details.
And in most object-oriented programs there is a mix of styles. Some code will be written purely functional, transforming immutable data structures. Other code will be coordinating the behaviour of objects that change their hidden, internal state over time.
As for high test coverage, it really doesn't tell you that much. Low test coverage shows you where you have inadequate testing, but high test coverage doesn't show you that the code is adequately tested. Tests can, for example, run through code paths and so increase the coverage stats but not actually make any assertions about what those code paths did. Also, what really matters is how different parts of the program behave in combination, which unit test coverage won't tell you. If you want to verify that your tests really are testing your system's behaviour adequately you could use a Mutation Testing tool. It's a slow process, so it's something you'd run in a nightly build rather than on every check-in.
I'd asked a similar question How Much Unit Testing is a Good Thing, which might help give an idea of the variety of levels of testing people feel are appropriate.
What is the probability that during your code's maintenance some junior employee will break the part of code that runs "controller calls such and such method in particular order"?
What is the cost to your organization if such a thing occurs - in production outage, debugging/fixing/re-testing/re-release, legal/financial risk, reputation risk, etc...?
Now, multiply #1 and #2 and check whether your reluctance to achieve a reasonable amount of test coverage is worth the risk.
Sometimes, it will not be (this is why in testing there's a concept of a point of diminishing returns).
E.g. if you maintain a web app that is not production critical and has 100 users who have a workaround if the app is broken (and/or can do easy and immediate rollback), then spending 3 months doing full testing coverage of that app is probably non-sensical.
If you work on an app where a minor bug can have multi-million-dollar or worse consequences (think space shuttle software, or guidance system for a cruise missile), then the thorough testing with complete coverage becomes a lot more sensical.
Also, i'm not sure if i'm reading too much into your question but you seem to be implying that having mocking-enabled unit testing somehow excluds application/integration functional testing. If that is the case, you are right to object to such a notion - the two testing approaches must co-exist.