I took the time to set up some Unit Tests and set up the targets in XCode, etc., and they're pretty useful for a few classes. However:
I want to test small UI pieces for which I don't want to launch the entire application. There is no concept of pass/fail: I need to "see" the pieces, and I can make dummy instances of all the relevant classes to do this. My question is: how can I set this up in XCode?
I realize I could use another XCode project for each class (or groups of classes), but that seems a bit cumbersome. Another target for each?
I know that you're looking for an approach to testing UI components that doesn't require a fully functional application, but I've been impressed with what the new UI Automation instrument introduced in iOS 4.0 lets you do.
This instrument lets you use Javascript scripts to interactively test your application's interface, and it does so in a way that does not require checking exact pixel values or positions on a screen. It uses the built-in accessibility hooks present in the system for VoiceOver to identify and interact with components.
Using this instrument, I have been able to script tests that fully exercise my application as a user would interact with it, as well as ones that hammer on particular areas and look for subtle memory buildups.
The documentation on this part of Instruments is a little sparse, but I recently taught a class covering the subject for which the video is available on iTunes U for free (look for the Testing class in the Fall semester). My course notes (in VoodooPad format) cover this as well. I also highly recommend watching the WWDC 2010 video session 306 - "Automating User Interface Testing with Instruments".
Well, you cannot call showing a piece of some GUI a testing even if that GUI is a part of a large application. What you can do here is create a separate executable target and write a small tool that reuses GUI components from your application and shows them to you basing on input parameters. This will eliminate the need for many many different targets.
If you still insist on using unit tests, you can show your GUI for some period of time, for example, 10 seconds. So the test case will run until GUI is closed or timeout elapses and each test will take up to N seconds to execute.
This is a good question. I think you actually do not want to use unit tests for those 'visual confirmations'. Personally I usually write little test apps to do this kind of testing or development. I don't like separate targets in the same project so I usually just create a test project next to the original one and then reference those classes and resources using relative paths. Less clutter. And it is really nice to be able to test more complex user interface elements in their own little test environment.
I would take a two-level approach to UI "unit testing":
lthough Cocoa/CocoaTouch are still closer to the Model-View-Controller than the Model-View-ViewModel paradigm, you can gain much of the testability advantage by breaking your "View" into a "view model" and a "presenter" view (note that this is somewhat along the lines of the NSView/NSCell pair; Cocoa engineers had this one a long time ago). If the view is a simple presentation layer, than you can test behavior of the view by unit testing the "view model".
To test the drawing/rendering of your views, you will have to either do human testing or do rendering/pixel-based tests. Google's Toolbox for Mac has several tools for doing pixel-by-pixel comparison of rendered NSViews, CALayers, UIViews, etc. I've written a tool for the Core Plot project to make dealing with the test failures and merging the reference files back into your unit test bundle a little easier.
Related
I have been making the native iPhone app and I have never used a xib, I mainly used to make the application in run time based, but know with the introduction of Main Story Board. I just want to know which one will be easier for using run time based which I mainly used to or start using story board.
Both have its own advantage and Storyboard has more options and ease when used in the right way (that is why it is introduced!).
By using storyboard
You can avoid so many line of code for page navigation and all
Complete picture of project is visible
By xib
Used a lot and hence more easy
Independent,so can be worked better
For working as a team xib is preferrable since the conflicts can be less if each team member working in seperate files
MY SOLUTION
Since it is possible to use both in same project. I use both in my projects. To take the both advantages according to my requirement.
You'll probably want to use a Storyboard(s) in the long run. However, there are pros and cons:
Pros
You can see your whole project
You avoid a whole lot of nasty code
You can visually 'hook' stuff together
Cons
Only available >iOS 5.0 <<<<Huge!
In my personal experience, I usually use Storyboards - the percentage of devices running < iOS 5 is actually quite low.
Locked. This question and its answers are locked because the question is off-topic but has historical significance. It is not currently accepting new answers or interactions.
What are the best technologies to use for behavior-driven development on the iPhone? And what are some open source example projects that demonstrate sound use of these technologies? Here are some options I've found:
Unit Testing
Test::Unit Style
OCUnit/SenTestingKit as explained in iOS Development Guide: Unit Testing Applications & other OCUnit references.
Examples: iPhoneUnitTests, Three20
CATCH
GHUnit
Google Toolbox for Mac: iPhone Unit Testing
RSpec Style
Kiwi (which also comes with mocking & expectations)
Cedar
Jasmine with UI Automation as shown in dexterous' iOS-Acceptance-Testing specs
Acceptance Testing
Selenium Style
UI Automation (works on device)
UI Automation Instruments Guide
UI Automation reference documentation
Tuneup js - cool library for using with UIAutomation.
Capturing User Interface Actions into Automation Scripts
It's possible to use Cucumber (written in JavaScript) to drive UI Automation. This would be a great open-source project. Then, we could write Gherkin to run UI Automation testing. For now, I'll just write Gherkin as comments.
UPDATE: Zucchini Framework seems to blend Cucumber & UI Automation! :)
Old Blog Posts:
Alex Vollmer's UI Automation tutorial
O'Reilly Answers UI Automation tutorial
Adi Saxena's UI Automation tutorial
UISpec with UISpecRunner
UISpec is open source on Google Code.
UISpec has comprehensive documentation.
FoneMonkey
Cucumber Style
Frank and iCuke (based on the Cucumber meets iPhone talk)
The Frank Google Group has much more activity than the iCuke Google Group.
Frank runs on both device and simulator, while iCuke only runs in simulator.
Frank seems to have a more comprehensive set of step definitions than iCuke's step definitions. And, Frank also has a step definition compendium on their wiki.
I proposed that we merge iCuke & Frank (similar to how Merb & Rails merged) since they have the same common goal: Cucumber for iOS.
KIF (Keep It Functional) by Square
Zucchini Framework uses Cucumber syntax for writing tests and uses CoffeeScript for step definitions.
Additions
OCMock for mocking
OCHamcrest and/or Expecta for expectations
Conclusion
Well, obviously, there's no right answer to this question, but here's what I'm choosing to go with currently:
For unit testing, I used to use OCUnit/SenTestingKit in XCode 4. It's simple & solid. But, I prefer the language of BDD over TDD (Why is RSpec better than Test::Unit?) because our words create our world. So now, I use Kiwi with ARC & Kiwi code completion/autocompletion. I prefer Kiwi over Cedar because it's built on top of OCUnit and comes with RSpec-style matchers & mocks/stubs. UPDATE: I'm now looking into OCMock because, currently, Kiwi doesn't support stubbing toll-free bridged objects.
For acceptance testing, I use UI Automation because it's awesome. It lets you record each test case, making writing tests automatic. Also, Apple develops it, and so it has a promising future. It also works on the device and from Instruments, which allows for other cool features, like showing memory leaks. Unfortunately, with UI Automation, I don't know how to run Objective-C code, but with Frank & iCuke you can. So, I'll just test the lower-level Objective-C stuff with unit tests, or create UIButtons only for the TEST build configuration, which when clicked, will run Objective-C code.
Which solutions do you use?
Related Questions
Is there a BDD solution that presently works well with iOS4 and Xcode4?
SenTestingKit (integrated with XCode) versus GHUnit on XCode 4 for Unit Testing?
Testing asynchronous code on iOS with OCunit
SenTestingKit in Xcode 4: Asynchronous testing?
How does unit testing on the iPhone work?
tl;dr
At Pivotal we wrote Cedar because we use and love Rspec on our Ruby projects. Cedar isn't meant to replace or compete with OCUnit; it's meant to bring the possibility of BDD-style testing to Objective C, just as Rspec pioneered BDD-style testing in Ruby, but hasn't eliminated Test::Unit. Choosing one or the other is largely a matter of style preferences.
In some cases we designed Cedar to overcome some shortcomings in the way OCUnit works for us. Specifically, we wanted to be able to use the debugger in tests, to run tests from the command line and in CI builds, and get useful text output of test results. These things may be more or less useful to you.
Long answer
Deciding between two testing frameworks like Cedar and OCUnit (for example) comes down to two things: preferred style, and ease of use. I'll start with the style, because that's simply a matter of opinion and preference; ease of use tends to be a set of tradeoffs.
Style considerations transcend what technology or language you use. xUnit-style unit testing has been around for far longer than BDD-style testing, but the latter has rapidly gained in popularity, largely due to Rspec.
The primary advantage of xUnit-style testing is its simplicity, and wide adoption (amongst developers who write unit tests); nearly any language you could consider writing code in has an xUnit-style framework available.
BDD-style frameworks tend to have two main differences when compared to xUnit-style: how you structure the test (or specs), and the syntax for writing your assertions. For me, the structural difference is the main differentiator. xUnit tests are one-dimensional, with one setUp method for all tests in a given test class. The classes that we test, however, aren't one-dimensional; we often need to test actions in several different, potentially conflicting, contexts. For example, consider a simple ShoppingCart class, with an addItem: method (for the purposes of this answer I'll use Objective C syntax). The behavior of this method may differ when the cart is empty compared to when the cart contains other items; it may differ if the user has entered a discount code; it may differ if the specified item can't be shipped by the selected shipping method; etc. As these possible conditions intersect with one another you end up with a geometrically increasing number of possible contexts; in xUnit-style testing this often leads to a lot of methods with names like testAddItemWhenCartIsEmptyAndNoDiscountCodeAndShippingMethodApplies. The structure of BDD-style frameworks allows you to organize these conditions individually, which I find makes it easier to make sure I cover all cases, as well as easier to find, change, or add individual conditions. As an example, using Cedar syntax, the method above would look like this:
describe(#"ShoppingCart", ^{
describe(#"addItem:", ^{
describe(#"when the cart is empty", ^{
describe(#"with no discount code", ^{
describe(#"when the shipping method applies to the item", ^{
it(#"should add the item to the cart", ^{
...
});
it(#"should add the full price of the item to the overall price", ^{
...
});
});
describe(#"when the shipping method does not apply to the item", ^{
...
});
});
describe(#"with a discount code", ^{
...
});
});
describe(#"when the cart contains other items, ^{
...
});
});
});
In some cases you'll find contexts in that contain the same sets of assertions, which you can DRY up using shared example contexts.
The second main difference between BDD-style frameworks and xUnit-style frameworks, assertion (or "matcher") syntax, simply makes the style of the specs somewhat nicer; some people really like it, others don't.
That leads to the question of ease of use. In this case, each framework has its pros and cons:
OCUnit has been around much longer than Cedar, and is integrated directly into Xcode. This means it's simple to make a new test target, and, most of the time, getting tests up and running "just works." On the other hand, we found that in some cases, such as running on an iOS device, getting OCUnit tests to work was nigh impossible. Setting up Cedar specs takes some more work than OCUnit tests, since you have get the library and link against it yourself (never a trivial task in Xcode). We're working on making setup easier, and any suggestions are more than welcome.
OCUnit runs tests as part of the build. This means you don't need to run an executable to make your tests run; if any tests fail, your build fails. This makes the process of running tests one step simpler, and test output goes directly into your build output window which makes it easy to see. We chose to have Cedar specs build into an executable which you run separately for a few reasons:
We wanted to be able to use the debugger. You run Cedar specs just like you would run any other executable, so you can use the debugger in the same way.
We wanted easy console logging in tests. You can use NSLog() in OCUnit tests, but the output goes into the build window where you have to unfold the build step in order to read it.
We wanted easy to read test reporting, both on the command line and in Xcode. OCUnit results appear nicely in the build window in Xcode, but building from the command line (or as part of a CI process) results in test output intermingled with lots and lots of other build output. With separate build and run phases Cedar separates the output so the test output is easy to find. The default Cedar test runner copies the standard style of printing "." for each passing spec, "F" for failing specs, etc. Cedar also has the ability to use custom reporter objects, so you can have it output results any way you like, with a little effort.
OCUnit is the official unit testing framework for Objective C, and is supported by Apple. Apple has basically limitless resources, so if they want something done it will get done. And, after all, this is Apple's sandbox we're playing in. The flip side of that coin, however, is that Apple receives on the order of a bajillion support requests and bug reports each day. They're remarkably good about handling them all, but they may not be able to handle issues you report immediately, or at all. Cedar is much newer and less baked than OCUnit, but if you have questions or problems or suggestions send a message to the Cedar mailing list (cedar-discuss#googlegroups.com) and we'll do what we can to help you out. Also, feel free to fork the code from Github (github.com/pivotal/cedar) and add whatever you think is missing. We make our testing frameworks open source for a reason.
Running OCUnit tests on iOS devices can be difficult. Honestly, I haven't tried this for quite some time, so it may have gotten easier, but the last time I tried I simply couldn't get OCUnit tests for any UIKit functionality to work. When we wrote Cedar we made sure that we could test UIKit-dependent code both on the simulator and on devices.
Finally, we wrote Cedar for unit testing, which means it's not really comparable with projects like UISpec. It's been quite a while since I tried using UISpec, but I understood it to be focused primarily on programmatically driving the UI on an iOS device. We specifically decided not to try to have Cedar support these types of specs, since Apple was (at the time) about to announce UIAutomation.
I'm going to have to toss Frank into the acceptance testing mix. This is a fairly new addition but has worked excellent for me so far. Also, it is actually being actively worked on, unlike icuke and the others.
For test driven development, I like to use GHUnit, its a breeze to set up, and works great for debugging too.
Great List!
I found another interesting solution for UI testing iOS applications.
Zucchini Framework
It is based on UIAutomation.
The framework let you write screen centric scenarios in Cucumber like style.
The scenarios can be executed in Simulator and on device from a console (it is CI friendly).
The assertions are screenshot based. Sounds inflexible, but it gets you nice HTML report, with highlighted screen comparison and you can provide masks which define the regions you want to have pixel exact assertion.
Each screen has to be described in CoffeScript and the tool it self is written in ruby.
It is kind of polyglott nightmare, but the tool provides a nice abstraction for UIAutomation and when the screens are described it is manageable even for QA person.
I would choose iCuke for acceptance tests and Cedar for unit tests. UIAutomation is a step in the right direction for Apple, but the tools need better support for continuous integration; automatically running UIAutomation tests with Instruments is currently not possible, for example.
GHUnit is good for unit tests; for integration tests, I've used UISpec with some success (github fork here: https://github.com/drync/UISpec), but am looking forward to trying iCuke, since it promises to be a lightweight setup, and you can use the rails testing goodness, like RSpec and Cucumber.
I currently use specta for rspec like setups and it's partner (as mentioned above) expecta which has ton's of awesome matching options.
I happen to really like OCDSpec2 but I'm biased, I wrote OCDSpec and contribute to the second.
It's very fast even on iOS, in part because it's built from the ground up rather than being put on top of OCUnit. It has an RSpec/Jasmine syntax as well.
https://github.com/ericmeyer/ocdspec2
Is there any reason to not write iphone apps without useing the interface builder?
(or in other words, is it ok to write apps without the builder?)
Personally, my applications use no NIBs within them, but that's more of a matter of how I started doing development than anything else. I transitioned over from doing Mac development (where I use Interface Builder almost every day) to iPhone from when the first beta SDKs were released. Initially, there was no Interface Builder, and even when it came you couldn't do a lot of things with it, so I never took the time to really get to know it on the iPhone. It's more a matter of me doing what I'm familiar with.
Jeff LaMarche makes a convincing argument in his article "Don't Fear the Interface Builder" for you to use Interface Builder wherever you can, and I encourage new developers to learn to use it first before dropping down to programmatic user interface generation. It saves you a tremendous amount of time for interfaces using standard elements.
Some people have argued that there is a performance benefit to be had using purely programmatic interfaces, but Matt Gallagher ran a series of benchmarks and found that this speedup is only typically on the order of 5-10%. If you really want to shave that last bit off of your application's startup time, you might be able to have the best of both worlds with Adrian Kosmaczewski's nib2objc, which generates Objective-C from your NIB files.
However, there are plenty of times that you'll need to manipulate interfaces programmatically, such as for custom views and animations. This code can exist in parallel with Interface Builder without too much confusion. Again, it's more of a matter of personal preference at this point, but my recommendation is to use Interface Builder due to the time it can save you.
I've never written an iPhone app that uses Interface Builder (.xib files), but that's just personal taste: I like applications to be 100% code.
The reason is that I don't like applications that require design-time tools if you need to change something later on. It creates a dependency that may not be a problem today or next month, but it can become a problem in 3 years time.
So, the answer is: yes, it's fine to write iPhone apps without using Interface Builder.
Sure, it's absolutely fine to write apps without IB. It takes a little bit more setup (I believe there are some modifications you have to make to your Info.plist file), but it's certainly possible, and people do it all the time.
Personally, I prefer using Interface Builder for most tasks, but usually just to get the basic layout in place (so I don't have to deal with calculating rects, etc). Any further customization I prefer to do in code.
You can edit .XIB files by hand if you want to. They're XML, after all. Can't help having at least one of them in the project. Alternatively, you can create all your controls from code. The latter takes less typing, as far as I can tell.
Then, there are some apps where a XIB file would be of no use. I'm talking about games, where artistic/design considerations beat the convenience of having native widgets.
So yes, it's OK on both design level (no native widgets) and implementation level (yes native widgets, but sans IB).
I'm a big believer in testing, but not a very good practitioner. I've done pretty well at getting coverage on my model objects and programming them in a TDD style. I'm actually enjoying it so much I'd love to extend this to my controller layer, particularly my UIViewController subclasses.
Unfortunately, many UIKit classes don't function in independent tests. However, I'm unhappy with the restriction of only running my dependent tests on the device. It's really important to me to run all unit tests before every build, and it seems to me like its possible and worthwhile to unit test (as opposed to other types of testing) controller code.
My question is simply this: How do I test UIViewControllers in such a way that the tests run before every build? I am aware of a couple of different solutions to this problem, but don't know a whole lot about the various benefits of each one.
In general if you are stuck with an environmental problem where unconditional testing seems to be made impossible, you can work around it if the benefits are great enough to warrant your jumping through some hoops.
The situation here is a special case of unwanted outside dependencies, which I generally solve in my unit tests by either #ifdefing out tiny bits of dependent code, or by implementing stub classes to fill the expected dependency roles enough that my code can be tested.
So in this specific case, you might create a new source file, linked only in the test bundle, called "UIKitStubClasses.m" … within it you could implement the bare necessities to simulate UIKit dependent classes such as UIViewController, so that your tests link and thoroughly exercise their own logic.
The important thing to remember is this is usually not necessarily all that much work. The tests will let you know what you need to implement in the stub, for example by issuing exceptions about unimplemented methods. You just add what you need to quiet the errors and test your code, and then your stub class is as sufficient for testing as any of the legitimate system framework classes would be.
Can you not use the tips/methods in Chris Hanson's blog here: http://chanson.livejournal.com/120263.html
It seems that you would probably need to run your app as the test harness, but it seems doable. Just because you are running the 'app' doesn't mean it necessarily has to do the normal thing your application does. Alternatively, you could create another application target that is just a simple dummy application that will load your test bundles and run them.
I agree it might not be perfect, but it sounds like it might work, and it seems like it could be automated.
So I was unhappy with this situation as it stood, and both Daniel and Ed's answers spurred me on to improve things further. I decided to take matters into my own hands.
What I ended up writing was a small Cocoa Touch application that in -applicationDidFinishLaunching scans the class hierarchy for any SenTestCase subclasses and runs them. Unlike GoogleToolboxForMac's unit testing stuff, I tried to use as much of the built in SenTesting machinery as possible and as a result it didn't really end up as that much code. I haven't done too much testing beyond verifying that UILabel allocates without crashing the test rig (otest does in fact crash if you try), but it should work.
I've put the source online on both Bitbucket and Github.
While apple's tools have improved over time, they still leave much to be desired. GHUnit is much better, has a real test runner, and makes is dead simple to set breakpoints, etc. You can even leverage all your existing SenTests.
Get it on github.
You can instantiate all your UIKit code with no worries. I use it to test UIViewControllers, assert outlets, and do behavior testing.
Our company is currently writing a GUI automation testing tool for compact framework applications. We have initially searched many tools but none of them was right for us.
By using the tool you can record test-cases and group them together to test-suites. For every test-suite there is generated an application, which launches the application-under-test and simulates user-input.
In general the tool works fine, but as we are using window handles for simulation user input, you can't do very many things. For example it is impossible for us to get the name of a control (we just get the caption).
Another problem using window handles is checking for a change. At the moment we simulate a click on a control and depending on the result we know if the application has gone to the next step.
Is there any other (simpler) way for doing such things (for example the message queue or anything else)?
Interesting problem! I've not done any low-level (think Win32) Windows programming in a while, but here's what I would do.
Use a named pipe and have your application listen to it. Using this named pipe as a communication medium, implement a real simple protocol whereby you can query the application for the name of a control given its HWND, or other things you find useful. Make sure the protocol is rich enough so that there is sufficient information exchanged between your application and the test framework. Make sure that the test framework does not yield too much "special behavior" from the app, because then you wouldn't really be testing the features, but rather your test framework.
There's probably way more elegant and cooler ways to implement this, but this is what I remember from the top of my head, using only simple Win32 API calls.
Another approach, which we have implemented for our product at work, is to record user events, such as mouse clicks and key events in an event script. This should be rich enough so that you can have the application play it back, artificially injecting those events into the message queue, and have it behave the same way it did when you first recorded the script. You basically simulate the user when you play back the script.
In addition to that, you can record any important state (user's document, preferences, GUI controls hierarchy, etc.), once when you record the script, and once when you play it back. This gives you two sets of data you can compare, to make sure for instance that everything stays the same. This solution gives you tests that not easy to modify (you have to re-record if your GUI changes), but that provide awesome regression testing.
(EDIT: This is also a terrific QA tool during beta testing, for instance: just have your users record their actions, and if there's a crash, you have a good chance of easily reproducing the problem by just playing back the script)
Good luck!
Carl
If the Automated GUI testing tool has knowledge about the framework the application is written in it could use that information to make better or more advanced scripts. TestComplete for example knows about Borland's VCL and WinForms. If you test applications build using Windows Presentation Foundation has advanced support for this build in.
use NUnitForms. I've used them with great success for single and multi threading apps and you don't have to worry about handles and stuff like that
Here are some posts about NUnitForms worth reading
NUnitForms and failed DragDrop registration - problem of MTA vs STA
Compiled application exe GUI testing with NUnitForms
I finally found a solution to communicate between the testing-application and the application-under-test: Managed Spy. It's basically a .NET application build on top of ManagedSpyLib.
ManagedSpyLib allows programmatic access to the Windows Forms controls of another process. For this it uses Window Hooks and memory-mapping files.
Thanks for all who helped me to get to this solution!
Managed Spy does not provide a solution for compact framework applications.
The company Jamo Solutions (www.jamosolutions.com) meets the requirements for automation testing on mobile devices, including .net compact framework applications.