Related
As many of you are aware as of the release of MatLab 2013a, xUnit a popular unit testing framework for MatLab is canceling further development.
Is MatLab's new and native unit testing framework comparable to xUnit? what features is it lacking when compared to xUnit? Is it better or worse than xUnit?
MATLAB xUnit has been an excellent contribution to the test focused development efforts of those writing MATLAB code. It has a solid implementation, it follows the xUnit paradigm very well, and has been invaluable as a file exchange contribution.
The MATLAB Unit Test framework has indeed learned from this submission as well as decades of requirements and test focused development for the MathWorks' internal code base. We have also learned and extended upon frameworks in other languages such as JUnit, NUnit, and python's unittest framework. As such there certainly are many more features in the R2013a-beyond framework, and it is designed to scale and extend.
There are too many other features to go into in a simple answer, but perhaps one way to describe some of the differences are that the 13a framework is what I loosely call an "xUnit 2.0" and the file exchange submissions is an "xUnit 1.0" framework. If you are familair with JUnit, this is like the difference between JUnit 3 and JUnit 4.
There are also other intangible or as yet unrealized benefits, such as:
The framework is included directly in MATLAB so you can share tests with others and know that they can run the tests even if they are not familiar with testing and do not want to download the file exchange framework.
The framework is under active development with a pipeline of additional features and capabilities in the works for future releases.
Hope that helps. I would be happy to go over any questions you have about specific functionality or features.
I don't believe MathWorks are planning at all to stop making xUnit available, so you can continue using it if you like. xUnit had not seen any large changes for quite a while in any case, and even though it won't be developed further in terms of features, it may receive an occasional fix if any are needed.
I have tried out the new framework quite a bit, but have not used it on any large projects yet. Previously I have used xUnit on large projects. However, I'm no expert on unit testing - so please read the following opinions in that context.
I'm pretty sure there's nothing you can do in xUnit that you can't do in the new framework. In general it's much more flexible and powerful than xUnit, providing additional features and a better way to organise and structure your tests. It's a lot easier to set up and tear down suites of tests, to manage and close resources (files, figure windows, database connections etc), and to carry out tricky tests such as checking that the right number of arguments are returned.
However, whereas a typical xUnit test was implemented as a fairly simple MATLAB function, tests in the new framework are typically implemented (in 13a, but see below for 13b) as classes using MATLAB's OO syntax, and if you're not comfortable with that it may seem like a big leap.
I should also add that although the documentation for the testing framework is excellent as reference material, I haven't found it to be great as a tutorial.
In 13b, the need to use classes has been offset a bit with the introduction of the functiontests command, which creates a test suite for you from a file containing tests implemented as local functions. That will make things much easier if you're not comfortable with class syntax. But I would think that if you want to take advantage of everything, you'd probably still want to use the main framework.
Hope my experience is of help - if you're lucky, perhaps #AndyCampbell will chime in...
I recently joined a Perl project and I need to start being productive with the codebase fairly quickly. However, I'm finding that I'm getting stuck because I don't know where I need to change or how all the parts of the code fit together.
What are your tips and tools for becoming familiar with a Perl codebase that you have no experience with?
(Note: I realize that there's already a similar question. I'm wondering if there's any Perl-specific strategies.)
First, if the previous maintainers were doing their job well, you should have an extensive test suite and perldoc documentation for each module and script in the codebase. If so, read through the perldoc, and read through the tests. The perldoc should give you an overview of what things do, and the test suite will give you examples of the code being used in context.
Depending on the author, the internal comments may be useful in understanding the intention of the code, so looking through the actual source my provide insights into algorithms, bugs, and intended use as well.
If you don't have any of these, proceed as you would for any badly-maintained codebase: start small, writing programs that try to use the code, and use Test::More and the like to start turning these into a test suite.
In the first case, you may find it to be very simple, in the second, very hard. Peter Scott's Perl Medic can be very useful in assisting you in turning such a codebase into something usable and useful if you're stuck with the second case, and Mike Thomsen's recommendation of Effective Perl Programming is also a good one.
I work on Melody which is written primarily in Perl. It's a rather large code base, and I've found the process of learning the Melody code base is identical to any Java system I've worked on.
It really comes down to just working with it, googling when you see behavior you've never seen before and experimentation.
This book is a great reference for picking up Perl in a serious way. It's not very dense and it will teach you a lot about proper Perl development.
Besides the "similar question", http://perldoc.perl.org/ and an empty test.pl file is a good starting point!
I would like to see a real answer here. The only thing I have is more questions (you don't have to provide answers here, just ask yourself):
Goal
What is the goal of the project, what it is supposed to do?
Who knows the workflow?
Environment
Can you set up the project in a clean test environment?
Does it use a versioning system?
Where are the entry points (i.e. executable files)?
Does it rely on external programs?
Does it require additional system tweaks (i.e. cron scripts)?
Perl code
Is your project using strict and warnings everywhere?
Which CPAN modules are used?
Are there any frameworks used (Moose, Catalyst, probably some ORM, ...)?
Are there any perldocs in the project's modules?
Are there any tests (notably t/*.t)?
I usually start with working on some simple bug report or a simple feature I want to add. While working on code I write comments for code and commit them. Writing tests also helps.
Locked. This question and its answers are locked because the question is off-topic but has historical significance. It is not currently accepting new answers or interactions.
What are the best technologies to use for behavior-driven development on the iPhone? And what are some open source example projects that demonstrate sound use of these technologies? Here are some options I've found:
Unit Testing
Test::Unit Style
OCUnit/SenTestingKit as explained in iOS Development Guide: Unit Testing Applications & other OCUnit references.
Examples: iPhoneUnitTests, Three20
CATCH
GHUnit
Google Toolbox for Mac: iPhone Unit Testing
RSpec Style
Kiwi (which also comes with mocking & expectations)
Cedar
Jasmine with UI Automation as shown in dexterous' iOS-Acceptance-Testing specs
Acceptance Testing
Selenium Style
UI Automation (works on device)
UI Automation Instruments Guide
UI Automation reference documentation
Tuneup js - cool library for using with UIAutomation.
Capturing User Interface Actions into Automation Scripts
It's possible to use Cucumber (written in JavaScript) to drive UI Automation. This would be a great open-source project. Then, we could write Gherkin to run UI Automation testing. For now, I'll just write Gherkin as comments.
UPDATE: Zucchini Framework seems to blend Cucumber & UI Automation! :)
Old Blog Posts:
Alex Vollmer's UI Automation tutorial
O'Reilly Answers UI Automation tutorial
Adi Saxena's UI Automation tutorial
UISpec with UISpecRunner
UISpec is open source on Google Code.
UISpec has comprehensive documentation.
FoneMonkey
Cucumber Style
Frank and iCuke (based on the Cucumber meets iPhone talk)
The Frank Google Group has much more activity than the iCuke Google Group.
Frank runs on both device and simulator, while iCuke only runs in simulator.
Frank seems to have a more comprehensive set of step definitions than iCuke's step definitions. And, Frank also has a step definition compendium on their wiki.
I proposed that we merge iCuke & Frank (similar to how Merb & Rails merged) since they have the same common goal: Cucumber for iOS.
KIF (Keep It Functional) by Square
Zucchini Framework uses Cucumber syntax for writing tests and uses CoffeeScript for step definitions.
Additions
OCMock for mocking
OCHamcrest and/or Expecta for expectations
Conclusion
Well, obviously, there's no right answer to this question, but here's what I'm choosing to go with currently:
For unit testing, I used to use OCUnit/SenTestingKit in XCode 4. It's simple & solid. But, I prefer the language of BDD over TDD (Why is RSpec better than Test::Unit?) because our words create our world. So now, I use Kiwi with ARC & Kiwi code completion/autocompletion. I prefer Kiwi over Cedar because it's built on top of OCUnit and comes with RSpec-style matchers & mocks/stubs. UPDATE: I'm now looking into OCMock because, currently, Kiwi doesn't support stubbing toll-free bridged objects.
For acceptance testing, I use UI Automation because it's awesome. It lets you record each test case, making writing tests automatic. Also, Apple develops it, and so it has a promising future. It also works on the device and from Instruments, which allows for other cool features, like showing memory leaks. Unfortunately, with UI Automation, I don't know how to run Objective-C code, but with Frank & iCuke you can. So, I'll just test the lower-level Objective-C stuff with unit tests, or create UIButtons only for the TEST build configuration, which when clicked, will run Objective-C code.
Which solutions do you use?
Related Questions
Is there a BDD solution that presently works well with iOS4 and Xcode4?
SenTestingKit (integrated with XCode) versus GHUnit on XCode 4 for Unit Testing?
Testing asynchronous code on iOS with OCunit
SenTestingKit in Xcode 4: Asynchronous testing?
How does unit testing on the iPhone work?
tl;dr
At Pivotal we wrote Cedar because we use and love Rspec on our Ruby projects. Cedar isn't meant to replace or compete with OCUnit; it's meant to bring the possibility of BDD-style testing to Objective C, just as Rspec pioneered BDD-style testing in Ruby, but hasn't eliminated Test::Unit. Choosing one or the other is largely a matter of style preferences.
In some cases we designed Cedar to overcome some shortcomings in the way OCUnit works for us. Specifically, we wanted to be able to use the debugger in tests, to run tests from the command line and in CI builds, and get useful text output of test results. These things may be more or less useful to you.
Long answer
Deciding between two testing frameworks like Cedar and OCUnit (for example) comes down to two things: preferred style, and ease of use. I'll start with the style, because that's simply a matter of opinion and preference; ease of use tends to be a set of tradeoffs.
Style considerations transcend what technology or language you use. xUnit-style unit testing has been around for far longer than BDD-style testing, but the latter has rapidly gained in popularity, largely due to Rspec.
The primary advantage of xUnit-style testing is its simplicity, and wide adoption (amongst developers who write unit tests); nearly any language you could consider writing code in has an xUnit-style framework available.
BDD-style frameworks tend to have two main differences when compared to xUnit-style: how you structure the test (or specs), and the syntax for writing your assertions. For me, the structural difference is the main differentiator. xUnit tests are one-dimensional, with one setUp method for all tests in a given test class. The classes that we test, however, aren't one-dimensional; we often need to test actions in several different, potentially conflicting, contexts. For example, consider a simple ShoppingCart class, with an addItem: method (for the purposes of this answer I'll use Objective C syntax). The behavior of this method may differ when the cart is empty compared to when the cart contains other items; it may differ if the user has entered a discount code; it may differ if the specified item can't be shipped by the selected shipping method; etc. As these possible conditions intersect with one another you end up with a geometrically increasing number of possible contexts; in xUnit-style testing this often leads to a lot of methods with names like testAddItemWhenCartIsEmptyAndNoDiscountCodeAndShippingMethodApplies. The structure of BDD-style frameworks allows you to organize these conditions individually, which I find makes it easier to make sure I cover all cases, as well as easier to find, change, or add individual conditions. As an example, using Cedar syntax, the method above would look like this:
describe(#"ShoppingCart", ^{
describe(#"addItem:", ^{
describe(#"when the cart is empty", ^{
describe(#"with no discount code", ^{
describe(#"when the shipping method applies to the item", ^{
it(#"should add the item to the cart", ^{
...
});
it(#"should add the full price of the item to the overall price", ^{
...
});
});
describe(#"when the shipping method does not apply to the item", ^{
...
});
});
describe(#"with a discount code", ^{
...
});
});
describe(#"when the cart contains other items, ^{
...
});
});
});
In some cases you'll find contexts in that contain the same sets of assertions, which you can DRY up using shared example contexts.
The second main difference between BDD-style frameworks and xUnit-style frameworks, assertion (or "matcher") syntax, simply makes the style of the specs somewhat nicer; some people really like it, others don't.
That leads to the question of ease of use. In this case, each framework has its pros and cons:
OCUnit has been around much longer than Cedar, and is integrated directly into Xcode. This means it's simple to make a new test target, and, most of the time, getting tests up and running "just works." On the other hand, we found that in some cases, such as running on an iOS device, getting OCUnit tests to work was nigh impossible. Setting up Cedar specs takes some more work than OCUnit tests, since you have get the library and link against it yourself (never a trivial task in Xcode). We're working on making setup easier, and any suggestions are more than welcome.
OCUnit runs tests as part of the build. This means you don't need to run an executable to make your tests run; if any tests fail, your build fails. This makes the process of running tests one step simpler, and test output goes directly into your build output window which makes it easy to see. We chose to have Cedar specs build into an executable which you run separately for a few reasons:
We wanted to be able to use the debugger. You run Cedar specs just like you would run any other executable, so you can use the debugger in the same way.
We wanted easy console logging in tests. You can use NSLog() in OCUnit tests, but the output goes into the build window where you have to unfold the build step in order to read it.
We wanted easy to read test reporting, both on the command line and in Xcode. OCUnit results appear nicely in the build window in Xcode, but building from the command line (or as part of a CI process) results in test output intermingled with lots and lots of other build output. With separate build and run phases Cedar separates the output so the test output is easy to find. The default Cedar test runner copies the standard style of printing "." for each passing spec, "F" for failing specs, etc. Cedar also has the ability to use custom reporter objects, so you can have it output results any way you like, with a little effort.
OCUnit is the official unit testing framework for Objective C, and is supported by Apple. Apple has basically limitless resources, so if they want something done it will get done. And, after all, this is Apple's sandbox we're playing in. The flip side of that coin, however, is that Apple receives on the order of a bajillion support requests and bug reports each day. They're remarkably good about handling them all, but they may not be able to handle issues you report immediately, or at all. Cedar is much newer and less baked than OCUnit, but if you have questions or problems or suggestions send a message to the Cedar mailing list (cedar-discuss#googlegroups.com) and we'll do what we can to help you out. Also, feel free to fork the code from Github (github.com/pivotal/cedar) and add whatever you think is missing. We make our testing frameworks open source for a reason.
Running OCUnit tests on iOS devices can be difficult. Honestly, I haven't tried this for quite some time, so it may have gotten easier, but the last time I tried I simply couldn't get OCUnit tests for any UIKit functionality to work. When we wrote Cedar we made sure that we could test UIKit-dependent code both on the simulator and on devices.
Finally, we wrote Cedar for unit testing, which means it's not really comparable with projects like UISpec. It's been quite a while since I tried using UISpec, but I understood it to be focused primarily on programmatically driving the UI on an iOS device. We specifically decided not to try to have Cedar support these types of specs, since Apple was (at the time) about to announce UIAutomation.
I'm going to have to toss Frank into the acceptance testing mix. This is a fairly new addition but has worked excellent for me so far. Also, it is actually being actively worked on, unlike icuke and the others.
For test driven development, I like to use GHUnit, its a breeze to set up, and works great for debugging too.
Great List!
I found another interesting solution for UI testing iOS applications.
Zucchini Framework
It is based on UIAutomation.
The framework let you write screen centric scenarios in Cucumber like style.
The scenarios can be executed in Simulator and on device from a console (it is CI friendly).
The assertions are screenshot based. Sounds inflexible, but it gets you nice HTML report, with highlighted screen comparison and you can provide masks which define the regions you want to have pixel exact assertion.
Each screen has to be described in CoffeScript and the tool it self is written in ruby.
It is kind of polyglott nightmare, but the tool provides a nice abstraction for UIAutomation and when the screens are described it is manageable even for QA person.
I would choose iCuke for acceptance tests and Cedar for unit tests. UIAutomation is a step in the right direction for Apple, but the tools need better support for continuous integration; automatically running UIAutomation tests with Instruments is currently not possible, for example.
GHUnit is good for unit tests; for integration tests, I've used UISpec with some success (github fork here: https://github.com/drync/UISpec), but am looking forward to trying iCuke, since it promises to be a lightweight setup, and you can use the rails testing goodness, like RSpec and Cucumber.
I currently use specta for rspec like setups and it's partner (as mentioned above) expecta which has ton's of awesome matching options.
I happen to really like OCDSpec2 but I'm biased, I wrote OCDSpec and contribute to the second.
It's very fast even on iOS, in part because it's built from the ground up rather than being put on top of OCUnit. It has an RSpec/Jasmine syntax as well.
https://github.com/ericmeyer/ocdspec2
There's Test::Simple, Test::More, Test::Builder (all part of the Test::Simple distribution), Test::Class, Test::Unit, Test::Moose...
I'm starting a new project using Moose—which module should I use for writing my tests?
You can eliminate Test::Builder from the list for a while. Test::Builder is the base module that other Test:: modules are built on. So until you want to start writing your own test modules you won't need it.
I'd also ignore Test::Simple. Test::More does all that Test::Simple does - and plenty more.
Test::Class is a nice way to write unit tests in a really object oriented way. I'd recommend it for complex OO-based systems.
Test::Moose is for testing various Moose-related features in your code. You say that you're using Moose so it might well be useful for you. It can be used in conjunction with Test::More.
So my recommendation would be to start with Test::More and Test::Moose. But also take a look at Test::Class to see if it fits with the way that you want to write tests.
Perl Testing: A Developers Notebook is a great introduction to this topic.
In addition to davorg's excellent answer, I'd like to note that I'm still mostly using Test::More (with some help from Test-Differences, Test-WWW-Mechanize-LibXML, and other modules). I can recommend against using Test.pm which is old and stupid, and Test::Simple, which is a small subset of the Test::More functionality.
There are also Test::Most (an extension of Test::More), Test::Class and Test::Class::Most, which some people prefer, but I didn't take the time to learn them yet.
There's an ongoing debate about whether a plan (= tests' count) is a good thing or not. Personally, I've already noticed a case in someone else's CPAN module where the count of tests was different on my system's than on them (and varied based on different versions of DBI (IIRC)) and which convinced me that the plan is a good thing. As a result I've created Test-Count which is a way to count and update the assertions count based on annotations inside well-formed comments (and which supports source code in other languages besides Perl 5). I'm still supporting it, so if you need anything, give me a shout.
I'd recommend Test::Class as a basis for your test framework. It makes for better structured, more modular code. And you can still use Test::More and other Test modules with it.
Also check Test::Exception.
Others have suggested Test::Class; I have found the following PDF overview from $foo Magazin (I didn't write it, just found it) was quite helpful for some exampled beyond what the POD documentation provides.
It is not really essential for moose development.
However, if you're into Web Development , I think Test::WWW::Selenium is becoming indespensable for testing javascript-heavy web pages and their behavior in the most common web browsers (firefox, iexplorer, googlechrome, etc.)
Every time I see Ruby or Python discussed in the context of web development, it's always with a framework (Rails for Ruby, Django for Python). Are these frameworks necessary? If not, is there a reason why these languages are often used within a framework, while Perl and PHP are not?
I can only speak towards Ruby - but, no, you don't need a framework to run Ruby based pages on the web. You do need a ruby enabled server, such as Apache running eruby/erb. But, once you do, you can create .rhtml files just like RoR, where it processes the inline ruby code.
The short answer is no, they are not necessary. In ruby you have .erb templates that can be used in a similar way as you use PHP pages. You can write a site in ruby or Python using several technologies (Rails-like frameworks, Templates or even talking directly with the HTTP library and building the page CGI-style).
Web frameworks like Python's Django or Ruby's Rails (there are many) just raise the level of abstraction from the PHP's or ASP's, and automate several process (like login, database interaction, REST API's) which is always a good thing.
"Need" is a strong word. You can certainly write Python without one, but I wouldn't want to.
Python wasn't designed (like PHP was, for example) as a direct web scripting language, so common web-ish things like connecting to databases isn't native, and frameworks are handy.
EDIT: mod_python exists for Apache, so if you're merely looking to write some scripts, then Python doesn't need a framework. If you want to build an entire site, I'd recommend using one.
From a Pythonic point of view, you'd absolutely want to use one of the frameworks. Yes, it might be possible to write a web app without them, but it's not going to be pretty. Here's a few things you'll (probably) end up writing from scratch:
Templating: unless you're writing a really really quick hack, you don't want to be generating all of your HTML within your Python code -- this is a really poor design that becomes a maintainability nightmare.
URL Processing: splitting a URL and identifying which code to run isn't a trivial task. Django (for example) provides a fantastic mechanism to map from a set of regular expressions to a set of view functions.
Authentication: rolling your own login/logout/session management code is a pain, especially when there's already pre-written (and tested) code available
Error handling: frameworks already have a good mechanism in place to a) help you debug your app, and b) help redirect to proper 404 and 500 pages.
To add to this, all of the framework libraries are all heavily tested (and fire tested). Additionally, there are communities of people who are developing using the same code base, so if you have any questions, you can probably find help.
In summary, you don't have to, but unless your project is "a new web framework", you're probably better off using one of the existing ones instead.
Framework? Heck, you don't even need a web server if you're using Python, you can make one in around three lines of code.
As to the why:
The most plausible thing I can think of is that Perl and PHP were developed before the notion of using frameworks for web apps became popular. Hence, the "old" way of doing things has stuck around in those cultures. Ruby and Python became popular after frameworks became popular, hence they developed together. If your language has a good framework (or more than one) that's well supported by the community, there's not much reason to try to write a Web App without one.
A framework isn't necessary per se, but it can certainly speed development and help you write "better" code. In PHP, there are definitely frameworks that get used like CakePHP, and in Perl there are many as well like Mason and Catalyst.
The frameworks aren't necessary. However, a lot of developers think frameworks ease development by automating a lot of things. For example, Django will create a production-ready backend for you based on your database structure. It also has lets you incorporate various plugins if you choose. I don't know too much about Rails or Perl frameworks, but PHP frameworks such as Zend, Symfony, Code Igniter, CakePHP, etc are used widely.
Where I work at we rolled our own PHP framework.
Are these frameworks necessary?
No. They, like any 'framework', are simply for speeding up development time and making the programmer's job easier.
If not, is there a reason why these languages are often used within a framework, while Perl and PHP are not?
PHP and Perl were popular languages for building web sites well before the idea of using frameworks was. Frameworks like Rails are what gave Ruby it's following. I'm not sure that Python or Ruby were that common as web languages before they were backed by frameworks.
These days, even PHP/Perl web developement should be backed by a framework (of which there are now many).
By no means are those development frameworks required. But as with most development environments, your productivity will increase exponentially if you have a supported framework to reference and build your applications on. It also decreases the training needed to bring others up to speed on your applications if they already have a core understanding of the framework that you use.
For python, the answer is No you don't have to. You can write python directly behind your web server very easily, take a look at mod_python for how to do it.
A lot of people like frameworks because they supply a lot of the boilerplate code in a reliable form so you don't have to write it yourself. But, like any code project, you should choose the tools and frameworks on their merit for your problem.
You can certainly write CGI scripts in either language and do things "raw".
The frameworks (ideally) save the trouble of writing a pile of code for things that other people have already handled (session handling, etc.).
The decision probably comes down to what you need to do. If the framework has the features you need, why not use it. If the framework is going to require extensive modifications, it might be easier to roll your own stuff. Or check out a different framework.
The python library has numerous modules for doing cgi, parsing html, cookies, WSGI, etc:
http://docs.python.org/library/index.html
PHP has a lot of frameworks. Probably more then most. In Ruby most use Rails so thats what you hear, and Django for Python is mentioned more then not.
But with PHP you have many to choose from.
List of web application frameworks
Any language that can "print" can be used to generate web pages, but frameworks handle a lot of the HTML generation for you. They let you concentrate more on the content and less on the details of coding the raw HTML.