What would be the best means/tools for testing an Eclipse plugin? Are there some tools for testing the GUI features of an application created with Eclipse plugins?
You should try to keep GUI tests to a minimum since they are slow to run and takes time to create. If your code is well structured in a Model-view-controller pattern then the GUI specific code should be minimal.
Thats the theory in a perfect world atleast. Until we get there, I prefer to use SWTbot
Eclipse Jubula is a very nice tool for GUI based testing for Eclipse plugins. It provides automated functional GUI testing for various types of applications. It is aimed at teams who want their automated tests to be written by test experts from the user perspective, without requiring any coding effort. Jubula tests incorporate best practices from software development to ensure long-term maintainability of the automated tests.
It'll be useful to work with Jenkins as your CI build system.
You can connect to a real database as your wish for test result storage purposes.
You can save screenshots taken by Jubula upon test failures directly in the test report
I'd suggest you to take a look at RCP Testing Tool. This tool lets you to develop dozens of UI tests per day per engineer, and do not have stability and incorrect-recording problems. It's designed specially and only to test Eclipse-based apps. It is official Eclipse project and it's free.
After analyzing the possibilities, I have opted for using Jubula for testing the GUI part:
Eclipse Jubula Project
It's a good tool for creating tests specific for an RCP application. Moreover, it's not code based and it allows the creation of tests from a user point of view.
Related
Locked. This question and its answers are locked because the question is off-topic but has historical significance. It is not currently accepting new answers or interactions.
What are the best technologies to use for behavior-driven development on the iPhone? And what are some open source example projects that demonstrate sound use of these technologies? Here are some options I've found:
Unit Testing
Test::Unit Style
OCUnit/SenTestingKit as explained in iOS Development Guide: Unit Testing Applications & other OCUnit references.
Examples: iPhoneUnitTests, Three20
CATCH
GHUnit
Google Toolbox for Mac: iPhone Unit Testing
RSpec Style
Kiwi (which also comes with mocking & expectations)
Cedar
Jasmine with UI Automation as shown in dexterous' iOS-Acceptance-Testing specs
Acceptance Testing
Selenium Style
UI Automation (works on device)
UI Automation Instruments Guide
UI Automation reference documentation
Tuneup js - cool library for using with UIAutomation.
Capturing User Interface Actions into Automation Scripts
It's possible to use Cucumber (written in JavaScript) to drive UI Automation. This would be a great open-source project. Then, we could write Gherkin to run UI Automation testing. For now, I'll just write Gherkin as comments.
UPDATE: Zucchini Framework seems to blend Cucumber & UI Automation! :)
Old Blog Posts:
Alex Vollmer's UI Automation tutorial
O'Reilly Answers UI Automation tutorial
Adi Saxena's UI Automation tutorial
UISpec with UISpecRunner
UISpec is open source on Google Code.
UISpec has comprehensive documentation.
FoneMonkey
Cucumber Style
Frank and iCuke (based on the Cucumber meets iPhone talk)
The Frank Google Group has much more activity than the iCuke Google Group.
Frank runs on both device and simulator, while iCuke only runs in simulator.
Frank seems to have a more comprehensive set of step definitions than iCuke's step definitions. And, Frank also has a step definition compendium on their wiki.
I proposed that we merge iCuke & Frank (similar to how Merb & Rails merged) since they have the same common goal: Cucumber for iOS.
KIF (Keep It Functional) by Square
Zucchini Framework uses Cucumber syntax for writing tests and uses CoffeeScript for step definitions.
Additions
OCMock for mocking
OCHamcrest and/or Expecta for expectations
Conclusion
Well, obviously, there's no right answer to this question, but here's what I'm choosing to go with currently:
For unit testing, I used to use OCUnit/SenTestingKit in XCode 4. It's simple & solid. But, I prefer the language of BDD over TDD (Why is RSpec better than Test::Unit?) because our words create our world. So now, I use Kiwi with ARC & Kiwi code completion/autocompletion. I prefer Kiwi over Cedar because it's built on top of OCUnit and comes with RSpec-style matchers & mocks/stubs. UPDATE: I'm now looking into OCMock because, currently, Kiwi doesn't support stubbing toll-free bridged objects.
For acceptance testing, I use UI Automation because it's awesome. It lets you record each test case, making writing tests automatic. Also, Apple develops it, and so it has a promising future. It also works on the device and from Instruments, which allows for other cool features, like showing memory leaks. Unfortunately, with UI Automation, I don't know how to run Objective-C code, but with Frank & iCuke you can. So, I'll just test the lower-level Objective-C stuff with unit tests, or create UIButtons only for the TEST build configuration, which when clicked, will run Objective-C code.
Which solutions do you use?
Related Questions
Is there a BDD solution that presently works well with iOS4 and Xcode4?
SenTestingKit (integrated with XCode) versus GHUnit on XCode 4 for Unit Testing?
Testing asynchronous code on iOS with OCunit
SenTestingKit in Xcode 4: Asynchronous testing?
How does unit testing on the iPhone work?
tl;dr
At Pivotal we wrote Cedar because we use and love Rspec on our Ruby projects. Cedar isn't meant to replace or compete with OCUnit; it's meant to bring the possibility of BDD-style testing to Objective C, just as Rspec pioneered BDD-style testing in Ruby, but hasn't eliminated Test::Unit. Choosing one or the other is largely a matter of style preferences.
In some cases we designed Cedar to overcome some shortcomings in the way OCUnit works for us. Specifically, we wanted to be able to use the debugger in tests, to run tests from the command line and in CI builds, and get useful text output of test results. These things may be more or less useful to you.
Long answer
Deciding between two testing frameworks like Cedar and OCUnit (for example) comes down to two things: preferred style, and ease of use. I'll start with the style, because that's simply a matter of opinion and preference; ease of use tends to be a set of tradeoffs.
Style considerations transcend what technology or language you use. xUnit-style unit testing has been around for far longer than BDD-style testing, but the latter has rapidly gained in popularity, largely due to Rspec.
The primary advantage of xUnit-style testing is its simplicity, and wide adoption (amongst developers who write unit tests); nearly any language you could consider writing code in has an xUnit-style framework available.
BDD-style frameworks tend to have two main differences when compared to xUnit-style: how you structure the test (or specs), and the syntax for writing your assertions. For me, the structural difference is the main differentiator. xUnit tests are one-dimensional, with one setUp method for all tests in a given test class. The classes that we test, however, aren't one-dimensional; we often need to test actions in several different, potentially conflicting, contexts. For example, consider a simple ShoppingCart class, with an addItem: method (for the purposes of this answer I'll use Objective C syntax). The behavior of this method may differ when the cart is empty compared to when the cart contains other items; it may differ if the user has entered a discount code; it may differ if the specified item can't be shipped by the selected shipping method; etc. As these possible conditions intersect with one another you end up with a geometrically increasing number of possible contexts; in xUnit-style testing this often leads to a lot of methods with names like testAddItemWhenCartIsEmptyAndNoDiscountCodeAndShippingMethodApplies. The structure of BDD-style frameworks allows you to organize these conditions individually, which I find makes it easier to make sure I cover all cases, as well as easier to find, change, or add individual conditions. As an example, using Cedar syntax, the method above would look like this:
describe(#"ShoppingCart", ^{
describe(#"addItem:", ^{
describe(#"when the cart is empty", ^{
describe(#"with no discount code", ^{
describe(#"when the shipping method applies to the item", ^{
it(#"should add the item to the cart", ^{
...
});
it(#"should add the full price of the item to the overall price", ^{
...
});
});
describe(#"when the shipping method does not apply to the item", ^{
...
});
});
describe(#"with a discount code", ^{
...
});
});
describe(#"when the cart contains other items, ^{
...
});
});
});
In some cases you'll find contexts in that contain the same sets of assertions, which you can DRY up using shared example contexts.
The second main difference between BDD-style frameworks and xUnit-style frameworks, assertion (or "matcher") syntax, simply makes the style of the specs somewhat nicer; some people really like it, others don't.
That leads to the question of ease of use. In this case, each framework has its pros and cons:
OCUnit has been around much longer than Cedar, and is integrated directly into Xcode. This means it's simple to make a new test target, and, most of the time, getting tests up and running "just works." On the other hand, we found that in some cases, such as running on an iOS device, getting OCUnit tests to work was nigh impossible. Setting up Cedar specs takes some more work than OCUnit tests, since you have get the library and link against it yourself (never a trivial task in Xcode). We're working on making setup easier, and any suggestions are more than welcome.
OCUnit runs tests as part of the build. This means you don't need to run an executable to make your tests run; if any tests fail, your build fails. This makes the process of running tests one step simpler, and test output goes directly into your build output window which makes it easy to see. We chose to have Cedar specs build into an executable which you run separately for a few reasons:
We wanted to be able to use the debugger. You run Cedar specs just like you would run any other executable, so you can use the debugger in the same way.
We wanted easy console logging in tests. You can use NSLog() in OCUnit tests, but the output goes into the build window where you have to unfold the build step in order to read it.
We wanted easy to read test reporting, both on the command line and in Xcode. OCUnit results appear nicely in the build window in Xcode, but building from the command line (or as part of a CI process) results in test output intermingled with lots and lots of other build output. With separate build and run phases Cedar separates the output so the test output is easy to find. The default Cedar test runner copies the standard style of printing "." for each passing spec, "F" for failing specs, etc. Cedar also has the ability to use custom reporter objects, so you can have it output results any way you like, with a little effort.
OCUnit is the official unit testing framework for Objective C, and is supported by Apple. Apple has basically limitless resources, so if they want something done it will get done. And, after all, this is Apple's sandbox we're playing in. The flip side of that coin, however, is that Apple receives on the order of a bajillion support requests and bug reports each day. They're remarkably good about handling them all, but they may not be able to handle issues you report immediately, or at all. Cedar is much newer and less baked than OCUnit, but if you have questions or problems or suggestions send a message to the Cedar mailing list (cedar-discuss#googlegroups.com) and we'll do what we can to help you out. Also, feel free to fork the code from Github (github.com/pivotal/cedar) and add whatever you think is missing. We make our testing frameworks open source for a reason.
Running OCUnit tests on iOS devices can be difficult. Honestly, I haven't tried this for quite some time, so it may have gotten easier, but the last time I tried I simply couldn't get OCUnit tests for any UIKit functionality to work. When we wrote Cedar we made sure that we could test UIKit-dependent code both on the simulator and on devices.
Finally, we wrote Cedar for unit testing, which means it's not really comparable with projects like UISpec. It's been quite a while since I tried using UISpec, but I understood it to be focused primarily on programmatically driving the UI on an iOS device. We specifically decided not to try to have Cedar support these types of specs, since Apple was (at the time) about to announce UIAutomation.
I'm going to have to toss Frank into the acceptance testing mix. This is a fairly new addition but has worked excellent for me so far. Also, it is actually being actively worked on, unlike icuke and the others.
For test driven development, I like to use GHUnit, its a breeze to set up, and works great for debugging too.
Great List!
I found another interesting solution for UI testing iOS applications.
Zucchini Framework
It is based on UIAutomation.
The framework let you write screen centric scenarios in Cucumber like style.
The scenarios can be executed in Simulator and on device from a console (it is CI friendly).
The assertions are screenshot based. Sounds inflexible, but it gets you nice HTML report, with highlighted screen comparison and you can provide masks which define the regions you want to have pixel exact assertion.
Each screen has to be described in CoffeScript and the tool it self is written in ruby.
It is kind of polyglott nightmare, but the tool provides a nice abstraction for UIAutomation and when the screens are described it is manageable even for QA person.
I would choose iCuke for acceptance tests and Cedar for unit tests. UIAutomation is a step in the right direction for Apple, but the tools need better support for continuous integration; automatically running UIAutomation tests with Instruments is currently not possible, for example.
GHUnit is good for unit tests; for integration tests, I've used UISpec with some success (github fork here: https://github.com/drync/UISpec), but am looking forward to trying iCuke, since it promises to be a lightweight setup, and you can use the rails testing goodness, like RSpec and Cucumber.
I currently use specta for rspec like setups and it's partner (as mentioned above) expecta which has ton's of awesome matching options.
I happen to really like OCDSpec2 but I'm biased, I wrote OCDSpec and contribute to the second.
It's very fast even on iOS, in part because it's built from the ground up rather than being put on top of OCUnit. It has an RSpec/Jasmine syntax as well.
https://github.com/ericmeyer/ocdspec2
I would like to start a new project which will make extensive use of plugins. I know that both Eclipse and Netbeans have their respective Rich Client Platforms, both with their respective strengths and weaknesses.
I would like some comments on which the Stack Overflow community prefers.
Also, and most importantly, how easy it is with the respective platforms to write plugins for already existing applications. For example, if I finish my application, and would like to enable 3rd parties to extend it with their own plugins, how does each of these platforms provide functionality for this? Would they need my source to do it, or do these platforms provide plugin APIs towards which 3rd parties can code?
I would like some comments on which the Stack Overflow community prefers.
I lean my preference to Eclipse RCP, mostly because I still think that the IDE itself is the best right now. Eclipse RCP is also more mature, and has more books and documentations on the web. Netbeans RCP is slightly behind with only three books I can find on Amazon.com regarding the platform.
I'm also very eager to see the platform growing with its e4 projects which will simplify a lot of things (from dependency injection to UI customization)
Also, and most importantly, how easy it is with the respective platforms to write plugins for already existing applications. For example, if I finish my application, and would like to enable 3rd parties to extend it with their own plugins, how does each of these platforms provide functionality for this? Would they need my source to do it, or do these platforms provide plugin APIs towards which 3rd parties can code?
I can imagine that the answer for this question will not really be satisfying while both platform are designed to be extensible. They are pretty equals in this department.
The most important thing is to design your application to also be extensible. That is, providing extension points. You don't have to provide source code for that but you can document the extension points. Also, in case you need to provide interfaces, you can just provide the javadoc without the real source.
I repeat my point, designing application using RCP doesn't mean your application will automatically be extensible. You have to also design your application to be so. This won't come easily as you learn the platform for the first time but you will eventually learn about it from experience.
(Note: I'm not speaking for the community here ;) )
The two major differences between Netbeans and Eclipse RCP are:
OSGi (Netbeans 6.9 is only beginning to support it): it is designed to support extensions
Swing vs. SWT (can you app benefit from the native look supported by SWT)
Another factor, as you can read in this blog post is RAP (Rich Ajax Platform), which could help deploy your app on many platform based on one source code. But that may not concern you.
The maven integration is quite good from both side (see the Netbeans-RCP-Maven Hello World article), but the Maven3-Tycho integration is primarily developed on Eclipse.
Again, you might not need those new maven features (or you could use other build management system entirely, like Ivy or Graddle)
Consider if your plugins does really need such complex architecture as eclipse RCP or NetBeans provide. Plugins for RCP or NetBeans can be only created by Java Developers with strong RCP/NetBeans experience. Be aware that you can do (very easy) RCP application that is not pluggable at all. In some cases is better to write your own, maybe less flexible but more friendly way to attach new plugins. You can even do some wizards for them.
I recently added GWT to our project to implement an Ajax feature of our web app. The other devs are complaining about the extra time GWT compile adds to the build and are asking why I didn't use JSON and jQuery instead. What should I tell them?
Try to make the build smarter, if it isn't already: The GWT (client) part should only be re-compiled, when the client source changes. I assume, it's mostly you who changes that source, so the other developers won't experience the pain.
Caveat: This doesn't work of course, if your client source shares code with the existing project (I assume, it's a Java project on the server side?). But maybe you should avoid shared code in your case: Even though that's violating the DRY (Don't Repeat Yourself) principle, realize that you'd violate it anyway, if you didn't use GWT.
However, if you do reuse code from the server project, then you have a good argument, why you used GWT.
If developers have to compile the whole GWT stuff (all the permutations) in order to develop application it is real pain. Starting from GWT 2 you can configure the webapp project to be run in "development mode". It can be started directly from eclipse (Google plugin) thanks to built in jetty container. In such scenario only requested resources are compiled, and the process is incremental. I find this very convenient - GWT compilation overhead in our seam+richfaces+GWT application is very small during development cycle.
When it comes to application builds there are several options which can speed up GWT compilation. Here is checklist:
disable soyc reports
enable draftCompile flag which skips some optimization
by adjusting localWorkers flag you can speed things a bit when building on multi-core CPU
compile limited set of permutation: e.g. only for browsers used during development and only for one language
Release builds of the webapp should have draftCompile disabled though. Also all the laguage variants should be enabled. Maven profiles are very useful for parametrization of builds.
What was the reason you used GWT instead of JSON/jQuery?
I would ask the same question since for what you need, GWT may not be legitimately needed.
In my experience, I totally understand the complaints you are getting. GWT is a wonderful technology, and it has many benefits. It also has downsides and one of them is long compile time. The GWT compiler does lots of static code analysis and it's not something that has an order-of-magnitude solution.
As a developer, the most frustrating thing in the world is long development-deploy-test cycles. I know how your developers feel.
You need to make an architectural decision if the technological benefits of GWT are worth it. If they are, your developers will need to get used to the technology, and there are many solutions which can make the development much easier.
If there was a good reason for using GWT instead of pure javascript, you should tell them this reason (skills, debugging for a very hard to implement problem, you didn't want to deal with browser compatibility, etc). If there is no good reason, maybe they're right to be upset.
I use GWT myself and I know about this compile time :-)
If you used GWT for an easy-to-implement-in-javascript widget or something like that, maybe you should have consider using javascript instead.
What tool you're using to compile project?
Long time ago I've used ant and it was smart enough to find out that, when none of source files for GWT app (client code) has changed, the GWT compiler task was not called.
However, after that I've used maven and it was real pain, because it's plugin didn't recognize the code hasn't changed and GWT compilation was run all and over, no matter if it was needed or not.
I would recommend ant for GWT projects. Alternative would be rewriting the maven plugin or getting developers used to long compile time.
I need to write a GUI for a Java desktop application and I want to use something more / better than Swing. What is the learning curve for the NetBeans RCP like? The learning curve for Eclipse RCP is so bad that it's not worth considering:
"Right now, after one week of training, a RCP newbie just touched the very basic things of RCP. This is one reason why many big industry companies have problems adopting RCP. "
http://wiki.eclipse.org/E4/RCP_Future
Is the learning curve for NetBeans RCP considerably less?
Thanks.
Dean
We've had a summer intern working on an application based on the Netbeans RCP and it's been going really well. Because it's mainly swing, lots of your existing knowledge is directly applicable. You can still use tools such as the Matisse GUI builder and just need to learn a few patterns to get lots of benefit from the RCP.
The book The definitive guide to the Netbeans Platform is an excellent reference/tutorial and is reasonably up to date.
It's also worth checking out Geertjan's Blog - he's the technical author of the Rich Client Platform documentation and his regular blog posts form a Netbeans RCP cookbook.
I spent about four days writing a swing application and was able to port it into a NetBeans Platform app in about three days. I've done some Eclipse RCP and I can say it is much easier using the NetBeans Platform. I'm still new at NetBeans Platform but i'm pretty sure at this point if i were to write a complex application wo putting too much time into it I'd start with the NetBeans platform first. I was also using Heiko Bock's "The Definitive Guide to NetBeans Platform 7"
I doubt that the inherent complexity is dramatically different between any GUI libraries of substantial size. Since Netbeans RCP is actually Swing-based (with a better platform-look-and-feel on Windows, I think), the learning curve might be somewhat flatter for you, since you already know Swing.
"Right now, after one week of training, a RCP newbie just touched the very basic things of RCP. This is one reason why many big industry companies have problems adopting RCP. "
This may or may not be correct, but consider that in some cases (i.e. no fancy graphics/canvas/3D/bitmap manipulation), just plain old dialogs/forms go a long way. The critical parts to master are (in my opinion) the list/tree-models and the general data-binding mechanisms and to find, learn and stick to a layout manager that works (MigLayout is superb and has backends for Swing and SWT).
I am about to start on a college project (a web application) and I have never used a full-fledged IDE such as Eclipse.
Turbo C/C++, Visual Basic 6,Java Basic, a bit of SQL, ASP, etc is the sort of exposure I have.
What things should I keep in mind before starting my project using Eclipse? Are the tools mentioned appropriate for the project?(If not please give a detailed answer)
Designing - UML (Rational Rose)
Language - Google Web Toolkit
Server - Google App Engine
IDE - Eclipse
Version Control - Subversion or Mercurial?
I would definitely recommend Googling first. There are a number of tutorials regarding Eclipse as it is a very popular IDE. A quick Google search of my own brought up all these results: developing web applications in eclipse
As for other things to consider, if you are developing a web application, you'll need:
web server (Tomcat is a popular one)
possibly a database (MySQL is an open source, easy to use DB)
language (I'm assuming you're going with Java since you are using Eclipse)
Of course, you'll also need to consider how you hook everything together and what technologies you want to use to do that. (Hibernate, Spring, etc) Eclipse itself has a ton of plugins to help bring together all these various aspects.
That list from Wikipedia is a good and comprehensive list, but if you are learning or developing on your own machine, you may not necessarily need all of that.
Hope that helps.
Since it is a web application, then start with the looks of it.
First, layout a template of how your pages should look, that is what users see. If it does not look good, the users will think the application (in its entirety) is also not good (it does not matter if the code behind the view is perfect; it will just make a first bad impression).
Start with that and be consistent with the design in all your pages. SiteMesh is a nice tool to dissociate the looks of the page from the functionality you put in it.
Then think what the application will do and what it will use:
you have a database? (use something like MySQL). With what are you going to access it? (IBatis is nice; Hibernate I think is a bit heavy weight for a first project)
you need a server: Tomcat is easy to use;
are you going for a simple Servlet/JSP approach or you want to use a framework (look at Spring or Struts);
try to find the good ways of writing the application, look at service layers, DAO pattern, DTO, MVC. Also, you must understand how HTTP works.
A lot more could be said.
Ah.. and also use a source repository. It’s a must (even if you work alone on this project).
Eclipse can handle all of these tools, but then so can Netbeans.
For your first project with GWT you should read through this tutorial:
http://code.google.com/webtoolkit/tutorials/1.6/gettingstarted.html
If you don't know Java then you will have some learning to do, and unlike Visual Studio your UI won't be just drag and drop, so it will be a bit harder than you are used to.
Subversion is fine, it is a nice source control, and any IDE will work with it.
Depending on your project would determine if GWT is the best choice though.
I expect UML may be overkill, and if you were following an agile methodology you wouldn't use it.
Your best bet is to get the UI done first, just have it appear as you want, and have some fake results, until you are happy with the look and feel.
Then, start to do the wiring to whatever you need on the backend.
Don't mean to be harping on you, but is Eclipse a must? For myself, in the beginning NetBeans turned out to be a really painless introduction to getting a web project up a running fast. I believe in the beginning one will spend a lot less time fighting the IDE with NetBeans.
A lot of the items from your bullet list NB makes super easy to ramp up as well. Just my $0.2