Related
We are planing to generate JBehave stories dynamically by entering the when then commands (?) in a simple web form. I am not sure, if I like that idea.
I mean, I could programatically save the .story file before starting the test and point to this file in an overwritten StoryPathResolver.resolve method.
But, do you think that this would make a lot sense?
Thanks
I'm not sure if I understood your planned feature, but based on my assumptions *), I say: no, it does not make sense for the following reasons.
My assumption on this here is that you want to perform (test) the system on-the-fly and/or you don't know how beforehand what should be tested.
But either way, whatever is typed in wont be reproducible for later regression tests nor any other typical benefit that comes with automated tests would apply, first and foremost: speed.
Instead of implementing such a framework that would present you the options available at each step as well as the stop-and-go system that would be needed, it would be way better to optimize the path of writing the .story files and inject them into a running system. This way the writer still can take any time to specify the examples as well as having them reproducible from the start.
*) Should I be wrong, please rephrase your question. There are a few other questions in your first post that I could deduct.
I'm a big believer in testing, but not a very good practitioner. I've done pretty well at getting coverage on my model objects and programming them in a TDD style. I'm actually enjoying it so much I'd love to extend this to my controller layer, particularly my UIViewController subclasses.
Unfortunately, many UIKit classes don't function in independent tests. However, I'm unhappy with the restriction of only running my dependent tests on the device. It's really important to me to run all unit tests before every build, and it seems to me like its possible and worthwhile to unit test (as opposed to other types of testing) controller code.
My question is simply this: How do I test UIViewControllers in such a way that the tests run before every build? I am aware of a couple of different solutions to this problem, but don't know a whole lot about the various benefits of each one.
In general if you are stuck with an environmental problem where unconditional testing seems to be made impossible, you can work around it if the benefits are great enough to warrant your jumping through some hoops.
The situation here is a special case of unwanted outside dependencies, which I generally solve in my unit tests by either #ifdefing out tiny bits of dependent code, or by implementing stub classes to fill the expected dependency roles enough that my code can be tested.
So in this specific case, you might create a new source file, linked only in the test bundle, called "UIKitStubClasses.m" … within it you could implement the bare necessities to simulate UIKit dependent classes such as UIViewController, so that your tests link and thoroughly exercise their own logic.
The important thing to remember is this is usually not necessarily all that much work. The tests will let you know what you need to implement in the stub, for example by issuing exceptions about unimplemented methods. You just add what you need to quiet the errors and test your code, and then your stub class is as sufficient for testing as any of the legitimate system framework classes would be.
Can you not use the tips/methods in Chris Hanson's blog here: http://chanson.livejournal.com/120263.html
It seems that you would probably need to run your app as the test harness, but it seems doable. Just because you are running the 'app' doesn't mean it necessarily has to do the normal thing your application does. Alternatively, you could create another application target that is just a simple dummy application that will load your test bundles and run them.
I agree it might not be perfect, but it sounds like it might work, and it seems like it could be automated.
So I was unhappy with this situation as it stood, and both Daniel and Ed's answers spurred me on to improve things further. I decided to take matters into my own hands.
What I ended up writing was a small Cocoa Touch application that in -applicationDidFinishLaunching scans the class hierarchy for any SenTestCase subclasses and runs them. Unlike GoogleToolboxForMac's unit testing stuff, I tried to use as much of the built in SenTesting machinery as possible and as a result it didn't really end up as that much code. I haven't done too much testing beyond verifying that UILabel allocates without crashing the test rig (otest does in fact crash if you try), but it should work.
I've put the source online on both Bitbucket and Github.
While apple's tools have improved over time, they still leave much to be desired. GHUnit is much better, has a real test runner, and makes is dead simple to set breakpoints, etc. You can even leverage all your existing SenTests.
Get it on github.
You can instantiate all your UIKit code with no worries. I use it to test UIViewControllers, assert outlets, and do behavior testing.
I have this grand idea to basically employ some brute force attack to test/verify that my web application doesn't crash.
Don't get me started on unit testing, and IoC stuff, this is something else entirely.
What I'm doing, and what I'm asking for help with is to create an intelligent exhaustive search, that explore parts of the program state.
What I have is a web page with things I can do, clicking is one thing, text input is another, some inputs like radio buttons and drop down lists are constrained to certain values. Pretty basic things. What I end up with a finite set of events and values and what I want to model is a progression of state. Maybe this is FSM optimization in a way, but the goal is to systematically go through arbitrary permutations of events and values and see what happens.
When a problem is found I want to try and provoke that error with as little effort as possible to be able to present a clear test case.
This relates to formal verification methods and I'm asking for help or insight from people with experience.
What you want to do sounds a little like model-checking, on the one hand, and automated test case generation on the other hand (in the latter category check out Concolic testing, a technique to avoid wasting time with unfeasible execution paths).
Model-checking would be the preferred method if you assume your web application is correct and want to prove that it is. But in the case of a warning, you may have to work to understand if the problem is real or not. Test case generation is oriented towards bug-finding: it does not prove that you app is correct, but if it finds a problem, it gives you an input vector to produce it so you don't need to wonder if the problem is real.
I am not aware of any existing tools for web apps, but that doesn't mean that they don't exist.
It sounds like you want a fuzzer. Peach is one such tool.
Exhaustive search can be non trivial task for limited resource (memory,space) ,but with many techniques the problem can be reduced ,like abstracting you code (ex: replacing database driver classes with stubs), an experience is presented in this paper: Abstract Model Checking of Web Applications Using Java PathFinder (Vinh Cuong Tran, Yoshinori Tanabe, Masami Hagiya, University of Tokyo).
If you look to a kind of formal verification of FSM like models, Java PathFinder has an extension to verify UML state charts written in Java+annotation (it depends on the Javapathfinder VM):
http://babelfish.arc.nasa.gov/trac/jpf/wiki/projects/jpf-statechart
I recently wrote a small tool to generate a class for each tier I hand write for the boring "forms over data" work where I spend almost 90% of my time (depressing I know) ... more on this as the economy improves ;)
My question is this - will using this tool instead of hand typing all this code from day to day actually hurt me as a developer? I feel like I will always be making changes to this tool and thus I "should" stay on top of the patterns used/ choices made/ etc... but some small part of me feels like I might lose my edge ... am I wrong?
If the tool can spit the code out without thought, then it probably saves you lots of thoughtless typing.
Writing the tool in the first place requires thinking, so I'd guess you'd be more "on the edge" maintaining and writing the tool.
That's good! Of course writing a tool to do all the job for you is impossible and wrong.
But automating repeatable tasks is always good - and sometimes writing specific types of code is repeatable.
It is even encouraged in the "Pragmatic Programmer" book.
Make sure that in the source control you have checked in a code generator and not its output (unless you have to modify the code later by hand)!
You are most definitely not wrong. I use code generators anywhere I can - I currently use CodeSmith to create my DAO's by looking at the database.
What edge are you afraid of losing? In my mind going to code generation is actually giving you an edge.
Larry Wall (of Perl fame) describes the three cardinal virtues of programming as Laziness, Impatience, and Hubris.
Congratulations! You have shown good laziness, in that you have identified some work you can pass off to an automated process and done so. (Bad laziness leads to cutting corners, procrastination, and generally postponing rather than eliminating work.) If you can successfully palm off some work onto another program, you are spending less time on annoying triviality and more on accomplishing things and learning.
Generate what you can. Code generation is one of the best tools I've picked up over the last 2 or 3 years. Typing the same code over and over (or copy and pasting it) is prone to error.
Spending less time doing something by having something/someone else do it, and more time researching better ways to do it will generally lead to doing it in a better way.
This doesn't have to just apply to programming....
Your code generator (at least in principle - I haven't looked at it myself) is The Right Thing, at least as far as it goes.
The next step would be to see whether you can, instead of generating all this redundant code, create a base class whose functionality matches the generated code and then derive your application code from it. Using inheritance rather than generation will allow you to benefit from improvements without needing to re-run the generator on all your projects. Perhaps more importantly, if you customize the generated code, the customizations would be lost if you re-run the generator, but customizations in a derived class will be preserved when the base class is changed.
No. Why do you think IDE's are so popular. Imagine if all the people who use Visual Studio had to programmatically create the GUI's without help from the IDE, it'd be terrible. I would be willing to bet most people who use VisualStudio won't know how to manualy create the forms they're creating in the IDE. But there's nothing wrong with that.
I believe in code generation wherever possible to remove the rote tasks of programming. You will not lose your edge, you will probably become a better programmer because you will spend more time working on the important and interesting stuff.
BTW, your tool sounds interesting. Have you released it anywhere?
Code generation is fine as long as you understand what you are generating. Physicists use calculators because they understand the formulas they are automating and realize that their precious time is better spent on important tasks.
Code generation is one of those invaluable DO:s that The Pragmatic Programmer advocates. I truly recommend that book. Here's a Pragmatic Programmer quick ref.
Its almost hypocritical not to code generate. Here we are automating all of these tasks that were traditionally done by hand... and yet many of us still hand crank all of our code, even if it can be easily generated.
My only experience with code generation is the macros of Common Lisp. They are used all the time. Everything that automats repetitive tasks is beneficial; that is what programming is about.
Read the story of Mac.
Imagine that each time you made a change to the tool and regenerated your code, that you made that design change by hand on all of your modules.
Since I've started generating code and gotten up to speed, I've found that I rarely get bugs in the generated code.
I find that writing code gen does help me learn the nuances of good architecture. You start seeing common patterns as opposed to a narrow view of your design. That said, don't use code gen as a substitute for good object-oriented code, and don't love your code gen so much you ignore new technologies. For example, if you're in .NET and are writing code-gen for data access, you'd better have a good excuse for not using Linq to SQL or NHibernate. Similarly, Dynamic Data can help in many forms-on-data scenarios. So, my advice: spike new stuff and code gen as needed.
My 2cents on code gen is that it is also critical for use in refactoring. I have found that partial classes and a good file comparison utility (Araxis or BeyondCompare) are essential.
Keep your generated code in one file and the custom Tweaks you made for that class in another file.
This practice will allow you to make those comprehensive framework changes implemented quickly and will also help you move to a new paradigm while easily being able to save your custom logic.
CodeSmith FTW!
While build servers are great to make sure all your code compiles, it doesn't address the differences in signatures with your stored procs or the like. If you routinely run the code gen you can more easily identify when those changes occur. A unit test will tell you the SP is wrong, code gen will tell you how to make it right.
I am coding a client-server application using Eclipse's RCP.
We are having trouble testing the interaction between the two sides
as they both contain a lot of GUI and provide no command-line or other
remote API.
Got any ideas?
I have about 1.5 years worth of experience with the RCP framework, I really liked it. We simply JUnit for testing...
It's sort of cliche to say, but if it's not easy to test, maybe the design needs some refactoring?
Java and the RCP framework provide great facilities for keeping GUI code and logic code separate. We used the MVC pattern with the observer, observable constructs that are available in Java...
If you don't know about observer / observable construct that are in Java, I would HIGHLY recommend you take a look at this: http://www.javaworld.com/javaworld/jw-10-1996/jw-10-howto.html, you will use it all the time and your apps will be easier to test.
As a former Test & Commissioning manager, I would strongly argue for a test API. It does not remove the need for User Interface testing, but you will be able to add automated tests and non regression tests.
If it's absolutely impossible, I would setup a test proxy, where you will be able to:
Do nothing (transparent proxy). Your app should behave normally.
Spy / Log data traffic. Add a filter mechanism so you don't grab everything
Block specific messages. Your filter system is very useful here
Corrupt specific messages (this is more difficult)
If you need some sort of network testing:
Limit general throughput (some libraries do this)
Delay messages (same remark)
Change packet order (quite difficult)
Have you considered using a UI functional testing tool? You could check out HP's QuickTest Professional which covers a wide varieties of UI technologies.
we are developing one client-server based application using EJB(J2EE) technology, Eclips and MySQL(Database). pl suggest any open source testing tool for functional testing .
thanks
Hitesh Shah
Separate your client-server communication into a pure logic module (or package). Test this separately - either have a test server, or use mock objects.
Then, have your UI actions invoke the communications layer. Also, have a look at the command design pattern, using it may help you.