I have a standard set of scenarios (about 100 request messages) that need to run against a service.
I want to run this same set against a variety of configurations with possibly different expected outcomes.
Ideally I would be able to say...
Here's my scenarios and default configuration and default results.
Here are other configurations and different expected results. Best if I only need to specify results that differ from the default.
Can I do this in cucumber without having to repeat the requests for each configuration?
I know I won't get exactly what I want, but any way to approximate this?
I have a hard time to see that you are describing a problem Cucumber would be good at supporting.
Behaviour-Driven Development, BDD, is all about communication. Communicating how something should behave in such a way that it is
Understandable by non techies
Thought provoking so you explore and find unknown unknowns
Executable
What you are describing is something I personally would use any testing framework to solve. You wouldn't be using scenarios to discuss a desired behavior for a system. You would use the tool more like a test runner. And using Cucumber like this is possible, but other tooling would solve the problem just as good and with less work.
Related
I am working on JPA project and I want to have unit tests (although as a database is required, in this case it will be more as integration tests.)
What is the best way to test JPA project? jUnit can do that ? Is there other better way ?
Thank you very much
You have given limited information on the tools/frameworks you are using and a very general question, but I will give a quick answer on the points you raise. These are just pointers however as I believe you need to do a good bit more leg-work in order for you to figure out what is best for your particular project.
Junit allows you to target your class methods with specific parameters and to examine the return values. The returned values maybe an entity that should have certain field at certain values, a list of entities with certain expected field values, exceptions etc., etc. (Whatever you methods are). You can run your test as you introduce new functionality, and re-run them to test for regression as development proceeds. You can easily test edge cases and non-nominal stuff. Getting Junit up and running in Java SE/EE is quite straight forward so that could be a good option for you to get stick-in with testing. It is one of the quicker ways I use to test new functionality.
Spring/MVC – Using an MVC framework can certainly be useful. I have used JSF/Primefaces. But that is principally because the application was to be a JSF application and such development tests gave confidence that the ‘Model’ layer provided what was needed to the rest of the framework. So this provides some confidence in the model/JPA/DB layers (it is certainly nice to see the data that is delivered) but does not provide for flexible, nimble and targeted testing you might expect from Junit.
I think Dbunit might be something to look at when you’ve made some progress with JUnit.
See http://dbunit.sourceforge.net/
DbUnit is a JUnit extension (also usable with Ant) targeted at
database-driven projects that, among other things, puts your database
into a known state between test runs. This is an excellent way to
avoid the myriad of problems that can occur when one test case
corrupts the database and causes subsequent tests to fail or
exacerbate the damage.
We are planing to generate JBehave stories dynamically by entering the when then commands (?) in a simple web form. I am not sure, if I like that idea.
I mean, I could programatically save the .story file before starting the test and point to this file in an overwritten StoryPathResolver.resolve method.
But, do you think that this would make a lot sense?
Thanks
I'm not sure if I understood your planned feature, but based on my assumptions *), I say: no, it does not make sense for the following reasons.
My assumption on this here is that you want to perform (test) the system on-the-fly and/or you don't know how beforehand what should be tested.
But either way, whatever is typed in wont be reproducible for later regression tests nor any other typical benefit that comes with automated tests would apply, first and foremost: speed.
Instead of implementing such a framework that would present you the options available at each step as well as the stop-and-go system that would be needed, it would be way better to optimize the path of writing the .story files and inject them into a running system. This way the writer still can take any time to specify the examples as well as having them reproducible from the start.
*) Should I be wrong, please rephrase your question. There are a few other questions in your first post that I could deduct.
I am looking for a black-box type program/script/Selenium add-on/etc. that will generate different kinds of data, input it into a web form, hit SUBMIT, and test whether the correct response occurred.
I have found programs that would test for one kind of data or another, but I'm looking for a more comprehensive program that would test a wide variety of possible user inputs, so you can verify that the form is accepting or rejecting the inputs in accordance with program specifications.
For example, the program could verify that an error occurs when:
a number is input into a field that is supposed to be text
a required field is left blank
a SQL injection is present
(checks for other things I have not thought of)
Note 1: I started using Selenium IDE to insert random data, but realized I would have to write a great deal of loops of test cases to test a variety of combinations of different kinds of data. It seems like this would be a very common issue, so I suspected there must be some solution already out there. After much Googling, I could not find one.
Note 2: I realize this can be done via unit testing or other white box methods, but I am looking for a black box solution here.
Thanks!
Selenium IDE is great for prototyping your tests, but for anything past trivial solutions you really need to look to tools/frameworks like Watin or Webdriver (Selenium 2) to create useful tests which are maintainable over time. You should also have a look at Telerik's Test Studio (Disclosure: I'm their evangelist for that tool) which is another great functional test solution.
Regardless of which toolset you decide on, it sounds like you're looking for an approach to data drive your various tests: validating combinations of given inputs create expected output conditions. There are a number of posts and articles around on that. I have a blog post with a brief example of data driving an AJAX menu using C# and Webdriver which might be of help.
I usually made php forms and "try" to use "good practices" in them.
I'm concerned about the real safety and error-free of that forms and I want to do some tests simulating the customer behavior, and I do it manually, but I find that is a hard work specially when the form is large and I know that there are a lot of combinations that I can't test, so usually I find bugs in the production phase.
Is there a tool that do this? I listened about Selenium, did somebody use it in the way I need? Or how can I create my own test tools that simulate user inputs at random?
User inputs implies: not filling/checking all the fields, putting in invalid data, using differents setups (no javascript, browser versions, ...), SQL injections, and I don't know so more...
You'll need to consider a combination of approaches here: good test case design, data driving those tests with various input combinations, and an automation tool such as Selenium, WebDriver, Telerik's Test Studio (commercial tool I help promote), or some other automation tool.
Design your test cases such that you're focusing on groups of behavior (a successful path case, a case validating invalid input, a case validating protection against SQL injection, etc.). Then you can look to (perhaps) data drive those test cases with sets of inputs and expected results. You can randomize that as needed through your test case code.
Most good functional automation tools support multiple browsers running the same test script, so that's a good help for hitting multi-browser testing.
Above all, start your automation efforts with small steps and focus first on high-value tests. Don't spend time trying to automate everything because that costs you a lot of time.
Selenium is used to automate browsers in exactly the way you described.
Its used for what is called Functional Testing. Where you test the external aspects of an application to ensure that they meet the specifications.
Its is most often combined with unit tests that test the internal aspects. For example to test that your application is safe against different forms of SQL injection.
Each programming language usually has several different frameworks for writing unit tests.
This are often used together this with an approach called test driven development (TDD) where you write the tests before the application code.
I have this grand idea to basically employ some brute force attack to test/verify that my web application doesn't crash.
Don't get me started on unit testing, and IoC stuff, this is something else entirely.
What I'm doing, and what I'm asking for help with is to create an intelligent exhaustive search, that explore parts of the program state.
What I have is a web page with things I can do, clicking is one thing, text input is another, some inputs like radio buttons and drop down lists are constrained to certain values. Pretty basic things. What I end up with a finite set of events and values and what I want to model is a progression of state. Maybe this is FSM optimization in a way, but the goal is to systematically go through arbitrary permutations of events and values and see what happens.
When a problem is found I want to try and provoke that error with as little effort as possible to be able to present a clear test case.
This relates to formal verification methods and I'm asking for help or insight from people with experience.
What you want to do sounds a little like model-checking, on the one hand, and automated test case generation on the other hand (in the latter category check out Concolic testing, a technique to avoid wasting time with unfeasible execution paths).
Model-checking would be the preferred method if you assume your web application is correct and want to prove that it is. But in the case of a warning, you may have to work to understand if the problem is real or not. Test case generation is oriented towards bug-finding: it does not prove that you app is correct, but if it finds a problem, it gives you an input vector to produce it so you don't need to wonder if the problem is real.
I am not aware of any existing tools for web apps, but that doesn't mean that they don't exist.
It sounds like you want a fuzzer. Peach is one such tool.
Exhaustive search can be non trivial task for limited resource (memory,space) ,but with many techniques the problem can be reduced ,like abstracting you code (ex: replacing database driver classes with stubs), an experience is presented in this paper: Abstract Model Checking of Web Applications Using Java PathFinder (Vinh Cuong Tran, Yoshinori Tanabe, Masami Hagiya, University of Tokyo).
If you look to a kind of formal verification of FSM like models, Java PathFinder has an extension to verify UML state charts written in Java+annotation (it depends on the Javapathfinder VM):
http://babelfish.arc.nasa.gov/trac/jpf/wiki/projects/jpf-statechart