Would like to know if anyone has implemented a process on how to identify related test scripts for regression when a feature has changed.
Assuming that in BDD, one would group tests under a feature, I believe its not sufficient to run only the tests of that feature for Regression as there would be some other related feature that would have been impacted.
Technically is it possible for a tool like visual studio to guide on where the code in the Application has changed and recommend the tests from the regression pack ?
Related
As many of you are aware as of the release of MatLab 2013a, xUnit a popular unit testing framework for MatLab is canceling further development.
Is MatLab's new and native unit testing framework comparable to xUnit? what features is it lacking when compared to xUnit? Is it better or worse than xUnit?
MATLAB xUnit has been an excellent contribution to the test focused development efforts of those writing MATLAB code. It has a solid implementation, it follows the xUnit paradigm very well, and has been invaluable as a file exchange contribution.
The MATLAB Unit Test framework has indeed learned from this submission as well as decades of requirements and test focused development for the MathWorks' internal code base. We have also learned and extended upon frameworks in other languages such as JUnit, NUnit, and python's unittest framework. As such there certainly are many more features in the R2013a-beyond framework, and it is designed to scale and extend.
There are too many other features to go into in a simple answer, but perhaps one way to describe some of the differences are that the 13a framework is what I loosely call an "xUnit 2.0" and the file exchange submissions is an "xUnit 1.0" framework. If you are familair with JUnit, this is like the difference between JUnit 3 and JUnit 4.
There are also other intangible or as yet unrealized benefits, such as:
The framework is included directly in MATLAB so you can share tests with others and know that they can run the tests even if they are not familiar with testing and do not want to download the file exchange framework.
The framework is under active development with a pipeline of additional features and capabilities in the works for future releases.
Hope that helps. I would be happy to go over any questions you have about specific functionality or features.
I don't believe MathWorks are planning at all to stop making xUnit available, so you can continue using it if you like. xUnit had not seen any large changes for quite a while in any case, and even though it won't be developed further in terms of features, it may receive an occasional fix if any are needed.
I have tried out the new framework quite a bit, but have not used it on any large projects yet. Previously I have used xUnit on large projects. However, I'm no expert on unit testing - so please read the following opinions in that context.
I'm pretty sure there's nothing you can do in xUnit that you can't do in the new framework. In general it's much more flexible and powerful than xUnit, providing additional features and a better way to organise and structure your tests. It's a lot easier to set up and tear down suites of tests, to manage and close resources (files, figure windows, database connections etc), and to carry out tricky tests such as checking that the right number of arguments are returned.
However, whereas a typical xUnit test was implemented as a fairly simple MATLAB function, tests in the new framework are typically implemented (in 13a, but see below for 13b) as classes using MATLAB's OO syntax, and if you're not comfortable with that it may seem like a big leap.
I should also add that although the documentation for the testing framework is excellent as reference material, I haven't found it to be great as a tutorial.
In 13b, the need to use classes has been offset a bit with the introduction of the functiontests command, which creates a test suite for you from a file containing tests implemented as local functions. That will make things much easier if you're not comfortable with class syntax. But I would think that if you want to take advantage of everything, you'd probably still want to use the main framework.
Hope my experience is of help - if you're lucky, perhaps #AndyCampbell will chime in...
What would be the best means/tools for testing an Eclipse plugin? Are there some tools for testing the GUI features of an application created with Eclipse plugins?
You should try to keep GUI tests to a minimum since they are slow to run and takes time to create. If your code is well structured in a Model-view-controller pattern then the GUI specific code should be minimal.
Thats the theory in a perfect world atleast. Until we get there, I prefer to use SWTbot
Eclipse Jubula is a very nice tool for GUI based testing for Eclipse plugins. It provides automated functional GUI testing for various types of applications. It is aimed at teams who want their automated tests to be written by test experts from the user perspective, without requiring any coding effort. Jubula tests incorporate best practices from software development to ensure long-term maintainability of the automated tests.
It'll be useful to work with Jenkins as your CI build system.
You can connect to a real database as your wish for test result storage purposes.
You can save screenshots taken by Jubula upon test failures directly in the test report
I'd suggest you to take a look at RCP Testing Tool. This tool lets you to develop dozens of UI tests per day per engineer, and do not have stability and incorrect-recording problems. It's designed specially and only to test Eclipse-based apps. It is official Eclipse project and it's free.
After analyzing the possibilities, I have opted for using Jubula for testing the GUI part:
Eclipse Jubula Project
It's a good tool for creating tests specific for an RCP application. Moreover, it's not code based and it allows the creation of tests from a user point of view.
Few days back I went to an interview there they asked me What all modules you will test in Regression Testing? How you find out which test cases need to be executed in Regression testing?
Those modules that are having little or more modification in existing modules or code are required regression testing
The best way to do this is to have some insight into which test cases cover which parts of the product. Then when a part of the product changes, you can run just the cases that cover the change. This isn't always easy. In a complex piece of software, a change in one part can have an effect in a seemingly disconnected part.
The best solution I have seen to this problem is to use code coverage data. If you know which blocks are hit by each test and you know which blocks were changed by the fix, you can know exactly which test cases to run.
If you don't have a lot of data, your best bet is to think about the change and what things it could affect and then run cases that are in those areas.
I usually made php forms and "try" to use "good practices" in them.
I'm concerned about the real safety and error-free of that forms and I want to do some tests simulating the customer behavior, and I do it manually, but I find that is a hard work specially when the form is large and I know that there are a lot of combinations that I can't test, so usually I find bugs in the production phase.
Is there a tool that do this? I listened about Selenium, did somebody use it in the way I need? Or how can I create my own test tools that simulate user inputs at random?
User inputs implies: not filling/checking all the fields, putting in invalid data, using differents setups (no javascript, browser versions, ...), SQL injections, and I don't know so more...
You'll need to consider a combination of approaches here: good test case design, data driving those tests with various input combinations, and an automation tool such as Selenium, WebDriver, Telerik's Test Studio (commercial tool I help promote), or some other automation tool.
Design your test cases such that you're focusing on groups of behavior (a successful path case, a case validating invalid input, a case validating protection against SQL injection, etc.). Then you can look to (perhaps) data drive those test cases with sets of inputs and expected results. You can randomize that as needed through your test case code.
Most good functional automation tools support multiple browsers running the same test script, so that's a good help for hitting multi-browser testing.
Above all, start your automation efforts with small steps and focus first on high-value tests. Don't spend time trying to automate everything because that costs you a lot of time.
Selenium is used to automate browsers in exactly the way you described.
Its used for what is called Functional Testing. Where you test the external aspects of an application to ensure that they meet the specifications.
Its is most often combined with unit tests that test the internal aspects. For example to test that your application is safe against different forms of SQL injection.
Each programming language usually has several different frameworks for writing unit tests.
This are often used together this with an approach called test driven development (TDD) where you write the tests before the application code.
How can I create a wrapper around a testing framework? We still doesn't know which testing framework are going to use, but I need to start writing unit tests. With this question I want to know how can I switch from NUnit to mbUnit, xUnit or even MSTest.
You could create a wrapper - but I think you can utilise your time much better. I'd say pick the simplest one (My personal favourite would be the war-horse NUnit) that fits your needs - the newer frameworks add functionality that help you write more complex test fixtures.
However I value simplicity over "flexibility". In the future, if you find yourself wanting that "cool feature X in Y testing framework", you could either write that particular test fixture using Y. (you could also migrate the entire test fixture to use Y for consistency - but time is always scarce) - Switch between 2 unit testing framework is usually monotonous work (rename attributes) however some might be more work (disclaimer: no flying time with MbUnit)
Your comment however worries me a bit.
Why is the customer deciding the testing framework that you'd use for development - it should be a choice of the development team. The customer wouldn't want you to define product requirements - would he/she ? The quality of tests doesn't depend on the framework used so I don't see how this affects the customer.
You could use an existing wrapper that allows you to run multiple unit test frameworks, so even if you switch frameworks you can still use the old unit tests. For the unit test frameworks you listed, I would recommend taking a look at Gallio.
From http://www.gallio.org/...
At present Gallio can run tests from MbUnit versions 2 and 3, MSTest, NBehave, NUnit, xUnit.Net, csUnit, and RSpec. Gallio provides tool support and integration with AutoCAD, CCNet, MSBuild, NAnt, NCover, Pex, Powershell, Resharper, TestDriven.Net, TypeMock, and Visual Studio Team System.