As it currently stands, this question is not a good fit for our Q&A format. We expect answers to be supported by facts, references, or expertise, but this question will likely solicit debate, arguments, polling, or extended discussion. If you feel that this question can be improved and possibly reopened, visit the help center for guidance.
Closed 9 years ago.
I would like to learn more about how to use Selenium IDE and RC for creating good automation tests. Is there anybody who is interested in sharing info. or discussing this? Especially for things like:
1) What's a good way to organize the UI tests? Right now I am executing these tests thru NUnit and that executes the tests in the order of their alphabetical names. Is there a better way. (I am writing the selenium scripts in C# and not java)
2) How are the results logged?
3) Modularizing the Selenium RC tests? I am having a problem here. Since I am executing my tests thru NUnit, the only way I could figure was to give IDs to the tests so that they get run in the right order. Also I am having to write all my tests one after the other in the textfixture class. Again is there a better way? I tried creating additional files but when I create tests there and select them to run in NUnit, it looks like the selenium server doesn't even get started.
4) What are the best practices for automation testing? Any pointers to any sites/books/ etc.?
These might seem very simple things but I've spent weeks trying to come up with better ways, so if somebody is willing to offer suggestions or input I'll really appreciate it!
Thanks!
I have answered each of the questions for you below.
Tests shouldn't matter about the order that they run in. If you start worrying about that you are going to start introducing flakiness of tests. If you have to login with each test do so, if you need to register a number of users during your tests do so. Tests should always be able to run on their own.
Nunit logs its results in a XML file. This will have a list of the passes and fails and is normally rendered in a pretty fashion by nearly all Continuous
See my answer for Best Practices for modularizing Selenium RC test scripts
Tests should always have a known starting point. In the context of Selenium this could mean opening a certain page to start a workflow.
Tests should not have to rely on any other tests to run. If a test is going to add something do not have a separate test to delete it. This is to ensure that if something goes wrong in one test it will not mean you have a lot of unnecessary failures to check.
Tests should only test one thing at a time.
Tests should clean up after themselves.
If your desires is to use Selenium then I can suggest instead use the TestPlan front-end instead of another language. In additional to a domain specific language it offers nice test case management. You can organize your tests into a nice hierarchy and it will execute all of them and report the results.
It takes care of a lot of details of launching Selenium and smooths over many difficulties in the RC API.
I've just went through this. In terms of modularizing and useful tools. I think it's really important to think more about building an agile, well maintained automated ui testing framework than simply modularizing selenium RC scripts. Selenium 2: Testing Tools Begginner's Guide was a great start for idea how to start building a good testing framework. I had MUCH success with the (Page Object pattern)[https://code.google.com/p/selenium/wiki/PageObjects] after failing 2-3 times getting an automated testing framework going. In terms of proccess, I found it best to start by writing code (you are writing tests, so it should be pretty easy to test), but aggressively refactoring using good OOP as patterns and structures begin to build up. I'd also reccommend How Google Tests Software for motivation of Software QA in general, good testing practices on an organizational level, and when and when not to use automation.
Related
As many of you are aware as of the release of MatLab 2013a, xUnit a popular unit testing framework for MatLab is canceling further development.
Is MatLab's new and native unit testing framework comparable to xUnit? what features is it lacking when compared to xUnit? Is it better or worse than xUnit?
MATLAB xUnit has been an excellent contribution to the test focused development efforts of those writing MATLAB code. It has a solid implementation, it follows the xUnit paradigm very well, and has been invaluable as a file exchange contribution.
The MATLAB Unit Test framework has indeed learned from this submission as well as decades of requirements and test focused development for the MathWorks' internal code base. We have also learned and extended upon frameworks in other languages such as JUnit, NUnit, and python's unittest framework. As such there certainly are many more features in the R2013a-beyond framework, and it is designed to scale and extend.
There are too many other features to go into in a simple answer, but perhaps one way to describe some of the differences are that the 13a framework is what I loosely call an "xUnit 2.0" and the file exchange submissions is an "xUnit 1.0" framework. If you are familair with JUnit, this is like the difference between JUnit 3 and JUnit 4.
There are also other intangible or as yet unrealized benefits, such as:
The framework is included directly in MATLAB so you can share tests with others and know that they can run the tests even if they are not familiar with testing and do not want to download the file exchange framework.
The framework is under active development with a pipeline of additional features and capabilities in the works for future releases.
Hope that helps. I would be happy to go over any questions you have about specific functionality or features.
I don't believe MathWorks are planning at all to stop making xUnit available, so you can continue using it if you like. xUnit had not seen any large changes for quite a while in any case, and even though it won't be developed further in terms of features, it may receive an occasional fix if any are needed.
I have tried out the new framework quite a bit, but have not used it on any large projects yet. Previously I have used xUnit on large projects. However, I'm no expert on unit testing - so please read the following opinions in that context.
I'm pretty sure there's nothing you can do in xUnit that you can't do in the new framework. In general it's much more flexible and powerful than xUnit, providing additional features and a better way to organise and structure your tests. It's a lot easier to set up and tear down suites of tests, to manage and close resources (files, figure windows, database connections etc), and to carry out tricky tests such as checking that the right number of arguments are returned.
However, whereas a typical xUnit test was implemented as a fairly simple MATLAB function, tests in the new framework are typically implemented (in 13a, but see below for 13b) as classes using MATLAB's OO syntax, and if you're not comfortable with that it may seem like a big leap.
I should also add that although the documentation for the testing framework is excellent as reference material, I haven't found it to be great as a tutorial.
In 13b, the need to use classes has been offset a bit with the introduction of the functiontests command, which creates a test suite for you from a file containing tests implemented as local functions. That will make things much easier if you're not comfortable with class syntax. But I would think that if you want to take advantage of everything, you'd probably still want to use the main framework.
Hope my experience is of help - if you're lucky, perhaps #AndyCampbell will chime in...
As it currently stands, this question is not a good fit for our Q&A format. We expect answers to be supported by facts, references, or expertise, but this question will likely solicit debate, arguments, polling, or extended discussion. If you feel that this question can be improved and possibly reopened, visit the help center for guidance.
Closed 9 years ago.
I understand how to implement unit tests, I'm just struggling figure out when to use them.
Let's say I have a basic Reminders App. A user can add/edit/delete reminders and view them in a tableview. What parts of the app would I want to set up unit tests for?
Ideal world answer would say that every line of code you wrote should be unit tested.
But let's forget about that for a moment and move back to real world. Write tests for code that is important and having another line of defense is worth it. In other words, does it make much sense testing constructor that simply assigns value to one field? Most likely not. Is it worth to unit test parser extracting account data from complex XML your client provides? Probably yes.
Where does this difference comes from? Two major reasons:
it's less likely that constructor code will suffer from unpredictable changes (vs. much-evolving parser code to meet up with changing requirements/optimizations/refactorings)
constructor code is fairly simple and you've wrote such code many times already, tests might not offer you huge advantage in spotting issues; quick glance at such code and you'll be most likely able to tell what's going on (vs. complex XML parser code)
Why making the distinction? Why testing this and not that? Wouldn't it be easier to simply test everything (as ideal world answer would suggest)?
No. Because of time and money constraints. Writing code takes both. And there's only certain amount of money somebody is willing to pay for your product just as there's only certain amount of time he's going for wait for it to be delivered. Some tests are simply not worth it (again, constructor code example). Remember that unit tests are not immune to diminishing returns (having 80% code base covered with tests might take extra 20% development time and later save 20% time spent on debugging/maintenance, while going for another 10% might be twice as time consuming yet yield much lesser gains).
Again, you probably want to ask "Where's the line?" When do you decide "Ok, unit tests for this piece of code are not really needed"? Unfortunately, this kind of judgement comes with experience. Write code, read code, see what others (possibly more experienced developers) do and learn.
If I were to give couple of generic advises (what to unit test), those would be:
start with business/domain logic code
make sure to test all kind of converters/parsers/calculators (those are fairly easy to test, tend to change often [either due to changing requirements or refactorings] and by their nature are error prone)
avoid testing simple one-liner methods, unless that one line is crucial in some way
write tests for bugs that you discover (and keep them!)
don't follow magic fairy-tales of "good code must have 99.99% test coverage" blindly
reading questions on topic at programmers.stackexchange.com can often give you different perspective to approach problems
Test all the code you write. And if you want to be really cool, write the test first. If you have a method on a model or controller, you should also have a test for it.
Without knowing more about your code, its hard to advise. But it sounds like you would have a controller (like RemindersController) and a model (like Reminder). This would be a basic outline I would start with:
RemindersController
should add a new reminder
should update an existing reminder
should delete an existing reminder
Reminder
initWithMessage:atTime: should set a message
initWithMessage:atTime: should set a time
Assuming that you're storing your reminders somewhere, perhaps in the plist. You could write a unit test to generate a Reminder object, store it, retrieve the data, and finally generate a usable Reminder class object.
That way you know several things:
A: Your Reminder generation is working
B: Your method of storing the data is working
C: Going from Data to your Reminder object is working
However, you should not expect to be able to Unit test the actual "functionality" of your app. Such as touch events or navigation controls. These should be left to Acceptance testing which is an entirely different discussion.
I follow these principles in choosing what types of tests to write and when:
Focus on writing end-to-end tests. You cover more code per test than with unit tests and so get more testing bang for the buck. Make these the bread-and-butter automated validation of your system as a whole.
Drop down to writing unit tests around nuggets of complicated logic. Unit tests are worth their weight in situations where end-to-end tests would be difficult to debug or unwieldy to write for adequate code coverage.
Wait until the API you are testing against is stable to write either type of test. You want to avoid having to refactor both your implementation and your tests.
Rob Ashton has a fine article on this topic, which I drew heavily from to articulate the principles above.
As it currently stands, this question is not a good fit for our Q&A format. We expect answers to be supported by facts, references, or expertise, but this question will likely solicit debate, arguments, polling, or extended discussion. If you feel that this question can be improved and possibly reopened, visit the help center for guidance.
Closed 11 years ago.
What do you think about both?
I began reading a book about Catalyst, and found it pretty complex as compared to Dancer.
so now I'm giving Dancer a try, and it looks easier to learn and more "human friendly".
I think David's comment is very accurate and excellent. However, as someone who has done development in both but is not a developer on either perhaps I can be slightly more objective (and technical) in what the differences are.
Both frameworks provide a variation on the Web MVC paradigm.
Catalyst's main level of abstraction is the Controller. Catalyst expects you to break separate logic out into separate packages in some logical fashion (Login code goes here, Registration code goes there, Search functionality over here). This works incredibly well if you have a team of programmers since each of you can work on separate files and not step all over each other during merges. Catalyst provides a lot of tools for making the Controller logic extensible and flexible, I think the premier example of this is Chained actions which let you split up and build a complex flow for any given request. The downside is that it becomes very seductive to put your business logic into the Controllers and you end up with very fat logic in the Controllers where it (theoretically) belongs in the Model.
Dancer's main level of abstraction is the Route. My experience with Dancer is this leads to much smaller applications. Partly my experience here is tinged with the fact that I have dealt with several thousand line applications in Catalyst but I have yet to write a Dancer app that is longer than 200 lines (with a much smaller scope). I think however that this experience holds true. The push in Dancer is in keeping the Controller logic very thin because it doesn't have the same tools for managing complex behaviors there that Catalyst does.
Honestly I've enjoyed working in both of them. They both provide different opinions on what writing a web application is supposed to be. I would, given the time and inclination, recommend learning both ultimately.
This is a somewhat subjective question, but I'll try to give you an answer in an objective way. First things first, a disclaimer: I'm part of the Dancer development team, so my opinion should of course be considered somewhat biased :)
Catalyst is more widely used than Dancer, and so there's more community support behind it - if you were to look for contractors with experience working with either framework, say, you'd be more likely to find developers who've used Catalyst. So, if you're looking for commercial support, that would be a good reason to choose Catalyst.
Dancer is a younger project, and targeted more towards smaller projects, making getting up and running quick and easy, and trying to stay out of your way. That's not to say that Dancer isn't suitable for larger projects, however; the same habit of staying out of your way means you can organise your project in the way that suits you.
However, it has picked up a lot of support, and there's a growing community of helpful users and developers on IRC and the mailing list, and more and more useful plugins being released all the time. As with Catalyst, Dancer is designed so that you can pick and choose your preferred template engine, session storage backend etc, and it's easy to extend the framework by writing your own plugins if you need to.
For user testimonials to see what people say about Dancer, see the section at the bottom of the homepage on the new website: http://www.perldancer.org/
In the interests of showing other options, there's also Mojolicious, another modern Perl web framework which has been gaining in popularity lately.
Catalyst provides the same abstraction that Dancer does, Dancer's strength or rather Catalyst's weakness or rather Dancer's weakness is in how Catalyst forces the developer to adhere to Perl OO best practices and the MVC design pattern. After doing webapps for a while, this will all become apparent.
As it currently stands, this question is not a good fit for our Q&A format. We expect answers to be supported by facts, references, or expertise, but this question will likely solicit debate, arguments, polling, or extended discussion. If you feel that this question can be improved and possibly reopened, visit the help center for guidance.
Closed 12 years ago.
I know that Powershell is generally considered the IT management language for windows, but why/when would you choose F# instead, which is strongly typed and 'intellisensed' ?
I'd choose F# when it was a better fit for the company I was working for, the team who'll be using it, and the task at hand. Otherwise I'd choose something else.
I can't be more specific than that without more details, but I can suggest a few questions you should ask:
What is the problem you're solving?
In which language is it easier to express the solution to the problem?
Are existing staff F# developers themselves (i.e. consider the learning curve involved)?
Will the people maintaining this code be administrators (who are usually quite familiar with Powershell, & not at all familiar with F#)?
A related hint: don't allow a profusion of languages to develop in your production code and infrastructure.
By all means experiment to find the best language to solve the problem you're having, but if you find in five years time that someone has written a core component of your solution in a language that is no longer understood by anyone in the company, you may have a serious problem on your hands.
My personal rule of thumb (of course there are exceptions) is that a project should have a core language and a scripting language to hold it all together, & that those languages should be chosen first to fit the problem domain, and secondly to maximise proficiency across the team. 'Pet languages' should be strongly discouraged in production code or infrastructure.
It depends on your problem and on who will maintain the scripts.
PowerShell is very strong in administration areas (working with files, active directory, csv/xml, computers, etc.), but it's programming abilities are quite limited (e.g. case of generics). Threading in PowerShell is not as easy as it could be. But nobody expects from administrators that they will use threads. Instead there are background jobs that satifisfy administrator's needs.
On the other hand, F# is real programming language, which means it is quite low level and therefore verbose (compared to PowerShell). Some tasks written in PowerShell are one-liners, whereas in F# you would write a lot of code. Besides that - how would you achieve running command on remote computers (Invoke-Command -computer myserver ...)?
As you see, it depends on your problem.
Very important is also who will maintain the scripts. You might be on holiday and your colleagues will need to edit the scripts. Both languages need some time to learn. IMHO it is easier to learn PowerShell; thinking in functional style needs to bend your head.
It is probably a bad idea, at this point in time, to choose to write scripts in F# if they might be needed by anyone other than yourself ...
The cadre of F# speaking people is very small, so you're creating scripts which very few people can read or edit.
The fsi executable doesn't come with the .Net Framework, nor is it part of the F# runtime redistribution package (as far as I can tell). This means the scripts can only be executed where Visual Studio with F# tools have been installed.
The obvious exception, of course, is if you're a developer working on a project where F# is a primary language, since that will mean that everyone that matters will be able to run and edit your scripts.
However, I have to say that I think even in this ideal/exceptional case, it would probably be more work to use F# if your scripting task involves tasks for which there are pre-existing cmdlets in PowerShell (ie: filesystem tasks, registry, database maintenance, active directory, exchange, the list goes on and on). Of course, you might still choose to do it if you're using these scripts as an opportunity to learn the language, but bear in mind points 1 & 2 ;-)
A job came in to me that's built with CodeCharge - had a look at it and seems to be a pretty basic point-and-click site builder tool. Has anyone got any in-depth experience with it? My first reaction is one of horror and to just rebuild the code in Rails or PHP but I thought I'd ask the question first, maybe i'm missing something...
I'm currently evaluating it for use in quickly producing a back-office environment, and it seems a very comfortable way of getting an interface up and running quickly.
Once the code is generated it is simple, but easily modified for any special needs you have.
In short it is a very good tool (though it would seem that Iron Speed Designer is much better though much more expensive) for what it does - fast prototyping and almost no coding approach to developing a web application. In my opinion, not much different than a Ruby On Rails application in terms of functionality, and, I can generate the code in any language I want.
You have to realize, it is all about speed - some quality is thrown in, but it is a very generic of quality - this is NOT a custom application, mind you, the resulting code you get here might not be pretty but it is a few level higher than your average script kiddie code.
I'm seriously considering this tool for creating back-office applications for sites I develop - a fast and easy solution instead of mucking around in tables of data and useless and repetitive SQL code.
Codecharge is a powerful tool that I have used for over 10 years to build very large content management systems, CRMs and many other management type tools.
Its far from simple once you get into it, and honestly when you use a tool like Codecharge to cleanly generate your user interfaces, you end up with a healthier application that can last many many years.
For instance, I have three clients that have been running Codecharge created portals for over 10 years and they always comment how bug free they have been.
There is a learning curve to learning CodeCharge but it will also teach you what entire applications should have in place and it will please the executives every time because they can get functionality within hours or days rather than weeks or never.
Development teams will often not like it though, because they would rather hand code everything or use the latest and greatest approach to development.