I'm building tests for a suite of enterprise-ware. The good advice I had from the author of Test::WWW::Selenium was to build routines of larger functions and then parameterise them. So we have: add_user, post_blog and so on. These routines can then be enriched to check that the right text appears on pages and so on. This software is very configurable and we have tens of sites, all different. But these building blocks can be strung together and the driver data modified appropriately on a per-site basis.
Everything I've found out there on Selenium has been very beginner, nothing about building larger Test Suite. Is there anything beyond this or is this as good as it gets?
It is very much possible to run very large number of selenium tests. In my organization we run around 200,000 to 300,000 tests per day across multiple websites. So yes its possible.
Note:- I code in JAVA and all the info below is from a JAVA perspective.
For a large scale testing using selenium to be successful, I would say it needs 3 basic components
The infrastructure
A good framework and easily maintainable code
Easy and clear reporting
Infrastructure
Your infrastructure should be able to support the huge load. We use selenium grid (if you are using Selenium 2 then its called as Grid 2.0) to achieve this. Selenium Grid allows us to run multiple tests in parallel and across multiple browsers. We use our own servers in which virtual machines are deployed to support this testing. There are vendors like Saucelabs.com to whom you can outsource the infrastructure maintenance.
Framework and testcode
Your framework must support multithreading and it should be threadsafe to utilize the selenium grid features. We use JAVA to make sure this happens. To run the tests in parallel testNG is used. testNG can run multiple methods in parallel and all these tests will point to one single Hub. The hub would then distribute all these tests against multiple Remote Controls(RCs) connected to it.
When you have large set of tests, maintenance is inevitable. To reduce the rework effort due to a change in the application, its always better to follow the Page Object Model. Page object essentially means - each page in your application should have a corresponding class in your code. You will define all the elements and functions that can happen in that page in this class. This is a very modular and reusable structure. You can google for pageobject model and you will find many articles.
Reporting
Its great to run 200,000 tests per day, but if you cannot showcase what failed and what passed in an easily accessible method your automation tests will not be of much use. There are multiple methods for this. From HTML reporting to building custom dashboards to show the results.
Once you have all these in place, then the only thing you will need is a tool to keep these tests running continuously. You can use any CI (continuous integration) tool to achieve this. Jenkins , Hudson , cruisecontrol are few of the CI tools that are mostly used.
Related
We use a fairly standard CI deployment pipeline in Teamcity to package our application. We started out with the following pipeline. Each of the steps below represent a gate that the build must pass in order to advance to the next step:
Compile
Unit Tests
Back-end component integration tests
Front-end acceptance tests (Selenium based)
Package
This worked alright in the beginning of the project when the front-end test suite was small and relatively quick ( <2 mins). However, as the suite grew in size and length (15:00 minutes and growing), firing it off on every check-in quickly became untenable. We have since removed the suite from our main pipeline and kick it off four times daily on an independent pipeline:
Compile
Unit Tests
Back-end component integration tests
Package
The problem with this approach is clear, in that it is quite likely a build will make it all the way to the package stage even though it caused regressions in the front-end test suite. I suppose I could invalidate an already created package if the front-end test suite fails, but this seems clunky. We've looked at optimizing the test suite further but I think that's a dead end unless we can run the tests in parallel, something which TeamCity doesn't support.
Suggestions/critiques welcome.
Not sure what development platform you are working here, but I see two general options.
Use selenium grid or something like testNG
Leverage any teamcity build agents you have, and section up tests to be run in parallel on an array of these agents. I have in the past had dedicated build agents running tests, but even so this method will eventually fall over if your test sets increase too much in size.
You should be aiming to reduce the amount of testing you do directly through the UI. Look at how putting more emphasis on testing code further down the stack can reduce your direct UI test burden. In a recent project we refactored the javascript in our app to make it testable. These faster running tests meant that we could remove a large amount of our slower running webdriver tests. Its a bit of an investment in time and effort, but its causes much less pain than your current situation.
Scheduled builds should be avoided where possible as, like you mentioned, you could end up with packaged software that has bundled defects, plus the feedback time increases massively.
A common goal of software design seems to be to structure an application so that changes impact a minimal amount of components (i.e compiled assemblies) which can then be published individually. Dependency Inversion Principle is applied so stable components don't depend on volatile ones, classes are packaged in a way again to limit a deployment to a minimal set of components.
So my question is, what is wrong with publishing an entire application in its entirety for each change? Especially if a publish is a completely automated 1-click solution. Surely deploying components individually means I then have to version and manage them individually as mini projects with their own dependencies? A large chunk of 'good design' seems to hang on this one principle that publishing each component separately is a good thing, but why?
What is wrong with publishing an entire application in its entirety
for each change?
I don't think there is anything wrong if you can manage it. It seems this is the approach that Facebook takes:
Because Facebook's entire code base is compiled down to a single
binary executable, the company's deployment process is quite different
from what you'd normally expect in a PHP environment. Rossi told me
that the binary, which represents the entire Facebook application, is
approximately 1.5GB in size. When Facebook updates its code and
generates a new build, the new binary has to be pushed to all of the
company's servers.
Partial deployments can sometimes be useful to maintain a degree of autonomy between component teams but they can result in unexpected behaviours if the particular version-combination of components hasn't been tested.
However, the motivation for modular design is more around ease of change (evolvability, maintainability, low coupling, high cohesion) than the ability to do partial deployments.
I'm in a QA department of an internal development group. Our production database programmers have been building an SSIS package to create a load file from various database bits for import into a third-party application (we are testing integration with this).
Once built, it was quickly discovered that it had dependencies on the version of SQL Server and Visual Studio that it was created with, and had quite of few dependencies on the production environment as well (this is not an SSIS problem, just describing the nature of our setup).
Getting this built took several days of solid effort, and then would not run under our QA environment.
After asking that team for the SQL queries that their package was running (it works fine in the production environment), I wrote a python script that performed the same task without any dependencies. It took me a little over two hours (note that I already had a custom library for handling our database interaction), and I was able to write out a UTF-16LE file that I needed.
Now, our production database programmers are not SSIS experts, but they use it a fair bit in their workflows -- I would readily call all of them competent in their positions.
Thus, my question -- given the time it appears to take and the dependencies on the versions of SQL Server and Visual Studio, what advantage or benefits does an SSIS package bring that I may not see with my python code? Or a shell script, or Ruby or code-flavor-of-the-moment?
I am not an expert in SSIS by any means but an average developer who has experience working with SSIS for little over three years. Like any other software, there are short comings with SSIS as well but so far I have enjoyed working with SSIS. Selection of technology depends on one's requirement and preferences. I am not going to say SSIS is superior over other technologies. Also, I have not worked with Python, Ruby or other technologies that you have mentioned.
Here are my two cents. Please take this with a grain of salt.
From an average developer point of view, SSIS is easy to use once you understand the nuances of how to handle it. I believe that the same is true for any other technology. SSIS packages are visual work flows rather than a coding tool (of course, SSIS has excellent coding capabilities too). One can easily understand what is going on within a package by looking at the work flows instead of going through hundreds of lines of code.
SSIS is built mainly to perform ETL (Extract, Transform, Load) jobs. It is fine tuned to handle that functionality really well especially with SQL Server and not to mention that it can handle flat files, DB2, Oracle and other data sources as well.
You can perform most of the tasks with minimal or no coding. It can load millions of rows from one data source to another within few minutes. See this example demonstrating a package that loads a million rows from tab delimited file into SQL Server within 3 minutes.
Logging capabilities to capture every action performed by the package and its tasks. It helps to pinpoint the errors or track information about the actions performed by the package. This requires no coding. See this example for logging.
Check Points help to capture the package execution like a recorder and assists in restarting the package execution from the point of failure instead of running the package from the beginning.
Expressions can be used to determine the package flow depending on a given condition.
Package configurations can be set up for different environments using database or XML based dtsconfig files or Machine based Environment variables. See this example for Environment Variables based configuration. Points #4 - #7 are out-of-the-box features which require minor configuration and requires no coding at all.
SSIS can leverage the .NET framework capabilities and also developers can create their own custom components if they can't find a component that meets their requirement. See this example to understand how .NET coding can be best used along with different data source. This example was created in less than 3 hours.
SSIS can use the same data source for multiple transformations without having to re-read the data. See this example to understand what Multicasting means. Here is an example of how XML data sources can be handled.
SSIS can also integrate with SSRS (Reporting Services) and SSAS (Analysis Services) easily.
I have just listed very basic things that I have used in SSIS but there are lot of nice features. As I mentioned earlier, I am not sure if Python, Ruby or other languages can handle these tasks with such ease.
It all boils down to one's comfort with the technology. When the technology is new, people are very much skeptical and unwilling to adapt it.
In my experience, once you understand and embrace SSIS it is really a nice technology to use. It works really well with SQL Server. I don't deny the fact that I faced obstacles during development of my packages but mostly found a way to overcome them.
This may not be the answer that you were expecting but I hope this gives an idea.
I do a sort of integration/stress test on a very large product (think operating-system size), and recently my team and I have been discussing ways to better organize our test workloads. Up until now, we've been content to have all of our (custom) workload applications in a series of batch-type jobs each of which represent a single stress-test run. Now that we're at a point where the average test run involves upwards of 100 workloads running across 13 systems, we think it's time to build something a little more advanced.
I've seen a lot out there about unit testing frameworks, but very little for higher-level stress type tests. Does anyone know of a common (or uncommon) way that the problem of managing large numbers of workloads is solved?
Right now we would like to keep a database of each individual workload and provide a front-end to mix and match them into test packages depending on what kind of stress we need on a given day, but we don't have any examples of the best way to do more advanced things like ranking the stress that each individual workload places on a system.
What are my fellow stress testers on large products doing? For us, a few handrolled scripts just won't cut it anymore.
My own experience has been initially on IIS platform with WCAT and latterly with JMeter and Selenium.
WCAT and JMeter both allow you to walk through a web site as a route, completing forms etc. and record the process as a script. The script can then be played back singly or simulating multiple clients and multiple threads. Randomising the play back to simulate lumpy and unpredictable use etc.
The scripts can be edited and or can be written by hand once you know whee you are going. WCAT will let you play back from the log files as well allowing you simulate real world usage.
Both of the above are installed on a PC or server.
Selenium is a FireFox add in but works in a similar way recording and playing back scripts and allowing for scaling up.
Somewhat harder is working out the scenarios you are testing for and then designing the tests to fit them. Also Interaction with the database and other external resouces need to be factored in. Expcet to spend a lot of time looking at log files, so good graphical output is essential.
The most difficult thing for me was to collect and organized performance metrics from servers under load. Performance Monitor was my main tool. When Visual Studio Tester edition came up I was astonished how easy it was to work with performance counters. They pre-packaged a list of counters for a web server, SQL server, ASP.NET application, etc. I learned about a bunch of performance counters I did not know even exist. In addition you can collect your own counters too. You can store metrics after each run. You can also connect to production servers and see how they are feeling today. Then I see all those graphics in real time I feel empowered! :) If you need more load you can get VS Load Agents and create load generating rig (or should I call it botnet). Comparing to over products on the market it is relatively inexpensive. MS license goes per processor but not per concurrent request. That means you can produce as much load as your hardware can handle. On average, I was able to get about 3,000 concurrent web requests on dual-core, 2GB memory computer. In addition you can incorporate performance tests into your builds.
Of course it goes for Windows. Besides the price tag of about $6K for the tool could be a bit high, plus the same amount of money per additional load agent.
Hi in my project we have hundreds of test cases.These test cases are part of build process which gets triggered on every checkin and sends mail to our developer group.This project is fairly big and is been for more than five years.
Now we have so many test cases that build takes more than an hour .Some of the test cases are not structured properly and after refactoring them i was able to reduce the running time substantially,but we have hundreds of test cases and refactoring them one by one seems bit too much.
Now i run some of the test cases(which takes really long to execute) only as part of nightly build and not as part of every checkin.
I am curious as how other guys manage this .
I believe it was in "Working Effectively with Legacy Code" that he said if your test suite takes longer than a couple minutes it will slow developers down too much and the tests will start getting neglected. Sounds like you are falling into that trap.
Are your test cases running against a database? Then that's most likely your biggest source of performance problems. As a general rule, test cases shouldn't ever be doing I/O, if possible. Dependency Injection can allow you to replace a database object with mock objects that simulate the database portion of your code. That allows you test the code without worrying whether the database is setup correctly.
I highly recommend Working Effectively with Legacy Code by Michael Feathers. He discusses how to handle a lot of the headaches that you seem to be running into without having to refactor the code all at once.
UPDATE:
A another possible help would be something like NDbUnit. I haven't used it extensively yet, but it looks promising: http://code.google.com/p/ndbunit/
Perhaps you could consider keeping your oracle database but running it from a ram drive? It wouldn't need to be large because it would only contain test data.
We have about 1000 tests, large percentage of those communicating through REST and hitting database. Total run time is about 8 minutes. An hour seems excessive, but I don't know what you are doing and how complex your tests are.
But I think there is a way to help you. We are using TeamCity and it has a nice ability to have multiple build agents. What you could do is split your test project into subprojects with each subproject containing just a number of tests. You could use JNunit/NUnit Categories to separate them. Then you would configure TeamCity so that each agent would build just one type of subproject. This way, you'd get parallel execution of tests. With few agents (you get 3 for free), you should be able to get to 20 minutes, which might even be acceptable. If you put each agent into VM, you might not even require additional machines, you just need lots of RAM.