The software I worked on has both unit tests and system tests. System tests can take minutes to run, they take input values and we validate the results against expected output. There are hundreds of system tests. The software must be built (done this) and tested on both windows and Linux.
How can I automate testing with VSTS? I'd like to avoid doing this at build stage, because it would slow the builds down. I can't see how to automate this in the Test stage. Do I need additional extensions to do this? Everything seems so geared up for web development, e.g. selenium tests, how do we run automated tests for good old binary programs?
I would suggest using Release Management to deploy your application to a test environment and then run your tests as a part of your Release Definition. You can then choose to run tests in parallel to make sure that your system tests don't take days to run.
On a side note, having so many system tests is a code smell. I would suggest looking into building as many fast running unit tests as possible and only using system tests when absolutely necessary.
Related
I have 3 sets of related questions:
1.Are both TestNg and Jenkins competitive tools in the industry. Do they have the same function?
2. Are both used primarily to run Selenium Test Cases?
3. Can both be used to run test cases on IDE's like Eclipse?
1.Are both TestNg and Jenkins competitive tools in the industry. Do they have the same function?
No they are not. Jenkins is an Orchestrator tool that is predominantly used for Continuous Integration (As and when developer checks-in code, it must be compiled, unit tests should be run on it and if the tests pass, it should be deployed to some place) and Continuous Delivery (All of Continuous Integration + run functional tests on the deployed code and if the functional tests pass, promote the code to the next environment and eventually ship to production in an automated fashion). Jenkins cannot do this by itself. So it makes use of build tools such as Ant/Gradle/Maven to help it compile the code and run tests on top of it.
TestNG is a test framework, that lets developers write unit tests and also lets test engineers build their functional automation using it. Its pretty much like JUnit, except that it lets a test engineer visualise a test just as he/she would do in the real world. So they both have totally different functions.
Are both used primarily to run Selenium Test Cases?
Jenkins is agnostic to what it orchestrates. So it doesn't care whether you are using it to build selenium tests or production code. To Jenkins a build is a build. TestNG on the other hand can be used to run Selenium Tests as well. TestNG merely cares about running tests. What goes inside the test is left to the engineer who is building those tests.
Can both be used to run test cases on IDE's like Eclipse?
Jenkins is usually started and left running as a standalone server and one usually interacts with it via a Web UI. So it has got nothing to do with an IDE. TestNG can be run via the below 3 mechanisms :
Via IDE
Via command line
Via build tools.
My project at work used Urban Code Deploy (UCD) for its continuous deployment process. My code runs locally and passes all unit tests, but the build group says that my code is failing the FVT test being run by UCD. Is there any way to run this FVT test locally, or at least attempt to run it, so I can hopefully figure out what is failing?
Mike
UC Deploy isn't a testing tool. So the team that has set it up, has it running some other testing tool at the end of hte deployment (which is pretty normal).
So you'll need to ask them what testing tool they're using and go from there.
If you can see how the build group is deploying your code, you should be able to see what testing they are doing, and then be able to replicate that in your own environment. Often the code changes and changes in requirements will not be reflected in the FVT tests, and you need to deliver updated FVT test scripts in conjunction with your code changes.
I have spent a couple of days going through a lot of sites and reading about Nant , Rake etc.
please forgive my Noob question but I still cannot find what is the difference between Nant and CruiseControl.
As far as I can see Nant can do automated builds , run tests .
so what extra does cruisecontrol do ?
Also there was mention of Teamcity . there too from the documents I can see it can do builds but it also can use Nant but I fail to understand why it needs to use Nant when it can do the builds itself
I am basically trying to follow proper software practices by introducing automated builds at my workplace
Appreciate all help
Nant by itself can do builds and test, but it needs to be launched by some other mechanism such as a windows scheduled job. There is not a capability of launching the build only when source code changes, at least without
an amount of additional scripting.
Nant itself is just a script runner, not a scheduler - it requires some other software (or a manual user action) to launch it.
Continuous integration (CI) tools such as CruiseControl or TeamCity provide monitoring of source control to launch a build process in addition to other things. The build itself could the be a single nant script which runs the build and tests as you suggest, or the build could be done using a series of tasks which are built in to the CI server. The difference is not how the builds are done, but how they are initiated and reported.
CI servers additionally usually provide web-based reporting of the details of the build runs and unit tests.
In summary, Ci tools provide monitoring, scheduling, and reportingin addition to scripting of the build process.
Nant is a scripting language
CruiseControl is a free continuse integration tools
Teamcity in another contiuse integration tools,
regards,
We are currently working on a Salesforce.com custom APEX project that involves a lot of apex classes, triggers and Visualforce pages. We also have numerous applications from AppExchange that are part of the system.
We develop all the Apex Classes, Visualforce pages, etc in test environment and then deploy it to the live environment using Eclipse IDE. What happens is that every time we deploy changes to the live environment, all the test methods of all the classes (including those from AppExchange Apps) seems to be executing. So deployment of a simple change could end up taking couple of minutes.
Is there a way in apex to "package" classes by namespace or something like that so that when we try to deploy a change, only the test methods relevant to that package are executed. If something like that exists, our deployment can happen much faster.
Unfortunately no, there is no partial testing for deployment of apex code, every change, no matter how minute or self-contained triggers a full test run. This among other things enforces code metrics (minimum total code coverage for instance)
IMHO, this is proving to be a two-sided coin when it comes to enforcing code reliability. When we started using apex all of our tests were very comprehensive performing actual testing of the code with lots of asserts and checks. Then we started having very very long deploy times so now our tests serve one and only function, satisfying minimum code coverage, and even with that simplification it takes almost 3 minutes to deploy anything and we only use 20% of our apex code allowance.
IMHO2, Apex is way too slow of a coding platform to be enforcing this kind of testing. I cant even imagine how long the tests would run if we reach 50% allowance, not to mention any more.
This is possible but you'll need to learn about Apache Ant and have a look at the Force.com Migration Toolkit. You can then use a Build file to determine which files are deployed as well as which tests are run.
I'm busy writing a whitepaper that'll touch on this and other related development strategies... I'll post to my blog when it's done.
If we use the apache ant migration tool we have many options for deployment
like
deployCodeFailingTest which will skip the test classes
and if you want to run only specific test classes
please use : something similar to this in ur build.xml
<target name="deployCode">
`<sf:deploy`
username="${sf.username}"
password="${sf.password}"
serverurl="${sf.serverurl}"
deployroot="codepkg">
<runTest>SampleDeployClass</runTest>
</sf:deploy>
</target>
for detailed reference please use this link
http://www.salesforce.com/us/developer/docs/daas/salesforce_migration_guide.pdf
I would recommend the following approach:
Git as repository for all your sf code
jenkins to deploy your code as CI/CD
PMD as the static code analyser
sfdx as the deployment method in jenkins for deployment.
Refer the trailhead link: https://trailhead.salesforce.com/users/strailhead/trailmixes/architect-dev-lifecycle-and-deployment
I'm using TeamCity for my CI builds, and I'd like to set up a second build for running automated UI tests on Windows XP and Windows 7 virtual machines.
I imagine the build working as follows:
Compile, run unit tests, etc.
Prepare MSI using WiX
Copy MSI to target test machines
Remotely execute MSI's
Copy test harness code to remote machine
Run tests
Build finishes
The automated UI tests are written using NUnit and would need to be run directly on the test virtual machine (they can't run remotely). It's important that if the tests fail, it appears in the TeamCity build log and the build fails. I'd rather not install VS or the TeamCity build agents on either of the test virtual machines.
It seems that most of this should be possible using psexec.exe. Are there any alternative (preferably open source) tools that I should look at?
takes a deep breath
We were looking into something to help us out with our automated UI tests. We use ranorex to test the UI and TeamCity/Msbuild to execute the tests.
We never found any tools to help us out (I’m constantly keeping an eye out for some so will monitor this thread) but here is what we did instead.
The CI server copies the setup files and test scripts to the Testing Host Server.
The CI server then launches a custom app on the Testing Host Server providing the name of the VM to launch.
The Test Host Server then launches the VM software, using Virtual PC.exe -singlepc -pc vhdname.vhd -launch , and waits for it to shutdown (after it’s run its tests).
The VM grabs the setup files and scripts from the network location and executes.
After the tests are run it then returns the results to a networked location and shuts itself down.
Control is returned to the custom app.
Control is returned to the CI server which determines from the results if it has passed or failed (and updates the UI so developers are made aware of the result).
Results are collection as artifacts in TeamCity and tagged in Svn.
I think that's everything. Its convoluted, however, it works. Hope someone of that helps you.
Jeff Brown of the Gallio team has been talking about a tool called Archimedes that he's planning to write to support this kind of requirement. It sounds promising, but I don't think there has been much progress on it so far.
In the mean time though, there is something in the Gallio project called VM Tool that may do what you want. It provides commands to stop, start and snapshot virtual machines, and more importantly, to copy files back and forth and execute commands.
I presume you have also considered Powershell Remoting?