Im using Eclipse - TestNG / Webdriver, and doing regression testing my application and getting the test results, so is there any way that i can store the test results(i.e, whaterver the printed output), can i export in any format like XL file or HTML whenever i run the tests?
If your tests executed successfully, than you can see output in location:: project-directory/test-output
Refer to these files:
index.html => Full TestNg HTML reports
emailable-report.html => Reports that can be easily printed and emailed
testng-results.xml => TestNg results in xml format
If you don't mind, why do you want to store them?
If they pass, test results should be ignoreable.
And if they don't pass, don't you need to fix the one that failed? (and therefore just run the test that failed manually while looking through it?).
Related
Background:
We have a regression test suite that tests the generation of some large xml files by comparing them field-by-field to the corresponding baseline files.
This is implemented using a junit4 parameterized test running the test for each file and assertj soft assertions to collect the field comparison errors.
Problem:
When I run this test from my IDE, I can see the assertion errors output after each test (each file), but when run from maven, surefire collects all the errors in memory and outputs them at the end (when all the tests for the class have finished). Now, running this for 2000+ files, comparing hundreds of fields in each and having a lot of differences results in OutOfMemoryError, even with 8GB of heap allocated.
Question:
I'm trying to find out if there's any option in surefire to either output the errors after each test or not collect & output them at all (we're logging them into a file and generating custom reports from there anyway).
I've tried <redirectTestOutputToFile>true</redirectTestOutputToFile> but this only redirects stdout (the logs produced during test execution), the assertion errors are still spit to console after the tests finish.
Other options I see are:
Split the test parameters into smaller batches and run the test suite
for each batch, then aggregate the results - this could be done in
the Jenkins job.
Remove the detailed error reporting using soft assertions and only have a single assertion at the end of the test. This is what we had
before and obviously didn't help in finding errors. I wouldn't like
to go back there.
Add an option to run the tests in two modes:
use soft assertions to provide detailed error reporting when run locally (with a smaller set of data)
use a single assertion when run on Jenkins (with the full set of data) - here we're only interested in the logs, not the console output
The third solution would result in some ifs in the code and would make the code less readable, that's why I'm trying to solve this from configuration first.
Any other ideas? Thanx!
I wrote an integration test suite using NUnit. Since we're talking integration tests, test code uses configuration files, the file system, database and so on.
However, I noticed that it would be nice to change the test environment (i.e. change a value inside a configuration file - this would change the code behavior in some cases), and then run the full test suite again but using this new environment.
Is there a way to automate this using NUnit? I have code that updates the file, so if I can somehow set things up programatically, great.
Im using TeamCity with Gallio/XUnit/Specflow and trying to display our unit/acceptance test results. I believe the output is NUnit xml test result format.
Our unit tests auotmatically display using the command runner with Gallio. The acceptance tests are run through a large Powershell script which calls Gallio (Run-Gallio).
Results are output to acceptance-test-results.xml. Is there a way to display the acceptance-test-results.xml in TeamCity (6.5.5)?
To do this in a PowerShell script add the following
Write-Output "##teamcity[importData type='nunit' path='C:\SomeDirectory\YourResults.xml']"
You can use built in TeamCity feature importdata service message:
<!-- Send to TeamCity a service message using MSBuild -->
<Message Text="##teamcity[importData
type='nunit'
path='$(OutputPath)\UnitTestsReport.xml']"
Importance="High" />
For more details see: Build Script Interaction with TeamCity - Importing XML Reports
Thought this might be useful also for other people that came to this SO post with slightly different requirements. It's the public wiki for TC that covers this particular area.
http://confluence.jetbrains.com/display/TCD8/Build+Script+Interaction+with+TeamCity#BuildScriptInteractionwithTeamCity-ImportingXMLReports
I'm trying to integrate Robot Framework (an acceptance testing framework) with TeamCity. In order to do this it needs to send messages to the console output which TeamCity will then read and return realtime test progress/results. I'm doing this by calling the command line to run the tests with a simple exec task. Everything seemed to be working other than I was only getting the results at the end of the run and not on the fly.
After a bit of struggling with NAnt I swapped to using MSBuild and everything worked first time.
I have what I need now, but for completeness I'd like to find out why I couldn't get it working with NAnt. As far as I can tell the issue is that NAnt is prefixing all console output with [exec]. Is it possible to suppress this?
I don't think the console output is configurable.
NAnt is open source: you could fork your own copy and/or submit a feature patch.
Nunit works quite well with CruiseControl.NET, but there is one thing that irritates me a lot.
If there is a test that causes Nunit to crash, I would only get little information about the crash because the XML report of Nunit doesn't get a chance to be created and be merged into the CruiseControl report.
I need a way to report the progress even when Nunit crashes during the execution.
I have been tried to force each test to output some information to the console to resolve this problem. I have thought about using SetUp method, but I haven't found any good way to get the name of the current running test.
I think a better answer would be to create an NUnit Add-in that implements EventListener interface to capture the TestStarted event to output the progress to the console or a file.
The EventListener interface is documented on NUnit website: http://nunit.org/index.php?p=eventListeners&r=2.5
In addition, we can make the Dashboard report better even when NUnit crashes during its execution. We can use the following procedure to ensure that the DashBoard always shows something about the tests.
Run tests with the EventListener which outputs the progress to a separate file
After running tests, use another program to check the file
If the file does not contain a specific "end line", generate a special XML report based on the file and merge it into the CruiseControl log
If getting the name of the current running test is what you're after you could grab it with the following:
using System.Diagnostics;
...
[Test]
public void SomeTestThatWillCrash()
{
StackFrame sf = new StackFrame();
Console.WriteLine("Now running method: " + sf.GetMethod().Name);
...
}
CruiseControl.net recommends that you use NUnit through your builder (i.e. NAnt/MSBuild). See here: http://confluence.public.thoughtworks.org/display/CCNET/NUnit+Task. As they describe - it will allow you to run these tests locally first - which should give you an exception that you can clear up.
That being said - are your developers running these unit tests prior to checking in code? That could ease this issue. If its an integration issue - I would suggest grabbing the latest code base and running the tests locally to see what is out of sorts.
I don't know if NUnit is able to create the results file even when it crashes. Even if it did - you could run into problems if that file is not well formed due to the crash.
You could use #jpoh's approach but do it in the TestSetup method which would require you do it per-fixture. If really needed, you could write a base class that all your test fixtures inherit from that implement this method.
Another solution is to use MSBuild to run NUnit and use the task in the MSBuildCommunityTasks library. This allows you to continue on error and also get the error code back from NUnit. You won't get what method caused the problem, but might help some. Here is my MSBuild target:
<Target Name="UnitTest"
DependsOnTargets="BuildIt">
<NUnit Assemblies="#(TestAssemblies)"
ToolPath="$(NUnitx86Path)"
WorkingDirectory="%(TestAssemblies.RootDir)%(TestAssemblies.Directory)"
OutputXmlFile="#(TestAssemblies->'%(FullPath).$(NUnitFile)')"
Condition="'#(TestAssemblies)' != ''"
ExcludeCategory="$(ExcludeNUnitCategories)"
ContinueOnError="true">
<Output TaskParameter="ExitCode" ItemName="NUnitExitCodes"/>
</NUnit>
<!-- Copy the test results for the CCNet build before a possible build failure (see next step) -->
<CallTarget Targets="CopyTestResults" Condition="'#(TestAssemblies)' != ''"/>
<Error Text="Test error(s) occured" Code="%(NUnitExitCodes.Identity)" Condition=" '%(NUnitExitCodes.Identity)' != '0' And '#(TestAssemblies)' != ''"/>
</Target>
This probably won't fit your needs as is, but is something to try out and play with.
That said, I would agree with #rifferte that it sounds like you need to debug the problem locally and not rely on CC.NET to handle the reporting.