I am generating xml reports via postman in the azure devops pipeline. My Postman collection structure is below:
Collection name->Folder name->Requests(Containing test cases)
When the xml report gets generated then it looks like below:
<?xml version="1.0" encoding="UTF-8"?>
<testsuites name="Suite1" tests="2" time="1.317">
<testsuite name="Folder1 / Request1" id="abc" timestamp="time/date" tests="1" failures="1" errors="0" time="0.658">
<testcase name="Verify that the success status code is 200" time="0.179" classname="ABC">
<failure type="AssertionFailure" message="expected response to have status code 200 but got 401">
I expect from Azure devops to show the generated & categorize xml report as:
Suite1->Folder1->Test cases inside Request1
but it actually shows like below:
Suite1->Test cases inside Request1
I expect all the test cases to be inside "Folder1" instead of Suite1. SomeHow it is skipping the "Folder1" inside "Suite1". Let me know if needed more explanation.
Related
How to send the XML report to SonarQube? I mean, while the TEST-report.xml file contains any failures, the import operation fails. I got an error:
Running SonarQube using SonarQube Runner.17:18:18.452 ERROR: Error during SonarScanner execution
java.lang.NullPointerException
at java.text.DecimalFormat.parse(DecimalFormat.java:2030)
at java.text.NumberFormat.parse(NumberFormat.java:383)
...
The TEST-report.xml file (JUnit) contains something like:
<?xml version='1.0' encoding='UTF-8'?>
<testsuites name='X UITests.xctest' tests='12' failures='3'>
<testsuite name='X.MoreViewTests' tests='1' failures='1'>
<testcase classname='X.MoreViewTests' name='testSucceed_allAction'>
<failure message='XCTAssertTrue failed'>X UITests/Cases/Bottom Navigation/MoreViewTests.swift:212</failure>
</testcase>
</testsuite>
<testsuite name='X.OrderViewTests' tests='1' failures='0'>
<testcase classname='X.OrderViewTests' name='testSucceed_allAction' time='68.570'/>
</testsuite>
</testsuites>
But when I removed the failure lines, becomes:
<?xml version='1.0' encoding='UTF-8'?>
<testsuites name='X UITests.xctest' tests='12' failures='3'>
<testsuite name='X.OrderViewTests' tests='1' failures='0'>
<testcase classname='X.OrderViewTests' name='testSucceed_allAction' time='68.570'/>
</testsuite>
</testsuites>
It works. Any idea? Thanks.
Your XML report is invalid (doesn't meet the Surefire XML format requirements).
testsuite required parameters:
name - Full class name of the test for non-aggregated testsuite documents. Class name without the package for aggregated testsuites documents
tests - The total number of tests in the suite
failures - The total number of tests in the suite that failed. A failure is a test which the code has explicitly failed by using the mechanisms for that purpose. e.g., via an assertEquals
errors - The total number of tests in the suite that errored. An errored test is one that had an unanticipated problem. e.g., an unchecked throwable; or a problem with the implementation of the test
skipped - The total number of ignored or skipped tests in the suite
The exception starts with DecimalFormat.parse
java.lang.NullPointerException
at java.text.DecimalFormat.parse(DecimalFormat.java:2030)
at java.text.NumberFormat.parse(NumberFormat.java:383)
which means that the XML report parser couldn't parse a number. You have two missing parameters: errors and skipped. I don't know how you generate these reports (which JUnit version etc.), but probably the used tool is outdated or miss-configured.
I am running 5 API test scenarios using karate. When I run the test in non-parallel mode using #RunWith(Karate.class) then in xml generated by surefire-reports, all scenarios are reported individually as .
<testcase classname="[healthCheck]" name="[1:3] Check health check API returns status code 200" time="2.846"/>
<testcase classname="[healthCheckHeader]" name="[1:6] Check health check API returns status code 200" time="0.285"/>
<testcase classname="[userLogin]" name="[1:3] Check User Login API returns status code 200" time="0.108"/>
<testcase classname="[requestChaining]" name="[1:7] chain request demo" time="0.521"/>
<testcase classname="[viewRequests]" name="[1:10] Check View Requests API returns status code 200" time="0.278"/>
However, when I use karate parallel runner, then each scenario is not reported individually.
<testcase classname="demoTest.AutomationSuiteParallelCucRunner" name="testParallel" time="10.917"/>
I want to have similar report for parallel runner as generated when tests executed in non-parallel mode.
Here is the code for running tests in non-parallel mode:
#RunWith(Karate.class)
public class AutomationSuiteTest {
}
Here is the code for running tests in parallel mode:
#CucumberOptions(tags = {"~#ignore"})
public class AutomationSuiteParallelCucRunner {
#Test
public void testParallel() {
String karateOutputPath = "target/surefire-reports";
KarateStats stats = CucumberRunner.parallel(getClass(), 3, karateOutputPath);
assertTrue("SCENARIO FAILURES!!", stats.getFailCount() == 0);
}
}
Having struggled with this myself, the solution was surprisingly simple, though not really covered in the karate docs (as of this time).
The point is: when using the parallel runner in this manner, there is from JUnit's perspective just one unit test - the single method annotated with #Test. So this single test is all that's listed in target/surefire-reports. Karate doesn't seem to hook into the JUnit execution or so, so that's it.
But - and that was the missing piece for me - the karate runner itself can generate JUnit XML files. You just have to call .outputJunitXml(true) when creating it. This causes it to generate JUnit XML files in the karate report dir along with the HTML report and (optionally) cucumber JSON files:
import com.intuit.karate.Results;
import com.intuit.karate.Runner;
public class KarateTestRunner {
#Test
public void runParallel() {
final Results results = Runner.path("classpath:api/karate")
.outputCucumberJson(true)
.outputJunitXml(true) # <----
.parallel(8);
assertEquals(0, results.getFailCount());
}
}
I've got an NUNIT test suite that properly generates an xml file with attachments, but can't find if there is a way to get ReportUnit to show those attachments in the rendered HTML.
Example - TestResult.xml contains:
<test-case id="0-1091" name="TestName" fullname="NUnitTestProject1.TestName" methodname="MethodName" classname="ClassName" runstate="Runnable" seed="92684126" result="Passed" start-time="2018-01-30 22:21:55Z" end-time="2018-01-30 22:21:55Z" duration="0.030017" asserts="5">
<properties />
<attachments>
<attachment>
<filePath>C:\Output\Result1.json</filePath>
<description><![CDATA[Request]]></description>
</attachment>
<attachment>
<filePath>C:\Output\Result2.png</filePath>
<description><![CDATA[Response]]></description>
</attachment>
</attachments>
</test-case>
It shows up when I run it in Visual Studio with the Name Outcome and Attachments
Visual Studio Result
But when I run ReportUnit on it the Attachments don't show up (in either passed tests or failed tests.)
As mentioned in frisby official document (http://frisbyjs.com/) , I am using --junitreport something like following
jasmine-node ./demo/validation_spec.js/ --junitreport --output
C:\Users\Administrator\Documents\script/Reports
which is generating 15 xml files. As I am calling 15 time post using for loop. File contains data like following
<?xml version="1.0" encoding="UTF-8" ?>
<testsuites>
<testsuite name="Frisby Test:blank action " errors="0" tests="1" failures="0" time="0.284" timestamp="2017-03-21T13:55:15">
<testcase classname="Frisby Test: blank action " name="
[ POST https://localhost:8443/api/v2/settings/pointwise ]" time="0.282"></testcase>
</testsuite>
</testsuites>
so my difficulty is that I need to read each and every file to check if tests is pass or not. I want some kind of overview page like http://maven.apache.org/surefire/maven-surefire-report-plugin/surefire-report.html
which contain information how many test passed, fail etc. Is it possible to get that.
I'm building the set up to deploy my SSRS reports through Octopus Deploy, I found out one Octopus Library and I'm working on it, but I've had some issues:
1º ---- Message error: (The path is alright, but it keeps with the same warning)
WARNING: Unable to find datasource SalesDrivers in /Sales Drivers/Data Sources
2º ---- The method doesn't exist
Method invocation failed because [Microsoft.PowerShell.Commands.NewWebserviceProxy.AutogeneratedTypes.WebServiceProxy3er_ReportService2005_asmx_wsdl.ReportingService2005] doesn't contain a method named 'LoadReportDefinition'.
The powershell function from the template\library that is throwing the error can been seen below:
#region Update-ReportParamters()
Function Update-ReportParameters($ReportFile)
{
# declare local variables
$ReportParameters = #();
# necessary so that when attempting to use the report execution service, it doesn't puke on you when it can't find the data source
$ReportData = (Remove-SharedReferences -ReportFile $ReportFile)
# get just the report name
$ReportName = $ReportFile.SubString($ReportFile.LastIndexOf("\") + 1)
$ReportName = $ReportName.SubString(0, $ReportName.IndexOf("."))
# create warnings object
$ReportExecutionWarnings = $null
# load the report definition
Write-Host "*********************************************"
#Write-Host $ReportData
#(Remove-SharedReferences -ReportFile $ReportFile)
#Write-Host $ReportExecutionWarnings
$ExecutionInfo = $ReportExecutionProxy.LoadReportDefinition($ReportData, [ref] $ReportExecutionWarnings);
# loop through the report execution parameters
foreach($Parameter in $ExecutionInfo.Parameters)
{
# create new item parameter object
$ItemParameter = New-Object "$ReportServerProxyNamespace.ItemParameter";
# fill in the properties except valid values, that one needs special processing
Copy-ObjectProperties -SourceObject $Parameter -TargetObject $ItemParameter;
# fill in the valid values
$ItemParameter.ValidValues = Convert-ValidValues -SourceValidValues $Parameter.ValidValues;
# add to list
$ReportParameters += $ItemParameter;
}
# force the parameters to update
Write-Host "Updating report parameters for $ReportFolder/$ReportName"
if ($IsReportService2005) {
$ReportServerProxy.SetReportParameters("$ReportFolder/$ReportName", $ReportParameters);
}
elseif ($IsReportService2010) {
$ReportServerProxy.SetItemParameters("$ReportFolder/$ReportName", $ReportParameters);
}
else { Write-Warning 'Report Service Unknown in Update-ReportParameters method. Use ReportService2005 or ReportService2010.' }
}
Anyone knows how I could sort it out?
I have solved a similar problem but took a slightly different approach. Rather than using powershell and octopus directly I used the useful open source tool RSBuild to deploy the reports. It is pretty easy to bundle up the rsbuild.exe executable (it is tiny) and a deploy.config along with your reports inside the octopus package. Then you can use octopus's substitution feature to rewrite the config file and Powershell function to execute the executable. This also has the advantage that you can deploy easily without octopus, the config for data sources and reports is declarative in XML rather than procedural in Powershell and the smarts of your scripted deployment can live alongside your reports rather than buried in Octopus.
So my config looks a bit like:
<?xml version="1.0" encoding="utf-8" ?>
<Settings>
<Globals>
<Global Name="CollapsedHeight">0.5in</Global>
</Globals>
<ReportServers>
<ReportServer Name="RS1" Protocol="http" Host="${ReportServer}" Path="${ReportServerPath}" Timeout="30" />
</ReportServers>
<DataSources>
<DataSource Name="Source1" Publish="true" Overwrite="true" TargetFolder="Data Sources" ReportServer="RS1">
<ConnectionString>data source=${ReportServer};initial catalog=${DatabaseName}</ConnectionString>
<CredentialRetrieval>Store</CredentialRetrieval>
<WindowsCredentials>False</WindowsCredentials>
<UserName>${RepotrUser}</UserName>
<Password>${ReportsPassword}</Password>
</DataSource>
</DataSources>
<Reports>
<ReportGroup Name="Details" DataSourceName="Source1" TargetFolder="Reports"
ReportServer="RS1" CacheTime="10080">
<Report Name="BusinessReportABC">
<FilePath>reports\BusinessReportABC.rdl</FilePath>
</Report>
<!--More reports here-->
</ReportGroup>
</Reports>
</Settings>
My deployed octopacked artefacts contain RSBuild.Core.dll, RSBuild.exe, deploy.config and the reports files
Then I simply call the executable using powershell:
PS> rsbuild deploy.config