I'm new on DevOps Azure. I'm trying to link my automated tests for Azure DevOps platform, but outcome is always "In progress". I don't get failed or passed for any test. I created my tests in Visual Studio C#.
I created app and CheckIn it in Visual Studio and it's showed in Repos File, also my test folder is shown too. If I do changes in my code it's updated in Repos. So it is fine.
Then, I created Pipeline:TestMirjana-.NET Desktop-CI
and build is successful for this one. Than, I created Release linked for this pipeline:
and it is successful too.
I linked my test case for Visual Studio:
also linked Test Plan Settings for these pipeline and release
but when I run my test case Outcome is always "In progress", no other results shown (not failed or passed)
validation is good
but Outcome is always None
.
So, I'm not sure why it's not working properly.
Maybe I didn't link something properly, can you please support?
Many thanks in advance.
The 'In Progress' result means it has some automation associated but there are no incoming results (having the required test configuration).
Try to run the vstest task in the Release pipeline (instead of the build pipeline) and use "Select tests using=Test plan" and pick the Test plan where you expect to see the results from automated tests. It works for me.
Related
I have an .Net Core application along with unit test cases. For that I have configured the Build pipeline in Azure DevOps. In that pipeline I have integrated SonarQube tasks (prepare analysis, run code analysis, and publish quality gate results).
I can see the report in SonarQube server after successful run. But in that report, I didn’t see the Code Coverage Results and Unit test results. Even though I used Cobertura for unit tests.
Your pipeline seems to contain the right steps, so there can be two issues:
1. Code coverage file is not generated (correctly)
The easiest way to validate if the code coverage file is generated correctly, is by publishing it as an artifact. Now check what format the output file is. If there is no output file, please check if you did include /p:CollectCoverage=true --logger trx to the test command. If you are running the build pipeline on Linux, you should also add /p:CoverletOutputFormat=opencover and install the coverlet.collector NuGet package in the .NET Test Project.
2. Code coverage file is not sent to Sonarqube
If you configured step 1 correctly, it is still possible that the generated files are not sent to Sonarqube. The best way to see what is going wrong, is by checking the build logs of the Run Code Analysis and Publish Quality Gate Results steps.
The most common issue, is that Sonarscanner is checking the wrong directory. In the prepare step, please specify where the files are located, like:
sonar.cs.opencover.reportsPaths=$(Build.SourcesDirectory)/**/coverage.opencover.xml
sonar.cs.vstest.reportsPaths=$(Agent.TempDirectory)/*.trx
Build pipeline generates code coverage files would it be coverlet or opencover, the problem is that Azure Devops pipeline creates the reports outside the working directory on the build agent, it uses the _temp folder inside _work whereas soanrquube searches inside the working directory. I am facing this issue with coverlet as well as Visual studio coverage tool. You can read the following 2 threads
https://github.com/coverlet-coverage/coverlet/issues/1399
https://github.com/microsoft/azure-pipelines-tasks/issues/11536.
I posted an issue on snoarqube community site but did not get any response so far. The solution I think is either make the test step in the build pipeline change the directory or configure somewher sonarqube to search the agent build directory _temp which is above the working directory
I'm currently using cTest to publish test results from google test into azure.
Is there any way i can decorate my test so it comes up already linked to a requirement. without manually having to link it in azure?
Edit:
I'm using gtest recordproperty to add data to the xml output of the test case. I have tried with #1234( as in a commit), 1234 and several combinations. I cannot find anything in the documentation.
Currently, you could refer to this doc: Run automated tests from test plans to automate test cases in your test plans and run them directly from Azure Test Plans. And then open the Test Plans page and select the test plan to see the status of your tests if tests are updated after test execution is complete.
In addition, if you turn on below “Automatically link new work in this build” option in build pipeline page, and then you can link work item to Git commit, the new build queued by this commit will link to this work item when it completes successfully.
Also, you could enable “Create work item on failure” option to create a new work item when this build is failed.
BTW, you could link work items to deployments refer to this doc.
I'm using Microsoft Unit Tests to validate my solutions via Azure DevOps Pipelines. I am able to see tests being executed and the % of code coverage
However, I can't see the code coverage results (or they won't render)...
Here is my publish code coverage task - am I missing something to have it render here?
This is a known issue on Azure devops. Now, we could only download the report, and open it with Visual Studio.
Azure devops only support the download link for .coverage files currently. The white page you see is a UI glitch. This scenario is only supposed to render a download link to the coverage file.
Besides, this issue has been submitted in this earlier suggestion ticket linked here:
support vstest .coverage "code coverage" build results tab
This feature request is On Roadmap, I believe it will be released soon, you can follow this thread to know its latest feedback.
In addition, we found a similar case and I have tested via the answer, set the code coverage tool to Cobertura, then I get the code coverage report in the Azure DevOps pipeline.
I have a automation script which built in Java/selenium. I want to run my test cases from VSTS by triggering my script and also want to get the result back in VSTS. Can anybody give me the path how can I make that happen. Also where should I keep by project?
I was doing research on it. But doesn't make sense to me
https://learn.microsoft.com/en-us/azure/devops/pipelines/test/continuous-test-selenium?view=azure-devops
run my test cases from VSTS by triggering my script and also want to
get the result back in VSTS
For this issue ,there is a blog that gives a path to do this , please refer to it to see if it helps .
In this path , you need to do stuff both to your project, and to your vsts build definition. The jest-junit package is needed in the project and then create a build with three tasks(“npm”,“Publish Test Results”, “Publish Code Coverage Results” ).
I am trying to add Azure Pipelines configuration to an existing project, bundler/bundler. Here is the PR that adds the configuration:
https://github.com/bundler/bundler/pull/6899
As one of the maintainers set up the bundler/bundler project on Azure Pipelines, this PR already triggers a build:
https://dev.azure.com/bundler/bundler/_build/results?buildId=11
Note that the build has a green checkmark and is marked as finished.
(Also note that there are loads of tests failing in the build, as this wasn't tested on Windows before. To make the build succeed anyway - and not all PRs and commits get the red "x" on Github while I am working on fixing the tests, I added || exit 0 at the end of the test command - which works fine on Azure Pipelines)
A feature of Azure Pipelines' GitHub integration is that the build results are shown in Github via a feature called "Check":
https://github.com/bundler/bundler/pull/6899/checks
(A shorter version of that is also included at the end of the PR page: https://github.com/bundler/bundler/pull/6899#partial-pull-merging)
Unfortunately, this check doesn't reflect the build status on Azure Pipelines and is still shown as "in progress":
and
Any idea why the GitHub check doesn't reflect the build status on Azure Pipelines?
What is confusing me further, is that the integration with Azure Pipelines actually worked just fine (check correctly reflects the build status) in the Pull Request that was automatically created by Azure Pipelines when creating the bundler/bundler project: https://github.com/bundler/bundler/pull/6955
But: It also can't really be the Azure Pipelines configuration I created in my PR, because the same configuration also works just fine in my fork: https://github.com/janpio/bundler/pull/6#partial-timeline (see the green checkmark for the bundler task). (On the other hand: Here Azure Pipelines doesn't use the "Check" feature of Github at all)
Great question. The most likely reason is that there some was glitch in the communication between Azure Pipelines and GitHub. It's very rare but sometimes a webhook between GitHub and Azure Pipelines doesn't trigger. There's no way to tell why it happened; it could have been a fault on either side.
Unfortunately, there's no way to re-send a webhook that didn't get delivered. Your only recourse is to rebuild that pull request. If you select the "Rebuild" option (in the ... menu):
Then a new build will be queued and, when it finishes, the status update will be sent back to GitHub. The check in the pull request will then be updated.
A less likely (but definitely possible) reason is that there's a bug in either Azure Pipelines or GitHub. And in this particular case, there was a bug with the code that uploads test results from Azure Pipelines to the test case manager API.
(Thanks for reporting the issue, we're sorry that we had a bit of a problem here, but we're glad that we were able to resolve this.)
Setting the following configuration worked for me: