I'm currently running xUnit unit tests in by build pipeline to prevent releasing builds with failing tests. I see that there's an option to associated automated tests with test plans within Azure DevOps. Is this a good practice for unit tests, or should this only be done with functional and/or integration tests?
Run unit, function and integration test suites in Azure Pipelines. Azure Test plans is supposed to improve overall code quality by using manual and exploratory testing services for your applications.
Related
I want to populate test plan with result of tests that are done by our CI pipeline on github. I am thinking of using a release pipeline like this. (Azure DevOps Test Plans - Synchronize JUnit test results in Runs with test cases on Tests Plan)
But as the author was not able to populate test plans with it. I am looking for a better solution.
As I need the project report for the test plan.
I've created CI&CD on Azure repo, and it is doing code deployment on App service.
Now created Automation that opens the browsers and do actions and generates reports for test cases. (C#+Selenium+report package).
Now I need to the setup Test plan, I know, needs to run headless testing, my the questions are
need to do without buying a Test plan package, then how we should do? (I have a subscription but not with test plans)
How to do using Azure Test plans (any recent blog pls, and no MS docs links).
any help appreciated
As far as I know, you can execute Selenium tests directly in the pipeline(Without test plan).
Here are the steps:
Add Visual Studio Test Task to run the Selenium tests and the test result will be output to the Test tab(Build Pipeline or Release Pipeline).
You also could use Publish Test Results task to publish test result.
Here is a blog about the detailed steps: Automating Selenium Tests in Azure Pipelines.
For headless testing,
You could refer to UI testing considerations.
By the way, since you have already created CI/CD, you can also consider integrating UI testing steps into CI/CD.
If you still want to configure test plan, you could refer to this doc.
I'm using Team Foundation Version Control as a source control for my .NET Core 2.1 project.
AzureDevOps is configured in continuous integration to checkout the code and build it.
We have 3 environments (Staging, PreProd, Prod). The Staging is not isometric with Prod so it is untrustworthy and we have to execute our integration tests on each environment with environmental data.
My build is generated by an agent in AzureDevOps on an OnPremise server which can only reach Prod environment.
I'd like to automate my XUnit integration tests in an AzureDevOps pipeline, however, I don't know where and how to do it. Am I supposed to execute the integration test step after building? or after releasing?
It looks like I need to deploy my binaries first on my environments, then execute the integration tests, and, if they go wrong, rollback the release.
Weird?!?
How can unblock this situation?
Regards,
If you want to run integration tests you need to first deploy your binaries to environment. You can do it as a separate:
step,
stage
pipeline
after deploying code.
Here it is up to you how you will do it. (To achieve last option you need to use pipeline triggers)
If you follow approach shift left, it means you detect issues as quickly as possible, you should don't worry about breaking them. If it happens on staging I would rather encourage you to fix the issue instead of roll backing code. Especially if it involves data model change.
And on production you can run only smoke tests, which are kind of integration tests which doesn't impact on state. They are like GET in REST - smoke tests should be idempotent, so you can run them without worrying bout changing state.
Since you use TFVC version, you could define a build pipeline to build and test your code, and then to publish artifacts. You also define a release pipeline to consume and deploy those artifacts to deployment targets.
As you have to execute integration tests on each environment with environmental data, you can run your XUnit integration tests in Release pipeline via VSTest task.
Righ now, I am setting a CI/CD pipeline on Azure devops 2019. I have implemented some Autometed test, but I would like to know if there is a way to set a stage in a pipeline with a task that allow me to trigger a manual test created on test plans ?
To do this first we need to Associate automated tests with test cases. Then you can use Visual Studio Test task by selecting tests using Test Plan to run tests from your test plan that have an automated test method associated with it.
A similar question has been answered on the Comments in Azure DevOps Labs.Please refer to it.
Are there any built-in options to perform Regression testing and Performance testing of APIs in Azure API Management? If not, are there any options to do that within the Azure ecosystem (like VSTS Performance tests tool etc.)? Thanks!
I think there are no built-in options to perform Regression testing that available in Azure. But for performance testing, there are some.
Regression Testing
For regression.I'd recommend going with manual testing like creating Test Plan, Test Cases execute/test them manually using VSTS
Creating a Test Plan in VSTS
Creating a Test Case in VSTS
Run Manual Test in VSTS
Load/Performance Testing
There are 3 ways you can easily do this
With Visual Studio
With VSTS
Azure Portal
I prefer going with VSTS