I am a fairly new engineer (about 1 year) and I was tasked with figuring out a solution to shift performance testing as far left as possible.. to fit into a DevOps model. What I am trying to achieve is:
- Add additional scripting into the existing Newman to record the response time of EACH request in the collection.
- Generate a report each time the pipeline (testing for pass or fail of API scripts) is invoked to run, and save it in ADO along with the pass/fail results.
I do NOT want to have to create a collection for each individual API, but rather have large collections run daily (ex; Web services API's, app API's, UI API's)
ANY direction or help would be super helpful, I've googled a lot but just can't come up with a lightweight solution that isn't going to negatively impact the overall testing or dev process.
In azure devops pipeline, you can use Newman the cli Companion for Postman to analyze the response time by install extension Newman the cli Companion for Postman first:
In the configuration of task:
Add your collection url which get from Postman into the
Collection Source Type:
Then fill your local Newman path and the html report export
path in the task.
Note: Don't forget install Newman locally first.
After you run the pipeline, the task will run Newman and export the
report with .html to the path you specified. As the below pic
shown(this is my report), you can see the response time and the
results of all APIs in the collection.
Since you want the report saved in ADO repos, I' afraid you need to use git command to push local .html report to Azure Devops.
Related
I have all my test cases in Azure Devops Test Plans and now I'm executing them manually against a build.
How do I view manual test results by build?
I can view and filter test results by build on the "Runs" page but I would prefer to see test results grouped by build not filtered to a single build.
Any options for this?
I had hoped the the Test Plans - Progress Report would provide this but I don't see a way.
Maybe there is a way to group all my manual tests executions into a Test Run?
I researched and attempted with the related Azure DevOps REST API and Azure CLI, however, I did not find any available way.
It seems that we can only directly filter the test results to the build run level, and cannot to the build pipeline level.
Maybe, you can indirectly get the associated build pipeline via the following way:
Use the REST API "Runs - Query" to list the test runs associated with a specified build pipeline.
Use the REST API "Results - List" in a loop to list the test results of each test run.
I'm currently using cTest to publish test results from google test into azure.
Is there any way i can decorate my test so it comes up already linked to a requirement. without manually having to link it in azure?
Edit:
I'm using gtest recordproperty to add data to the xml output of the test case. I have tried with #1234( as in a commit), 1234 and several combinations. I cannot find anything in the documentation.
Currently, you could refer to this doc: Run automated tests from test plans to automate test cases in your test plans and run them directly from Azure Test Plans. And then open the Test Plans page and select the test plan to see the status of your tests if tests are updated after test execution is complete.
In addition, if you turn on below “Automatically link new work in this build” option in build pipeline page, and then you can link work item to Git commit, the new build queued by this commit will link to this work item when it completes successfully.
Also, you could enable “Create work item on failure” option to create a new work item when this build is failed.
BTW, you could link work items to deployments refer to this doc.
I have an Azure DevOps pipeline build that has several steps and the build is long. Every time there is something wrong with the build we review the logs and identify issues or come up with theories, then in case of a theory we have to insert a diagnostic command line (such as get directory, show contents of a file, etc) in between the steps; and in case of a fix we add a fix but we have to wait for the whole pipeline to rerun and find out. This is causing us to take a lot of time to fix build issues.
If we had access to the state of the agent of an unfinished build and we could just log on using RDP or any other terminal and checkout the contents, and the state of the files on disk that would have saved us a lot of hours.
Is there any way with Azure DevOps to do any diagnostic of this type?
No, if you are using hosted agent. If you are using self-hosted agent you can obviously log in to that one. You can, however, implement steps that only work if the build failed and those steps can attempt to capture information you are interested in (say publish the state of the build directory).
If you are using Azure DevOps Services, there is a new REST API version out that will let you do a "preview" run of changes to the YAML definitions: https://learn.microsoft.com/en-us/azure/devops/release-notes/2020/sprint-165-update#preview-fully-parsed-yaml-document-without-committing-or-running-the-pipeline
I run UI tests for my app in a release definition in Azure DevOps. I generate test report. I decided that it is convenient to save it in the build directory(wrong assumption?). The directory where the report is is:
browserName + DateTime.Now.ToString("ddMMyyyyHHmmss", CultureInfo.InvariantCulture) + #"\";
so the directory regex would be for instance : Chrome\d+
I build the test project in the release pipeline, run the tests, then I try to publish my report. The goal is to make it available in the Azure DevOps, or send a link to download it or any other way that makes it accessible.
To do so I added a step Publish Build Artifact
but then I get an error:
but then I don't have a fileshare available(I am able to create Azure Storage Account for instance) additionaly Publish Build Artifact doesn't support wildcards so I can't use regex Chrome\d+ to pin down the report directory.
Question: How can I make a file that is generated during release pipeline accessible?
EDIT: I found out in the meantime that I have Sharepoint available with enough storage.
Unfortunately publishing from release pipeline is not allowed.
Can we publish artifacts in release pipeline - Azure devOps?
One way to get around this is to try to publish Universal package in release pipeline but it has limitations. Create feed and publish there your files so you can share URL to others. It is not best option but if your test results files are not large you can publish to the feed and clean it sometimes(manually, because REST API provides way to delete package but does not provide function to get list of all packages published).
https://learn.microsoft.com/en-us/azure/devops/pipelines/tasks/package/universal-packages?view=azure-devops
The disadvantage of this option is that for free users it has limit up to 2GB. You can delete old packages when required but it takes around 24 hours to free space. But you can forget to free space and your pipeline will fail with not enough storage error and by next 24 hours you will have to disable this task to let pipeline pass.
"I decided that it is convenient to save it in the build directory"
Remember that agent working directory is cleaned depending on option you choose.
https://learn.microsoft.com/en-us/azure/devops/pipelines/repos/pipeline-options-for-git?view=azure-devops#clean-the-local-repo-on-the-agent
I recently got to know Dockup and while I really love this kind of integration in Github, I am wondering if the same thing can be achieved in Gitlab, for free.
Basically, what I would like to achieve is:
On merge request, build tests and deploy in a dedicated environment which would use the name of the merge request
Send a message to a given slack about the environment or the failure of the build / tests in the pipeline with the related link of the pipeline
It seems that since Gitlab 11.6 it is possible to have Pipelines for merge requests but I don't really see how to get the information of the merge request or even who has submitted the merge request to use it for creating the dedicated deployment environment in my pipeline script?
How can I get that?
Note: It seems only a webhook can provide the information about the user and all the details.
Found the information I wanted in https://docs.gitlab.com/ee/ci/variables
CI_MERGE_REQUEST_TITLE
CI_MERGE_REQUEST_PROJECT_URL
GITLAB_USER_NAME
CI_MERGE_REQUEST_ASSIGNEES
Probably won't even need to create a webservice for the webhook since everything can be found in the environment variables when running the pipeline script, this is pretty awesome!