We are currently checking the test plan module on our Azure Devops server and I'm facing an issue that the community may help.
According to this once a test run has been marked as failed or blocked there is no way to resume the test once the raised defects have been resolved. You either have to mark the test run as paused (so as to be granted the "resume" option) or reset the whole test to active in which case you loose all your progress and need to start over. Two questions:
This "workaround" works as long as you are careful and save the test run as paused. Otherwise if a failed step exists the run is saved as failed. Once this is done I have found no way of changing it to "paused" so as to get the resume option. Is there a way I'm missing?
In the test list under a specific suite you get the current test run outcome and state of test case. In case of "Paused" you don't have a way to know if the test execution is paused or you are utilizing the aforementioned workaround (e.g. a bug is registered). I cannot see a way that I could enhance the test run with some information that would signal this (there is no wit template I have found) on the grid. So is the standard query module (bringing testcases and associated bugs with state) the only way?
thanks
For the first question, if the test case is on status Failed and you retest the test case, all test steps need to be redone. Also there is no resume-button. This is by design. For now, set the status to Paused can resume the test, but it cannot truly reflect the result. As far as I know, there is no other way.
You could add your request for this feature(For example: when status = Failed/Block, the restore button is enable) on our UserVoice site , which is our main forum for product suggestions.After suggest raised, you can vote and add your comments for this feedback. The product team would provide the updates if they view it.
For the second question, as workaround , you can also add tags to paused test cases for labeling.
Related
My understanding is if the bug was in Activity, the new code will be used since Activity does not have the concept of version. But what if there is a bug in Workflow code and causing some workflows to fail (or even worse, not fail but stuck), we fixed the bug and deployed the new version, but those failing workflows are stuck on old versions. Is there a way to automatically move those failed / stuck workflows to new version and re-run them?
This happening a lot to us because we just started to learn to use Cadence/Temporal
Thank you in advance!
It depends on the bug.
For bugs that cause workflow to get stuck (in Go a nil pointer dereferencing causes panic which blocks the workflow progress by default) deploying the new version of the code is usually enough to unblock them.
If the bugfix requires backwards incompatible change then the best option would be to reset workflow to the point before the bug. This way workflow will be rolled back and continue through the new code.
For situation when the breakage of workflow code is caused by a new bad build you can rollback the workers to the previous build and mark that build as broken and all workflows going to roll back their state to before that build automatically. See "Recovery from bad deployment" section of the Temporal documentation.
If you are using ElasticSearch integration then you can also perform batch reset of multiple workflows using a predicate to select workflows that match some criteria. See "Signal, cancel, terminate workflows as a batch job" section. Besides the name of the section the batch jobs apply to reset operations as well.
In Azure Devops, After creating the Test plan, it's possible to run the tests manually and take screenshots, video, or add a comment during the test run. But when I try to export the Test plan after the execution is completed, it's not possible to export the screenshots, comments or videos as attachments. Is there a way to do this ? We work in regulated Industry and need to provide evidence for the test results.
Please go through below link and each steps:
Link:https://www.azuredevopslabs.com/labs/azuredevops/testmanagement/
Navigate to your team project on Azure DevOps.
Select Test Plans to navigate to the Test Hub. The test hub provides a central place for all test planning, execution, and analysis.
In general, every major milestone in a project should have its own test plan. Within each test plan are test suites, which are collections of test cases (and optionally other test suites) designed to validate a work item, such as a feature implementation or bug fix. Each test case is designed to confirm a specific behaviour and may belong to one or more test suites. The Parts Unlimited project has one test plan, which is under the Parts Unlimited Team and called Parts Unlimited_TestPlan1. Select the Parts Unlimited_TestPlan1 test plan.
Select the suite of tests for the story As a customer, I would like to store my credit card details securely. This suite of tests focuses on that work item, which happens to be a feature. Note that the work item numbers will vary every time you generate demo data for a lab.
On the right side you can see that this test suite has three test cases designed to confirm expected behaviour of the feature implementation. Double-click the Verify that user is allowed to save his credit card detail test case.
This dialog provides all the info you need on this test case. Locate the Related Work panel and note that this test case is linked to the suite it belongs to. Click the work item to navigate to it.
In the test suite, we can see all of the linked work items, which happen to be the test cases.
However, it’s not yet associated with the feature it’s designed to test, which we can link now. Click Add link | Existing item.
Set the Link type to Parent and search for “credit card”.
Select the Feature for Credit Card Purchase.
Click OK.
The parent feature is now associated with the suite that tests it and anyone can navigate between them to view their relationship relative to the other work items involved.
Click Save & Close.
Dismiss the original test case dialog.
For this issue ,if you want to download attachment from test run ,you can go into Tests Plan -> Runs. Then you will be able to click on Tests Runs, select the run you want to download, and simply download the attachment.
You can also export test plans, test suites, and test cases from Test Plans. Select the details that you want in the report. Then email or print this report for review.
You can refer to this case with the similar issue.
Can you explain your questions in more detail? Are you using any tools for writing test plans or taking screenshots? There are various tools for writing test cases.
If you are looking for some test plan/test cases management tools:
Hiptest
TestRail
The organization I am with has transitioned to Azure DevOps for source control and for Sprint Management. I am responsible for investigating moving the organization to Azure DevOps for testing off of another solution. There is one question I have received from multiple people after doing some demonstrations that I have not been able to get an answer on.
What I have been asked if the system can do, is that when a Bug is entered during a Test Execution, for it to automatically have the User Story that the script is related to be set as a parent.
For example:
If I have User Story 123, which has a Tested By relationship to Test Case Test1. Test Case Test1 is in Test Plan ABC, which was created with the Area Path of my team and Iteration of the current Iteration. If I create a Bug while executing Test1 (in a Test Suite in Test Plan ABC), it currently adds the Bug to the current Iteration, but under an Unparented story. We would like to see that bug automatically appear under User Story 123.
Is this possible? Thanks for any information regarding this.
I think you can use Create work item task behind your test task in your build/release pipeline.
In this task, you can set the work item link type and target work item.
And other conditions about the work item.
You can set the Control Options as below.
It means, only when the test task has failed, this task will run.
Hops this will help.
So, for anyone else searching, I was able to resolve my issue.
What ended up being the solution was using the Create Test Suite from Requirement option, within a Test Plan. This is a very useful method for our use, as once the Test Suite is setup, and Test Cases which are created with a Tested By/Test Relationship (User case should show Tested By in the Related Work section, the test case will show a Tests entry to the User story) will automatically be added into the Test Suite. Then, when testing, when you create a new Bug from within a Test Execution, will automatically be created with the User Story as a Parent to the Bug.
I'm currently using AppCenter to Test my Xamarin app. The complete suite is being run at night. But I'm currently looking for a way to make this part of my CI builds to introduce an extra quality gate before features get merged.
However, we would need to slim down the amount of tests that need to be run otherwise this would become unmanageable. For this we can specify categories on our App Center test build step. When queued manually I'm able to specify the categories, however when we make this CI Build part of a branch policy and use a manual trigger for our build validation we are unable to specify the categories as we don't get the popup when queuing.
Any suggestions on how to adres this issue differently?
Seems you want to set build variables in pull requests when you queue the build under policies.
Unfortunately, this could not be achieved by now. It will not pop up the dialog when you queue the build. There has been a related uservoice, you could vote up and follow it, TFS PM will kindly review your suggestion.
Allow Setting Build Variables in Pull Requests
https://developercommunity.visualstudio.com/idea/365725/allow-setting-build-variables-in-pull-requests.html
The only workaround for now is editing the build definitions and add the variable under variables directly, then queue again. You could clone the original build definition first.
I have set up Jenkins to use the GitHub pull request builder. It's working great! Except one little thing. My test suite creates HTML Reports (which I have beautifully posted using the HTML result poster plugin) but I can't seem to change the status message that gets posted to GitHub. My commits all get marked like:
The status of the commit gets updated properly, but the message part always says "Build finished. No test results found."
I can't seem to figure out how to change this message based on the status of the test suite. So, basically the only feedback I'm getting is the exit status of my shell script. This script does all of the building, testing, and handling of results, etc. This is reasonably acceptable in the event of a pass, but in the event of a failure, this message could be much more helpful! Even just knowing whether it was a build failure vs. a test suite failure would be nice, so then I can adequately verbally abuse the person who made the pull request :).
Everything else is working so good, that I can't figure out what seems like a trivial change is driving me crazy! Any ideas?
FWIW, the logic was added here: https://github.com/janinko/ghprb/commit/ffbc581d2712d6b99b9c6d0b081d7895a5ee8039
I believe you just need to generate JUnit formatted test output, and you may or may not need to ad a post-build step to publish the JUnit XML output.