I am trying to build a custom solution on the top of pytest framework to handle async assertions and for that, I need test cases to be marked as PENDING.
Related
I have a React component which allows users to submit potentially long-running queries to a remote service. While the query is running, the component shows a cancel button. I want to test that this button shows up when expected, that its click handler cancels the previous API request, and so on.
Since the button is only present while the async API call is active, the tests I wrote for this purpose make their assertions about the button in the mock implementation of the async API itself. They're not super elegant but I confirmed that they do go red as I expect when I remove parts of the production code.
On upgrading #testing-library/react from 8.0.1 to 9.3.2, although the tests still pass, I now get the following warning several times:
console.error node_modules/#testing-library/react/dist/act-compat.js:52
Warning: You seem to have overlapping act() calls, this is not supported. Be sure to await previous act() calls before making a new one.
I have reproduced the issue in the following CodeSandbox (be sure to selected the "Tests" tab on the right-hand side and then view the "Console" messages at the bottom right).
The final comment on this GitHub issue says that I shouldn't need to worry about act() as long as I'm using the React Testing Library helpers and functions (which I am). What am I missing?
Raising an issue (here) against react-testing-library has helped me reach the conclusion that the best approach seems to be not to worry about it too much. The library's author was kind enough to propose a simpler test pattern of just making assertions directly after the action that causes the component to enter the transient state that you're trying to test. For example:
fireEvent.click(getByText("Submit")); // <-- action which triggers an async API call
expect(getByText("Fetching answers")).toBeInTheDocument();
Previously, I had the same expect statement in my mock implementation of the API call triggered by the click. My reasoning was that this was the only way I could be certain that the assertions would run while the component was in the transient state I was trying to test.
AFAICT, these tests are not strictly correct with respect to asynchronous actions because the promise returned by the mock implementation could resolve before the expectation was checked. In practice though, I have observed that the tests rewritten to the simpler approach:
do not provoke the warning about overlapping act() calls in the OP
can be made to fail as expected if I change the non-test code to break the behaviour under test
have not so far shown any intermittent failures
are far easier to read and understand
This question has never attracted much attention but I hope recording the answer I eventually arrived at myself might help others out in future.
I am using the Invoke REST API task in as a pre-deployment gate in my environment. The task has been configured to wait for a callback from my service (external to VSTS).
The problem is that when I try to call into VSTS to mark the task as completed I always get an error saying orchestration session xxxxxx_xxxxxx_xxxxxx not found for hub Gates. The same code when used with release or build definitions work fine but fails with this error when used with gates.
Here is a snippet of my code that makes the API call
var taskCompletedEvent = new TaskCompletedEvent(jobId, taskInstanceGuid, TaskResult.Succeeded);
taskClient.RaisePlanEventAsync(projectGuid, HUBNAME, planGuid, taskCompletedEvent).SyncResult();
This problem occurs because of a slight deviation on how gates are executed when compared to builds or releases. In general, the safest way to update such server side tasks using the callback mode would be by using the TaskClient maintained by the VSTS team itself, which takes care of all such quirks.
The slight change that can be made in the original code to make it work would be -
var taskCompletedEvent = new TaskCompletedEvent(taskInstanceId, Guid.Empty, TaskResult.Succeeded);
taskClient.RaisePlanEventAsync(projectGuid, HUBNAME, planGuid, taskCompletedEvent).SyncResult();
The difference lies in how the event is initialised. The TaskId parameter is not defined and JobId is not used anywhere. The recommendation is still to use the TaskClient on GitHub to ensure everything continues to work fine even when the VSTS Release Management team decides to fix this rather annoying difference.
we are triggering the runscope test using triggerid of the specific test. How can we learn the status of the test so we can progress our builds ?
We have a blog post that describes how to do this with Codeship, but the same methods (especially the polling done in the Python sample script) should be applicable to any CI environment.
Apologies for bumping an old post, but I was looking for something similar, that is, checking the Runscope results to automatically approve the next step in a release (as opposed to a build that OP asked about).
It seems that VSTS has the concept of Gates that can do some action and only progress the release when the action succeeds, with configurable timeout and retry. One of the actions is Invoke REST API, which would probably do the job.
https://learn.microsoft.com/en-us/vsts/pipelines/release/approvals/gates?view=vsts
Note, I haven't actually tried this yet, so YMMV
I have read that "Protractor can automatically execute the next step in your test the moment the webpage finishes pending tasks, so you don’t have to worry about waiting"
But, I had to implement waiting(s) or sleeps in my test script to make them all PASS.
Can anyone help to understand this waiting.
Read At :http://www.protractortest.org/#/
Automatic Waiting:
You no longer need to add waits and sleeps to your test. Protractor can automatically execute the next step in your test the moment the webpage finishes pending tasks, so you don’t have to worry about waiting for your test and webpage to sync.
Right, I find this description as confusing as you. I think it describes the ideal world with no network delays and timeouts, no animations and layout issues.
The description originates from the following:
Protractor runs an extra command before performing any action on the
browser to ensure that the application being tested has stabilized.
This extra command is an async script which asks Angular to respond when the application is done with all timeouts and asynchronous requests, and ready for the test to resume.
Now, what does that "application is ready" statement mean? It basically means that, there are no pending requests, promises and "macro tasks" inside the Angular running application (source for angular testability).
From what I understand, this helps to cover most of the timing and waiting issues, but, if there is a pending JS code executed outside of Angular, or if there are any pending animations or other UI-related changes - this may potentially have an effect on your test stability - for instance, an element might not be yet visible or clickable, an input may not yet get enabled etc.
And, this does not actually contribute to the feedback from the end-to-end tests being stable and helpful - for example, in our project we often find ourselves adding browser.wait()s here and there to tackle occasionally failing tests. Also, here is a set of things that helped us to tackle this flakiness:
Protractor flakiness
I use the TeamCity (7.0) REST API to allow developers to trigger custom builds. I add the build to the queue like this:
http://teamcity/httpAuth/action.html?add2Queue=[buildTypeId]&name=[propName]&value=[propValue]
My question is how I best can track the progress of the build just triggered. The REST call does not return any info about build ID assigned to the build, so even if I poll the list of builds (running/finished) I will not know if one of them is the one I triggered. There could potentially be several builds for the same buildTypeId in the queue, so I need a way to separate out the one I am after.
I read somewhere a suggestion that you could add a build property with a unique value to each build you put in the queue, and then later poll the build list and look for one with that exact property value. I have however not found a way of listing the properties for the builds, so I am still stuck. This REST call does not provide information about properties:
http://teamcity/httpAuth/app/rest/builds/?locator=buildType:[buildTypeId]
Any suggestions on how to solve this? I would ideally like to know if the build is in the queue, if it is running, and when it's done I would like to get the status. The most important is however to know if it is done and what the status has.
After some further investigation I came up with a solution for this which seems to work fine:
I found out that even though you did not get any information about the custom build properties using the "/builds/?locator=buildType:x" call, you could extract the build ID for each one of the builds in that list and then do another REST call to get more details about one specific build. The rest call looks like this:
http://teamcity/httpAuth/app/rest/builds/id:{0}
The response from this call will give you a "build object" which contains a list of build properties, among other things.
My solution for tracking the build progress was then like this:
When a build is added to the TeamCity queue, I first add a property to the URL called "BuildIdentifier". The value is just a GUID. I pass this identifier back to the client application, and then the client starts polling the server, asking for the status of the build with this specific identifier. The server then goes through some steps to identify the current stage of the build:
1: Check if the build is running. I get the list of running builds with the call "/builds?locator=running:true", iterate through the builds and use the build ID to query the REST API for details. I then go through the details for each running build looking for a build with a matching "BuildIdentifier" property to the one I received from the client. If there is a match in one of the running builds I send a response with a message that the build is running at x percent (PercentageComplete property of the build object) to the client who is tracking the progress. If a match is not found I move on to step 2.
2: Check if it is finished: First get the latest build list using the "/builds/?locator=buildType:x" call. Then do the same thing as in step 1, and extract the X latest builds from the list (I chose 5). In order to limit the number of REST calls I set an assumption that the build would be in the latest 5 builds if it was finished. I then look for a match on the BuildIdentifier, and if I get one I return the status of the build (FAILED, SUCCESS, etc.).
3: If there was no match for the BuildIdentifier in step 1 or 2 I can assume that the build is in the queue, so I return that as the current status.
On the client side I poll the server for the status every x seconds as long as the status is saying that the build is either in the queue, or running.
Hope this solution could be helpful if there are someone else with the same problem out there! I would think that tracking the progress of a triggered build is a pretty common task if you use the TeamCity REST API.
The Queued Builds rest api is the best way to accomplish this but has only been available since version 8.1.
To start the build send a POST request to ~/httpAuth/app/rest/buildQueue like this
{
buildType: { id: "bt667" },
branchName: "master",
properties: {
property: [
{ "name": "Property", "value": "test" }
]
}
}
The response contains a href which can be used to check the status of the build.
{
...
"href": "/httpAuth/app/rest/buildQueue/taskId:49337",
...
}
To check the status of the build queued send a GET request to the href specified in the response from step 1.
This is a huge improvement over the previous API.
Since TeamCity 8.1, REST API got a dedicated way to trigger a build and tracking the queued build results is much easier as the build queue request returns the link to the queued build which can later be used to track the build's current status.
See details in the TeamCity documentation.