We are using the Publish Test Results task PublishTestResults#2 to publish junit type results in a pipeline in Azure Devops. It has previously worked fine but is now hanging for 10mins (I think this is the default job timeout) and then failing, even though the results have been published. If I try to cancel the job when it starts hanging, the cancel request is ignored and the job continues to hang. Has anyone else experienced similar?
This is the log output whilst the task is hanging
Starting: PublishTestResults
==============================================================================
Task : Publish Test Results
Description : Publish test results to Azure Pipelines
Version : 2.160.0
Author : Microsoft Corporation
Help : https://learn.microsoft.com/azure/devops/pipelines/tasks/test/publish-test-results
==============================================================================
##[warning]An error occurred while sending the request.
Publishing test results to test run '1033544'.
TestResults To Publish 11, Test run id:1033544
Test results publishing 11, remaining: 0. Test run id: 1033544
Async Command Start: Publish test results
We eventually nailed this down to only occurring on agents running as a service, rather than interactively. The problem is that our agents are behind a proxy. We provided the proxy settings as per the instructions but it appears the Publish Test Results task doesn't use that settings, so we had to provide it to the environment that runs the service as well by editing runsvc.sh:
export HTTP_PROXY=http://ourproxy:8080/
export NO_PROXY=localhost,127.0.0.1,localaddress
export HTTPS_PROXY=http://ourproxy:8080/
Publishing test results in AzureDevops Hangs
According to the error message:
[warning]An error occurred while sending the request.
It shows that an error was encountered while sending the request, you could enable the debug log by changing the default variables system.debug to true.
If it worked fine previously and you haven't changed your build definition, then the problem should be caused by your network or Azure devops agent server. You could use private agent to check if it related to the hosted agent.
Besides, for the cancel request is ignored and the job continues to hang, you could set the Build job cancel timeout in minutes in the build options:
So, it won't hang your job all the time.
Hope this helps.
We had the same issue and it really looks like a proxy problem. Since we don't need result files uploaded, the following 'input' helped:
publishRunAttachments: false
Related
Azure Data Factory pipeline have been working fine for 2 years with Self-hosted Integration runtime (Azure VM 16GB)
In last few weeks pipeline have got very unreliable and data is no longer processed correctly.
Data Factory Activity using "MyAzureIntegrationRuntime" instance is failing with "timeout" error.
Some of activities get successfully completed, but most of them get failed.
ADF Monitor tell that node is "Unavailable". Typical ADF activity is Azure SQL lookup.
There is no event log errors in Virtual Machine. There seems to be enough CPU/RAM to execute IR activities.
Reboot of VM has once helped to recover connectivity and pipelines.
However latest VM reboot restored status to "Running" from “Unavailable”, but many pipeline activities get failed.
Integration Runtime is currently not in High Available Cluster.
There is single VM serving Sandbox, Dev, Test and Prod ADF. It has worked fine for last years except last 2 weeks.
How could I find what is the problem and fixed it?
How could I find what is the problem and fixed it?
For failed activities that are running on a self-hosted IR or a shared IR, the service supports viewing and uploading error logs. To get the error report ID, follow the instructions here, and then enter the report ID to search for related known issues.
On the Monitor page for the service UI, select Pipeline runs.
Under Activity runs, in the Error column, select the highlighted button to display the activity logs, as shown in the following screenshot:
The activity logs are displayed for the failed activity run.
For further assistance, select Send logs.
The Share the self-hosted integration runtime (IR) logs with Microsoft window opens.
Select which logs you want to send.
For a self-hosted IR, you can upload logs that are related to the failed activity or all logs on the self-hosted IR node.
For a shared IR, you can upload only logs that are related to the failed activity.
When the logs are uploaded, keep a record of the Report ID for later use if you need further assistance to solve the issue.
I'm new to the whole Azure DevOps world and just got transferred to a new team that does just that.
One of my assignments is to fix an issue with a pipeline where one of the steps runs a shell script that installs an application. Currently, the step seems to run without any issue shown on the log, but when we connect to the container's pod, the app is not there.
If we run the script directly inside the pod, the application is installed correctly. I'm not sure how to track this. One of the things I've tried was to check the event log to see if there's any error while the installation is executed:Get-Eventlog -LogNmae "Windows PowerShell" -Newest 20, so far no luck here. Again, kinda of new at this, not sure what other tools are out there to track the reason why the script is not installing during the pipeline execution.
To troubleshoot your pipeline run, you can configure your pipeline logs to be more verbose.
1, To configure verbose logs for a single run, you can start a new build by choosing Run pipeline and selecting Enable system diagnostics, Run.
2,To configure verbose logs for all runs, you can add a variable named system.debug and set its value to true.
You can also try logging into your agent server and check for the event log. See this blog for view event log on windows.
The issue was related to how the task was awaited. Adding this piped params helped us solve the issue:
RUN powershell C:\dev\myprocess.ps1 -PassThru | Wait-Process;
I am trying to run the Test Compelete Scripts on a virtual machine through Azure DevOps pipeline.
When I run the Azure Devops pipeline release. Execution starts in VM. But when I check the logs I see that the user action in the Test scripts are not performed.
Eg: My application is launched, and next steps when I have to select a radio button execution fails and also in
TE log I can see "User session is disconnected and also I can see that it is not able to identify the object)
You typically get this error when the test is run from a scheduled task with the Run whether user is logged on or not option. The error occurs because in this case the test is run in a non-interactive session and cannot interact with the GUI.
To resolve the problem, do the following:
Modify properties of the scheduled task to use the Run only when the
user is logged on option and specify the user account under which
the task will be run.
Make sure that the specified user is logged into the system when the
task is triggered.
Here is the document you can refer to.
In our dev environment we have lots of repos, lots of builds and lots of buildservers, and most of the time things work just like they should - however, we are seeing an increase in builds that fail because of timeouts.
These timeouts are not happening because we are getting close to the limit, but because something "gets stuck/blocked" in the pipeline and it stays on that step until timeout kills the build.
To better debug why that happens, we need to be able to query what builds fails because of this timeout, so we for instance can see, if it is a particular build server or agent that has this problem.
We can not find anything in the API that would give us the timeout error, but we can see that the UI is able to deduct it somehow:
So far we have narrowed it down to query all builds with completed status (through this API), but we get no completion reason, and buildtimes are never exact the same as the timeout of the build defintion, so "guessing" it from the execution plan will also be a bit shaky.
How can we filter our builds down to only the builds that have timed out?
We can use the below API to get details for a build.
Note: do not add timelineId, we should list all info
GET https://dev.azure.com/{organization}/{project}/_apis/build/builds/{buildId}/timeline?api-version=6.1-preview.2
If the build is canceled because of the timeout setting, we can get the message: The job running on agent Hosted Agent ran longer than the maximum time of xxx minutes. For more information, see https://go.microsoft.com/fwlink/?linkid=2077134
By the way, we can use the API Builds - List to filter all failed build. if the build is canceled due to a timeout setting. the result is failed instead of cancel.
I have a powershell script that runs for ~30 minutes (waiting on various process' to finish). At the end, it writes a message to the event log, determining if the process was a success or failure. I plan on hosting this script on teamcity and want the build to fail, but don't know how to handle the interaction between the script and teamcity in order for this to happen.
I'm looking for a way to make a powershell script that is ran remotely to communicate to teamcity whether it was a failure or success. I've read up on a lot of the teamcity documentation and I'm still not sure how to start going about this.
You should probably consider using TeamCity Service Messages, or specifically Reporting Build Problems.
An example of how to emit a service message using PowerShell (assuming you're using the PowerShell build step):
Write-Output "##teamcity[buildStatus text='I am a successful build']"
or
Write-Output "##teamcity[buildProblem description='$powershell_error_message']"
where you can inject the captured powershell error message.
To intentionally fail a build in teamcity you should use "[Environment] :: Exit (1)" instead of "Exit 1".
More information can be obtained at the link below:
https://confluence.jetbrains.com/display/TCD9/PowerShell