Azure DevOps Services - self hosted agents have gone Offline - azure-devops-self-hosted-agent

Question

When starting up the windows service for the agent following error is logged in the event log:
Failed to create session. VS30063: You are not authorized to access https://dev.azure.com.
(Level: Error; Source: VstsAgentService)
The agents are configured using a PAT.
The windows services are run as Network Service.
Configured according to: https://learn.microsoft.com/en-us/azure/devops/pipelines/agents/v2-windows?view=azure-devops
The agents have been running successfully for weeks but s omething has broken our agents. Of course it will afterwards be connected to some type of change. Thanks in advance for any troubleshooting hints.
Extractions from Agent_20200421-125954-utc.log.
[2020-04-21 12:59:58Z INFO RSAEncryptedFileKeyManager] Loading RSA key parameters from file D:\agent5\.credentials_rsaparams
[2020-04-21 12:59:58Z INFO RSAEncryptedFileKeyManager] Loading RSA key parameters from file D:\agent5\.credentials_rsaparams
[2020-04-21 12:59:58Z INFO VisualStudioServices] AAD Correlation ID for this token request: Unknown
[2020-04-21 12:59:58Z WARN VisualStudioServices] Authentication failed with status code 401.
...
[2020-04-21 12:59:58Z ERR VisualStudioServices] POST request to https://dev.azure.com/{organization}/_apis/distributedtask/pools/1/sessions is not authorized. Details: VS30063: You are not authorized to access https://dev.azure.com.
[2020-04-21 12:59:58Z ERR MessageListener] Catch exception during create session.
[2020-04-21 12:59:58Z ERR MessageListener] Microsoft.VisualStudio.Services.Common.VssUnauthorizedException: VS30063: You are not authorized to access https://dev.azure.com.
[2020-04-21 12:59:58Z INFO MessageListener] Non-retriable exception: VS30063: You are not authorized to access https://dev.azure.com.
[2020-04-21 12:59:58Z ERR Terminal] WRITE ERROR: Failed to create session. VS30063: You are not authorized to access https://dev.azure.com.

Answer 1

Something has broken our agents.
Since completely know nothing to what are the changes and how's the broken of agent now, just give some possible suggestion and you can have a try.
Go D:\agent5, and open .credentials with text.
Make sure its content is like below:
{
"scheme": "OAuth",
"data": {
"clientId": "xxxxxx", # This are generated automatically by
"authorizationUrl": "https://vssps.dev.azure.com/{org name}/_apis/oauth2/token",
"oauthEndpointUrl": "https://vssps.dev.azure.com/{org name}/_apis/oauth2/token"
}
}
Then save it, and try to re-run agent again.
If this way still could not make your agent available, or the .credentials file has contained the contents I mentioned above, now I guess the broken should caused on .credentials_rsaparams file.
.credentials_rsaparams is a file which has a RSA private key for OAuth token exchange between agent and server. The server hold the public key, and the server will send a new token to agent encrypted by the public key every 50 mins.
It is the file which generated by server automatically, which means you can not modified it manually since you don't know what you should input.
At present, you'd better re-configure this agent by following this steps:
1) Run Command line/Powershell line by using Run as Administrator, then cd D:\agent5.
2) Run .\config remove. And do agent configuration steps after the remove succeed.