How to get retry-after header in csom context? - powershell

Question

I am using PowerShell and CSOM to mirror Sharepoint Online and OneDrive sites with all their files.
Consequently, after a few thousand files/a few hours of file download, an "The operation has timed out" exception is thrown, as expected. This is due to Microsoft's throttling.
To prevent the timeout, I am using the RequestTimeOut Paramter of the CSOM context, and also doing incremental retry, and also throttling the amount of ExecuteQuery() calls to 2 per second, and also decorating the CSOM call. That is all not enough, though.
The http response header of the failed call is supposed to include a "Retry-After" line, which I would like to use to time the retry.
The Exception happens either during Microsoft.SharePoint.Client.ClientContext's ExecuteQuery() or [Microsoft.SharePoint.Client.File]::OpenBinaryDirect().
Here is some simplified code extract:
$Context = New-Object Microsoft.SharePoint.Client.ClientContext($WebURL)
$Context.Credentials = $spoCredential
$Context.RequestTimeout = 60000; # 1 min
$Context.add_ExecutingWebRequest({
param($Source, $EventArgs)
$request = $EventArgs.WebRequestExecutor.WebRequest
$request.UserAgent = "XXX|CsomPs|MyScript/1.0"
})
$Web = $Context.Web
$Context.Load($Web)
$Context.ExecuteQuery()
Which all works perfectly well, until the "The operation has timed out" exception is thrown. Say, $Context.ExecuteQuery() of the sample throws the exception.
How do I access the http response and especially the http response headers and even more especially the Retry-After header within my CSOM powershell script?
Thanks!

Answer 1

First thing, you can't access and custom HTTP response. This is handled by SharePoint Online side.
Second, you should avoid getting throttled or blocked in SharePoint Online.
Microsoft don't have exact throttling limit, so it's better for you to reduce the number of operations per request or reduce frequency of calls
Here is the code of decorating traffic in CSOM
// Get access to source site
using (var ctx = new ClientContext("https://contoso.sharepoint.com/sites/team"))
{
//Provide account and pwd for connecting to SharePoint Online
var passWord = new SecureString();
foreach (char c in pwd.ToCharArray()) passWord.AppendChar(c);
ctx.Credentials = new SharePointOnlineCredentials("contoso#contoso.onmicrosoft.com",
passWord);
// Add our User Agent information
ctx.ExecutingWebRequest += delegate (object sender, WebRequestEventArgs e)
{
e.WebRequestExecutor.WebRequest.UserAgent = "NONISV|Contoso|GovernanceCheck/1.0";
};
// Normal CSOM Call with custom User-Agent information
Web site = ctx.Web;
ctx.Load(site);
ctx.ExecuteQuery();
}
If this is not helpful, you can follow up CoreThrottling this demonstration.

Answer 2

You can access the headers from the failed CSOM response this way. This is interchangeable to Powershell.
try {
context.ExecuteQuery();
}
catch(WebException wex) {
var response = wex.Response as HttpWebResponse;
if (response != null && (response.StatusCode == (HttpStatusCode)429 || response.StatusCode == (HttpStatusCode)503))
{
// Reference the headers in the throttled / failed response
response.Headers
}
}
Full MS code sample (search page for .RetryQuery): https://learn.microsoft.com/en-us/sharepoint/dev/general-development/how-to-avoid-getting-throttled-or-blocked-in-sharepoint-online
After some extensive throttling, i have never actually seen the Retry-After value come back as anything other than 120 (seconds).