can't get ADLS Gen2 REST continuation token to work - rest

Question

I'm trying to retrieve list of files and folders form ADLS Gen2. I can get the first 5000 items, but when I use continuation to get the rest (about 17,000 items or so), I get Error 403 (Forbidden). According to documentation, I add the continuation token to URI and to Canonicalized Resource in signature string. However, I cannot get it to work.
I've read the documentation on ADLS Gen2 REST calls and whatever I could find on this, and I can't figure out the issue.
var date = System.DateTime.UtcNow.ToString("R");
string toSign = DefaultSignatureString(date);
toSign +=
$"/{storageaccountname}/{filesystemname}" + "\n" +
$"directory:{dir}" +"\n" +
"recursive:true" + "\n" +
"resource:filesystem";
var signedSignature = SignData(accessKey, toSign);
var uri = $"https://{storageaccountname}.dfs.core.windows.net/{filesystemname}?directory={dir}&recursive=true&resource=filesystem";
HttpWebResponse response = GetWebResponse(storageaccountname, date, signedSignature, uri);
var token_continuation = response.Headers["x-ms-continuation"];
//I get the token_continuation and repeat the previous steps, adding the continuation part:
while (token_continuation != null)
{
date = System.DateTime.UtcNow.ToString("R");
toSign = DefaultSignatureString(date);
toSign +=
$"/{storageaccountname}/{filesystemname}" + "\n" +
$"continuation:{token_continuation}" + "\n" +
$"directory:{dir}" + "\n" +
"recursive:true" + "\n" +
"resource:filesystem";
signedSignature = SignData(accessKey, toSign);
uri = $"https://{storageaccountname}.dfs.core.windows.net/{filesystemname}?directory={dir}&recursive=true&resource=filesystem&continuation={token_continuation}";
response = GetWebResponse(storageaccountname, date, signedSignature, uri);
token_continuation = response.Headers["x-ms-continuation"];
}
//this is my GetWebResponse method
private static HttpWebResponse GetWebResponse(string storageaccountname, string date, string signedSignature, string uri, string continuation = null)
{
WebRequest request = WebRequest.Create(uri);
if (continuation != null)
{
request.Headers.Add($"x-ms-continuation:{continuation}");
}
request.Headers.Add($"x-ms-date:{date}");
request.Headers.Add($"x-ms-version:2018-11-09");
request.Headers.Add($"Authorization:SharedKey {storageaccountname}:{signedSignature}");
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
return response;
}
As I said, I get the first response OK; when I get into while loop, I get the error. What am I doing wrong?

Answer 1

In most of the cases (if not all) continuation token returns with == at the end, which messes up the URI.
For URIs, escape them by replacing == with %3D%3D.
For canonicalized resources, leave the string as is.