Downloading files with a manually generated signed url errors with "SignatureDoesNotMatch" - google-cloud-storage

Question

I've followed this guide to implement a URL signer for cloud storage download URLs in Rust.
Bucket and service account have necessary permissions.
I'm using signBlob technique to sign the string.
Here is the output of my canonical request.
GET
/abc.pd
x-goog-algorithm=GOOG4%2DRSA%2DSHA256&x-goog-credential=basak%2Dservice%40alibasak%2Eiam%2Egserviceaccount%2Ecom%2F20220521%2Fauto%2Fstorage%2Fgoog4%5Frequest&x-goog-date=20220521T004156Z&x-goog-expires=3600&x-goog-signedHeaders=host
host:basak.storage.googleapis.com
host
UNSIGNED-PAYLOAD
My string to sign.
(Hash is calculated from the canonical request with SHA256)
GOOG4-RSA-SHA256
20220521T004156Z
20220521/auto/storage/goog4_request
43b27d5947adf3b915d8a5a51cfe2f5cf1344a12b3d3731287cf4741525eabef
The final signed url I produce.
https://basak.storage.googleapis.com/abc.pd?x-goog-algorithm=GOOG4%2DRSA%2DSHA256&x-goog-credential=basak%2Dservice%40alibasak%2Eiam%2Egserviceaccount%2Ecom%2F20220521%2Fauto%2Fstorage%2Fgoog4%5Frequest&x-goog-date=20220521T004156Z&x-goog-expires=3600&x-goog-signedHeaders=host&x-goog-signature=TBqhQ9edLBnGC0z8jhWPAt6NDGM87PHcdAZBt2bcVfd9N/zE1i/HY0jUi5XZoMUgABoBvU36dizS4lr8PrOjXG6GT9KgXbEBcrQqPb83outeAfhL2pgXgbQjXcetFX7cYzY3GSULRWs7+7wH0rxMWiQ6E3tahraBUXI9VZ2XqbUGLuZZXtOhExQ14dKWOnvVVEl0C5BehMEXpDzMFXSWUrsuDMpDlN86nwaJgcGlTNBBrot7J2gMde+xGcJ4zC/c3BADoKHGdjhyOzQh7zToQHnpkLHdEVILUD7k4CN6f9TNzvUGsqNABJ3H4t3fwDgZ/2OqSJ9Na6Xisi2OMMgaTQ==
Trying a get request with this URL results with this error,
<?xml version='1.0' encoding='UTF-8'?>
<Error>
<Code>SignatureDoesNotMatch</Code>
<Message>The request signature we calculated does not match the signature you provided. Check your Google secret key and signing method.</Message>
<StringToSign>GOOG4-RSA-SHA256
20220521T004156Z
20220521/auto/storage/goog4_request
e38b8ab7532e4fa3009a7234313086ed45551be91398a5eef856eef4bfd857cf</StringToSign>
<CanonicalRequest>GET /abc.pd x-goog-algorithm=GOOG4-RSA-SHA256&x-goog-credential=basak-service%40alibasak.iam.gserviceaccount.com%2F20220521%2Fauto%2Fstorage%2Fgoog4_request&x-goog-date=20220521T004156Z&x-goog-expires=3600&x-goog-signedHeaders=host
host:basak.storage.googleapis.com
host
UNSIGNED-PAYLOAD</CanonicalRequest>
</Error>
I've read that sometimes a header mismatch was causing this, I've tried to make this request in browser, curl and with a node script. Each of those give me the same error.
Could you point me to the right direction about debugging this or maybe you may think of causes which I'm not aware yet.
Thanks a lot.
Update
John Hanley suggested that the problem might be in the encoding.
I've applied the fixes he suggested such as sorting the headers and took the example in the end of this page as a reference to percent encode the URL.
Now the canonical request looks like this
GET
/abc.pd
X-Goog-Algorithm=GOOG4-RSA-SHA256&X-Goog-Credential=basak-service%40alibasak.iam.gserviceaccount.com%2F20220521%2Fauto%2Fstorage%2Fgoog4_request&X-Goog-Date=20220521T082931Z&X-Goog-Expires=3600&X-Goog-SignedHeaders=host
host:basak.storage.googleapis.com
host
UNSIGNED-PAYLOAD
and the final URL looks like this,
https://basak.storage.googleapis.com/abc.pd?X-Goog-Algorithm=GOOG4-RSA-SHA256&X-Goog-Credential=basak-service%40alibasak.iam.gserviceaccount.com%2F20220521%2Fauto%2Fstorage%2Fgoog4_request&X-Goog-Date=20220521T082931Z&X-Goog-Expires=3600&X-Goog-SignedHeaders=host&X-Goog-Signature=SEf8qecHjF+T+Uy71jygrEzrj2I1TXeKdUAOpm+BEf0Xm0W9e/LH5MC5BN+JA+OZ44EHN///Ai8ZO7GXqmE6+xbjrVhPlVETkLQSQ84GVecxj6onauRmpDTpxW8RXc+fXdzia75YndT5D1HKjvy8nI26Va8YMG2U3W/A7HMjg5YG+Cmcow3Pw1jpjIG+03gLLobhVMTnyp63S5AnycWT3Wzf7uo6l/WR7MxMK7pKA3isXXhQg7g9o8XUFFMvesDZUsI6mLYakxFHKHL42p1h6/P1SCFke1lpuYa9pV/EVnUMVfLp1ZfkgA1WBLqKZzyPNhMepfgIsUiUv2AKx/VoWQ==
I've also used the cli command,
gsutil -i basak-service#alibasak.iam.gserviceaccount.com signurl -d 10m -u gs://basak/abc.pd
to sign the url for comparison and the result is
https://storage.googleapis.com/basak/abc.pd?x-goog-signature=454801bc0d9ae19b5c5465a4e76846abbb1775549fe2532c839952125a54c5b1cb8f89385ac1cc1a8313e23945951d259a2ddc5a1c95890a205db1ab30a32d6efe8a2e706d03c68de6c4a502f50ff1a1b7fae5b94e3aa85768bfe473abf557eb8ae4e2b15ff9a5bb73ccb3d0bd1b8470cbe0bcb7ec6538fc575664672d641cb9f3c63ec04c41c13a6f2f6329290ce82bc57a700137edcf6fbade0885dd8130ebe2ba9bfe48f91ec94bf6e85b2ac8a7a26aeda77cbd5b0c30136d77defffeb5493f08bf9479f84522c1cb78503693e8ceab79fe0c6282ac4ecaa7e33b6355d2a7f870409b777512819ef54628a86a43b14ce8370477d11a9f857c2ec4ada90d6b&x-goog-algorithm=GOOG4-RSA-SHA256&x-goog-credential=basak-service%40alibasak.iam.gserviceaccount.com%2F20220521%2Fus%2Fstorage%2Fgoog4_request&x-goog-date=20220521T090829Z&x-goog-expires=600&x-goog-signedheaders=host
this url works perfectly.
The differences are:
Header order starts with the signature and continues sorted.
Headers are lowercase
The signature does not contain slashes. (It could be that I need to decode the signature? I'll check.)
I've implemented the first two in my code quickly and nothing changed.
On the other hand I suspect that maybe I'm doing something wrong with the signature since neither the example nor the gsutil generated url have slashes in the signature..
As John Hanley suggested I am sharing my code.
(The code is written in a sketchy way to debug and make it work first. Imports and use statements omitted.)
const SERVICE_ACCOUNT_EMAIL: &str = "basak-service#alibasak.iam.gserviceaccount.com";
#[derive(Serialize)]
struct SignRequest {
// The sequence of service accounts in a delegation chain. Each service account must be granted the roles/iam.serviceAccountTokenCreator role on its next service account in the chain. The last service account in the chain must be granted the roles/iam.serviceAccountTokenCreator role on the service account that is specified in the name field of the request.
// The delegates must have the following format: projects/-/serviceAccounts/{ACCOUNT_EMAIL_OR_UNIQUEID}. The - wildcard character is required; replacing it with a project ID is invalid.
delegates: Vec<String>,
// Required. The bytes to sign.
// A base64-encoded string.
payload: String,
}
#[derive(Deserialize)]
#[serde(rename_all = "camelCase")]
struct SignResponse {
key_id: String,
signed_blob: String,
}
fn credential_scope(date: &str, location: &str, service: &str, request_type: &str) -> String {
format!("{date}/{location}/{service}/{request_type}")
}
const FRAGMENT: &AsciiSet = &CONTROLS
// ?=!#$&'()*+,:;#[]."
.add(b'/')
.add(b' ')
.remove(b'=')
.remove(b'!')
.remove(b'#')
.remove(b'$')
.remove(b'&')
.remove(b'\'')
.remove(b'(')
.remove(b')')
.remove(b'*')
.remove(b'+')
.remove(b',')
.remove(b':')
.remove(b';')
.add(b'#')
.remove(b'[')
.remove(b']')
.remove(b'.')
.remove(b'"');
async fn generate_signed_url(
bucket_name: &str,
object_name: &str,
expiration: &str,
token: &gcp_auth::Token,
) -> Result<String, reqwest::Error> {
// We'll use google's signing service to generate a signature.
let sign_blob_url = format!("https://iamcredentials.googleapis.com/v1/projects/-/serviceAccounts/{SERVICE_ACCOUNT_EMAIL}:signBlob?alt=json");
let mut query_parameters = std::collections::BTreeMap::new();
let canonical_uri = format!("/{object_name}");
let date = chrono::offset::Utc::today().format("%Y%m%d").to_string();
let time_stamp = chrono::offset::Utc::now()
.format("%Y%m%dT%H%M%SZ")
.to_string();
let credential_scope = credential_scope(&date, "auto", "storage", "goog4_request");
let credential = format!("{SERVICE_ACCOUNT_EMAIL}/{credential_scope}");
let host = format!("{bucket_name}.storage.googleapis.com");
let algorithm = "GOOG4-RSA-SHA256";
query_parameters.insert("x-goog-algorithm", algorithm);
query_parameters.insert("x-goog-credential", &credential);
query_parameters.insert("x-goog-date", &time_stamp);
query_parameters.insert("x-goog-expires", expiration);
query_parameters.insert("x-goog-signedHeaders", "host");
let mut canonical_query_string =
query_parameters
.iter()
.fold("".to_owned(), |mut acc, (k, v)| {
//
let encoded_k = percent_encode(k.as_bytes(), FRAGMENT);
let encoded_v = percent_encode(v.as_bytes(), FRAGMENT);
acc.push_str(&format!("{}={}&", encoded_k, encoded_v));
acc
});
canonical_query_string.pop();
// let canonical_headers = format!("content-type:text/plain\nhost:{host}");
// let signed_headers = "content-type;host";
let canonical_headers = format!("host:{host}");
let signed_headers = "host";
// HTTP_VERB
// PATH_TO_RESOURCE
// CANONICAL_QUERY_STRING
// CANONICAL_HEADERS
// \n
// SIGNED_HEADERS
// PAYLOAD
let canonical_request = format!(
"GET\n{canonical_uri}\n{canonical_query_string}\n{canonical_headers}\n\n{signed_headers}\nUNSIGNED-PAYLOAD"
);
println!("canonical_request: {}", canonical_request);
let mut hasher = Sha256::new();
hasher.update(canonical_request);
let hashed_canonical_request = format!("{:x}", hasher.finalize());
let string_to_sign =
format!("{algorithm}\n{time_stamp}\n{credential_scope}\n{hashed_canonical_request}");
println!("string_to_sign: {}", string_to_sign);
let body = SignRequest {
delegates: vec![],
payload: base64::encode(string_to_sign),
};
let client = reqwest::Client::new();
let response = client
.post(sign_blob_url)
.bearer_auth(token.as_str())
.json(&body)
.send()
.await?
.bytes()
.await?;
dbg!(&response);
let sign_response: SignResponse = serde_json::from_slice(&response).unwrap();
let signed_url = format!(
"https://{host}{canonical_uri}?x-goog-signature={}&{canonical_query_string}",
// percent_encode(response.signed_blob.as_bytes(), FRAGMENT)
sign_response.signed_blob
);
Ok(signed_url)
}

Answer 1

First of all big thanks to John Hanley for motivating me to resolve this.
As he suggested there were multiple issues.
The percent encoding was incorrect and comparing it with the CLI output helped me to correct it.
The headers should be sorted and the same technique of comparing also helped me to find the right sorting.
The major issue was something else though.
It is my mistake that I've missed this in the docs.
In the doc of signedBlob it clearly states that the signature in the response is a Base64 encoded string. I was using it plainly instead.
After receiving the response doing,
#[derive(Deserialize)]
#[serde(rename_all = "camelCase")]
struct SignResponse {
key_id: String,
signed_blob: String,
}
let response: SignResponse = serde_json::from_slice(&response).unwrap();
// First decode the signature
let decoded_signature = base64::decode(response.signed_blob).unwrap();
// Then make a string with hex representation of bytes
let hex_encoded_signature = hex::encode(decoded_signature);
// Then construct the url
let signed_url = format!("https://{host}{canonical_uri}?x-goog-signature={hex_encoded_signature}&{canonical_query_string}");
resolved the issue and I can even use the resulting URL in the browser!
By experimentation I've also derived that using lowercase (x-goog-..) or uppercase (X-Goog..) headers or even mixing them do not change anything.
This comment also confused me a little throughout the process:
When defining the resource path, you must percent encode the following reserved characters: ?=!#$&'()*+,:;#[]." Any other percent encoding used in the URL should also be included in the resource path.
which is taken from this source
Because it says to percent encode that character set but does not include characters such as /, etc. which should be percent encoded also and such as . shouldn't be encoded.
I am still confused about it a little but maybe I should read more about percent encoding techniques.
Anyway I'm happy that it is resolved and hope it helps someone later.