Google Cloud Services put Fails when Using Signed URL - google-cloud-storage

Question

I am not able to PUT a file to google cloud services via a signed URL. When I try to do a PUT from a JS Client, I get:
"SignatureDoesNotMatch...The request signature we calculated does not match the signature you provided. Check your Google secret key and signing method."
When I try to post the file using CURL, I get the same error.
The curl command I use is:
#!/bin/bash
URL="https://storage.googleapis.com/..."
echo $URL
curl $URL -H "Content-Type: image/jpg" --upload-file b.jpg
I have configured the bucket I intend to post data to based on the documentation, I have generated a service account with the key, and this key is used to generate the signed url.
The request I sign is of the form:
PUT
image/jpg
1234567890
my-bucket/b.jpg
where the expiration and bucket names are set and computed.
I have the following Groovy code to generate signed urls:
public String sign(PrivateKey key, String toSign) {
Signature signer = Signature.getInstance("SHA256withRSA");
signer.initSign(key);
signer.update(toSign.getBytes("UTF-8"));
byte[] rawSignature = signer.sign();
String s = new String(Base64.encodeBase64(rawSignature), "UTF-8");
return s;
}
public String signUrl(PrivateKey key, String clientId, String method, String md5, String contentType,
long expiration, String gcsPath) {
String toSign = "${method}\n${md5}\n${contentType}\n${expiration}\n${gcsPath}";
String signature = sign(key, toSign);
String url = java.net.URLEncoder.encode(signature);
return url;
}
public String generateSignedUrl(PrivateKey key, String clientId, String method, String md5, String contentType,
long expiration, String gcsPath) {
String canonicalizedResource = "/${gcsPath}";
String signature = signUrl(key, clientId, method, md5, contentType, expiration, canonicalizedResource);
String finalUrl = "https://storage.googleapis.com/${gcsPath}?GoogleAccessId=${clientId}&Expires=${expiration}&Signature=${signature}"
finalUrl
}
This code is accompanied with the following passing unit test lifted straight out of the gsutils github project (https://github.com/GoogleCloudPlatform/gsutil/blob/master/gslib/tests/test_signurl.py):
#Test
void thatWeCanSignAPutUrlCorrectly() {
String expected = """https://storage.googleapis.com/test/test.txt?GoogleAccessId=test#developer.gserviceaccount.com&Expires=1391816302&Signature=A6QbgTA8cXZCtjy2xCr401bdi0e7zChTBQ6BX61L7AfytTGEQDMD%2BbvOQKjX7%2FsEh77cmzcSxOEKqTLUDbbkPgPqW3j8sGPSRX9VM58bgj1vt9yU8cRKoegFHXAqsATx2G5rc%2FvEliFp9UWMfVj5TaukqlBAVuzZWlyx0aQa9tCKXRtC9YcxORxG41RfiowA2kd8XBTQt4M9XTzpVyr5rVMzfr2LvtGf9UAJvlt8p6T6nThl2vy9%2FwBoPcMFaOWQcGTagwjyKWDcI1vQPIFQLGftAcv3QnGZxZTtg8pZW%2FIxRJrBhfFfcAc62hDKyaU2YssSMy%2FjUJynWx3TIiJjhg%3D%3D""";
long expiration = 1391816302;
String signedUrl = gsUtils.generateSignedUrl(privateKey, "test#developer.gserviceaccount.com","PUT", "", "", expiration, "test/test.txt")
assertEquals(expected, signedUrl);
}
Thank you for whatever insights you may be able to provide, I have been at this problem for a while.

Answer 1

Debugging signed URL logic is difficult. There is a useful trick that helps, though. An error response like the one you describe will look like this:
<?xml version='1.0' encoding='UTF-8'?><Error><Code>SignatureDoesNotMatch</Code>
<Message>The request signature we calculated does not match the signature you provided. Check your Google secret key and signing method.</Message>
<StringToSign>PUT
text/jpeg
1472720161
/myBucket/test.txt</StringToSign></Error>
That last bit, <StringToSign>, is critical. The string in there is exactly the string that GCS will sign, and it's the string that you should also be signing. Compare you string against this one; it will probably be different in some way.
Also, because implementing this signing logic is tricky, the gcloud-java library has a signUrl() method that I recommend you use instead of implementing the logic yourself.

Answer 2

One reason that might cause this error (happened to me before) is when you generate a base64 encoded signature with your signed string, the encoded signature may contain illegal url characters + and /. Make sure you replace them in the string with %2B and %2F, respectively.