In my app, I need to send to a server a base64-encoded string as a query parameter, eg.
&data=AwGZnyx+JUi0PFJoyYSEDpgtlrxP(cut...)==
Problem is, anything works just fine when there isn't a plus sign in my string: on the other hand, everytime there's one, the server code doesn't behave correctly.
I noticed there are escaping functions (eg. addingPercentEncoding) but they don't work with the plus sign.
Aside from removing all the pluses "manually" (eg. with a regex), is there anything else I can do?
At the moment anything works fine if I use:
string.replacingOccurrences(of: "+", with: "%2B")
The server is probably interpreting the + sign as a space because it is often used in query parameters as a substitute for a space. addPercentEncoding isn't going to help you because it only translates non ASCII characters.
You'll need to manually replace + with %2B as you are doing.
.... although
NSString has a version of addPercentEncoding that also takes a CharacterSet as a parameter. So you could create a Characterset with all the base64 characters in it except + using init(charactersIn:) i.e.
let safeChars = Characterset(charactersIn: "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrtuvwxyz0123456789/=")
One good way to construct a URL is to use URLComponents. This should work in almost all cases, and an example is shown below:
let base64 = "AwGZnyx+JUi0PFJoyYSEDpgtlrxP(cut...)=="
var components = URLComponents(string: "https://example.com")
components?.queryItems = [ URLQueryItem(name: "data", value: "AwGZnyx+JUi0PFJoyYSEDpgtlrxP(cut...)==") ]
let url = components?.url
Result:
https://example.com?data=AwGZnyx+JUi0PFJoyYSEDpgtlrxP(cut...)%3D%3D
However, in your particular case, it seems the the server is not correctly handling the + and you need to work around that issue similar to what you have above. The better fix would be for the server to be changed to correctly process the URL.
let base64 = "AwGZnyx+JUi0PFJoyYSEDpgtlrxP(cut...)=="
var components = URLComponents(string: "https://example.com")
components?.queryItems = [ URLQueryItem(name: "data", value: base64) ]
let urlString = components?.string.replacingOccurrences(of: "+", with: "%2B")
Result:
https://example.com?data=AwGZnyx%252BJUi0PFJoyYSEDpgtlrxP(cut...)%3D%3D
Related
I've followed this guide to implement a URL signer for cloud storage download URLs in Rust.
Bucket and service account have necessary permissions.
I'm using signBlob technique to sign the string.
Here is the output of my canonical request.
GET
/abc.pd
x-goog-algorithm=GOOG4%2DRSA%2DSHA256&x-goog-credential=basak%2Dservice%40alibasak%2Eiam%2Egserviceaccount%2Ecom%2F20220521%2Fauto%2Fstorage%2Fgoog4%5Frequest&x-goog-date=20220521T004156Z&x-goog-expires=3600&x-goog-signedHeaders=host
host:basak.storage.googleapis.com
host
UNSIGNED-PAYLOAD
My string to sign.
(Hash is calculated from the canonical request with SHA256)
GOOG4-RSA-SHA256
20220521T004156Z
20220521/auto/storage/goog4_request
43b27d5947adf3b915d8a5a51cfe2f5cf1344a12b3d3731287cf4741525eabef
The final signed url I produce.
https://basak.storage.googleapis.com/abc.pd?x-goog-algorithm=GOOG4%2DRSA%2DSHA256&x-goog-credential=basak%2Dservice%40alibasak%2Eiam%2Egserviceaccount%2Ecom%2F20220521%2Fauto%2Fstorage%2Fgoog4%5Frequest&x-goog-date=20220521T004156Z&x-goog-expires=3600&x-goog-signedHeaders=host&x-goog-signature=TBqhQ9edLBnGC0z8jhWPAt6NDGM87PHcdAZBt2bcVfd9N/zE1i/HY0jUi5XZoMUgABoBvU36dizS4lr8PrOjXG6GT9KgXbEBcrQqPb83outeAfhL2pgXgbQjXcetFX7cYzY3GSULRWs7+7wH0rxMWiQ6E3tahraBUXI9VZ2XqbUGLuZZXtOhExQ14dKWOnvVVEl0C5BehMEXpDzMFXSWUrsuDMpDlN86nwaJgcGlTNBBrot7J2gMde+xGcJ4zC/c3BADoKHGdjhyOzQh7zToQHnpkLHdEVILUD7k4CN6f9TNzvUGsqNABJ3H4t3fwDgZ/2OqSJ9Na6Xisi2OMMgaTQ==
Trying a get request with this URL results with this error,
<?xml version='1.0' encoding='UTF-8'?>
<Error>
<Code>SignatureDoesNotMatch</Code>
<Message>The request signature we calculated does not match the signature you provided. Check your Google secret key and signing method.</Message>
<StringToSign>GOOG4-RSA-SHA256
20220521T004156Z
20220521/auto/storage/goog4_request
e38b8ab7532e4fa3009a7234313086ed45551be91398a5eef856eef4bfd857cf</StringToSign>
<CanonicalRequest>GET /abc.pd x-goog-algorithm=GOOG4-RSA-SHA256&x-goog-credential=basak-service%40alibasak.iam.gserviceaccount.com%2F20220521%2Fauto%2Fstorage%2Fgoog4_request&x-goog-date=20220521T004156Z&x-goog-expires=3600&x-goog-signedHeaders=host
host:basak.storage.googleapis.com
host
UNSIGNED-PAYLOAD</CanonicalRequest>
</Error>
I've read that sometimes a header mismatch was causing this, I've tried to make this request in browser, curl and with a node script. Each of those give me the same error.
Could you point me to the right direction about debugging this or maybe you may think of causes which I'm not aware yet.
Thanks a lot.
Update
John Hanley suggested that the problem might be in the encoding.
I've applied the fixes he suggested such as sorting the headers and took the example in the end of this page as a reference to percent encode the URL.
Now the canonical request looks like this
GET
/abc.pd
X-Goog-Algorithm=GOOG4-RSA-SHA256&X-Goog-Credential=basak-service%40alibasak.iam.gserviceaccount.com%2F20220521%2Fauto%2Fstorage%2Fgoog4_request&X-Goog-Date=20220521T082931Z&X-Goog-Expires=3600&X-Goog-SignedHeaders=host
host:basak.storage.googleapis.com
host
UNSIGNED-PAYLOAD
and the final URL looks like this,
https://basak.storage.googleapis.com/abc.pd?X-Goog-Algorithm=GOOG4-RSA-SHA256&X-Goog-Credential=basak-service%40alibasak.iam.gserviceaccount.com%2F20220521%2Fauto%2Fstorage%2Fgoog4_request&X-Goog-Date=20220521T082931Z&X-Goog-Expires=3600&X-Goog-SignedHeaders=host&X-Goog-Signature=SEf8qecHjF+T+Uy71jygrEzrj2I1TXeKdUAOpm+BEf0Xm0W9e/LH5MC5BN+JA+OZ44EHN///Ai8ZO7GXqmE6+xbjrVhPlVETkLQSQ84GVecxj6onauRmpDTpxW8RXc+fXdzia75YndT5D1HKjvy8nI26Va8YMG2U3W/A7HMjg5YG+Cmcow3Pw1jpjIG+03gLLobhVMTnyp63S5AnycWT3Wzf7uo6l/WR7MxMK7pKA3isXXhQg7g9o8XUFFMvesDZUsI6mLYakxFHKHL42p1h6/P1SCFke1lpuYa9pV/EVnUMVfLp1ZfkgA1WBLqKZzyPNhMepfgIsUiUv2AKx/VoWQ==
I've also used the cli command,
gsutil -i basak-service#alibasak.iam.gserviceaccount.com signurl -d 10m -u gs://basak/abc.pd
to sign the url for comparison and the result is
https://storage.googleapis.com/basak/abc.pd?x-goog-signature=454801bc0d9ae19b5c5465a4e76846abbb1775549fe2532c839952125a54c5b1cb8f89385ac1cc1a8313e23945951d259a2ddc5a1c95890a205db1ab30a32d6efe8a2e706d03c68de6c4a502f50ff1a1b7fae5b94e3aa85768bfe473abf557eb8ae4e2b15ff9a5bb73ccb3d0bd1b8470cbe0bcb7ec6538fc575664672d641cb9f3c63ec04c41c13a6f2f6329290ce82bc57a700137edcf6fbade0885dd8130ebe2ba9bfe48f91ec94bf6e85b2ac8a7a26aeda77cbd5b0c30136d77defffeb5493f08bf9479f84522c1cb78503693e8ceab79fe0c6282ac4ecaa7e33b6355d2a7f870409b777512819ef54628a86a43b14ce8370477d11a9f857c2ec4ada90d6b&x-goog-algorithm=GOOG4-RSA-SHA256&x-goog-credential=basak-service%40alibasak.iam.gserviceaccount.com%2F20220521%2Fus%2Fstorage%2Fgoog4_request&x-goog-date=20220521T090829Z&x-goog-expires=600&x-goog-signedheaders=host
this url works perfectly.
The differences are:
Header order starts with the signature and continues sorted.
Headers are lowercase
The signature does not contain slashes. (It could be that I need to decode the signature? I'll check.)
I've implemented the first two in my code quickly and nothing changed.
On the other hand I suspect that maybe I'm doing something wrong with the signature since neither the example nor the gsutil generated url have slashes in the signature..
As John Hanley suggested I am sharing my code.
(The code is written in a sketchy way to debug and make it work first. Imports and use statements omitted.)
const SERVICE_ACCOUNT_EMAIL: &str = "basak-service#alibasak.iam.gserviceaccount.com";
#[derive(Serialize)]
struct SignRequest {
// The sequence of service accounts in a delegation chain. Each service account must be granted the roles/iam.serviceAccountTokenCreator role on its next service account in the chain. The last service account in the chain must be granted the roles/iam.serviceAccountTokenCreator role on the service account that is specified in the name field of the request.
// The delegates must have the following format: projects/-/serviceAccounts/{ACCOUNT_EMAIL_OR_UNIQUEID}. The - wildcard character is required; replacing it with a project ID is invalid.
delegates: Vec<String>,
// Required. The bytes to sign.
// A base64-encoded string.
payload: String,
}
#[derive(Deserialize)]
#[serde(rename_all = "camelCase")]
struct SignResponse {
key_id: String,
signed_blob: String,
}
fn credential_scope(date: &str, location: &str, service: &str, request_type: &str) -> String {
format!("{date}/{location}/{service}/{request_type}")
}
const FRAGMENT: &AsciiSet = &CONTROLS
// ?=!#$&'()*+,:;#[]."
.add(b'/')
.add(b' ')
.remove(b'=')
.remove(b'!')
.remove(b'#')
.remove(b'$')
.remove(b'&')
.remove(b'\'')
.remove(b'(')
.remove(b')')
.remove(b'*')
.remove(b'+')
.remove(b',')
.remove(b':')
.remove(b';')
.add(b'#')
.remove(b'[')
.remove(b']')
.remove(b'.')
.remove(b'"');
async fn generate_signed_url(
bucket_name: &str,
object_name: &str,
expiration: &str,
token: &gcp_auth::Token,
) -> Result<String, reqwest::Error> {
// We'll use google's signing service to generate a signature.
let sign_blob_url = format!("https://iamcredentials.googleapis.com/v1/projects/-/serviceAccounts/{SERVICE_ACCOUNT_EMAIL}:signBlob?alt=json");
let mut query_parameters = std::collections::BTreeMap::new();
let canonical_uri = format!("/{object_name}");
let date = chrono::offset::Utc::today().format("%Y%m%d").to_string();
let time_stamp = chrono::offset::Utc::now()
.format("%Y%m%dT%H%M%SZ")
.to_string();
let credential_scope = credential_scope(&date, "auto", "storage", "goog4_request");
let credential = format!("{SERVICE_ACCOUNT_EMAIL}/{credential_scope}");
let host = format!("{bucket_name}.storage.googleapis.com");
let algorithm = "GOOG4-RSA-SHA256";
query_parameters.insert("x-goog-algorithm", algorithm);
query_parameters.insert("x-goog-credential", &credential);
query_parameters.insert("x-goog-date", &time_stamp);
query_parameters.insert("x-goog-expires", expiration);
query_parameters.insert("x-goog-signedHeaders", "host");
let mut canonical_query_string =
query_parameters
.iter()
.fold("".to_owned(), |mut acc, (k, v)| {
//
let encoded_k = percent_encode(k.as_bytes(), FRAGMENT);
let encoded_v = percent_encode(v.as_bytes(), FRAGMENT);
acc.push_str(&format!("{}={}&", encoded_k, encoded_v));
acc
});
canonical_query_string.pop();
// let canonical_headers = format!("content-type:text/plain\nhost:{host}");
// let signed_headers = "content-type;host";
let canonical_headers = format!("host:{host}");
let signed_headers = "host";
// HTTP_VERB
// PATH_TO_RESOURCE
// CANONICAL_QUERY_STRING
// CANONICAL_HEADERS
// \n
// SIGNED_HEADERS
// PAYLOAD
let canonical_request = format!(
"GET\n{canonical_uri}\n{canonical_query_string}\n{canonical_headers}\n\n{signed_headers}\nUNSIGNED-PAYLOAD"
);
println!("canonical_request: {}", canonical_request);
let mut hasher = Sha256::new();
hasher.update(canonical_request);
let hashed_canonical_request = format!("{:x}", hasher.finalize());
let string_to_sign =
format!("{algorithm}\n{time_stamp}\n{credential_scope}\n{hashed_canonical_request}");
println!("string_to_sign: {}", string_to_sign);
let body = SignRequest {
delegates: vec![],
payload: base64::encode(string_to_sign),
};
let client = reqwest::Client::new();
let response = client
.post(sign_blob_url)
.bearer_auth(token.as_str())
.json(&body)
.send()
.await?
.bytes()
.await?;
dbg!(&response);
let sign_response: SignResponse = serde_json::from_slice(&response).unwrap();
let signed_url = format!(
"https://{host}{canonical_uri}?x-goog-signature={}&{canonical_query_string}",
// percent_encode(response.signed_blob.as_bytes(), FRAGMENT)
sign_response.signed_blob
);
Ok(signed_url)
}
First of all big thanks to John Hanley for motivating me to resolve this.
As he suggested there were multiple issues.
The percent encoding was incorrect and comparing it with the CLI output helped me to correct it.
The headers should be sorted and the same technique of comparing also helped me to find the right sorting.
The major issue was something else though.
It is my mistake that I've missed this in the docs.
In the doc of signedBlob it clearly states that the signature in the response is a Base64 encoded string. I was using it plainly instead.
After receiving the response doing,
#[derive(Deserialize)]
#[serde(rename_all = "camelCase")]
struct SignResponse {
key_id: String,
signed_blob: String,
}
let response: SignResponse = serde_json::from_slice(&response).unwrap();
// First decode the signature
let decoded_signature = base64::decode(response.signed_blob).unwrap();
// Then make a string with hex representation of bytes
let hex_encoded_signature = hex::encode(decoded_signature);
// Then construct the url
let signed_url = format!("https://{host}{canonical_uri}?x-goog-signature={hex_encoded_signature}&{canonical_query_string}");
resolved the issue and I can even use the resulting URL in the browser!
By experimentation I've also derived that using lowercase (x-goog-..) or uppercase (X-Goog..) headers or even mixing them do not change anything.
This comment also confused me a little throughout the process:
When defining the resource path, you must percent encode the following reserved characters: ?=!#$&'()*+,:;#[]." Any other percent encoding used in the URL should also be included in the resource path.
which is taken from this source
Because it says to percent encode that character set but does not include characters such as /, etc. which should be percent encoded also and such as . shouldn't be encoded.
I am still confused about it a little but maybe I should read more about percent encoding techniques.
Anyway I'm happy that it is resolved and hope it helps someone later.
I have a very simple GET Request in Swift 4 using URLSession for the dataTask and the URLComponents to create the URL/String.
The problem I have is as shown in the code snippet below
var url = URLComponents(string: "http://myServer.com:8086/query")
url?.queryItems = [
URLQueryItem(name: "db", value: "Database"),
URLQueryItem(name: "q", value: ("SELECT valueOne,valueTwo FROM \"TABLE\" WHERE \"valueOne\"='\(currValue)' ORDER BY time DESC LIMIT 1"))
]
NSLog((url?.url?.absoluteString)!)
/*
NSLog: http://myServer.com:8086/query?db=Database&q=SELECT2alueOne,valueTwo
0.000000ROM222TABLE220WHERE222valueOne23D'currValue'
0RDER2Y 0me -8ESC2MIT2
*/
I tried building the URL directly and applying the addingPercentEncoding with urlQueryAllowed on the query part of the string but I still got the same result. The only thing that partially worked was to substitute the empty spaces in the query with "+", but there was still a problem with the escaped inverted comas.
I'm quite new to Swift so my assumption is I'm missing something very obvious. Any and all assistance would be greatly appreciated.
I have created this function to generate URL. You can use this function to create your URL with safe ASCII characters.
func urlGenretor() -> URL{
var components = URLComponents()
components.scheme = "https"
components.host = "myServer.com:8086"
components.path = "/query"
let database = URLQueryItem(name: "db", value: "Database")
let queryItem2 = URLQueryItem(name: "q", value: ("SELECT valueOne,valueTwo FROM \"TABLE\" WHERE \"valueOne\"='\(currValue)' ORDER BY time DESC LIMIT 1"))
components.queryItems = [database,queryItem2]
let url = components.url
print(url!)
return url!
So the answer is that NSLog does not print out percent Escaped strings correctly, print on the other hand does. The best thing would have been just to set a breakpoint on the NSLog line and check the url value if it is correct, which it was.
I have a router which contains the lines
var url = URL(string: MyRouter.baseURLString)!
url.appendPathComponent(relativePath)
This replaces "?" with "%3F" which is rejected by the API server. How can I fix this? It's wrong to encode that character.
Because the ? isn't part of a path. It's a separator to signal the beginning of the query string. You can read about the different components of a URL in this article. Each component has its set of valid characters, anything not in that set needs to be percent-escaped. The best option is to use URLComponents which handles the escaping automatically for you:
var urlComponents = URLComponents(string: MyRouter.baseURLString)!
urlComponents.queryItems = [
URLQueryItem(name: "username", value: "jsmith"),
URLQueryItem(name: "password", value: "password")
]
let url = urlComponents.url!
EDIT
https://www.someurl.com/search?&access_token=1,84,848473938;848483,83&_json={"key1":"value1","key2":"value2"}
When declaring a URL that has a JSON string, I obviously need to use braces _json={ } and qoutes \"key1\":\"value1\"
NSURL(string: String), however, magically becomes nil if either of these characters are included in the string.
So as answered correctly here: NSURL is returning nil for a valid URL, I tried using:
let url = NSURL(string: url.stringByAddingPercentEncodingWithAllowedCharacters(NSCharacterSet.URLQueryAllowedCharacterSet())!)
But I believe that's deprecated since it was before Swift 2 was released and I am getting the error: cannot convert value of Type NSCharacterSet to expected argument type NSStringEncoding (aka UInt)
So I tried using
let url = NSURL(string: url.stringByAddingPercentEncodingWithAllowedCharacters(NSUTF8StringEncoding)!)!
and while that did allow NSURL to have a value instead of nil, it did not return the expected results, so something is still wrong.
I know that the format is correct, because if I type the URL string manually in a browser, I get the expected result. If i copy/paste the encoded version from Xcode, it gives me the wrong result as did Swift when encoding as shown above.
Any insight would be much appreciated.
You can modify a mutable character set to remove an allowed character: since you want the commas to be encoded, remove the comma from the URLQueryAllowedCharacterSet before using it.
In Swift 2, we need to dance with NSMutableCharacterSet like this:
let sourceURL = "https://www.someurl.com/search?&access_token=1,84,848473938;848483,83&_json={\"key1\":\"value1\",\"key2\":\"value2\"}"
let charSet = NSMutableCharacterSet()
charSet.formUnionWithCharacterSet(NSCharacterSet.URLQueryAllowedCharacterSet())
charSet.removeCharactersInString(",")
let url = NSURL(string: sourceURL.stringByAddingPercentEncodingWithAllowedCharacters(charSet)!)
print(url!)
Prints:
https://www.someurl.com/search?&access_token=1%2C84%2C848473938;848483%2C83&_json=%7B%22key1%22:%22value1%22%2C%22key2%22:%22value2%22%7D
To do the same thing with Swift 3 we're using the CharacterSet struct instead of NSMutableCharacterSet but it's the same idea:
var charSet = CharacterSet()
charSet.formUnion(.urlQueryAllowed)
charSet.remove(",")
if let encoded = sourceURL.addingPercentEncoding(withAllowedCharacters: charSet) {
if let url = URL(string: encoded) {
print(url)
}
}
As in Swift2 stringByAddingPercentEscapesUsingEncoding() is deprecated instead of stringByAddingPercentEncodingWithAllowedCharacters() is used.
But how to encode the especial character like ' % & in swift2
For example in iOS8(swift1.2) i used following code for encoding
NSURL(string: "myurl.php?deviceName=name’phone".stringByAddingPercentEscapesUsingEncoding(NSWindowsCP1250StringEncoding)!)
it work fine i.e. on server it decode correctly.
But in iOS9(Swift2.0) i used following code
NSURL(string: "myurl.php?deviceName=name ’phone".stringByAddingPercentEncodingWithAllowedCharacters(NSCharacterSet.URLFragmentAllowedCharacterSet())!)
It will not decode properly.
please tell me how i can encode special charater in swift2.0 ?
EDIT :
Eric D answer is right but when i encode below stringURL it will not encode properly.
Why?
let stringURL = "https://my.server.com/login.php?e=email&dn=my’s iPad&d=5&er=D550772E-34BB-4DCB-89C9-E746FAD83D24&tz=330"
print(stringURL)
let charSet = NSCharacterSet.URLPathAllowedCharacterSet()
let encoded = stringURL.stringByAddingPercentEncodingWithAllowedCharacters(charSet)!
let url = NSURL(string: encoded.stringByAddingPercentEncodingWithAllowedCharacters(NSCharacterSet.URLFragmentAllowedCharacterSet())!)!
print(url) //https%253A//my.server.com/login.php%253Fe=email&dn=my%25E2%2580%2599s%2520iPad&d=5&er=D550772E-34BB-4DCB-89C9-E746FAD83D ... 4&tz=330
EDIT 2:
How to encode NSWindowsCP1250StringEncoding in swift2.0 ?
Use URLPathAllowedCharacterSet() as character set:
let stringURL = "myurl.php?deviceName=name'phone"
let charSet = NSCharacterSet.URLPathAllowedCharacterSet()
let encoded = stringURL.stringByAddingPercentEncodingWithAllowedCharacters(charSet)!
let url = NSURL(string: encoded)!
print(url) // "myurl.php%3FdeviceName=name'phone"
Note that ' doesn't have to be encoded, it's a valid URL character.
Update: in the comments you state that it's still not working because your encoded URL is truncated and contains ..., but actually this is likely to be just a printed description truncated by NSURL; the actual URL contained by the NSURL object should be ok. Otherwise it would be nil. You should check on your server side for problems with very long but correct URLs.
let newURLEncodedString = urlString.stringByAddingPercentEncodingWithAllowedCharacters(NSCharacterSet.URLQueryAllowedCharacterSet())