I wrote a gateway application using Spring cloud Greenwich binaries. I'm seeing issues when special characters are present in URL. The request fails with below exception in Spring gateway when request URI contains special characters.
localhost:8080/myresource/WG_splchar_%26%5E%26%25%5E%26%23%25%24%5E%26%25%26*%25%2B)!%24%23%24%25%26%5E_new
When I hit above url, Spring fails with below exception. I'm not able to figure out why it's an invalid sequence and how things like these can be handled.
java.lang.IllegalArgumentException: Invalid encoded sequence "%^&#%$^&%&*%+)!$#$%&^_new"
at org.springframework.util.StringUtils.uriDecode(StringUtils.java:741) ~[spring-core-5.1.4.RELEASE.jar:5.1.4.RELEASE]
at org.springframework.http.server.DefaultPathContainer.parsePathSegment(DefaultPathContainer.java:126) ~[spring-web-5.1.4.RELEASE.jar:5.1.4.RELEASE]
at org.springframework.http.server.DefaultPathContainer.createFromUrlPath(DefaultPathContainer.java:111) ~[spring-web-5.1.4.RELEASE.jar:5.1.4.RELEASE]
at org.springframework.http.server.PathContainer.parsePath(PathContainer.java:76) ~[spring-web-5.1.4.RELEASE.jar:5.1.4.RELEASE]
at org.springframework.cloud.gateway.handler.predicate.PathRoutePredicateFactory.lambda$apply$2(PathRoutePredicateFactory.java:79) ~[spring-cloud-gateway-core-2.1.0.RC3.jar:2.1.0.RC3]
at org.springframework.cloud.gateway.support.ServerWebExchangeUtils.lambda$toAsyncPredicate$1(ServerWebExchangeUtils.java:128) ~[spring-cloud-gateway-core-2.1.0.RC3.jar:2.1.0.RC3]
at org.springframework.cloud.gateway.handler.AsyncPredicate.lambda$and$1(AsyncPredicate.java:35) ~[spring-cloud-gateway-core-2.1.0.RC3.jar:2.1.0.RC3]
at org.springframework.cloud.gateway.handler.RoutePredicateHandlerMapping.lambda$null$2(RoutePredicateHandlerMapping.java:112) ~[spring-cloud-gateway-core-2.1.0.RC3.jar:2.1.0.RC3]
at reactor.core.publisher.MonoFilterWhen$MonoFilterWhenMain.onNext(MonoFilterWhen.java:116) [reactor-core-3.2.5.RELEASE.jar:3.2.5.RELEASE]
at reactor.core.publisher.Operators$ScalarSubscription.request(Operators.java:2070) [reactor-core-3.2.5.RELEASE.jar:3.2.5.RELEASE]
at reactor.core.publisher.MonoFilterWhen$MonoFilterWhenMain.onSubscribe(MonoFilterWhen.java:103) [reactor-core-3.2.5.RELEASE.jar:3.2.5.RELEASE]
at reactor.core.publisher.MonoJust.subscribe(MonoJust.java:54) [reactor-core-3.2.5.RELEASE.jar:3.2.5.RELEASE]
at reactor.core.publisher.MonoFilterWhen.subscribe(MonoFilterWhen.java:56) [reactor-core-3.2.5.RELEASE.jar:3.2.5.RELEASE]
I answered the other question already and don't feel like retyping. The spirit of the answer is the exact same.
Write a unit test exercising this method off of the Spring cloud utils. This is what's breaking. You can try passing in more or less of the string you're concerned about to find where the breakage is. Use a binary search to figure out what's broken. Make sure you don't split the string in the middle of an encoded character or else you'll give yourself a false positive. When it says you have an invalid sequence I would expect you have something like %99 where 99 is does not map to any valid character (I'm just making one up)
https://docs.spring.io/spring/docs/current/javadoc-api/org/springframework/util/StringUtils.html#uriDecode-java.lang.String-java.nio.charset.Charset-
As an aside
Where is this encoded string coming from? Did someone at your company create their own solution to encode this string to begin with? Are you accepting user data? It's VERY POSSIBLE that whomever is responsible for producing this string encoded it incorrectly by homerolling their own encoder.
ALTERNATIVELY
spring.cloud.gateway.routes[7].predicates[0]=Path=/test/{testId}/test1/test_%26%5E%26%25%5E%26%25%26*%25%2B)!
When I look at this I see a path that is already encoded. For example, you've taken your ampersand & character and replaced it with %26
Have you tried inputting a path that is NOT already encoded?
For example
spring.cloud.gateway.routes[7].predicates[0]=Path=/test/{testId}/test1/test_&^&%^ < I only partially decoded it by hand using this chart. https://www.w3schools.com/tags/ref_urlencode.asp
Related
I'm making requests to Twitter, using the OAuth1.0 signing process to set the Authorization header. They explain it step-by-step here, which I've followed. It all works, most of the time.
Authorization fails whenever special characters are sent without percent encoding in the query component of the request. For example, ?status=hello%20world! fails, but ?status=hello%20world%21 succeeds. But the change from ! to the percent encoded form %21 is only made in the URL, after the signature is generated.
So I'm confused as to why this fails, because AFAIK that's a legally encoded query string. Only the raw strings ("status", "hello world!") are used for signature generation, and I'd assume the server would remove any percent encoding from the query params and generate its own signature for comparison.
When it comes to building the URL, I let URLComponents do the work, so I don't add percent encoding manually, ex.
var urlComps = URLComponents()
urlComps.scheme = "https"
urlComps.host = host
urlComps.path = path
urlComps.queryItems = [URLQueryItem(key: "status", value: "hello world!")]
urlComps.percentEncodedQuery // "status=hello%20world!"
I wanted to see how Postman handled the same request. I selected OAuth1.0 as the Auth type and plugged in the same credentials. The request succeeded. I checked the Postman console and saw ?status=hello%20world%21; it was percent encoding the !. I updated Postman, because a nice little prompt asked me to. Then I tried the same request; now it was getting an authorization failure, and I saw ?status=hello%20world! in the console; the ! was no longer being percent encoded.
I'm wondering who is at fault here. Perhaps Postman and I are making the same mistake. Perhaps it's with Twitter. Or perhaps there's some proxy along the way that idk, double encodes my !.
The OAuth1.0 spec says this, which I believe is in the context of both client (taking a request that's ready to go and signing it before it's sent), and server (for generating another signature to compare against the one received):
The parameters from the following sources are collected into a
single list of name/value pairs:
The query component of the HTTP request URI as defined by
[RFC3986], Section 3.4. The query component is parsed into a list
of name/value pairs by treating it as an
"application/x-www-form-urlencoded" string, separating the names
and values and decoding them as defined by
[W3C.REC-html40-19980424], Section 17.13.4.
That last reference, here, outlines the encoding for application/x-www-form-urlencoded, and says that space characters should be replaced with +, non-alphanumeric characters should be percent encoded, name separated from value by =, and pairs separated by &.
So, the OAuth1.0 spec says that the query string of the URL needs to be decoded as defined by application/x-www-form-urlencoded. Does that mean that our query string needs to be encoded this way too?
It seems to me, if a request is to be signed using OAuth1.0, the query component of the URL that gets sent must be encoded in a way that is different to what it would normally be encoded in? That's a pretty significant detail if you ask me. And I haven't seen it explicitly mentioned, even in Twitter's documentation. And evidently the folks at Postman overlooked it too? Unless I'm not supposed to be using URLComponents to build a URL, but that's what it's for, no? Have I understood this correctly?
Note: ?status=hello+world%21 succeeds; it tweets "hello world!"
I ran into a similar issue.
put the status in post body, not query string.
Percent-encoding:
private encode(str: string) {
// encodeURIComponent() escapes all characters except: A-Z a-z 0-9 - _ . ! ~ * " ( )
// RFC 3986 section 2.3 Unreserved Characters (January 2005): A-Z a-z 0-9 - _ . ~
return encodeURIComponent(str)
.replace(/[!'()*]/g, c => "%" + c.charCodeAt(0).toString(16).toUpperCase());
}
I'm writing app using Google custom search engine.
I received my search engine ID XXXXXXXX219143826571:7h9XXXXXXX (most interesting part bold).
Now I'm trying to use NSURLQueryItem to embed my ID into URL by using:
let params = ["cx" : engineID,...]
...
components.queryItems = parameters.map {
NSURLQueryItem(name: String($0), value: String($1))
}
It should percentage escape item to XXXXXXXX219143826571%3A7h9XXXXXXX (This value I'm getting when using Google APIs explorer while testing, it shows url dress that was used). It is not doing it. I'm getting url without escaping, no changes. If I use escaped string as engine ID in this mapping, I'm getting escaped string XXXXXXXX219143826571%253A7h9XXXXXXX (additional '25' is added to query).
Can someone tell me how to fix it? I don't want to use String and then convert it to URL by NSURL(string: str)!. It is not elegant.
Edit:
I'm using app Info.plist to save ID and I retrieve it by calling:
String(NSBundle.mainBundle().objectForInfoDictionaryKey("ApiKey")!)
Colons are allows in the query part of a URL string. There should be no need to escape them.
Strictly speaking, the only things that absolutely have to be encoded in that part of a URL are ampersands, hash marks (#), and (assuming you're doing a GET query with form encoding) equals signs. However, question marks in theory may cause problems, slashes are technically not allowed (but work just fine), and semicolons are technically allowed (but again, work in practice).
Colons, AFAIK, only have special meaning in the context of paths (if the OS treats it as a path separator) and in that it separates the scheme (protocol) from the rest of the URL.
So don't worry about the colon being unencoded unless the Google API barfs for some reason.
I am building a RESTful API for retrieving and storing comments in threads.
A comment thread is identified by an arbitrary URI -- usually this is the URL of the web page where the comment thread is related to. This design is very similar to what Disqus uses with their system.
This way, on every web page, querying the related comment thread doesn't require storing any additional data at the client -- all that is needed is the canonical URL to the page in question.
My current implementation attempts to make an URI work as a resource by encoding the URI as a string as follows:
/comments/https%3A%2F%2Fexample.org%2Ffoo%2F2345%3Ffoo%3Dbar%26baz%3Dxyz
However, before dispatching it to my application, the request URI always gets decoded by my server to
/comments/https://example.org/foo/2345?foo=bar&baz=xyz
This isn't working because the decoded resource name now has path delimiters and a query string in it causing routing in my API to get confused (my routing configuration assumes the request path contains /comments/ followed by a string).
I could double-encode them or using some other encoding scheme than URI encode, but then that would add complexity to the clients, which I'm trying to avoid.
I have two specific questions:
Is my URI design something I should continue working with or is there a better (best?) practice for doing what I'm trying to do?
I'm serving API requests with a Go process implemented using Martini 'microframework'. Is there something Go or Martini specific that I should do to make the URI-encoded resource names to stay encoded?
Perhaps a way to hint to the routing subsystem that the resource name is not just a string but a URL-encoded string?
I don't know about your url scheme for your application, but single % encoded values are valid in a url in place of the chars they represent, and should be decoded by the server, what you are seeing is what I would expect. If you need to pass url reserved characters as a value and not have them decoded as part of the url, you will need to double % encode them. It's a fairly common practice, the complexity added to the client & server will not be that much, and a short comment will do rightly.
In short, If you need to pass url chars, double % encode them, it's fine.
I am trying to support arbitrary unicode from a variety of international users. They have already put a bunch of data into sqlite databases on their iPhones, and now I want to capture the data into a database, then send it back to their device. Right now I am using a php page that is sending data back to from an internet mysql database. The data is saved in the mysql database properly, but when it's sent back it comes out as unicode text, such as
Frank\u00e2\u0080\u0099s iPad
instead of just
Frank's iPad
where the apostrophe should really be a curly apostrophe.
The answer posted to another question indicates that there is no built-in Cocoa methods to convert the "\u00e2\u0080\u0099" portion of the unicode string from the webserver to an NSString object. Is this correct?
That seems really surprising (and scarily disappointing), since Cocoa definitely allows input from many different Unicode characters, and I need to support any arbitrary language that I have never heard of, and all of the possible characters. I save them to and from the local sqlite database just fine now, but once I send it to a web server, then perhaps pull down different data, I want to ensure the data pulled from the web server is correctly formatted.
[...] there is no built-in Cocoa methods to convert [...]. Is this
correct?
It's not correct.
You might be interested in CFStringTransform and it's capabilities. It is a full blown ICU transformation engine, which can (also) perform your requested transformation.
See Using Objective C/Cocoa to unescape unicode characters, ie \u1234
All NSStrings are Unicode.
The problem with the “Frank\u00e2\u0080\u0099s iPad” data isn't that it's Unicode; it's that it's escaped to ASCII. “Frank’s iPad” is valid Unicode in any UTF, and is what you need.
So, you need to see whether the database is returning the data escaped or the PHP layer is escaping it at some point. If either of those is the case, fix it if you can; the PHP resource should return UTF-8/16/32. Only if that approach fails should you seek to unescape the string on the Cocoa side.
You're correct that there is no built-in way to unescape the string in Cocoa. If you get to that point, see if you can find some open-source code to do it; if not, you'll need to do it yourself, probably using NSScanner.
Check that your web service response has Content type and charset. Also that xml has encoding specified. In PHP you need to add the following before printing XML:
header('Content-type: text/xml; charset=UTF-8');
print '<?xml version="1.0" encoding="UTF-8"?>';
I guess there is just no encoding specified.
I'm using a Google Gears Worker to submt a POST httprequest (using var request = google.gears.factory.create('beta.httprequest'); )
with a parameter containing the string
"bford%20%24%23%26!%3F%40%20%E5%BE%B3%E5%8A%9B%E5%9F%BA%E5%BD%A6"
but the Django HttpRequest is receiving it as "bford $#&!?# å¾³å\u008a\u009bå\u009fºå½¦"
How do I specify to one or the other of the parties in the transaction to leave it untranslated?
Check the HttpRequest.encoding and the DEFAULT_CHARSET settings. Judging by the encoded value, this should be UTF-8 (which is indeed usually the right thing).
You can get the ‘untranslated’ (with %s still in) value by looking at the input stream (for POST) or environ QUERY_STRING (for GET) and decoding it manually, but it would be better to fix Django's incorrect string-to-unicode decoding really.
As I understand it, Django 1.0 should default to using UTF-8, so I'm not sure why it's not in your case.