Lets take this URL for example
https://<ip>/restconf/data/Cisco-IOS-XE-native:native/interface/GigabitEthernet=1
If i send a request to a similar endpoint using the internal library we've been using the server doesn't understand it since the = is encoded to %3D but making the same request on the CLI with curl works fine.
I'm not understanding why it's an issue, isn't = supposed to be encoded in a URL anyway? Is it something to do with the library treating it as a query?
The = represents a special character with special meaning in a RESTCONF URI. If you encode it as %3D, you take that special meaning away and reduce it to just a character. The only case where you would do that is when a list's key value or a leaf-list's value contain this character and you are referencing such an instance in your URI.
You are only allowed to use = when referencing list and leaf-list instances - and you do not percent encode it when you do use it.
In your case, GigabitEthernet would be expected to represent a list/leaf-list, with its key/type accepting 1 as a valid value. Both native and interface would be (and can only be) containers (or anydata).
RFC8040, Section 3.5.3, Encoding Data Resource Identifiers in the Request URI describes this in detail.
Here are the examples from the same section:
Examples:
container top {
list list1 {
key "key1 key2 key3";
...
list list2 {
key "key4 key5";
...
leaf X { type string; }
}
}
leaf-list Y {
type uint32;
}
}
For the above YANG definition, the container "top" is defined in the
"example-top" YANG module, and a target resource URI for leaf "X"
would be encoded as follows:
/restconf/data/example-top:top/list1=key1,key2,key3/list2=key4,key5/X
For the above YANG definition, a target resource URI for
leaf-list "Y" would be encoded as follows:
/restconf/data/example-top:top/Y=instance-value
The following example shows how reserved characters are
percent-encoded within a key value. The value of "key1" contains
a comma, single-quote, double-quote, colon, double-quote, space,
and forward slash (,'":" /). Note that double-quote is not a
reserved character and does not need to be percent-encoded. The
value of "key2" is the empty string, and the value of "key3" is the
string "foo".
Example URL:
/restconf/data/example-top:top/list1=%2C%27"%3A"%20%2F,,foo
Note the last example in particular. You would need to percent encode a =, if it appears after list1= in its key value.
Note that also that these are not parts of a query - they are path segments (RFC3986 terms).
Related
I'm making requests to Twitter, using the OAuth1.0 signing process to set the Authorization header. They explain it step-by-step here, which I've followed. It all works, most of the time.
Authorization fails whenever special characters are sent without percent encoding in the query component of the request. For example, ?status=hello%20world! fails, but ?status=hello%20world%21 succeeds. But the change from ! to the percent encoded form %21 is only made in the URL, after the signature is generated.
So I'm confused as to why this fails, because AFAIK that's a legally encoded query string. Only the raw strings ("status", "hello world!") are used for signature generation, and I'd assume the server would remove any percent encoding from the query params and generate its own signature for comparison.
When it comes to building the URL, I let URLComponents do the work, so I don't add percent encoding manually, ex.
var urlComps = URLComponents()
urlComps.scheme = "https"
urlComps.host = host
urlComps.path = path
urlComps.queryItems = [URLQueryItem(key: "status", value: "hello world!")]
urlComps.percentEncodedQuery // "status=hello%20world!"
I wanted to see how Postman handled the same request. I selected OAuth1.0 as the Auth type and plugged in the same credentials. The request succeeded. I checked the Postman console and saw ?status=hello%20world%21; it was percent encoding the !. I updated Postman, because a nice little prompt asked me to. Then I tried the same request; now it was getting an authorization failure, and I saw ?status=hello%20world! in the console; the ! was no longer being percent encoded.
I'm wondering who is at fault here. Perhaps Postman and I are making the same mistake. Perhaps it's with Twitter. Or perhaps there's some proxy along the way that idk, double encodes my !.
The OAuth1.0 spec says this, which I believe is in the context of both client (taking a request that's ready to go and signing it before it's sent), and server (for generating another signature to compare against the one received):
The parameters from the following sources are collected into a
single list of name/value pairs:
The query component of the HTTP request URI as defined by
[RFC3986], Section 3.4. The query component is parsed into a list
of name/value pairs by treating it as an
"application/x-www-form-urlencoded" string, separating the names
and values and decoding them as defined by
[W3C.REC-html40-19980424], Section 17.13.4.
That last reference, here, outlines the encoding for application/x-www-form-urlencoded, and says that space characters should be replaced with +, non-alphanumeric characters should be percent encoded, name separated from value by =, and pairs separated by &.
So, the OAuth1.0 spec says that the query string of the URL needs to be decoded as defined by application/x-www-form-urlencoded. Does that mean that our query string needs to be encoded this way too?
It seems to me, if a request is to be signed using OAuth1.0, the query component of the URL that gets sent must be encoded in a way that is different to what it would normally be encoded in? That's a pretty significant detail if you ask me. And I haven't seen it explicitly mentioned, even in Twitter's documentation. And evidently the folks at Postman overlooked it too? Unless I'm not supposed to be using URLComponents to build a URL, but that's what it's for, no? Have I understood this correctly?
Note: ?status=hello+world%21 succeeds; it tweets "hello world!"
I ran into a similar issue.
put the status in post body, not query string.
Percent-encoding:
private encode(str: string) {
// encodeURIComponent() escapes all characters except: A-Z a-z 0-9 - _ . ! ~ * " ( )
// RFC 3986 section 2.3 Unreserved Characters (January 2005): A-Z a-z 0-9 - _ . ~
return encodeURIComponent(str)
.replace(/[!'()*]/g, c => "%" + c.charCodeAt(0).toString(16).toUpperCase());
}
The Responses object contains a {HTTP Status Code: Response} mapping.
In all the examples I found, the status code is provided as a string:
{"200": {"description": "a pet to be returned"}}
I couldn't find any requirement for it to be a string and integers are accepted by the validators I tried.
All I found was a PR changing from integer to string in all YAML examples.
Should I only use strings?
Edit: In JSON, only strings are valid keys. So the question could be rephrased as "which of the following two assumptions is correct"?
OpenAPI doesn't specify that HTTP Status Codes should be strings because that's implicit (JSON format). However, validation and display tools are being loose about that requirement.
OpenAPI uses some kind of "JSON superset" in which integer keys are considered valid.
From this GH issue, the keys must be strings:
OpenAPI can be represented canonically in either JSON or YAML, as you say in JSON only strings can be keys. With regard to YAML:
This field MUST be enclosed in quotation marks (for example, "200") for compatibility between JSON and YAML.
This has the effect that the key is always a string type.
This is not really a specification, but rather a requirement of the JSON format.
For the request
http://xyz/resource?articleid=232&name=John
Response getDetails(#QueryParam("articleid") String articleid,(#QueryParam("name") String name){}
Is the above Query Parameter correct for the given URL?
Executive summary: there is no "integer" in the URL; there's only a string. The implementation is does extra work to convert the string into an integer if you ask it to do so.
Is the above Query Parameter correct for the given URL?
That should be perfectly acceptable.
https://www.rfc-editor.org/rfc/rfc3986#section-3.4
query is a just a sequence of pchar (plus '/' and '?'), which is to say it's just data.
A query in that form is usually an expression of an application/x-www-form-urlencoded resource. The key hint in the specification is
Let output be an initially empty list of name-value tuples where both name and value hold a string.
The JAX-RS specification describes the transformation of these string to other types, but it defers to the java doc for the annotation. Of course, QueryParam is in close alignment with the specification, so both places give the same answer.
We got a get request that sends string characters in url, so we use path variables to receive them. Apparently there is no way that the calling service would change its method of calling backend so we need to be able to accept a url with the following unencoded characters:
When percentage sign % is sent a http 400 is returned. It does go through if the two characters following % make up an UTF-encoded character
Backslash is converted into a forward slash. I need it to stay backslash.
I'm guessing these might be Tomcat or servlet configuration issues.
(spring boot version 1.5.14.RELEASE)
Percent signs (%) should be no problem if you properly URL encode them (%25). However, slashes and backslashes will not work with Tomcat, even if you encode them (%2F and %5C).
You could set the following properties when running the application:
-Dorg.apache.tomcat.util.buf.UDecoder.ALLOW_ENCODED_SLASH=true
-Dorg.apache.catalina.connector.CoyoteAdapter.ALLOW_BACKSLASH=true
However, this won't fix the issue, because in this case, those encoded slashes will be recognized as real ones. So, let's say you have the following controller:
#ResponseBody
#RequestMapping("/api/{foo}")
public String getFoo(#PathVariable String foo) {
return foo;
}
Well, then if you call /api/test%5Ctest, it won't be able to find the correct path. A solution to this problem is to use wildcard matchers and to parse the URL itself from the incoming HttpServletRequest:
#RequestMapping("/api/**")
public String getFoo(HttpServletRequest request) {
// ...
}
Another solution is to use a completely different web container. For example, when using Jetty, this isn't a problem at all, and URL encoded slashes and backslashes will both work.
Spring 5 now blocks encoded percent signs by default. To enable them, create a new Bean that calls setAllowUrlEncodedPercent()
#Bean
public HttpFirewall allowEncodedParamsFirewall() {
StrictHttpFirewall firewall = new StrictHttpFirewall();
firewall.setAllowUrlEncodedPercent(true);
return firewall;
}
There are similar method-calls for forward- and backwards-slash
What you are experiencing is not specific to Spring Boot. Instead, it's a restriction of HTTP.
The HTTP standard requires that any URL containing the percent characters must be decoded by the web server (cf page 36):
If the Request-URI is encoded using the "% HEX HEX" encoding [42], the
origin server MUST decode the Request-URI in order to properly
interpret the request.
As a result, it's not possible to escape the slash character reliably.
Therefore, when the slash is used in a URL – with or without encoding – it will be treated as a path separator. So it cannot be used in a Spring Boot path variable. Similar problem exist for the percent sign and backslash.
Your best options are to use query parameters or a POST request.
In the following URL, the value test_with_/and_% is transmitted:
https://host/abc/def?text=test_with_%2F_and%25
final String path =
request.getAttribute(HandlerMapping.PATH_WITHIN_HANDLER_MAPPING_ATTRIBUTE).toString();
final String bestMatchingPattern =
request.getAttribute(HandlerMapping.BEST_MATCHING_PATTERN_ATTRIBUTE).toString();
String arguments = new AntPathMatcher().extractPathWithinPattern(bestMatchingPattern, path);
if (null != arguments && !arguments.isEmpty()) {
pattern = pattern + '/' + arguments;
}
I also faced similar problem and I have used this so hope this might help
I'm writing app using Google custom search engine.
I received my search engine ID XXXXXXXX219143826571:7h9XXXXXXX (most interesting part bold).
Now I'm trying to use NSURLQueryItem to embed my ID into URL by using:
let params = ["cx" : engineID,...]
...
components.queryItems = parameters.map {
NSURLQueryItem(name: String($0), value: String($1))
}
It should percentage escape item to XXXXXXXX219143826571%3A7h9XXXXXXX (This value I'm getting when using Google APIs explorer while testing, it shows url dress that was used). It is not doing it. I'm getting url without escaping, no changes. If I use escaped string as engine ID in this mapping, I'm getting escaped string XXXXXXXX219143826571%253A7h9XXXXXXX (additional '25' is added to query).
Can someone tell me how to fix it? I don't want to use String and then convert it to URL by NSURL(string: str)!. It is not elegant.
Edit:
I'm using app Info.plist to save ID and I retrieve it by calling:
String(NSBundle.mainBundle().objectForInfoDictionaryKey("ApiKey")!)
Colons are allows in the query part of a URL string. There should be no need to escape them.
Strictly speaking, the only things that absolutely have to be encoded in that part of a URL are ampersands, hash marks (#), and (assuming you're doing a GET query with form encoding) equals signs. However, question marks in theory may cause problems, slashes are technically not allowed (but work just fine), and semicolons are technically allowed (but again, work in practice).
Colons, AFAIK, only have special meaning in the context of paths (if the OS treats it as a path separator) and in that it separates the scheme (protocol) from the rest of the URL.
So don't worry about the colon being unencoded unless the Google API barfs for some reason.