I recently received an email containing the following chunk (don't click!):
<A HrEf="/#/0X0a290d92b/UALI=28389-UI=176738575-OI=279-ONI=5477-SI=0-CI=0-BI=577-II=27913-IDSP=1-KLEM=11-TIE=A-IDE=276135-MID=572-FID=0-DIOM=0" sTyLe=color:#000;font-size:10px;font-family:arial;>
<span>UNS</span></a>
Here is a link to the raw email: https://gist.github.com/anonymous/16963a230cab0a3a1bcfc81209f297f1
As far as I know, /# is not a valid url. How is my browser able to resolve it to a site?
As it was already mentioned in comments # is allowed in URL paths.
Regarding URL resolving. I guess that attacker uses <base> tag to explicitly set default URL for all relative links in email body and hopes that your browser/email client will resolve it for you.
UPDATE
The original guess might be correct since it is not supported by majority of mail clients
After a bit of investigation I realized that 0x0A290D92B is actually is hex-encoded IPv4 address 162.144.217.43. The only thing which I do not yet understood is how it is supposed to be transformed to http(s)://0x0A290D92B in browser. It seems like the attacker is targeting specific browser/mail client behavior.
It's treating everything before the # as auth information that gets passed to the URL. The "real" url starts after the #, which is the encoded IP address that vsminkov mentioned. So the leading forward slash is discarded.
An easier to read example: http://username:password#example.com/
It's all just layers of obfuscation.
Here's an interesting link that goes over it in more detail:
http://www.pc-help.org/obscure.htm
and here's RFC 2396 describing that part of the URL:
URL schemes that involve the direct use of an IP-based protocol to a
specified server on the Internet use a common syntax for the server
component of the URI's scheme-specific data:
<userinfo>#<host>:<port>
where may consist of a user name and, optionally, scheme-
specific information about how to gain authorization to access the
server. The parts "#" and ":" may be omitted.
server = [ [ userinfo "#" ] hostport ]
The user information, if present, is followed by a commercial at-sign
"#".
userinfo = *( unreserved | escaped |
";" | ":" | "&" | "=" | "+" | "$" | "," )
Related
I'm making requests to Twitter, using the OAuth1.0 signing process to set the Authorization header. They explain it step-by-step here, which I've followed. It all works, most of the time.
Authorization fails whenever special characters are sent without percent encoding in the query component of the request. For example, ?status=hello%20world! fails, but ?status=hello%20world%21 succeeds. But the change from ! to the percent encoded form %21 is only made in the URL, after the signature is generated.
So I'm confused as to why this fails, because AFAIK that's a legally encoded query string. Only the raw strings ("status", "hello world!") are used for signature generation, and I'd assume the server would remove any percent encoding from the query params and generate its own signature for comparison.
When it comes to building the URL, I let URLComponents do the work, so I don't add percent encoding manually, ex.
var urlComps = URLComponents()
urlComps.scheme = "https"
urlComps.host = host
urlComps.path = path
urlComps.queryItems = [URLQueryItem(key: "status", value: "hello world!")]
urlComps.percentEncodedQuery // "status=hello%20world!"
I wanted to see how Postman handled the same request. I selected OAuth1.0 as the Auth type and plugged in the same credentials. The request succeeded. I checked the Postman console and saw ?status=hello%20world%21; it was percent encoding the !. I updated Postman, because a nice little prompt asked me to. Then I tried the same request; now it was getting an authorization failure, and I saw ?status=hello%20world! in the console; the ! was no longer being percent encoded.
I'm wondering who is at fault here. Perhaps Postman and I are making the same mistake. Perhaps it's with Twitter. Or perhaps there's some proxy along the way that idk, double encodes my !.
The OAuth1.0 spec says this, which I believe is in the context of both client (taking a request that's ready to go and signing it before it's sent), and server (for generating another signature to compare against the one received):
The parameters from the following sources are collected into a
single list of name/value pairs:
The query component of the HTTP request URI as defined by
[RFC3986], Section 3.4. The query component is parsed into a list
of name/value pairs by treating it as an
"application/x-www-form-urlencoded" string, separating the names
and values and decoding them as defined by
[W3C.REC-html40-19980424], Section 17.13.4.
That last reference, here, outlines the encoding for application/x-www-form-urlencoded, and says that space characters should be replaced with +, non-alphanumeric characters should be percent encoded, name separated from value by =, and pairs separated by &.
So, the OAuth1.0 spec says that the query string of the URL needs to be decoded as defined by application/x-www-form-urlencoded. Does that mean that our query string needs to be encoded this way too?
It seems to me, if a request is to be signed using OAuth1.0, the query component of the URL that gets sent must be encoded in a way that is different to what it would normally be encoded in? That's a pretty significant detail if you ask me. And I haven't seen it explicitly mentioned, even in Twitter's documentation. And evidently the folks at Postman overlooked it too? Unless I'm not supposed to be using URLComponents to build a URL, but that's what it's for, no? Have I understood this correctly?
Note: ?status=hello+world%21 succeeds; it tweets "hello world!"
I ran into a similar issue.
put the status in post body, not query string.
Percent-encoding:
private encode(str: string) {
// encodeURIComponent() escapes all characters except: A-Z a-z 0-9 - _ . ! ~ * " ( )
// RFC 3986 section 2.3 Unreserved Characters (January 2005): A-Z a-z 0-9 - _ . ~
return encodeURIComponent(str)
.replace(/[!'()*]/g, c => "%" + c.charCodeAt(0).toString(16).toUpperCase());
}
I'm using the Actions on Google OAUTH authorization flow. In the authorization URL, Google passes along a very long "state" parameter (500+ char) in the URL, yielding an error on the OAauth client/server (Bunq to be precise).
I've seen more cases where this is an issue, also including Amazon Skills.
If I manually shorten the length of the 'state' parameter, the error disappears. But since Google is using the 'state' parameter as a "bookmarking value", they do not allow for any alterations. There seems to be no way to resolve this issue on my side.
Here is an example of the Authorization URL that is generated by Actions on Google:
https://oauth.bunq.com/auth?redirect_uri=https%3A%2F%2Foauth-redirect.googleusercontent.com%2Fr%2F[MY PROJECT NUMBER]&client_id=[CLIENT ID]&response_type=code&state=[567 CHARACTERS]
Is there a way to shorten the state parameter, or to resolve this issue otherwise?
While there is no specified limit on the characters forming a URL specified by the Network Working Group in RFC 2616, the defacto convention used by many internet systems and browsers is up to 2k characters. You should check with the internal codebase as well if there is any limit on the number of characters that can be included in a query string.
I'm writing app using Google custom search engine.
I received my search engine ID XXXXXXXX219143826571:7h9XXXXXXX (most interesting part bold).
Now I'm trying to use NSURLQueryItem to embed my ID into URL by using:
let params = ["cx" : engineID,...]
...
components.queryItems = parameters.map {
NSURLQueryItem(name: String($0), value: String($1))
}
It should percentage escape item to XXXXXXXX219143826571%3A7h9XXXXXXX (This value I'm getting when using Google APIs explorer while testing, it shows url dress that was used). It is not doing it. I'm getting url without escaping, no changes. If I use escaped string as engine ID in this mapping, I'm getting escaped string XXXXXXXX219143826571%253A7h9XXXXXXX (additional '25' is added to query).
Can someone tell me how to fix it? I don't want to use String and then convert it to URL by NSURL(string: str)!. It is not elegant.
Edit:
I'm using app Info.plist to save ID and I retrieve it by calling:
String(NSBundle.mainBundle().objectForInfoDictionaryKey("ApiKey")!)
Colons are allows in the query part of a URL string. There should be no need to escape them.
Strictly speaking, the only things that absolutely have to be encoded in that part of a URL are ampersands, hash marks (#), and (assuming you're doing a GET query with form encoding) equals signs. However, question marks in theory may cause problems, slashes are technically not allowed (but work just fine), and semicolons are technically allowed (but again, work in practice).
Colons, AFAIK, only have special meaning in the context of paths (if the OS treats it as a path separator) and in that it separates the scheme (protocol) from the rest of the URL.
So don't worry about the colon being unencoded unless the Google API barfs for some reason.
In my application I have URIs that include the userid to identify user private resources. And userids are emails, for example:
/users/user2#example.com/private-resource
It's a good practice to put email in the URI, including characters like . and #?
Or should I use some other type of userid? Like an hash for example?
If the email can function as fixed identifier it would be ok.
The thing is that most of the time applications will allow users to change emails; in this case it would be more bullet proof to control the ID-space yourself, e.g. by using a surrogate key. (Because the email (if users can change them) is not the identity but merely a property of the resource).
Another argument against emails is - as #Rob points out - a potential security issue.
There's no problem with that... just make sure you URL-encode the appropriate characters.
(Email's kind of sensitive information, though, you might want to pass a hash or surrogate key instead just to protect your users' email addresses if the URLs get passed around.)
though having '-' in uri is absolutely fine.
use of '#' , '#' are discouraged
however, as long as email id is an id e.g. http://example.org/user/mail/{email}/1234
guess thats fine.
One of my routes in an MVC project serves documents that have been uploaded.
For SEO and user friendliness purposes I want the document title to be included in the URL, the route will take the ID from the incoming URL, match it with the document then redirect to a URL with the filename appended to the ID. As document titles can have a wide variety of characters in the title including ones that are used to break up parameters the filename is a catchall parameter.
This works fine for almost all characters in the title including reserved ones such as "/" but when the title include the combination "/ " routing breaks. Not just in terms of not match this route but apparently bypassing the entirety of the application and returning a 404, I tried to use Phil Haack's RouteDebugger but that was also giving a 404 rather than catching the request.
My web.config has requestvalidation turned off and I can't seem to find anyway to get the application to catch the request.
I would recommend you filtering those character or you will have many headaches. Here's a function you might find interesting.