I'm seeing mangled URL parameters coming from IE9 desktop clients. The links are sent via email, and all of the mangled URLs come from the plain-text version of the email.
I'm almost sure that it has nothing to do with my stack (django, nginx, mandrill) The values for the parameters have characters exactly transposed. The original character is the mangled one minus 13 places (eg. rznvy_cynva = email_plain, ubgryfpbz = hotelscom).
Here is one example of a mangled request that came through:
GET /book/48465?sid=rznvy_cynva&order=q09362qs55-741722-442521-98n2-n88s4nnr87192n&checkOut=07-17-15&affiliate=ubgryfpbz&checkIn=07-16-15 HTTP/1.1" 302 5 "-" "Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident/5.0)"
All of the requests with mangled URLs have the same user-agent as the example.
The IP addresses associated with the mangled URLs aren't restricted to any location.
Looking up the user-agent, this seems to be restricted to desktop Windows 7, IE9 users.
It is anti-malware software on your recipients' computers. It gets the links and scans your pages for any possible vulnerabilities. It uses rot13 obfuscation to ensure that it doesn't take any unwanted actions ("buy now", etc.).
https://security.stackexchange.com/questions/48684/help-investigating-potential-website-attack-url-rewriting-and-rot-13-obfuscatio
The solution is to track down what anti-malware software / company is performing the scans, and get your site whitelisted if possible.
This is going into the realm of speculation, but I'm also guessing you cannot get any answers which don't, so here goes ...
The rot13 encryption does not look like an accident. I have two guesses to offer;
Somebody is sharing their email and obfuscating query parameters in links so as to break the "order now", "unsubscribe" etc links while maintaining the overall integrity of the email messages. Maybe this is a feature of a spam-reporting tool or similar?
Alternatively, the queries are made from within a test network where users are not supposed to click on links, but the tools in there need pretty much unrestricted Internet access; so the admin set up an HTTP proxy which rewrites the query URLs to dismantle most GET transactions with parameters. (POST requests I guess would still probably work?)
Your observation that the IP addresses seem to be nonlocalized somewhat contradicts these hypotheses, but it could just mean that you are looking at TOR endpoints or similar.
Related
If you google "How to check an email address for existence" question, you will find, basically, only solutions using SMTP protocol what is not reliable. I tried this approach and found that Gmail SMTP server says "Yes, this email is registered here" on each and every email address I ask about. I suspect such strategy is used on the majority of popular email servers.
The method I would like to share is used in Gmail registration form to ensure you are going to register a brand new email. It uses AJAX request to ask Gmail server if given email exists or not
Request URL:https://accounts.google.com/InputValidator?resource=SignUp
Request Method:POST
Status Code:200
Remote Address:173.194.222.84:443
Response Headers
alt-svc:quic=":443"; ma=2592000; v="37,36,35"
cache-control:private, max-age=0
content-encoding:gzip
content-type:application/json; charset=utf-8
date:Wed, 29 Mar 2017 21:06:06 GMT
expires:Wed, 29 Mar 2017 21:06:06 GMT
server:GSE
set-cookie:GAPS=1:<redacted>;Path=/;Expires=Fri, 29-Mar-2019 21:06:06 GMT;Secure;HttpOnly;Priority=HIGH
status:200
strict-transport-security:max-age=10893354; includeSubDomains
x-content-type-options:nosniff
x-frame-options:DENY
x-xss-protection:1; mode=block
Request Headers
Provisional headers are shown
Content-type:application/json
Origin:https://accounts.google.com
Referer:https://accounts.google.com/SignUp?hl=en-GB
User-Agent:Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36
Query String Parameters
resource=SignUp
Request Payload
{"input01":{"Input":"GmailAddress","GmailAddress":"andy.v.che","FirstName":"","LastName":""},"Locale":"en-GB"}
Response
{"input01":{"Valid":"false","ErrorMessage":"Someone already has that username. Note that we ignore full stops and capitalisation in usernames. Try another?","Errors":{"GmailAddress":"Someone already has that username. Note that we ignore full stops and capitalisation in usernames. Try another?"},"ErrorData":["andyvche959"]},"Locale":"en_GB"}
As you can see, there is "Valid":"false" in the response if such an email does exist, and (spoilers) "Valid":"true" if it doesn't.
Throttling queries down
Guys from Gmail do understand this method could be used by spammers to look for existing emails. That's why they don't allow massive scans using it. I was doing such a scan for some time and could scan only 200 emails a day approximately.
More details
I was scanning 1 email a minute, and if I was getting response "No, this email doesn't exist", I also asked if my own email exists. If I got "No, your email doesn't exist as well" answer, I could clearly understand that I got ban from Gmail server by my IP address. Then, I took a break for 45 minutes to get unbanned, then continued the loop. The number af emails scanned a day was fluctuating around 200.
You may ask: you did a scan like a spammer would perform, for what purpose did you do that scan then?
My answer is: I was trying to find a guy who wrote his email unclearly (bad cursive). There was no other option to find him.
There were 3 unclear letters in his written email but it was clear the domain of it is gmail.com, so I came up with an idea to find a way to check an email address for existence on Gmail, generate a list of all possible emails (trying to substitute unknown symbols with all possible English letters) and check them all for existence. Then, send a letter to all existing ones.
The right of this information to be published is discussed in this question. I understand this article will be very useful for spammers so I'm open to deleting it partially or even completely for the sake of security.
I'm getting a bunch of errors on my application with the user agent string being:
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)
Looking this up on useragentstring.com, this is supposed to be internet explorer 6 while the user claims he is using internet explorer 9.
I'm no expert in user agents, can someone tell me why IE9 would be disguising as IE6, or what else am I missing here? Is there a way to "really" detect the browser server-side? Can I do a redirect server-side (using Coldfusion) or in htaccess?
Thanks!
This is what I could find from an archive of almost all user agent strings.
Explanation: This string has bit of history. We originally published it as a EudoraWeb string - since it was self identified by a site user as being that. However:
We got some email about this string suggesting that it was not eudora since it had no Eudora in it. To be fair the supplier of the string also voiced some doubt since it was left to the user to identify the string. If anyone can shed some more light on this topic - please email us and we'll publish.
We got some more comment which says it looks so much like a normal Win 2K that we've moved it. The suggestion is that both .NET strings are added when the MS WindowsUpdate system is used. Explanation from Matt Hair - thanks.
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)
http://www.zytrax.com/tech/web/msie-history.html
I've read several documents on the merits of the different HTTP redirect status codes, but those've all been very SEO-centric. I've now got a problem where search-engines don't factor in, because the section of the site in question is not publicly viewable.
However, we do want our website to be as accurate / helpful with meta-data as possible, especially for accessibility reasons.
Now, our application takes external links provided by third parties and routes them across an anti-spoofing page with a disclaimer. Since this redirector page can effectively also be embedded via an Ajax call in certain constellations, we also want to strip any query parameters from the referer (for privacy purposes; the target site has no business finding out what internal page the user was on before).
To do this, the confirmation button triggers a server-side script which in turn redirects (rather than just opening the page for the user).
So much as to why our anti-spoofing disclaimer page ends up triggering a redirect.
The question is:
Does it effectively make any difference which status code I use? Do non-typical browsers (e.g. screen-readers) care? If so, what's the best practise for such redirects? The most semantically sound, if you so will? They all seem various degrees of insincere to me.
I'm thinking of a 302 - but as it makes no sense trying to bookmark the page (it's protected with a crsf token), so there's probably no harm in a 301, either, is there? So I'm wondering if there's a reason for me to prefer the one over the other.
Hmm. Here's the list. 301 sounds okay (emphasis mine):
The requested resource has been assigned a new permanent URI and any future references to this resource SHOULD use one of the returned URIs. Clients with link editing capabilities ought to automatically re-link references to the Request-URI to one or more of the new references returned by the server, where possible.
302 doesn't fit n my opinion:
The requested resource resides temporarily under a different URI
However, my favourite is 303 see other:
The response to the request can be found under a different URI and SHOULD be retrieved using a GET method on that resource. This method exists primarily to allow the output of a POST-activated script to redirect the user agent to a selected resource. The new URI is not a substitute reference for the originally requested resource.
But that might be so rare (I've never seen it used in the wild) that some clients may not understand it - which would render your desire for maximum compatibility moot. 301 is probably the closest choice.
I have used referrer before in foo.php to decide whether the page iframing foo.php is of a particular URL. (using $_SERVER['HTTP_REFERER'])
It turned out that most of the time, it worked (about 98% of the time), but it also seemed like some users arrived the page and $_SERVER['HTTP_REFERER'] was not set in foo.php and therefore broke the code. [update: These user claimed that they followed the usual page flow and didn't use the URL of foo.php all by itself on the browser (that they let it be an iframe) and the users never altered their browser settings.]
I wonder what the reasons are that it could happen?
The HTTP/1.1 RFC does not make it mandatory to send an HTTP referer header. You can't make any assumptions about its presence when writing robust code; perfectly conforment browsers may not include it.
Moreoever, the RFC advises that "The Referer field MUST NOT be sent if the Request-URI was obtained from a source that does not have its own URI, such as input from the user keyboard", and "We suggest, though do not require, that a convenient toggle interface be provided for the user to enable or disable the sending of From and Referer information".
The later is not very common (though some browsers have a "Private" mode that fulfils the requirements). More likely for your 2% is that people Bookmarked the URL, which fulfils the first criteria (URI obtained from a source without a URI), and so the browser sends no referer.
Not by default AFAIK, but it's easy to turn it off (for privacy) e.g. in Firefox via about:config, and surely some users could be using browsers distributed to them (e.g. by their IT department) with such kinds of setting. So you should try to avoid relying on REFERER for any important functionality (also because it's mis-spelled, of course;-).
Why does using Fiddler break my site sometimes on page transitions.
After a server side redirect -- in the http response (as found in Fiddler) I get this:
Object moved
Object moved to here.
The site is an ASP.NET 1.1 / VB.NET 1.1 [sic] site.
Why doesnt Fiddler just go there for me? i dont get it.
I'm fine with this issue when developing but I'm worried that other proxy servers might cause this issue for 'real customers'. Im not even clear exactly what is going on.
That's actually what Response.Redirect does. It sends a 302 - Object moved response to the user-agent. The user-agent then automatically goes to the URL specified in the 302 response. If you need a real server-side redirect without round-tripping to the client, try Server.Transfer.
If you merely constructed the request using the request builder, you're not going to see Fiddler automatically follow the returned redirect.
In contrast, if you are using IE or another browser, it will generally check the redirect header and follow it.
For IE specifically, I believe there's a timing corner case where the browser will fail to follow the redirect in obscure situations. You can often fix this by clicking Tools / Fiddler Options, and enabling both the "Server" and "Client" socket reuse settings.
Thanks user15310, it works with Server.Transfer
Server.Transfer("newpage.aspx", true);
Firstly, transferring to another page using Server.Transfer conserves server resources. Instead of telling the browser to redirect, it simply changes the "focus" on the Web server and transfers the request. This means you don't get quite as many HTTP requests coming through, which therefore eases the pressure on your Web server and makes your applications run faster.
But watch out: because the "transfer" process can work on only those sites running on the server, you can't use Server.Transfer to send the user to an external site. Only Response.Redirect can do that.
Secondly, Server.Transfer maintains the original URL in the browser. This can really help streamline data entry techniques, although it may make for confusion when debugging.
That's not all: The Server.Transfer method also has a second parameter—"preserveForm". If you set this to True, using a statement such as Server.Transfer("WebForm2.aspx", True), the existing query string and any form variables will still be available to the page you are transferring to.
Read more here:
http://www.developer.com/net/asp/article.php/3299641/ServerTransfer-Vs-ResponseRedirect.htm