I'm getting a bunch of errors on my application with the user agent string being:
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)
Looking this up on useragentstring.com, this is supposed to be internet explorer 6 while the user claims he is using internet explorer 9.
I'm no expert in user agents, can someone tell me why IE9 would be disguising as IE6, or what else am I missing here? Is there a way to "really" detect the browser server-side? Can I do a redirect server-side (using Coldfusion) or in htaccess?
Thanks!
This is what I could find from an archive of almost all user agent strings.
Explanation: This string has bit of history. We originally published it as a EudoraWeb string - since it was self identified by a site user as being that. However:
We got some email about this string suggesting that it was not eudora since it had no Eudora in it. To be fair the supplier of the string also voiced some doubt since it was left to the user to identify the string. If anyone can shed some more light on this topic - please email us and we'll publish.
We got some more comment which says it looks so much like a normal Win 2K that we've moved it. The suggestion is that both .NET strings are added when the MS WindowsUpdate system is used. Explanation from Matt Hair - thanks.
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)
http://www.zytrax.com/tech/web/msie-history.html
Related
Someone help me with this using lwp::Useragent please
my $mech = WWW::Mechanize->new(autocheck => 0);
$mech->get($url);
my $content=$mech->content;
but getting Error 500 when trying to get https://camelcamelcamel.com/
It seems that the site blocks requests from "bad" (undesired) user agents. You may make WWW::Mechanize (LWP::UserAgent) present itself as another user agent using agent parameter for new or by calling agent method. Full IE8 identification string fixed the problem.
I have tested it using Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1.
[As listed by "User Agent Switcher" plugin for Firefox]
Short list of user agents (xml file)
Long list of user agents
WARNING
The site(s) may use or choose to use other means to block unwanted requests.
If you google "How to check an email address for existence" question, you will find, basically, only solutions using SMTP protocol what is not reliable. I tried this approach and found that Gmail SMTP server says "Yes, this email is registered here" on each and every email address I ask about. I suspect such strategy is used on the majority of popular email servers.
The method I would like to share is used in Gmail registration form to ensure you are going to register a brand new email. It uses AJAX request to ask Gmail server if given email exists or not
Request URL:https://accounts.google.com/InputValidator?resource=SignUp
Request Method:POST
Status Code:200
Remote Address:173.194.222.84:443
Response Headers
alt-svc:quic=":443"; ma=2592000; v="37,36,35"
cache-control:private, max-age=0
content-encoding:gzip
content-type:application/json; charset=utf-8
date:Wed, 29 Mar 2017 21:06:06 GMT
expires:Wed, 29 Mar 2017 21:06:06 GMT
server:GSE
set-cookie:GAPS=1:<redacted>;Path=/;Expires=Fri, 29-Mar-2019 21:06:06 GMT;Secure;HttpOnly;Priority=HIGH
status:200
strict-transport-security:max-age=10893354; includeSubDomains
x-content-type-options:nosniff
x-frame-options:DENY
x-xss-protection:1; mode=block
Request Headers
Provisional headers are shown
Content-type:application/json
Origin:https://accounts.google.com
Referer:https://accounts.google.com/SignUp?hl=en-GB
User-Agent:Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36
Query String Parameters
resource=SignUp
Request Payload
{"input01":{"Input":"GmailAddress","GmailAddress":"andy.v.che","FirstName":"","LastName":""},"Locale":"en-GB"}
Response
{"input01":{"Valid":"false","ErrorMessage":"Someone already has that username. Note that we ignore full stops and capitalisation in usernames. Try another?","Errors":{"GmailAddress":"Someone already has that username. Note that we ignore full stops and capitalisation in usernames. Try another?"},"ErrorData":["andyvche959"]},"Locale":"en_GB"}
As you can see, there is "Valid":"false" in the response if such an email does exist, and (spoilers) "Valid":"true" if it doesn't.
Throttling queries down
Guys from Gmail do understand this method could be used by spammers to look for existing emails. That's why they don't allow massive scans using it. I was doing such a scan for some time and could scan only 200 emails a day approximately.
More details
I was scanning 1 email a minute, and if I was getting response "No, this email doesn't exist", I also asked if my own email exists. If I got "No, your email doesn't exist as well" answer, I could clearly understand that I got ban from Gmail server by my IP address. Then, I took a break for 45 minutes to get unbanned, then continued the loop. The number af emails scanned a day was fluctuating around 200.
You may ask: you did a scan like a spammer would perform, for what purpose did you do that scan then?
My answer is: I was trying to find a guy who wrote his email unclearly (bad cursive). There was no other option to find him.
There were 3 unclear letters in his written email but it was clear the domain of it is gmail.com, so I came up with an idea to find a way to check an email address for existence on Gmail, generate a list of all possible emails (trying to substitute unknown symbols with all possible English letters) and check them all for existence. Then, send a letter to all existing ones.
The right of this information to be published is discussed in this question. I understand this article will be very useful for spammers so I'm open to deleting it partially or even completely for the sake of security.
I've already posted this in the MSDN forum and have no response. I'm trying to port an application that uses MS Translator that has been using the old authentication token mechanism (client ID, client secret) to use MS Azure. To that end, I set up an Azure account, added the MS Translator API through the cognitive services, and generated keys as per the instructions found here:
https://translatorbusiness.uservoice.com/knowledgebase/articles/1078534-microsoft-tran
I am able to call the Azure authentication page as documented here:
http://docs.microsofttranslator.com/oauth-token.html
However, when I use the token that I get from that method (both from my own request and from the swagger API docs that allows you to try the call), and call any of the methods in the API, I get a response code of 0 and a response error header saying 'no response from server'.
Part of my confusion is I'm not absolutely sure what key/subscription ID I should be using. When I subscribed, I got a subscription ID which doesn't work when calling the issueToken method. I also have two keys generated (a key1, and a key2) with the option to regenerate them. Both will give me back an authToken, but when I call any method in the API, I get the same empty response.
For example, when I called the GetLanguagesForTranslate method using their swagger API doc (i.e. the try it button), my request has the following headers:
Host: api.microsofttranslator.com
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:51.0) Gecko/20100101 Firefox/51.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate, br
Access-Control-Request-Method: GET
Access-Control-Request-Headers: authorization
origin: htee-tee-pee(slash-slash)docs.microsofttranslator.com
My response is empty.
Can anyone give me a clue to what is going on? Could it be because for whatever reason my subscription is just a trial? MS won't allow me to log a support ticket unless I buy a subscription. I don't want to do that because I'm working for a client and once this work is done, I won't be using the subscription anymore.
Thanks in advance
It sounds like you were able to get the access token with the key got from Azure Portal via http://docs.microsofttranslator.com/oauth-token.html.
Per my experience, I think the no response issue seems to be caused by setting incorrectly for parameters appid & Authorization. For example, setting the appid value with any string (such as your registed Azure service name) and setting the Authorization value with Bearer <access-token>.
However, as the description for appid below.
REQUIRED. If the Authorization header is used, leave the appid field empty else specify a string containing "Bearer" + " " + access token.
It means that you just need to set the appid value with Bearer <access-token> and put the Authorization value empty, as see the figure below.
Click Try it out!, then got as below.
I'm seeing mangled URL parameters coming from IE9 desktop clients. The links are sent via email, and all of the mangled URLs come from the plain-text version of the email.
I'm almost sure that it has nothing to do with my stack (django, nginx, mandrill) The values for the parameters have characters exactly transposed. The original character is the mangled one minus 13 places (eg. rznvy_cynva = email_plain, ubgryfpbz = hotelscom).
Here is one example of a mangled request that came through:
GET /book/48465?sid=rznvy_cynva&order=q09362qs55-741722-442521-98n2-n88s4nnr87192n&checkOut=07-17-15&affiliate=ubgryfpbz&checkIn=07-16-15 HTTP/1.1" 302 5 "-" "Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident/5.0)"
All of the requests with mangled URLs have the same user-agent as the example.
The IP addresses associated with the mangled URLs aren't restricted to any location.
Looking up the user-agent, this seems to be restricted to desktop Windows 7, IE9 users.
It is anti-malware software on your recipients' computers. It gets the links and scans your pages for any possible vulnerabilities. It uses rot13 obfuscation to ensure that it doesn't take any unwanted actions ("buy now", etc.).
https://security.stackexchange.com/questions/48684/help-investigating-potential-website-attack-url-rewriting-and-rot-13-obfuscatio
The solution is to track down what anti-malware software / company is performing the scans, and get your site whitelisted if possible.
This is going into the realm of speculation, but I'm also guessing you cannot get any answers which don't, so here goes ...
The rot13 encryption does not look like an accident. I have two guesses to offer;
Somebody is sharing their email and obfuscating query parameters in links so as to break the "order now", "unsubscribe" etc links while maintaining the overall integrity of the email messages. Maybe this is a feature of a spam-reporting tool or similar?
Alternatively, the queries are made from within a test network where users are not supposed to click on links, but the tools in there need pretty much unrestricted Internet access; so the admin set up an HTTP proxy which rewrites the query URLs to dismantle most GET transactions with parameters. (POST requests I guess would still probably work?)
Your observation that the IP addresses seem to be nonlocalized somewhat contradicts these hypotheses, but it could just mean that you are looking at TOR endpoints or similar.
So I'm trying to parse from this website http://dl.acm.org/dl.cfm . This website doesn't allow web scrapers, so hence I get an HTTP error: 403 forbidden.
I'm using python, so I tried mechanize to fill the form (to automate the filling of the form or a button click), but then again I got the same error.
I can't even open the html page using urllib2.urlopen() function, it gives the same error.
Can anyone help me with this problem?
If the website doesn't allow web scrapers/bots, you shouldn't be using bots on the site to begin with.
But to answer your question, I suspect the website is blocking urllib's default user-agent. You're probably going to have to spoof the user-agent to a known browser by crafting your own request.
headers = {"User-Agent":"Mozilla/5.0 (X11; U; Linux i686) Gecko/20071127 Firefox/2.0.0.11"}
req = urllib2.Request("http://dl.acm.org/dl.cfm", headers=headers)
urllib2.urlopen(req)
EDIT: I tested this and it works. The site is actively blocking based on user-agents to stop badly made bots from ignoring robots.txt