Protect Email on Web Site From Robots and Crawlers [duplicate] - email

This question already has answers here:
Closed 14 years ago.
Closed as duplicate of What are some ways to protect emails on websites from spambots?
I am finally puting up my personal web site. I want to publish a webmaster/feedback email on every page, but I am concerned about SPAM crawlers extracting the email address and bombarding me. This is especially true because I can't use my normal whitelist oriented filtering in this case. Are there effective ways to communicate an email address to people which are hard for crawlers to extract?
My long term plan is to allow feedback via posted forms (and then I will have questions about captcha for y'all), but I don't have time for that now (it's not an immediate priority), and I don't want to go live with no means of feedback at all.

when you get to the forms perspective then look at something free like recaptcha.net
As for now, simply obfuscating the email address in some form should help. I would consider the common [my email (AT) some domain dot com] type of writeout. You could also add another layer of defense by doing that in an email using various fonts and sizes and other effects to beat very sophisticated crawlers with OCR abilities

See this Stackoverflow thread
I'm partial to using javascript to construct the mailto URL on the client side.

Some alternatives to the JavaScript method.
Spell out your address and avoid making it a link.
Example: someone (at) somewhere (dot) com.
HTML encode the email address in the href attribute and inside the anchor tag.

Related

How websites like Facebook are protected against bot without any captcha

How websites like Facebook and Twitter are protected against bot during registration? I mean, there's no captcha at all on the signup form?
I want to create a signup form for a project, and I don't want bot during registration and Captchas are often ugly..
edit:
My question is really during the registration because I know Facebook uses Captchas once registred for the first time.
Facebook uses some sort of hidden spam protection, if you view source of sign-up form you will see things like:
class="hidden_elem"><div class="fsl fwb">Security Check</div>This is a standard security test that we use to prevent spammers from creating fake accounts and spamming users.
so capture becomes visible when javascript will think that you are a bot.
Where is few methods of making it harder for bots to complete registration without capture, things
like timing to fill out form, originators of mouse clicks events ect.
also random session based values in form (to privent direct submissions without downloading of the form first)
also some people use hidden form elements with common names like 'email' that is styled invisible in css but common simple bots will try to fill out all form fields and so you can block them if this hidden element have any value
twitter and fb spend lot of time on developing tecniques to block spammers i don't think they will made it public as it will be counter productive for them to fight the spammers.
But all the client side javascripts you can download from fb or twitter and study them if you want, because most of the protection will happen inside client not on server.
server could only issue some random session variable, check for valid headers in request, overall time etc. its really limited.
some sites are also use ajax exchanges between server and client during the time when user is filling out the form , mostly just to make it harder for bot developer to do simular fake exchanges of data.
Anyway, unfortunatelly where is no easy solution to do decent protection , espesially without captcha or some kind of question
also,
for submit button you can use image map instead of button,
you can dynamically create big image with a submit botton image drawn on it at random position using things like GDI in PHP and using css to display only portion of that image with the actuall button, and on server side check X and Y position of where mouse was clicked, this will be hard for bots to break.
Unless they use real browsers and just emulate keyboard and mouse. Anyway , as i said unfortunatelly where is no easy solution.
One way would be to send a verification to the user's email address or cell phone and obtain verification (so in that case, you would have to allow only one email address or cell phone per account)
Another option is to use "Negative CAPTCHA" or "Honeypot Captcha"
I don't know how Facebook and Twitter do it, but if you want to create something simple and that doesn't interfere with your site aesthetics, I know that some websites just ask the user to enter an answer to a simple math problem like "what is 2 + 3?". This is not the most secure way to do it, but it's just a thought.
Well you can always deploy hardware solutions as well to create Layer 4-7 firewall rules. You can create specific rules to look for the well known agents of bots crawling the web. However to stop newly created bots you need to know what agent they are using for the bot.
Since you don't want CAPTCHA, you can use Keypic - keypic.com - which is an invisible protection, no CAPTCHA needed. It's an efficient antispam method for any web form. Site users don't pass any tests which is good for the site as it improves the quality of the user experience and thus raises user engagement. The solution is a kind of an expert system which analyses the behaviour of the users and checks the databases, then makes a conclusion if the request comes from a legitimate user or a robot.
BTW, Twitter and Facebook still use CAPTCHA for password verification which is a very disputable method in terms of efficiency of such protection.
I had a problem with tons of bots signing up for my Nintendo site so I put a single image of Mario on the sign-up page (making sure nothing in the image data said "Mario") with the text "Who is this? Answer in one word." Haven't had a single bot sign-up since. Not sure if this is actually a good solution though, not sure how smart bots are. I'm kind of surprised that it worked.
In theory it might be keeping out a few legitimate users, but it is hard to imagine many legitimate users of a Nintendo site not knowing who Mario is...

Best practice for handling SMS/email convergence

I'm writing a school administration software package, but it strikes me that many developers will face this same issue: when communicating with users, should you use email or SMS or try to combine them?
A previous version of this question was closed for being too subjective. The answers will be somewhat subjective but I do think this is a good question, topical and not yet debated widely. I'll try to narrow it down as much as I can:
Is it feasible to give users the choice of email versus SMS, and have the same business logic apply to both, with the help of a long form and short form message template?
Do users get annoyed when receiving the same message over SMS and email?
Is it common to present administrators with a single report listing message delivery failures combining both SMS and email?
Are there significant numbers of users who prefer to be contacted via facebook rather than SMS or email?
Is there a best practice for all this stuff?
Is there a place on the web where this stuff has been debated?
Are there any reputable commentators who have made predictions about the future of all this stuff?
I'm particularly interested in hearing from developers who have already grappled with these questions.
Is it feasible to give users the choice of email versus SMS, and have the same business logic apply to both, with the help of a long form and short form message template?
Yes, you can implement logic for the user to select which methods of communication he would like to use
Do users get annoyed when receiving the same message over SMS and email?
ABSOLUTELY! Unless the user has set up to receive messages in both the formats
Is it common to present administrators with a single report listing message delivery failures combining both SMS and email?
Yes
Are there significant numbers of users who prefer to be contacted via facebook rather than SMS or email?
Cannot comment on this, YMMV depending on the usage of the site, the target user group etc. You site stats & trends over a period of time will help fine-tune this aspect.
Is there a best practice for all this stuff?
Sure, see a lot of sites which give opt-in/opt-out options for getting communications from the sites. The tendency to spam/flood inboxes with a lot of email should be avoided & the user should be able to set how little/much data should be sent to him & in what format. Google is your friend here, there are tons of articles out there on different aspects.
Is there a place on the web where this stuff has been debated?
I am sure you will find all sorts of legal & other details around this aspect, google is your friend on this item

How to track analytics for email client usage?

I just found this link on Campaign Monitor's website that mentions they can provide detailed analytics of which email clients people are using. I was under the impression that this wasn't possible. How do they do this? How can I write my own script to be able to determine this in my own HTML email campaigns?
You will need to use an embedded image which links to a .aspx file that can then take a query string from the end of the image src and add information into a database. Its a bit too complicated to explain in a single question.

using ajax to make email addresses safe?

I'm sorry if this is duplicating another post. I have a possible answer to a question in another post but I'm not sure if its a good solution and I wanted to ask people for their views.
The problem is the one raised in this post, how to protect emails from spam bots.
Rather than have the addresses on the page, split into different vars and then assembled by JavaScript, I send an ajax request to the server (just a GET to the welcome_controller) with a key ie 'address_id_42' in the params and it returns a mailto link which is then inserted into the page.
Is there any gain by not having any address data on the page initially? Is any advantage immediately lost by the fact the server will just hand out mailto links if you send it the right address id?
I could easily extend it so the server replies with some custom structure which gets unraveled by the js, but I agree that really this is not the right place to focus and that better spam filtering is the way forward, but I'm interesting in what people think to using ajax as a level of obfuscation?
Cheers :)
It depends on the kind of website it is.
Is the page only accessible after authentication (login)?
Is there another (simpler) way of doing it rather than getting it using AJAX?
The answer to your question really depends on these things.
But in a general way, yes, it might help. But such AJAX requests should only be triggered by some "humanly" action like clicking on "show email" button or something like that.
Also you could convert the email text to an image (which I believe is pretty easy to do with PHP).
Also other solutions could involve separating the two parts of the email address (part before and after '#' symbol) by putting them in different 'spans' etc.
I think obfuscating content through AJAX is a great idea. However, you can also try ready to use third party implementations like Mailhide instead of building all of this yourself. You get an additional layer of security by making the user fill up a CAPTCHA before the email address is revealed.

Integrating dynamic content into mass email

We're looking for the best way to integrate dynamic content into emails sent by various individuals (or companies) using various mass mailing systems, some of them proprietary.
What are the options to do that and what are the advantages and disadvantages?
For example, I guess that one of the options is to add an iframe to the emails. In this case the url for the iframe content will carry a token which will identify the specific email, and our system will generate the iframe content.
Any advice on the subject is highly appreciated.
Most email readers won't allow you to input dynamic content into emails. Unfortunately there's no way round this other than asking the users to click a link to get to the content (something I've seen done before).
Almost all email clients nowadays do not even download images from remote locations unless explicitly told to do so, let alone displaying iframes. So your best bet will be to ask users to click some link.
I created a multi threaded email mailer that has to generate unique content for each user. Each thread is creating the body of the email. Each section in the email has reference links to the detail page of that record inside the system