I'm using Flask-Mail to send email notifications for events and currently I'm encountering a problem where if two emails of the same subject end with the same thing (which is a rather niche circumstance, but definitely necessary to consider nonetheless), the ending part of the second email will be collapsed by certain mail clients (gmail, for example) that think it's a sign-off or signature or whatever. How do I prevent this?
Currently what I'm doing is generating a random sequence of letters and putting it in a <span style="display:none">...</span> and appending that to the email so the client thinks it's a different footer, but that feels really hacky and I want to know if there's a proper way to do it.
After doing some research, it does not seem as though gmail gives you an option to turn this off. All of the articles written on this topic suggest making your footer unique, which is exactly what you are doing.
Furthermore, this option means that the content of the email does not change at all. This seems to be the best solution.
There is an option in Gmail which allows you to turn off conversation view.
How the grouping occurs is:
The subject must be similar
The sender must be a part of the thread OR in-reply-to must be used
Now to save space, Gmail condenses texts that are same, which is what you see. If you want to turn off conversation view in desktops just navigate to setting -- > generals (tab) for desktop and use the radio button select and press save.
If you are after other devices check this link out.
Related
Lots of people seem to recommend hidden preheader text. For those who don't know, preheader text is a way to control the e-mail content preview, like this:
An example of hiding it would be:
<div style="display:none;font-size:1px;color:#333333;line-height:1px;max-height:0px;max-width:0px;opacity:0;overflow:hidden;">
Wishing you a safe and merry holiday season!
</div>
I'm wondering if hiding this preheader text from humans reading the e-mail might increase spam score or impact deliverability? In the world of web crawlers, hiding content from users but not machines (so-called cloaking) is a big no-no, and it can really hurt you.
Does anyone know if spam checkers might employ similar logic? I've seen some conjecture online, but not much in the way of solid references. Any anecdotes, quotes, or links on this topic would be helpful.
Short answer: yes, it can.
More detailed answer: add hidden text is exactly what spammers do, to bypass spam filters. Every spam filter can detect this hidden-zero-height-zero-width text and - depends on its configuration - will take that into account for the spam score calculation. It's certainly not a single spam marker, but with all the rest of your email, that might brings you over the threshold.
SpamAssassin can be configured to detect that.
I understand, that this is great for marketing purposes, but to get my stuff delivered, I would rather avoid it.
Carsten is right, hiding text in your email and playing with the email client behaviour to get it displayed will increase the spaminess of your email.
I have been working as a developer on antispam filter for several years, hiding text is a very common spammer technique. It is used to outline several words (or even letters) of a block of random text to display the spammer's message, making it harder for antispams filters to identify a common pattern.
Same way, hiding text between a subject header and the first mime part (another kind of preheader) is a common spammer technique.
Alone, these criterias may not be enough to get your message blocked. But added to other spam criterias (ex : if you message has already been identified as a mailing list), that may give you some bad surprises.
My current honey pot on my registration form is an entry for phone number which is placed off screen using position: fixed;.
It tricked a standard breed form filler, but doesn't work as well as I though. Also, the spambots immediately responded to changes in the form, meaning that this is automatic, what can I do.
Note: I hate captchs since the make they users solve my problems which is never good.
Build a really smart honeypot
That may seem obvious, but here are a few tricks(Details later):
Think Like a spam bot
Assume that they are able to know what is on screen or behind other elements
have multiple traps.
Time Trap
Honey pot
1. Think like a spam bot:
Start going through your page like a spam bot, You can even write your own which can waist time but is quite fun :).
Most spam bots will crawl through the markup looking for a <form> element. Then they will look at your inputs and fill them in appropriately, which is the catch: how do they know what to fill in. They will prbably look at the Id, class, placeholder, and label. which brings us to our first method
Method #1:
Mis label inputs in your form code. Bascily your username input should have the Id of #Form_Email boom! spam bot fills out form incorrectly. Also hide and mislabel your inputs labels, use divs instead.*
Method #2 starts here
You've probably noticed that if you simply ignore hidden stuff, based on location what is in front of it and even the good old display: none;,visibility: hidden;,opacity: 0; or type='hidden'. This gives us a powerful weapon. I discovered this by accident while testing a time trap. I used a basic form filler to fill the form. On my site(I'm not talking about GiantCowFilms.com), the register form is in a dialog that opens when a user clicks a register button. By default it is hidden. This gave me an idea for
Method #2
Default: form is hidden. Basically, your form is hidden on page load, but is uncovered by some mouse based action(I don't think bots have mouses). If you wan't your form to be visible on page load, add a I identical decoy one which is above the real one in the markup.If the bot fills in and submits it, block its Ip for a few minuets.** For really users, simply when the mouse hovers over the decoy form, switch them around.
2. Assume that they know what your page looks like
Assuming that hiding honeypot with CSS is perfect is a grave mistake. Their are a lot of super smart screen readers like JAWS that could be repurposed for spaming. That is why you have multiple lines of defense.
3. Have multiple traps
Time Traps:
Going back to thinking like a bot, would you wan't to wait on a site instead of attacking others?
Method #3:Create a time trap.
The best way is to print a time in a hidden input when the page loads. when you submit the form, it tells you how long it took. Fill the form as fast as you can. That should be the minimum amount of time to fill your for.Note: encrypt your time stamp so bots cannot change it. If you wan't to get really fancy, measure the WPM of the bot typing. This is done on stack exchange( try copy and pasting then submitting and question/answer). Also if the rate of typing is very consistent, that is a red flag.
Honeypots (Method #4):
Use all of the above at once for best results. Make sure to trick dumb bots as well as smart bots (don't assume the bot is always trying hard.).
Now, in order to spam us, bots will have to have cursors, render the page, wait, type at a variable realist speed. If they make a bot like that, Then I guess it'll be Captcha time :(.
*People using screen readers will trigger or be confused by these defenses, and depending on your country you could get into trouble for discriminating against blind to semi-blind people. Therefor, when a user triggers the bot test, take them to a non loaded form with a disability friendly captcha like reCaptcha.
**People often share Ips and you can chase away valid users.
P.S. Use simple honey pots like you already have. Some bots are just too dumb to get tricked by what we have here.
For years, I've used my own 'form spammer trap' that uses a variety of techniques to block spambots. The most effective is to obscure fields (especially the form's 'action') for a short time period. Since the bot scrapes your form for the fields, and the 'action' page, and uses that info to "CURL" your form data, putting false info in those areas will fool just about all spambots.
After a timeout (JS-based), the obscured values are replaced with their actual values. By the time that happens, the spambot has moved onto another target, and using the bogus values will fail.
I've developed this whole process into a free solution available at my "Form Spammer Trap" site https://www.FormSpammerTrap.com . That site has been around for years, and the contact form there has yet to be successfully 'spam-botted'. (You bots are welcome to try.)
To implement in your site, you add one 'include' and three functions. And you can customize the form with some advanced techniques.
For those that are interested in how a spambot works, and why common 'tricks' such as hidden fields don't work, see here: https://www.securitydawg.com/how-bots-spam-contact-forms/ .
My 'FormSpammerTrap' solution is free (PHP-based), and is available via the Contact form (just check the box to get the free code). No obligations; I don't even save your email address.
But it is quite effective. IMHO.
Added 21 Jun 2020
The "FormSpammerTrap" solution is still around, and has had significant updates over the years. The latest version allows a lot more (and easier) customization of the form, although the basic use only requires one customization entry. The documentation is greatly improved.
And I am working on version 9, which will add even more customization. It should be released in a couple of weeks.
The https://www.FormSpammerTrap.com site has full details of the latest version, how it work, and a link to a blog entry I wrote about how form spammers work (and how they can be blocked).
And it is still free - just fill out the contact form on the site and check the box and get it immediately, with full documentation and samples. It's fully guaranteed or your money back! (well, it is free...)
(And thanks for the upvote. )
Added July 2022
The program is now up to version 15, with version 16 in the works. There are tons of new features, but the process still works - it blocks all bot attempts on a contact form. Take a look at the changelog on the web site.
And, it's still free.
I've always assumed that it's risky to identify users in urls within emails. For example, let's say my app is something like eventBrite. I'm inviting a set of users to an upcoming event. I create unique urls for each user's email which allows them to simply click those url's in the email to accept or decline. Ie, they will not have to authenticate with the website.
If they view the email on a mobile device or a public computer through webmail, then clicking the links will fully accept/decline.
Is this approach too risky? I had assumed you should avoid this as something could see those urls and make requests on them which would trigger false accepts/declines.
It'a an opinion but I would assume the link itself can be more secure than the email actually. You can make the accept link valid only through certain period of time (it would not make much sense otherwise anyways).
Moreover, you can make it pretty much arbitrary long. So it's basically arbitrarily hard to guess.
That would leave two options to "see" the link, that I can think of. Physically seeing it by eaves dropping. But you could generate a mail in the html form, which would allow you to hide the full link behind a hyper-ref text. Like Accept / Decline.
There are several parts to this answer:
Is it secure? Absolutely not. It's security through obscurity. You're betting somebody can't guess the link which, as long as it's a finite string then they totally can and as soon as they do, they can RSVP to your event.
Follow up Does it matter? Probably not. I imagine the chances of somebody trying to spoof an RSVP to an event are pretty slim. I absolutely wouldn't protect anything critical this way but if you're just doing something like event RSVP etc (no money changing hands) I don't see anything wrong with this approach. As luk32 said, you can also make the links valid for limited amounts of time etc.
The real issue here, (unless there's something you're not telling us and this is somehow a high value target) is how likely is somebody to accidentally stumble on one of these links and RSVP to an event they aren't going to? You can make the chances of that exceedingly unlikely by generating the links in a sufficiently random manner so that no two links are a like. In this case, I don't think security is the big concern so much as data integrity. That is, is the data you're receiving valid.
I'm doing a small newsletter software for my business, and I'm wondering what metrics should I collect. Obviously, bounces and clicks should be tracked, but I'm wondering should I track email opens (via an image or a bgsound element)?
Do popular webmail services and ISPs check for tracking images and possibly increase my spam score? I guess if it increases the chances of ending up in a spam filter, I'd rather not collect that metric.
Thanks.
It's generally bad form to try to track your users in that way. Email opens are a private thing.
If you have an image with a unique URL per message, yes you can track it, but IMO, you really shouldn't. Including unnecessary images in an email is bad for a number of reasons:
Images can increase your spam score. There's a time and place for images. They can improve a message, but used inappropriately, they can look spammy.
It is obvious what you are doing. Sooner or later, one of your customers is likely to get wise to it. Some people won't care; others will feel violated.
It's REALLY unreliable. Most email clients and webmails feature an option to block images by default. You will get massively understated results.
Also remember, some people open an email immediately before they click the "delete" button. You are much better off tracking clicks.
There may be some merit in tracking the images you want to include anyway, but I'd not treat it as anything more than a very basic indicator.
As always it depends on the individual ISP and Webmail services. However, I can share some anecdotal evidence: I periodically use mailchimp to send out mass email notifications, and email opens are tracked in mailchimp using the same approach you mentioned (See following link for reference: http://kb.mailchimp.com/reports/about-open-tracking). I never experienced any issues with ending up in the spam folder, I have only had challenges with bounce backs and ending up in the Gmail promotions tab.
So based on the fact that some companies are already doing this, I have to say it probably won't increase your spam score.
Aside from the visual splendor of HTML emails - links are the only thing keeping me from sending plain text emails. They are much simpler for users at times and reduce bandwidth by over 50%. However, forcing my users to copy/paste or (* shiver *) type the URL from the plain text email is not acceptable.
However, it seems like many services such as gmail and hotmail are converting URLs into HTML links. If that's true, then for some lighter emails I could finally switch to plain text (in certain cases) without bothering anyone.
Anyone know what percentage (or what systems or clients) convert text URLs into clickable links?
Some users access via the web (Hotmail/Yahoo/Gmail) while others use clients (Outlook/Thunderbird).
All email progams I know make links clickable, web-based and normal ones.
You should consider putting the links at the end of the mail, and use "[number]" to refer to them:
You should really visit PEAR[1] and friends, PHP[2]!
[1] http://pear.php.net/
[2] http://www.php.net/
That frees you from problems with longer URLs within the text, and it keeps the text readable.
You must not forget one simple fact and that is every mail client make this configurable. So I for one have the option of reading / sending html emails but I don't do that. So it's totally irrelevant how many mail clients support this, the relevant question is how many users have this enabled.