If I were to build a newsletter emailing system, I will need to be able to generate reports on how many emails bounced, flagged as spam, unsubscribed, read vs. unread, click through rates etc....
So how do you keep track of user activity after the email has been sent? Am I right in assuming that you CAN NOT embed javascript code into emails to monitor user activity? How else do I gather data for my reports?
Once you send the e-mail, it's free like a baby bird kicked out of the nest. The writers of e-mail clients go to great lengths to make sure that they block any feature that will give you that kind of feedback you're asking for. While there are legitimate uses for this sort of information, spammers use such information to verify and clean their e-mail lists.
Many ISPs also block bounces because they give spammers information.
The best you can do is try to give your readers an incentive to click through back to your site. Then, you can gather information not available to a sender of e-mail.
You can easily track click-through rates by including a tracking query string bit in the URLs and route them through your site.
So a link might be: http://mysite.com/?LinkID=foobar
As for read vs. unread you can get an idea for that by including a small transparent image from your site that includes a tracking URL http://mysite.com/track.gif?EmailID=email. However this is not foolproof since emails can be read offline and most modern email clients do not display images without a user action to display images in the email.
For bounced, you'll have to track those by reading from a mailbox for the From email.
Can't think of way to track emails flagged as spam except to send it to several mailboxes that use some of the common spam filtering products and check the results. However, this isn't likely to be accurate because most can and are customized/trained by individual users.
If you want to do click tracking you'll have to replace all links from your message with links
that point back to your tracking script.
To do efficient tracking that you can actually use later for segmenting your list and better targeting you would have to track the subscriber's id and message and/or campaign id.
Some email marketing systems even track the link position in the message so you know exactly if the recipient clicked on the same link that was at the top of the message or in the middle and in the system that I have built I even track if they clicked a link in the html part of the message or the text part.
The tracker script would record all this information then redirect to the actual link.
Bounce tracking is done by processing the bounce messages that your server will receive or generate when a message cannot be delivered. I recommend using VERP: http://en.wikipedia.org/wiki/Variable_envelope_return_path
Open tracking is done by including the image with tracking code in the url. This would normally point to a script on your site that would record the subscriber id and message and/or campaign id then output the binary date for a transparent 1x1 px wide gif.
You can also track messages that are flagged as spam by some users of some ISPs like hotmail, yahoo, aol, and a few others. they offer feedback loops so every time someone clicks that "Spam" button in their webmail application they will send you a message that you can parse and determine the subscriber that actually flagged the message as spam. VERP also helps with this because the feedback loop messages don't always include the actual email address of the subscriber so you need another way to identify them. This page on wikipedia has a list of feedback ISPS that offer feedback loops : http://en.wikipedia.org/wiki/Feedback_Loop_%28email%29
Related
I have a software that sends notifications, quotes and invoices to "clients of my clients" by email. Sometimes people don't answer it very fast, so someone needs to call by phone to confirm if they received and get the feedback. I would like to automate this, to know if them, at least, read the email. I know this is very difficult due to how email works, but some companies already try to do this in a satisfactory way, like:
mailgun.com
mailchimp.com
sendwithus.com (YCombinator funded).
In HTML mail messages we can create a resource that points to the server, like a image. But mail clients usually ask permission to the user to load the images. So, problem
here.
But for text mail messages? Is there any way to know the email was read? How companies these companies do?
PS: I don't know what tags is the best to classify my answer, I shall appreciate any edit.
There is no way to be 100% sure if a email was opened, because of its architecture. There are some techniques to do this, but it always depends of user actions and mail client configurations. But:
For HTML messages you can use images and/or the return receipts (RFC 3798).
For text based messages you can use only the return receipts (RFC 3798).
About opening tracking:
Opens are tracked by including a transparent .png file, which will
only work if there is an HTML component to the email (i.e., text only
emails will not track opens). You should note that many email service
providers disable images by default, so this data will only show up if
the recipient clicks on display images button in his/her email.
(Text extracted from mailgun.com user docs)
References:
MailGun.com documentation.
Previous discutions on this thread.
As arnt says, you're fighting the design and basic operation of e-mail. Whenever you send a mail, there is a boundary between a MTA you control (or at least have an account on) and a MTA that is responsible for your target user's mail. What you can know is whether the user's MTA accepted the mail for delivery. Whatever happens afterwards is outside of your control.
Consider an example of a snail mail. When the package enters the recipient's box, you won't know whether they put the whole unopened envelope to a trash bin, or whether they opened is and read the contents very carefully. You can approximate that goal by using crude measures (like embedding a webcam-and-a-computer which will activate upon envelope opening and send you the snapshot of the face of the opener via a cell phone), but doing so is unreliable, unethical, and probably illegal in plenty of countries.
The "return receipts" or embedded image links are similar -- because the whole e-mail is already in the hands of the user's SW, they can do anything with it. A good MUA will probably ask before sending out dumb return receipts, and it also won't load remote images in HTML mail (because it's easy to create an http://trackme.example.org/mail/for/user/12345/message/666/image.png and have a database which says "hey, this URL belongs to Mr. Pichler, and is used in the first message we sent him). The most you can do is to ask nicely, and return receipts (RFC 3798) are a machine-readable way of doing just that.
Does anyone know how Sendgrid tracks email deliveries, junk folder designation, and email opens? The click tracking is pretty obvious, but the others leave me a bit perplexed. I'm assuming that if the email doesn't bounce they track it as delivered? Are they using web beacons for opens? If so, that doesn't seem too reliable.
1: Delivered means the receiving server has accepted receipt of the message at the SMTP level. It makes not claims as to the final destination of the message (Inbox, Promotions, Junk, Spam).
2: There's no way for a sender to know that a receiver has "bulked" a message, since the receiver doesn't send that information back. There are companies (such as Return Path) that partner between senders and receivers to provide aggregate Inboxing data, but nothing at the message level.
3: Opens are defined from the 1x1 pixel image. As you point out, they're limited to the reader displaying images, but I'm not sure of another way to do this. I encourage senders to design messages in such a way to encourage the reader to show images, though you don't want to force it. Also, these days Gmail pre-loads all images, which inflates their open rate.
I'm making a quick Google App Engine program that presents a publicly available form that users can fill out with their name and email address, then can enter a friend's name and e-mail address. The application pulls in the data via POST, then sends a pre-formatted e-mail like 'Hi, , your friend wants to invite you...'
What should I be doing to prevent spammers from exploiting this publicly facing e-mail sending program? Is there a good resource for best-practices in this field? I've spent a few hours searching, but I haven't really found anything definitive...
Principally creating a publicly available form that anyone can use to send[s] a pre-formatted e-mail is another name for creating a spam machine.
You can mitigate by making it harder for non-humans to use it, recaptcha is the typical way to achieve this.
You could send a confirmation email to the sender and require a secondary action (like clicking a link) before sending the email. Or, if you expect your users to return, ask them to sign up (with a similar confirmation) before allowing them to send email.
I would first impose some limit to the # of email addresses a specific user/IP can send. This won't solve the problem but will limit the damage in case someone does try to send spam to 1000 emails.
Second, you could try sending the emails in small batches if an user puts in a lot of email addresses. Send 5 at a time, and monitor to see if there's any spam complaints (you can probably automate this somehow). If no complaints after 2 days, keep sending the rest.
There are many articles and threads about guidelines while sending bulk emails. Most of the times it was mentioned that emails should be sent to the subscribed users. So that we can avoid "users clicking on spam in their mail boxes".
There are some features in famous social networking sites where we can send invitations to Yahoo contact list. In such scenarios the persons in our contact list actually did not subscribe with Linkedin to get invitations or other mails. But the hard thing to understand is how do Linkedin and Facebook mails don't go to spam?
Developers often attack this problem from "how can I do everything (DomainKeys, et al) to ensure the message will not end up in someone's spam box?".
I used to work on a website that mailed 20,000 users weekly.
The key point in keeping out of the spam box - beyond all the technical components laid out in the link above posted by Martin - is making sure that your recipients do not spam-can you.
Bayesian detection algorithms on mail servers source data from users who click "this is spam" in their mail clients, so if your users think you are SPAM, you are - and you'll get lower delivery percentages (be prepared: 100% is impossible).
Use your brand or website name as the From: field. Your customer is most likely to remember that, if nothing else. Attempts to "be close" to the customer by using a name of someone in your organization is usually a bad idea, even if you're consistent. Identify your organization UP FRONT, and in the subject line every time.
Make the unsubscribe link incredibly easy to find (not at the bottom!). Users will say "this is spam" to make the message go away if they can't figure out how to get off your list. You don't want these people on your list anyhow.
Double opt-in. Don't add anyone to your list without their explicit opt-in. Not only do you have their blessing, but the delivery of the opt-in message indicates that they're getting your mail. The ever-famous "call to action" of "please add us to your address book" is viable, but I personally never do this when a service asks me to.
Have a schedule on which you send, to whatever the user has agreed (weekly, monthly, etc). If you are not consistent, users will forget registering with you and the mail will seem random. Also, allow them to change the frequency as the next link right next to "unsubscribe" - many people unsubscribe because they are annoyed by too-frequent messages.
To avoid being considered as a SPAM, the easiest and most powerful method, is to not send a SPAM.
Check this to understand what is a SPAM and avoid sending one: http://en.wikipedia.org/wiki/Spam
Jeff Atwood has written quite a good article on this.
From now on, we are using a commercial provider. There are several on the market (like critsend, which is among the more inexpensive ones) which also do bounce handling and make sure that "permanent failures" are not sent out again in the next bulk.
I found How do you make sure email you send programmatically is not automatically marked as spam? to (hopefully) be a solid guide to avoiding being marked as spam. Are there any other important tips/suggestions?
How do I track bounces,opens,clicks?
These are features found in paid services like Mail Chimp and Campaign Monitor.
Do the same as Mail Chimp and Campaign Monitor then. LIE about your stats.
There is no accurate way to track emails. If there was it'd just get blocked again. Most people don't want you to know these things and most email software ensures you don't. The stats provided by email tracking services are bogus.
Consider:
Most spam services will detect image
'bugs' and flag you as spam.
Image bugs don't do anything until
the user clicks 'show images'. This
does not mean they didn't open or
read it without images. How can you tell if a mail service downloaded the image preemptively to cache it or check it for image spam?
It can be difficult to determine the difference between a bounce and a reply due to differences in mail servers.
Only clicks can be tracked by redirecting through your server. Even then who can say that mail services won't start processing links in emails to determine whether the email is spam?
Opens can be tracked using a 1x1 picture file in an email. However, this is the same tactic that spammers use to validate email address existence, so you'll be fighting on the same side in that regard, unfortunately.
Clicks can be tracked by assigning a unique identifier to each link, determined by two variables: the URL that was clicked and the email address that clicked it. You can, for example, determine these on-send and store them in a database with the same unique identifier.
Bounces should bounce back to you with the email address intact.
I was looking at the email facebook sends out. In addition to an image, they use a bgsound element as a tracking bug like this:
<bgsound src="http://www.facebook.com/email_open_log_pic.php?mid=99999999&s=a"
volume="-10000" />
I'm guessing the bgsound src is fetched by some readers when the images are off.
Check out Ask MailChimp: How do you track email opens?
if you really want to track bounces, use a service like Email Delivered (www.emaildelivered.com)
i also use Return Path (www.returnpath.com) for a really good reading on whats being delivered to the inbox vs spam box and what esp's are totally rejecting my mail.
Two ideas, clicking links, and statistical fudgery.
Clickthroughs
I would like to add that you can mark emails as read by a user clicking a "view this email online" or by tracking click-throughs. If a user clicks on any <a> tag in your email, send it to a script first that logs the email as read and marks which link they clicked on. This will give you can get a more accurate number.
Stats
I wonder if there is any research into how many users don't show images. That way you could 'statistically' correct for the lower open counts. Just did a bit of reading and found:
A 2009 report from Merkle states that only 48% of email recipients see
images automatically. This means that if an email campaign relies
heavily on images, it’s probably not being read by over half of its
intended recipients. Source
The same site says:
In the latest MarketingSherpa Email Marketing Benchmark Report (2010), a survey of email recipients found that only 33% have images turned on by default.
Somewhere in between there could be a useful figure (35-40%) of users not displaying images in emails. That doesn't necessarily say that those users are opening the emails. Just that auto-displaying images isn't enabled.
If anyone can come up with some more facts/stats, we could potentially get a correction factor. Just with this information I don't think you can do much other than marketing smoke-and-mirrors. For example, 30% opened the emails. Based on 35% of users not displaying images, that means ~9% of users didn't display images, but explicitly chose to turn them on for this email (not really, but just go with it). Let's say that leaves 26% to unaccounted for. You could "correct" your 30% to 56%! All with the magic of bogus stats and a touch of marketing.