I have a software that sends notifications, quotes and invoices to "clients of my clients" by email. Sometimes people don't answer it very fast, so someone needs to call by phone to confirm if they received and get the feedback. I would like to automate this, to know if them, at least, read the email. I know this is very difficult due to how email works, but some companies already try to do this in a satisfactory way, like:
mailgun.com
mailchimp.com
sendwithus.com (YCombinator funded).
In HTML mail messages we can create a resource that points to the server, like a image. But mail clients usually ask permission to the user to load the images. So, problem
here.
But for text mail messages? Is there any way to know the email was read? How companies these companies do?
PS: I don't know what tags is the best to classify my answer, I shall appreciate any edit.
There is no way to be 100% sure if a email was opened, because of its architecture. There are some techniques to do this, but it always depends of user actions and mail client configurations. But:
For HTML messages you can use images and/or the return receipts (RFC 3798).
For text based messages you can use only the return receipts (RFC 3798).
About opening tracking:
Opens are tracked by including a transparent .png file, which will
only work if there is an HTML component to the email (i.e., text only
emails will not track opens). You should note that many email service
providers disable images by default, so this data will only show up if
the recipient clicks on display images button in his/her email.
(Text extracted from mailgun.com user docs)
References:
MailGun.com documentation.
Previous discutions on this thread.
As arnt says, you're fighting the design and basic operation of e-mail. Whenever you send a mail, there is a boundary between a MTA you control (or at least have an account on) and a MTA that is responsible for your target user's mail. What you can know is whether the user's MTA accepted the mail for delivery. Whatever happens afterwards is outside of your control.
Consider an example of a snail mail. When the package enters the recipient's box, you won't know whether they put the whole unopened envelope to a trash bin, or whether they opened is and read the contents very carefully. You can approximate that goal by using crude measures (like embedding a webcam-and-a-computer which will activate upon envelope opening and send you the snapshot of the face of the opener via a cell phone), but doing so is unreliable, unethical, and probably illegal in plenty of countries.
The "return receipts" or embedded image links are similar -- because the whole e-mail is already in the hands of the user's SW, they can do anything with it. A good MUA will probably ask before sending out dumb return receipts, and it also won't load remote images in HTML mail (because it's easy to create an http://trackme.example.org/mail/for/user/12345/message/666/image.png and have a database which says "hey, this URL belongs to Mr. Pichler, and is used in the first message we sent him). The most you can do is to ask nicely, and return receipts (RFC 3798) are a machine-readable way of doing just that.
Related
Does anyone know how Sendgrid tracks email deliveries, junk folder designation, and email opens? The click tracking is pretty obvious, but the others leave me a bit perplexed. I'm assuming that if the email doesn't bounce they track it as delivered? Are they using web beacons for opens? If so, that doesn't seem too reliable.
1: Delivered means the receiving server has accepted receipt of the message at the SMTP level. It makes not claims as to the final destination of the message (Inbox, Promotions, Junk, Spam).
2: There's no way for a sender to know that a receiver has "bulked" a message, since the receiver doesn't send that information back. There are companies (such as Return Path) that partner between senders and receivers to provide aggregate Inboxing data, but nothing at the message level.
3: Opens are defined from the 1x1 pixel image. As you point out, they're limited to the reader displaying images, but I'm not sure of another way to do this. I encourage senders to design messages in such a way to encourage the reader to show images, though you don't want to force it. Also, these days Gmail pre-loads all images, which inflates their open rate.
I need to be able to display ads on email forwarded through a server (preferably postfix) based on the demographic information of the recipient. Basically a message will arrive for someuser#fakedomain.com and be forwarded to realuser#theirdomain.com with a small advertisement at the bottom.
I would like to use postfix because it appears to be able to use mysql data for various tasks, which would be beneficial so that the system could be controlled by a web app that feeds the database.
I'd prefer to use OpenX for the ad server due to targetting channels (used to select ads based on demographic information), the ability to do text ads as well as email zones, and the ability to run it locally. Other ad servers that are better suited to this problem are acceptable, of course.
The core of the problem, as I see it, is being able to write something at the bottom of any given email message. Of course html messages make this even trickier, but I'd settle for having a solution that works for plain text and work up from there.
Commercial software is an option as well, but a few days of intermittent searching hasn't turned anything up.
Simply writing something to the bottom of the email message will fail miserably in a LOT of cases, particularly with HTML email encapsulated in a MIME multipart message (or anything else in a MIME multipart message) because anything after the last MIME section marker is explicitly supposed to be ignored by any MIME parser. If you want your app to work it needs to
determine if the target message is a MIME message other than TEXT/PLAIN
If it is not, append your text only ad at the end and you're done
if it is, determine if its multipart
if not, then you need to determine the content type of the whole message
If the content type is HTML then you can attempt to insert your ad HTML somewhere appropriate. This will be very hit or miss since you have no idea what the HTML layout will be like.
if the content type is anything OTHER than HTML (or maybe RTF), you're best off not touching the message.
If the message is multipart, determine the subtype
if its 'mixed', then you need to determine which part if any is the primary readable content, and then modify that portion as if it were the whole message based on the above rules
If its 'alternative' then you need to find ALL the readable portions and modify each of them in turn according to the above rules
Finally, and most importantly
Be prepared for the massive ill will you will receive from everyone who gets mail routed through your server.
Let me see... People sign up for your service, which I assume is free. They provide you their real email address and in return you provide them a different email address which they can use to e.g. subscribe to mailinglists and for other services. And maybe some spamfilter functionality? That sounds like a legit service to me, especially if you only provide the ads to the person who subscribed. I do wonder if you'd get many subscriptions, though, since many people can just as easy use a Google, Yahoo or Hotmail account for these purposes. So, what is the added value those subscribers will get?
Anyway, you would need to modify existing emails, which is a bit complex when they are HTML mails or if they're digitally signed. (Especially in the latter case, you would actually block the recipient from receiving those emails, since their email system would detect that the email has been tampered with. You might also be at risk of possible legal problems, although I don't think there's anything illegal as long as the recipient agrees with the terms of your service.
I found How do you make sure email you send programmatically is not automatically marked as spam? to (hopefully) be a solid guide to avoiding being marked as spam. Are there any other important tips/suggestions?
How do I track bounces,opens,clicks?
These are features found in paid services like Mail Chimp and Campaign Monitor.
Do the same as Mail Chimp and Campaign Monitor then. LIE about your stats.
There is no accurate way to track emails. If there was it'd just get blocked again. Most people don't want you to know these things and most email software ensures you don't. The stats provided by email tracking services are bogus.
Consider:
Most spam services will detect image
'bugs' and flag you as spam.
Image bugs don't do anything until
the user clicks 'show images'. This
does not mean they didn't open or
read it without images. How can you tell if a mail service downloaded the image preemptively to cache it or check it for image spam?
It can be difficult to determine the difference between a bounce and a reply due to differences in mail servers.
Only clicks can be tracked by redirecting through your server. Even then who can say that mail services won't start processing links in emails to determine whether the email is spam?
Opens can be tracked using a 1x1 picture file in an email. However, this is the same tactic that spammers use to validate email address existence, so you'll be fighting on the same side in that regard, unfortunately.
Clicks can be tracked by assigning a unique identifier to each link, determined by two variables: the URL that was clicked and the email address that clicked it. You can, for example, determine these on-send and store them in a database with the same unique identifier.
Bounces should bounce back to you with the email address intact.
I was looking at the email facebook sends out. In addition to an image, they use a bgsound element as a tracking bug like this:
<bgsound src="http://www.facebook.com/email_open_log_pic.php?mid=99999999&s=a"
volume="-10000" />
I'm guessing the bgsound src is fetched by some readers when the images are off.
Check out Ask MailChimp: How do you track email opens?
if you really want to track bounces, use a service like Email Delivered (www.emaildelivered.com)
i also use Return Path (www.returnpath.com) for a really good reading on whats being delivered to the inbox vs spam box and what esp's are totally rejecting my mail.
Two ideas, clicking links, and statistical fudgery.
Clickthroughs
I would like to add that you can mark emails as read by a user clicking a "view this email online" or by tracking click-throughs. If a user clicks on any <a> tag in your email, send it to a script first that logs the email as read and marks which link they clicked on. This will give you can get a more accurate number.
Stats
I wonder if there is any research into how many users don't show images. That way you could 'statistically' correct for the lower open counts. Just did a bit of reading and found:
A 2009 report from Merkle states that only 48% of email recipients see
images automatically. This means that if an email campaign relies
heavily on images, it’s probably not being read by over half of its
intended recipients. Source
The same site says:
In the latest MarketingSherpa Email Marketing Benchmark Report (2010), a survey of email recipients found that only 33% have images turned on by default.
Somewhere in between there could be a useful figure (35-40%) of users not displaying images in emails. That doesn't necessarily say that those users are opening the emails. Just that auto-displaying images isn't enabled.
If anyone can come up with some more facts/stats, we could potentially get a correction factor. Just with this information I don't think you can do much other than marketing smoke-and-mirrors. For example, 30% opened the emails. Based on 35% of users not displaying images, that means ~9% of users didn't display images, but explicitly chose to turn them on for this email (not really, but just go with it). Let's say that leaves 26% to unaccounted for. You could "correct" your 30% to 56%! All with the magic of bogus stats and a touch of marketing.
If I were to build a newsletter emailing system, I will need to be able to generate reports on how many emails bounced, flagged as spam, unsubscribed, read vs. unread, click through rates etc....
So how do you keep track of user activity after the email has been sent? Am I right in assuming that you CAN NOT embed javascript code into emails to monitor user activity? How else do I gather data for my reports?
Once you send the e-mail, it's free like a baby bird kicked out of the nest. The writers of e-mail clients go to great lengths to make sure that they block any feature that will give you that kind of feedback you're asking for. While there are legitimate uses for this sort of information, spammers use such information to verify and clean their e-mail lists.
Many ISPs also block bounces because they give spammers information.
The best you can do is try to give your readers an incentive to click through back to your site. Then, you can gather information not available to a sender of e-mail.
You can easily track click-through rates by including a tracking query string bit in the URLs and route them through your site.
So a link might be: http://mysite.com/?LinkID=foobar
As for read vs. unread you can get an idea for that by including a small transparent image from your site that includes a tracking URL http://mysite.com/track.gif?EmailID=email. However this is not foolproof since emails can be read offline and most modern email clients do not display images without a user action to display images in the email.
For bounced, you'll have to track those by reading from a mailbox for the From email.
Can't think of way to track emails flagged as spam except to send it to several mailboxes that use some of the common spam filtering products and check the results. However, this isn't likely to be accurate because most can and are customized/trained by individual users.
If you want to do click tracking you'll have to replace all links from your message with links
that point back to your tracking script.
To do efficient tracking that you can actually use later for segmenting your list and better targeting you would have to track the subscriber's id and message and/or campaign id.
Some email marketing systems even track the link position in the message so you know exactly if the recipient clicked on the same link that was at the top of the message or in the middle and in the system that I have built I even track if they clicked a link in the html part of the message or the text part.
The tracker script would record all this information then redirect to the actual link.
Bounce tracking is done by processing the bounce messages that your server will receive or generate when a message cannot be delivered. I recommend using VERP: http://en.wikipedia.org/wiki/Variable_envelope_return_path
Open tracking is done by including the image with tracking code in the url. This would normally point to a script on your site that would record the subscriber id and message and/or campaign id then output the binary date for a transparent 1x1 px wide gif.
You can also track messages that are flagged as spam by some users of some ISPs like hotmail, yahoo, aol, and a few others. they offer feedback loops so every time someone clicks that "Spam" button in their webmail application they will send you a message that you can parse and determine the subscriber that actually flagged the message as spam. VERP also helps with this because the feedback loop messages don't always include the actual email address of the subscriber so you need another way to identify them. This page on wikipedia has a list of feedback ISPS that offer feedback loops : http://en.wikipedia.org/wiki/Feedback_Loop_%28email%29
How can I check if an e-mail has been read using POP3/SMTP?
I am able to read e-mails, but I can not figure out if the e-mail has been read or not. Any suggestions are appreciated.
There is no completely reliable way to do this, while some servers support Read receipts it is dependent on the client to respond to the receipt request.
Another way people do this is by embedding a tracking image into an HTML email that will get pulled from a server and that hit constitutes the read however this is often not accurate as most email reader block html external content by default.
Sign up for a free account on statcounter.com. Goto the install code options, choose invisible tracking button and HTML only counter. Statcounter will now provide you an HTML Image snippet that you have to insert inside the body of your HTML email message.
The image isn't visible in the email but the person will have to click "Display Images" when they open their email client.
This is about the only way you can do it if your server or client does not support read receipts.
With POP3, emails are almost always deleted from the server after they are read. When a client connects to a POP3 server, the server usually transfers emails to the client and then deletes the email from its own storage. So, if you can read an email, chances are that it hasn't been read.
As far as I know this is a client side only detail when it comes to POP3. If you wanted to have the status reflected on multiple clients you'd need to used IMAP. With web mail readers they keep track of the unique message ID and whether or not it has been read on the client, but if you were to load it on a desktop pop3 client, it would not be flagged as read.
store the latest read email's message-id somewhere and check when you run to pop
There is no guarantee an e-mail has been read or not, especially 2 cases we won't receive a Read Receipt,
When user opens an email for the message a pop-up confirmation window opens, if user selects No then end user wont receive a read receipt.
From email settings, If user selects Never send a read receipt then also end user wont receive a read receipt.
If user enabled Read Receipt then, the request for the receipt is sent as a header attached to the mail using the method
MimeMessage.setHeader("Disposition-Notification-To", "email-id#domain.com");