We have a requirement coming in to try to, as best we can, determine the progress an email made to the user. We know it's not 100%, and the solution I'm advising is to use an image/watermark in the email that is loaded from a URL that would record that the image was read...BUT there's a fair chance that they're going to rely on read/delivery receipts and bouncebacks. So I wanted to learn more about it, both to be ready and so I can argue against it in the meeting.
If we were to set up an email mailbox to receive bouncebacks, read receipts, and delivery receipts and then write a java program to poll said mailbox, get the messages and inspect them. How could I tell the bounces from the read receipts from the delivery receipts from the spam? I know that the read and delivery receipt REQUESTS are SMTP headers. Do the returning messages have a header that tells which they are? And do the bouncebacks? And if so, what are they? If not, am I parsing the message body? Does that differ from server to server? Is there any standard (or close to standard) thing in it? Like the word 'undeliverable' is always there?
I tried to google, but all the hits I could get were about REQUESTING the receipts.
How could I tell the bounces from the read receipts from the delivery receipts from the spam?
Jakarta Mail and JavaMail have com.sun.mail:dsn artifact that have support for parsing and creating messages containing Delivery Status Notifications.
Those classes have constructors that can parse the delivery notifications but it may not be able to parse any read receipt.
Related
I'm designing a capability to send emails out for my app.
I was wondering once I send out an email, is there anyway to find out whether
the email address exists and is real
the email was received
if not received, what the problem was? (mailbox full, email
address
invalid etc.)
the email was read (probably asking too much
but
would be good)
Do I get any feedback at all?
I'm using the SMTPClient in the .NET framework to do this.
no. You can find out if the target server accepts the address. but you can not find out if the account really exists. even if the server accepts the address it could be bounced later.
no. if you don't get a bounce, you have to assume it was delivered. there is no guarantee. it could have landed in a spam box etc.
if a mail is not received you either get a bounce message (or depending on how you send the message you get the error directly in the smtp transaction while sending it to the target server).
no. you can request a read-receipt or do fancy stuff like embedding links to external tracker images. but all this stuff is usually blocked by default.
I'm sending newsletter using Zend Mail. I have used setReturnPath() to put all undelivered mail notifications in one place.
And what now?
How to get the list of addresses which were unreachable?
How do I read and parse the returned notifications?
How to know whether the mail returned because of non existing email or just quota exceeded?
Which headers do I need to send and check?
Related:
Variable Envelope Return Pathwiki
Handling undelivered emails in web appso
This class may be helpful. Can determine whether the mail is a bounce and return a response code with description:
http://www.phpclasses.org/package/2691-PHP-Parse-bounced-e-mail-message-reports.html
Short Answer:
you can't do that in a simple way and not in your app.
Long Answer:
You should handle that in asynchronous way and outside your php app (at least in part). First of all you must setup the return address to something like sender+recipient=recipientdomain.com#senderdomain.com as in the TimB answer. At this point all the notification sent by receiving smtp server will go to that address.
Then you need to setup the smtp daemon at senderdomain.com mail exchanger to handle that kind of bounce messages and process them in some sort of pipe.
With a pipe you can forward the returned message to an external program which parse the message and extract the needed informations (i.e. the reason why the delivery is failed)
At that point in your program (which can be a cli script in your application) you can mark the address as failing and optionally can record why.
This is a pretty difficult task, which can't be handled in a simple application. Usually I use a dedicated software for large mailing list handling such as sympa which takes care of this task for you.
Otherwise you can use an external delivery service such as Sendgrid which will do the dirty job for you and report the failing addresses with a simple API. As a bonus with this solution, they are in the whitelists for all the major providers, so your email won't be marked as spam as far as you respect some simple rules (i.e. removing bouncing addresses and use an opt-in policy for your newsletter)
Well, the second link and especially the answer by TimB explains very well the procedure.
What may not be clear is that the return path is nothing other than a regular email account, i.e. you will get the email to that address. Zend_Mail is not waiting for a response and hence there is no list of addresses.
I work at a university, on a project for a web driven academic management system and I'm currently facing the following problem:
Sometimes the application needs to send e-mails, most of then are sent on demand (users ask for a password recovery link, for example). Many emails for this kind of service are sent daily, and if on a peak of access they are sent massively. This has caused our email server to be included in blacklists of common email providers (like yahoo and hotmail), resulting in failures on email delivery.
What are the common causes for this kind of problem? Is it possible to avoid these blacklists? Or at least is there any good practices to follow so I can "flag" these useful emails as non-spam or safe email?
thanks for reading.
first of all, check if those messages are really sent to email addresses in your account database. maybe there is a security hole in your application that allows sending messages to arbitrary recipients. an indicator of that would be if your domain or ip is blacklisted not only at specific providers like yahoo or hotmail, but also on public blacklists like spamhaus.
("most of then are sent on demand".. makes me think.. what about the others? could they be interpreted as spam by many recipients?)
then you need to find out if your server is blocked due to
the amount of messages sent or due to the content looking "spammy".
Check your logs from the time before the blacklisting happens. Do you see many deferred messages (4xx error code), do they contain error messages that indicate too many messages from your IP?
if so, configure your MTA to throttle message delivery to those providers.
also check your mailserver setup:
correct fully qualified HELO?
matching reverse dns?
If you have DKIM , SPF and the like... are the settings correct?
finally, examine the generated messages. Do they have all required headers? Run them through spamassassin and check the result. adapt the formatting of your messages accordingly.
Our clients sometimes don't get the emails that we send out. It's a BIG loss. How do I assure that they receive the emails so that if it's not received in the other end, the program can resend it or do something about it.
None of the suggestions above will work 100% of the time. Many email clients will (rightly so) refuse to load foreign images, negating the usefulness of "web bugs". They will also refuse (or be unable to) return Outlook-style "receipts". And many mail servers either deliberately (to curb spam) or mistakenly (due to misconfiguration) won't return bounce messages. Or possibly an over-aggressive spam filter ate your message, so it arrived but was never seen by the end user. Plus there is the little matter of mail taking hours or days to reach the end user or bounce, and how do you correlate these late notifications or bounces with the mail you sent 4 days ago?
So basically, you can catch some but not all, no matter what you do. I'd say that any design that relies on being able to know with certainty whether the end user got your mail is fatally flawed.
One thing that you can do is set up a bounceback address that receives any mail that is undeliverable. Use the bounceback address as the From address -- you may want a different one for Reply-To so that replies get directed properly.
Check the bounceback mailbox daily and contact customers to get updated email addresses for the ones that fail. You may be able to automate a couple of retries to failed addresses before resorting to the manual contact in case the failure is only intermittent.
This would take some code outside your application that scans the mailbox and keeps some state information about the number of contacts, etc. and attempts the resend.
Depending on how you generate the mails, you might be able to make this process easier: generate a unique bounce address for every single email you send out. You could use bounces+1234#example.com, for example.
Many SMTP servers will allow you to use the part after the + as a parameter to an external script, etc.
The problem is that many (broken) SMTP servers don't return enough info with a bounce to identify the original message -- sometimes, when there are forwardings involved, you don't even get back the original addressee...
With the above trick you can reliably correlate outgoing messages with incoming bounces.
There is no standard way to know whether the email reached the destination. Many email clients support different types of receipts though. You can use any of those if you want.
There are some ways to know when the user actually read the email.
There are many techniques like adding an image to your email that is to be fetched from your web server. When the user reads the email, the request for the image comes to your server and you can capture the event.
The problem is that there is no way to know that the mail did not reach the destination.
I worked on a bulk email system in a previous life. Deliverability was one of our major issues. The most common cause of undelivered emails is a spam filter.
Here are the steps we took to ensure the highest delivery rates:
We used Return Path to test emails for that spam-like smell.
If you send a lot of emails, you need to make sure your SMTP server is not blacklisted.
Remind your users to add your FROM address to their "safe senders" list.
Use a system that collects bouncebacks and use them to scrub your mailing list. This will also help keep you off the blacklists.
If the emails are critical, consider sending them return-receipt-requested. This will not really guarantee anything, but it might give you some metrics on actual deliverability.
There's not really a good way to determine if the email actually arrives in their inbox, you can only confirm that you sent it. Attach a receipt that lets you know when they open it perhaps?
Microsoft Outlook provides similar functionality, however it is based on the email client. I'm not sure if other clients, like Thunderbird, support this.
However, there is nothing in the protocols that specify receipts.
One option that may work: send a link to a generate web page and monitor that page for hits. This provides its own issues however: confidentiality, etc.
How can I check if an e-mail has been read using POP3/SMTP?
I am able to read e-mails, but I can not figure out if the e-mail has been read or not. Any suggestions are appreciated.
There is no completely reliable way to do this, while some servers support Read receipts it is dependent on the client to respond to the receipt request.
Another way people do this is by embedding a tracking image into an HTML email that will get pulled from a server and that hit constitutes the read however this is often not accurate as most email reader block html external content by default.
Sign up for a free account on statcounter.com. Goto the install code options, choose invisible tracking button and HTML only counter. Statcounter will now provide you an HTML Image snippet that you have to insert inside the body of your HTML email message.
The image isn't visible in the email but the person will have to click "Display Images" when they open their email client.
This is about the only way you can do it if your server or client does not support read receipts.
With POP3, emails are almost always deleted from the server after they are read. When a client connects to a POP3 server, the server usually transfers emails to the client and then deletes the email from its own storage. So, if you can read an email, chances are that it hasn't been read.
As far as I know this is a client side only detail when it comes to POP3. If you wanted to have the status reflected on multiple clients you'd need to used IMAP. With web mail readers they keep track of the unique message ID and whether or not it has been read on the client, but if you were to load it on a desktop pop3 client, it would not be flagged as read.
store the latest read email's message-id somewhere and check when you run to pop
There is no guarantee an e-mail has been read or not, especially 2 cases we won't receive a Read Receipt,
When user opens an email for the message a pop-up confirmation window opens, if user selects No then end user wont receive a read receipt.
From email settings, If user selects Never send a read receipt then also end user wont receive a read receipt.
If user enabled Read Receipt then, the request for the receipt is sent as a header attached to the mail using the method
MimeMessage.setHeader("Disposition-Notification-To", "email-id#domain.com");