Is there a "no-reply" email header? - email

I often see automated emails postfixed with a message like
Amazon:
*Please note: this e-mail was sent from an address that cannot accept incoming e-mail. Please use the link above if you need to contact us again about this same issue.
Twitter:
Please do not reply to this message; it was sent from an unmonitored email address. This message is a service email related to your use of Twitter.
Google Checkout:
Need help? Visit the Google Checkout help center. Please do not reply to this message.
Directly underneath this warning, Gmail shows me a reply input field. It seems to me that there should be some sort of header that could be attached to such automated emails that would tell the recipient's email client to not allow replies.
Is there such a header? If not, has it ever been discussed by the groups that control email formats?

Is there such a header?
No. I'm pretty sure there isn't anything like that; and even if there is, it'd be non-standard and not widely supported, so it'd be pretty much useless at the moment. Even if it were to become standard, any such header would presumably just be informational; and for backwards-compatibility, support would have to be entirely optional for email clients.
Clients would be slow to implement it, and many users would still be on old versions of mail clients.
If not, has it ever been discussed by the groups that control email formats?
Probably. People have had a long time to suggest all manner of things with email, but my gut feeling is that it would never be implemented; well... not unless there is a fundamental shift in the ideas of what email is designed to do.
I'm sure Google would be much happier if you didn't even have a "Reply" button when they email you, so if anyone is pushing for it, it'll be the people who are already sending from donotreply#...
Email is designed to be sent from real mailboxes. RFC 2822 and RFC 5322 say:
In all cases, the "From:" field SHOULD NOT contain any mailbox that
does not belong to the author(s) of the message.
To me, that is a clear indication that email is designed as a method for conversation, rather than broadcast.
Probably the biggest killer to any change would be the little bit above that line, which would need to be entirely redefined; which would cause more problems than would be solved:
The originator fields also provide the information required when
replying to a message. When the "Reply-To:" field is present, it
indicates the address(es) to which the author of the message suggests
that replies be sent. In the absence of the "Reply-To:" field,
replies SHOULD by default be sent to the mailbox(es) specified in the
"From:" field unless otherwise specified by the person composing the
reply.

RFC 6854 updates RFC 5322 to allow the group construct to be used in the From field as well (among other things). A group can be empty, which is likely the only way you've ever seen the group syntax being used: undisclosed-recipients:;.
Section 1 of the RFC explicitly lists "no-reply" among the motivations for allowing the group construct in the From field:
The use cases for the "From:" field have evolved. There are numerous instances of automated systems that wish to send email but cannot handle replies, and a "From:" field with no usable addresses would be extremely useful for that purpose.
It provides the following example: From: Automated System:;
However, at the end of the same section, the RFC also says:
This document recommends against the general use of group syntax in these fields at this time
In section 3, the RFC clarifies that the group syntax in the From field is only for Limited Use.
Personally, I think this method should not be used – unless we're certain that all relevant clients display the originating domain in some other way (reconstructed from the Return-Path or a new header). Otherwise, this defeats all the efforts towards domain authentication (SPF, DKIM, and DMARC). Introducing an additional header field which causes clients to simply hide the reply button seems the much better approach to me.
The RFC comments on this aspect in section 5:
Some protocols attempt to validate the originator address by matching the "From:" address to a particular verified domain (for one such protocol, see the Author Domain Signing Practices (ADSP) document [RFC5617]). Such protocols will not be applicable to messages that lack an actual email address (whether real or fake) in the "From:" field. Local policy will determine how such messages are handled, and senders, therefore, need to be aware that using groups in the "From:" might adversely affect deliverability of the message.
What a failed opportunity…

It seems that Thunderbird shows a built-in warning message if From address is of form no-reply#example.com. (The message I noticed this with also had To with no-reply#example.com and my email address in Cc field only. I haven't tested if this is important.)
As far as I know, the form no-reply#example.com has not been defined in any RFC.
Update: It appears that this behavior has been implemented in this bug: https://bugzilla.mozilla.org/show_bug.cgi?id=1342809
and the actual implementation is a regex
/^(.*[._-])?(do[._-]?not|no)[._-]?reply([._-].*)?#/
If that matches, a confirmation prompt is displayed:
Reply Not Supported
The reply address ({ $email }) does not appear to be a monitored
address. Messages to this address will likely not be read by anyone.
[Reply Anyway] [Cancel]
This seems sensible enough for me and maybe other vendors could agree here. Note that this causes all the following to show the warning before allowing a reply:
service-name-no-reply#example.com
donot-reply#example.com
noreply.xyz#example.com
no-reply-userid#example.com
Unfortunately, it doesn't match
no-reply+eventid#example.com
so you have to use something like
no-reply-productname+eventid#example.com
if you want to encode extra information in the tag part.
Update: Note that none of this is specified in any RFC related to email so this is about what works in practice instead of in theory.

Related

Routing Email Replies as Comments to Appropriate Discussions

Story is simple: one user creates a new discussion, and system sends out email notification to other users about that. When these users reply to a notification, their replies should be properly routed as comments to the particular discussion.
When system sends out email notification, it includes routing code in the subject. For example, subject of a notification may look like this: 'Discussion "Lets Talk" has been started {123}'. Since all email clients use Re: ORIGINAL SUBJECT we get {123} back as part of the subject, parse it and know where to put the comment.
We have this working already (had it for years actually), but current implementation looks a bit dirty (especially when codes become longer) so we would like to explore alternatives if there are any. Is there are a more elegant way to approach this, that works reliably across most email clients? Email header that we might be missing? Something similar?
Thanks so much
Since you didn't mention it I'm not sure if you looked into this:
There is a field in the email header called In-Reply-To which should contain the message id(s) of the email(s) that mail is replying to and one name References which should specify a thread this mail belongs to:
"In-Reply-To:" field may be used to identify the message (or
messages) to which the new message is a reply, while the
"References:" field may be used to identify a "thread" of
conversation.
According to the rfc the In-Reply-To field should contain the "parent"-message's Message-Id while the References field will quote the parent-message's References field.
The problem with this fields is that there is no guarantee that there is something useful in them because they are not required to be filled correctly for mail delivery so some mail clients might not fill them correctly or maybe not even at all.
I found this article about building a threading algorithm using the In-Reply-To field and claiming to be robust against garbage and malicious input in these fields.

Which headers are ALWAYS returned in a reply/forward?

I'm making a webstore with integrated customer service. Every few minutes, the system will retrieve emails into the database, parsing headers and relating the message to customers and orders.
Customer threading is fairly reliable via the messages From: header. But what about orders? It seems most people use the Reply-To: header for threading orders ...
From: <orders#company.com>
To: <person#place.com>
Subject: Company Order #314159
Reply-To: <order-314159#company.com>
But a messy Reply-to: obscures and uglifies things, and probably flags spam sensors or something. I definitely don't want to count on the Subject: field, people modify the subject all the time, even when replying. There are other headers that seem suited to the job, like ...
From: <orders#company.com>
To: <person#place.com>
Subject: Company Order #314159
Message-ID: <314159-2>
... or ...
In-Reply-To: <314159-1>
But are these sent back when the person reply? Are there any headers (other than Reply-To:) that are reliably copied into replies and forwards?
You can't rely entirely on headers being preserved. When replying or forwarding, a mail client creates a new message; that mail client can quite legitimately ignore or alter any content, as deemed appropriate.
You might be able to track by the following means, but all are vulnerable to alteration (mainly by the user, but also by a primitive mail client). You should really just use them to make a best-guess.
A disposable Reply-To address. Theoretically, you can also do it with "From" if you want, but Reply-To is better to ensure the user (and their mail server/client) recognise it's from you and act appropriately. I see no reason why a spam filter would care about disposable addresses. Seeing as most spam uses fake addresses anyway and doesn't care about replies, it is not really a spammer trick. It's unlikely to cause a substantial increase in spam filtering. Using a Reply-To for the same domain as your From address is also unlikely to look suspicious.
A unique subject. Yes, it can easily be changed, but usually the existing subject is appended to, rather than deleted (especially if it obviously contains some kind of reference number). You could apply a regular expression match - maybe only using it as a confirmation of your other detection methods.
A unique string in the body (possibly preceded by the words "DO NOT REMOVE THIS LINE")
The In-Reply-To and Reference headers are probably fine, when supported. There is a small chance that a user will copy their reply into a new blank message and trash the headers anyway.
Reply-To is sadly not entirely reliable. All replies should have References: which is better standardized than In-Reply-To: which is not easily machine readable.
Your best bet may be to set the envelope header to a unique identifier, perhaps with a From: and Sender: combo that directs replies to the right place but displays nicely.
See also tangentially Dan Bernstein's notes; http://cr.yp.to/immhf.html and in particular http://cr.yp.to/immhf/thread.html
I don't think you can count on anything when it comes to forwards.
Although you have already received some answers, however, we had a similar situation where we supposed to send emails to customers and read them back and associated them with various activities.
During the research the the only HEADER we found that does not get replaced or Removed by the various email clients (Outook, Yahoo, Gmail etc.) was "XREF".
We have thoroughly tested it and it has been working since we first introduced it.

Persist header data across reply emails

Am trying to determine the best way to persist information from an originating email, through to a reply back.
Essentially, it is to pass a GUID from the original email (c#), whereby when the receiver replies back, that GUID is also sent back for reference.
I have tried setting the MessageID, whereby using Outlook, the In-Reply-To value is set with the original ID, however using some webclient email systems, that value is not created on reply. Is there another way to sent this info through email headers?
Some variation on VERP is probably the most reliable...
http://en.wikipedia.org/wiki/Variable_envelope_return_path
Specifically, instead of having all your replies coming to the same address, encode the information you want to persist into the From address for the email.
For example, in the case of a helpdesk ticket, you could use something like:
From: Helpdesk <support-ticket-123#example.com>
To: End User <user#example.org>
Subject: Ticket #123 - problem with computer.
That way, regardless of what the user edits in the subject or text, you know what ticket is being referred to by the receiving email address.
I don't think you'll be able to do anything that is perfectly reliable by headers alone -- the number of clients that would have to cooperate is immense.
Most systems that do this work by including something in the body of the email that is sent that allows it to identify the message, and including text instructing the recipient to include that block of text in the response. You could also try including it in the subject (and including text in the body to leave the subject unchanged). That's how some mailing list managers I've seen do it.
I stumbled upon this question, and it's been very informative. This, however, leaves me with one question: Will using VERP, or a variation of editing the 'reply-to' or 'from address', cause the messages to be locked up in spam filters?
I have read that spam senders often change the bounce address to prevent their servers from getting clogged with bad email address bounces. Is it a spam risk to assume this approach?
The most reliable way is to put the ID in the subject, which should be preserved throughout replies.
(It doesn't hurt to tell your users that they should keep the subject intact.)
RT, a popular ticketing system, does this. They use a simple subject format like "[Ticket #123]" and key off of 123.

Guidelines for accepting email messages as input to application

A number of applications have the handy feature of allowing users to respond to notification emails from the application. The responses are slurped back into the application.
For example, if you were building a customer support system the email would likely contain some token to link the response back to the correct service ticket.
What are some guidelines, hints and tips for implementing this type of system? What are some potential pitfalls to be aware of? Hopefully those who have implemented systems like this can share their wisdom.
Some guidelines and considerations:
The address question: The best thing to do is to use the "+" extension part of an email (myaddr**+custom**#gmail.com) address. This makes it easier to route, but most of all, easier to keep track of the address routing to your system. Other techniques might use a token in the subject
Spam: Do spam processing outside the app, and have the app filter based on a header.
Queuing failed messages: Don't, for the most part. The standard email behavior is to try for up to 3 days to deliver a message. For an application email server, all this does is create giant spool files of mail you'll most likely never process. Only queue messages if the failure reasons are out of your control (e.g., server is down).
Invalid message handling: There are a multiple of ways a message can be invalid. Some are limitations of the library (it can't parse the address, even though its an RFC valid one). Others are because of broken clients (e.g., omitting quotes around certain headers). Other's might be too large, or use an unknown encoding, be missing critical headers, have multiple values where there should only be one, violate some semantic specific to your application, etc, etc, etc. Basically, where ever the Java mail API could throw an exception is an error handling case you must determine how to appropriately handle.
Error responses: Not every error deserves a response. Some are generated because of spam, and you should avoid sending messages back to those addresses. Others are from automated systems (yourself, a vacation responder, another application mail system, etc), and if you reply, it'll send you another message, repeating the cycle.
Client-specific hacks: like above, each client has little differences that'll complicate your code. Keep this in mind anytime you traverse the structure of a message.
Senders, replies, and loops: Depending on your situation, you might receive mail from some of the following sources:
Real people, maybe from external sources
Mailing lists
Yourself, or one of your own recipient addresses
Other mail servers (bounces, failures, etc)
Entity in another system (my-ldap-group#company.com, system-monitor#localhost)
An automated system
An alias to one of the above
An alias to an alias
Now, your first instinct is probably "Only accept mail from correct sources!", but that'll cause you lots of headaches down the line because people will send the damnedest things to an application mail server. I find its better to accept everything and explicitly deny the exceptions.
Debugging: Save a copy of the headers of any message you receive. This will help out tremendously anytime you have a problem.
--Edit--
I bought the book, Building Scalable Web Sites, mentioned by rossfabricant. It -does- have a good email section. A couple of important points it has are about handling email from wireless carriers and authentication of emails.
You can set the address that the email is sent from, what will be put into the To: address if someone just presses 'Reply-to'. Make that unique, and you'll be able to tell where it came from, and to where it must be directed back to.
When it comes to putting a name beside it though '"something here" ' - put something inviting to have them just reply to the mail. I've seen one major web-app, with Email capturing that has 'do not reply', which turns people off from actually sending anything to it though.
Building Scalable Web sites has a nice section on handling email. It's written by a Flickr developer.
(source: lsl.com.au)
EDIT: I misunderstood your question.
You could configure your email server to catch-all, and generate a unique reply-to address. E.g. CST-2343434#example.com.
A polling process on the server could read the inbox and parse out the relevant part from the received email, CS-2343434 could mean Customer Support ticket ID no. 2343434.
I implemented something like this using JavaMail API.
Just a thought.
The best way to achieve this will be to write a window service that acts like a mail client [pop3 or imap]. This windows service should execute a timed action triggered by a timer, which connects to the mail server and polls the server for any unread message(s) available in the email inbox. The email ID to check for is the email ID on which the users will give their input on/to. If the windows service client finds that there exists any new mail(s) then it should download and filter the email body and push further for processing based on the user input in the email. You can host the input processing in the same windows service but it is not advisable to do so. The windows service can put the inputs in a special application directory or database from where your main appication can read the user inputs received in email and process them as needed.
You will be required to develop a high performance TCP/IP client for doing so. I advise you not to use the default .Net library due to performance issues, instead use one of the best availabel open source TCP/IP implementations for .Net like XF.Server from kodart. we have used this in our applications and achieved remarkably grear results.
Hope this helps..
Bose has a pretty great system where they embed a Queue and Ticket ID into the email itself.
My company has the traditional Case # on the subject line, but when CREATING a case, require a specific character string "New Case" "Tech Support Issue" on the subject line to get through the spam filters.
If the email doesn't match the create or update semantics, the autoresponder sends an email back to the recipient demonstrating how to properly send an email, or directs them to our forums or web support site.
It helps eliminate the spam issue, and yet is still accessible to a wide technical audience that is still heavily email dependent.
Spam is going to be a bit of a concern. However since you are initiating the conversation you can use the presence of your unique identifier (I prefer to use the subject line - "Trouble ticket: Unable to log into web...[artf123456]") to filter out spam. Be sure to check the filter on occasion since some folks mangle the subject when replying.
Email is a cesspool of bad standards and broken clients. You need to be prepared to accept almost anything as input. You will need to be very forgiving about what kinds of input are tolerated. Anything easy for you to program will likely be difficult for your users to use correctly. Consider the old mailing list programs that require you to issue commands in the subject line. Only hardcore nerds can use those effectively. And some of those trouble-ticket CRM things you mentioned have bizarre requirements, such as forcing the user to reply between two specific text markers in the text. That sort of thing is confusing to people.
You'll need to deal with email clients that send you formatted text instead of plain text. Some email clients still don't handle HTML properly (cough GMail) so your replies will also need to be designed appropriately. There are various ways in which photos might be "uploaded" via email as well, especially when mobile phones are involved. You will need to implement various hacks and heuristics to deal with these situations.
It's also entirely possible that you will get email that is valid but unusable by the email parsing library you are using. Whether or not this is important enough to roll your own will be a judgement call.
Finally, others have mentioned using specific email addresses to uniquely identify a "conversation". This is probably the easiest way to do this, as the content of the mail will often not survive a round trip to a client. Be prepared, however, to get mail to old IDs from old customers who, instead of opening a new ticket somehow, reply to an old ticket. Your application will probably need some way to push emails with an old ID into a new case, either manually or automatically. For a CRM system it's very likely that a user would reply to an old email even if you already sent him a new email with a new ID in it. As for whether you should use some.email.address+some.id#yourdomain.com or just some.id#yourdomain.com, I'd go with the latter because the plus-sign confuses some email clients. Make your IDs guids or something and have some way to validate them (such as a CRC or something) and you'll get less junk. Humans should never have to type in the GUIDs, just reply to them. The downside is spam filtering: a user's computer might view such email addresses as spam, and there wouldn't be an easy way to whitelist the addresses.
Which reminds me: sending email these days is full of pitfalls. There are many anti-spam technologies which make it extremely hard for you to send email to your customers. You will need to research all of these and you need to be careful, and do some testing, to ensure that you can reach the major email providers. A website like Campaign Monitor
can help you if you are sending email.

How do you make sure email you send programmatically is not automatically marked as spam?

This is a tricky one and I've always relied on techniques, such as permission-based emails (i.e. only sending to people you have permission to send to) and not using blatantly spamish terminology.
Of late, some of the emails I send out programmatically have started being shuffled into people's spam folder automatically and I'm wondering what I can do about it.
This is despite the fact that these particular emails are not ones that humans would mark as spam, specifically, they are emails that contain license keys that people have paid good money for, so I don't think they're going to consider them spam
I figure this is a big topic in which I am essentially an ignorant simpleton.
Use email authentication methods, such as SPF, and DKIM to prove that your emails and your domain name belong together, and to prevent spoofing of your domain name. The SPF website includes a wizard to generate the DNS information for your site.
Check your reverse DNS to make sure the IP address of your mail server points to the domain name that you use for sending mail.
Make sure that the IP-address that you're using is not on a blacklist
Make sure that the reply-to address is a valid, existing address.
Use the full, real name of the addressee in the To field, not just the email-address (e.g. "John Smith" <john#blacksmiths-international.com> ).
Monitor your abuse accounts, such as abuse#yourdomain.example and postmaster#yourdomain.example. That means - make sure that these accounts exist, read what's sent to them, and act on complaints.
Finally, make it really easy to unsubscribe. Otherwise, your users will unsubscribe by pressing the spam button, and that will affect your reputation.
That said, getting Hotmail to accept your emails remains a black art.
Sign up for an account on as many major email providers as possible (gmail/yahoo/hotmail/aol/etc). If you make changes to your emails, either major rewording, changes to the code that sends the emails, changes to your email servers, etc, make sure to send test messages to all your accounts and verify that they are not being marked as spam.
A few bullet points from a previous answer:
Most important: Does the sender address ("From") belong to a domain that runs on the server you send the E-Mail from? If not, make it so. Never use sender addresses like xxx#gmail.com. User reply-to if you need replies to arrive at a different address.
Is your server on a blacklist (e.g. check IP on spamhaus.org)? This is a possibility when you're on shared hosting when neighbours behave badly.
Are mails filtered by a spam filter? Open an account with a freemailer that has a spam folder and find out. Also, try sending mail to an address without any spam filtering at all.
Do you possibly need the fifth parameter "-f" of mail() to add a sender address? (See mail() command in the PHP manual)
If you have access to log files, check those, of course.
Do you check the "from:" address for possible bounce mails ("Returned to sender")? You can also set up a separate "errors-to" address.
You can tell your users to add your From address to their contacts when they complete their order, which, if they do so, will help a lot.
Otherwise, I would try to get a log from some of your users. Sometimes they have details about why it was flagged as spam in the headers of the message, which you could use to tweak the text.
Other things you can try:
Put your site name or address in the subject
Keep all links in the message pointing to your domain (and not email.com)
Put an address or other contact information in the email
Confirm that you have the correct email address before sending out emails. If someone gives the wrong email address on sign-up, beat them over the head about it ASAP.
Always include clear "how to unsubscribe" information in EVERY email. Do not require the user to login to unsubscribe, it should be a unique url for 1-click unsubscribe.
This will prevent people from marking your mails as spam because "unsubscribing" is too hard.
In addition to all of the other answers, if you are sending HTML emails that contain URLs as linking text, make sure that the URL matches the linking text. I know that Thunderbird automatically flags them as being a scam if not.
The wrong way:
Go to your account now: http://www.paypal.com
The right way:
Go to your account now: http://www.yourdomain.org
Or use an unrelated linking text instead of a URL:
Click here to go to your account
You may consider a third party email service who handles delivery issues:
Exact Target
Vertical Response
Constant Contact
Campaign Monitor
Emma
Return Path
IntelliContact
SilverPop
Delivering email can be like black magic sometimes. The reverse DNS is really important.
I have found it to be very helpful to carefully track NDRs. I direct all of my NDRs to a single address and I have a windows service parsing them out (Google ListNanny). I put as much information from the NDR as I can into a database, and then I run reports on it to see if I have suddenly started getting blocked by a certain domain. Also, you should avoid sending emails to addresses that were previously marked as NDR, because that's generally a good indication of spam.
If you need to send out a bunch of customer service emails at once, it's best to put a delay in between each one, because if you send too many nearly identical emails to one domain at a time, you are sure to wind up on their blacklist.
Some domains are just impossible to deliver to sometimes. Comcast.net is the worst.
Make sure your IPs aren't listed on sites like http://www.mxtoolbox.com/blacklists.aspx.
I hate to tell you, but I and others may be using white-list defaults to control our filtering of spam.
This means that all e-mail from an unknown source is automatically spam and diverted into a spam folder. (I don't let my e-mail service delete spam, because I want to always review the arrivals for false positives, something that is pretty easy to do by a quick scan of the folder.)
I even have e-mail from myself go to the spam bucket because (1) I usually don't send e-mail to myself and (2) there are spammers that fake my return address in spam sent to me.
So to get out of the spam designation, I have to consider that your mail might be legitimate (from sender and subject information) and open it first in plaintext (my default for all incoming mail, spam or not) to see if it is legitimate. My spam folder will not use any links in e-mails so I am protected against tricky image links and other misbehavior.
If I want future arrivals from the same source to go to my in box and not be diverted for spam review, I will specify that to my e-mail client. For those organizations that use bulk-mail forwarders and unique sender addresses per mail piece, that's too bad. They never get my approval and always show up in my spam folder, and if I'm busy I will never look at them.
Finally, if an e-mail is not legible in plaintext, even when sent as HTML, I am likely to just delete it unless it is something that I know is of interest to me by virtue of the source and previous valuable experiences.
As you can see, it is ultimately under an users control and there is no automated act that will convince such a system that your mail is legitimate from its structure alone. In this case, you need to play nice, don't do anything that is similar to phishing, and make it easy for users willing to trust your mail to add you to their white list.
one of my application's emails was constantly being tagged as spam. it was html with a single link, which i sent as html in the body with a text/html content type.
my most successful resolution to this problem was to compose the email so it looked like it was generated by an email client.
i changed the email to be a multipart/alternative mime document and i now generate both text/plain and text/html parts.
the email no longer is detected as junk by outlook.
Yahoo uses a method called Sender ID, which can be configured at The SPF Setup Wizard and entered in to your DNS. Also one of the important ones for Exchange, Hotmail, AOL, Yahoo, and others is to have a Reverse DNS for your domain. Those will knock out most of the issues. However you can never prevent a person intentionally blocking your or custom rules.
You need a reverse DNS entry. You need to not send the same content to the same user twice. You need to test it with some common webmail and email clients.
Personally I ran mine through a freshly installed spam assassin, a trained spam assassin, and multiple hotmail, gmail, and aol accounts.
But have you seen that spam that doesn't seem to link to or advertise anything? That's a spammer trying to affect your Bayesian filter. If he can get a high rating and then include some words that would be in his future emails it might be automatically learned as good. So you can't really guess what a user's filter is going to be set as at the time of your mailing.
Lastly, I did not sort my list by the domains, but randomized it.
I've found that using the recipients real first and last name in the body is a sure fire way of getting through a spam filter.
In the UK it's also best practice to include a real physical address for your company and its registered number.
That way it's all open and honest and they're less likely to manually mark it as spam.
I would add :
Provide real unsubscription upon click on "Unsubscribe". I've seen real newsletters providing a dummy unsubscription link that upon click shows " has been unsubscribed successfully" but I will still receive further newsletters.
The most important thing you can do is to make sure that the people you are sending email to are not likely going to hit the "Spam" button when they receive your email. So, stick to the following rules of thumb:
Make sure you have permission from the people you are sending email to. Don't ever send email to someone who did not request it from you.
Clearly identify who you are right at the top of each message, and why the person is receiving the email.
At least once a month, send out a reminder email to people on your list (if you are running a list), forcing them to opt back in to the list in order to keep receiving communications from you. Yes, this will mean your list gets shorter over time, but the up-side is that the people on your list are "bought in" and will be less likely to flag your email.
Keep your content highly relevant and useful.
Give people an easy way to opt out of further communications.
Use an email sending service like SendGrid that works hard to maintain a good IP reputation.
Avoid using short links - these are often blacklisted.
Following these rules of thumb will go a long way.
I have had the same problem in the past on many sites I have done here at work. The only guaranteed method of making sure the user gets the email is to advise the user to add you to there safe list. Any other method is really only going to be something that can help with it and isn't guaranteed.
It could very well be the case that people who sign up for your service are entering emails with typing mistakes that you do not correct. For example: chris#gmial.com -or- james#hotnail.com.
And such domains are configured to be used as spamtraps which will automatically flag your email server's IP and/or domain and hurt its reputation.
To avoid this, do a double-check for the email address that is entered upon your product subscription. Also, send a confirmation email to really ensure that this email address is 100% validated by a human being that is entering the confirmation email, before you send them the product key or accept their subscription. The verification email should require the recipient to click a link or reply in order to really confirm that the owner of the mailbox is the person who signed up.
It sounds like you are depending on some feedback to determine what is getting stuck on the receiving end. You should be checking the outbound mail yourself for obvious "spaminess".
Buy any decent spam control system, and send your outbound mail through it. If you send any decent volume of mail, you should be doing this anyhow, because of the risk of sending outbound viruses, especially if you have desktop windows users.
Proofpoint had spam + anti-virus + some reputation services in a single deployment, for example. (I used to work there, so I happen to know this off the top of my head. I'm sure other vendors in this space have similar features.) But you get the idea. If you send your mail through a basic commerical spam control setup, and it doesn't pass, it shouldn't be going out of your network.
Also, there are some companies that can assist you with increasing delivery rates of non-spam, outbound email, like Habeas.
Google has a tool and guidelines for this. You can find them on: https://postmaster.google.com/ Register and verify your domain name and Google provides an individual scoring of that IP-address and domain.
From the bulk senders guidelines:
Authentication ensures that your messages can be correctly classified. Emails that lack authentication are likely to be rejected or placed in the spam folder, given the high likelihood that they are forged messages used for phishing scams. In addition, unauthenticated emails with attachments may be outrightly rejected, for security reasons.
To ensure that Gmail can identify you:
Use a consistent IP address to send bulk mail.
Keep valid reverse DNS records for the IP address(es) from which you send mail, pointing to your domain.
Use the same address in the 'From:' header on every bulk mail you send.
We also recommend the following:
Sign messages with DKIM. We do not authenticate messages signed with keys using fewer than 1024 bits.
Publish an SPF record.
Publish a DMARC policy.
I always use:
https://www.mail-tester.com/
It gives me feedback on the technical part of sending an e-mail. Like SPF-records, DKIM, Spamassassin score and so on. Even though I know what is required, I continuously make errors and mail-tester.com makes it easy to figure out what could be wrong.
First of all, you need to ensure the required email authentication mechanisms like SPF and DKIM are in place. These two are prominent ways of proving that you were the actual sender of an email and it's not really spoofed. This reduces the chances of emails getting filtered as spam.
Second thing is, you can check the reverse DNS output of your domain name against different DNSBLs. Use below simple command on terminal:
**dig a +short (domain-name).(blacklist-domain-name)**
ie. dig a +short example.com.dsn.rfc-clueless.org
> 127.0.0.2
In the above examples, this means your domain "example.com" is listed in blacklist but due to Domain Setting Compliance(rfc-clueless.org list domain which has compliance issue )
note: I prefer multivalley and pepipost tool for checking the domain listings.
The from address/reply-to-id should be proper, always use visible unsubscribe button within your email body (this will help your users to sign out from your email-list without killing your domain reputation)
The intend of most of the programmatically generated emails is generally transactional, triggered or alert n nature- which means these are important emails which should never land into spam.
Having said that there are multiple parameters which are been considered before flagging an email as spam. While Quality of email list is the most important parameter to be considered, but I am skipping that here from the discussion because here we are talking about important emails which are sent to either ourself or to known email addresses.
Apart from list quality, the other 3 important parameters are;
Sender Reputation
Compliance with Email Standards and Authentication (SPF, DKIM, DMARC, rDNS)
Email content
Sender Reputation = Reputation of Sending IP address + Reputation of Return Path/Envelope domain + Reputation of From Domain.
There is no straight answer to what is your Sender Reputation. This is because there are multiple authorities like SenderScore, Reputation Authority and so on who maintains the reputation score for your domain. Apart from that ISPs like Gmail, Yahoo, Outlook also maintains the reputation of each domain at their end.
But, you can use free tools like GradeMyEmail to get a 360-degree view of your reputation and potential problems with your email settings or any other compliance-related issue too.
Sometimes, if you're using a new domain for sending an email, then those are also found to land in spam. You should be checking whether your domain is listed on any of the global blocklists or not. Again GradeMyEmail and MultiRBL are useful tools to identify the list of blocklists.
Once you're pretty sure with the sender reputation score, you should check whether your email sending domain complies with all email authentications and standards.
SPF
DKIM
DMARC
Reverse DNS
For this, you can again use GradeMyEmail or MXToolbox to know the potential problems with your authentication.
Your SPF, DKIM and DMARC should always PASS to ensure, your emails are complying with the standard email authentications.
Here's an example of how these authentications should look like in Gmail:
Similarly, you can use tools like Mail-Tester which scans the complete email content and tells the potential keywords which can trigger spam filters.
To allow DMARC checks for SPF to pass and also be aligned when using sendmail, make sure you are setting the envelope sender address (-f or -r parameter) to something that matches the domain in the From: header address.
With PHP:
Using PHP's built-in mail() function without setting the 5th paramater will cause DMARC SPF checks to be unaligned if not done correctly. By default, sendmail will send the email with the webserver's user as the RFC5321.MailFrom / Return Path header.
For example, say you are hosting your website domain.com on the host.com web server. If you do not set the additional parameters parameter:
mail($to,$subject,$message,$headers); // Wrong way
The email recipient will receive an email with the following mail headers:
Return-Path: <your-website-user#server.host.com>
From: <your-website-user#domain.com>
Even though this passes SPF checks, it will be unaligned (since domain.com and host.com do not match), which means that DMARC SPF check will fail as unaligned.
Instead, you must pass the envelope sender address to sendmail by including the 5th parameter in the PHP mail() function, for example:
mail($to,$subject,$message,$headers, '-r bounce_email#domain.com'); // Right way
In this case, the email recipient will receive an email with the following mail headers:
Return-Path: <bounce_email#domain.com>
From: <your-website-user#domain.com>
Since both of these headers contain addresses from domain.com, SPF will pass and also be aligned, which means that DMARC will also pass the SPF check.