How much of a bad idea is it to allow users to send arbitrary emails via our servers - email

My company is developing a cloud contact management service and on our iOS app we're having some problems launching a particular enterprise email client app when the user presses the "Email" button on one of their contacts.
One member of our team came up with an idea to get around the problems with this enterprise app:
We let the user specify their email address in the app's settings and create our own email composing screen. Tapping the email button on a contact would open the composing screen, they would write their message and then we would send it on their behalf from our servers (or via service like mailchimp).
Basically, this would mean we would have to create an endpoint on our api that would accept a POST request with 'from', 'to', 'subject', and 'body' fields which would send the appropriate email.
This seems like a very bad idea as it's essentially creating a free, anonymous email service that could easily be abused send spam.
A few extra notes about our setup:
We don't verify an accounts email when they sign up
Even if we did verify the account's email, the user would need to be able to specify any email, as they may have signed up with personal email, but want to email someone from their work email.
Our API doesn't currently have any kind of rate limiting
Instead of having a from field in the request, we could instead send the id of the contact they want to email. This doesn't really change anything because if someone wants to abuse the send email endpoint they can also abuse the create contact endpoint.
So exactly how much of a bad idea is this, and how can I convince my team not to do this?

A few thoughts against doing it:
This is the perfect spamming service, which could damage the reputation of your company (reputational risk).
Your email servers would very quickly make it into blacklists (RBLs), making your outgoing emails land in spam folders in very many recipients' mailboxes.
Even if your servers are not yet in RBLs, if you send a forged email like that and proper email security is set up at the recipient end, your emails will still have a good chance to get classified as spam. Have a look at things like SPF and DKIM.
This could even have legal implications. Imagine the scenario when one of your users uses this service for something like blackmail. Would you be able to prove it was not you? Probably yes with the right controls, but would you want the hassle?
Still on the legal side, many countries (the EU, mainly) have data protection regulations which strictly control how personal data like email addresses can be used, especially for commercial advertisement. You probably want to adhere to that, but that would be hard with such a service (note that I'm not a lawyer, in such a case it's probably the abuser of your service that would offend these regulations and not you, I don't know, but it's something to consider).
If anyone can just send emails, it will be fairly easy to perform a denial of service attack against your services.
A few controls you could implement to mitigate some risks:
When adding a sender (from) address, you should validate that by for example sending a (cryptographically random) token and checking if the user can send it back (eg. by clicking a link in the email). If he can, that proves to some extent that he controls the email address and is probably a valid sender.
Limit the possible recipient addresses if you can. The best would be if recipients had to opt in to receive emails. If this is not possible, at least let recipients opt out from further emails. For this, you would have to add something like a footer to emails with "never again" links, and implement a facility to maintain recipients to which you must not send anymore.
Implement rate limiting. Depending on your exact scenario and use case, only allow to send the least number of emails acceptable for your application.
Implement proper logging so that you have an audit log of who exactly sent what email to whom. For this, log metadata like IP addresses as well. For this, you will likely have to authenticate your users.
On an operational level, have monitoring in place, and be prepared to ban offending users, based on a clear ToS shared with your users.

Related

Allow customers to send from their own domain in a SAAS application

I'm currently running a SAAS application and mails are being sent from our application using Mailjet.
Some of the larger customers have been asking to allow the emails to be sent from their domain (e.g. info#largehotel.com) instead of our system (notifications#saasapp.com).
Are there any initial pointers I will need to look at? I'm guessing they will need to add our SPF records to their SPF records too and that they will need add a DKIM key that we generate for them to add to their records too? Then do some validation on them on the DNS level and mark them as validated?
I have some understanding to have customers run their own domain against our SAAS domain but a bit lost on the sending from their email domain requirement.
First, for the record, my SaaS platform does this (vía option 2b). It’s an e-commerce marketplace and I need the receipts to be sent from the email address of the product seller, not from me (the marketplace)
You have two(ish) options
Send email through your client’s mail servers (instead of mailjet)
Verify the client’s domain on your Mailjet (or similar email) service
option 1
With option 1, you’ll need to ask your client’s IT team to setup a username and password for you to access their SMTP server. This is essentially just like them creating an email account for you to use. This may seem like the easiest path available for you, but there are potential pitfalls and disadvantages:
Doing this, you will lose the mail open/click/bounce tracking functionality you get with mailjet; because you’ll be using the company’s SMTP server instead.
If you’re sending out as a fairly common email address (eg info#your-client.com) the client may already have that account active on their mail servers. That would allow them to receive replies into the existing infrastructure but make them wary of the security issues with sharing a password to their mail server with you.
You might find that they don’t even have the ability to give you a username and password. Modern mail services don’t allow for SMTP access (which is what your web app will need); and security conscious companies require 2 factor authentication on mail accounts (which your web app can’t answer)
Option 2
For this, you will need to ask their IT team to configure some DNS records to prove to mailjet, and to the email recipient, that you’re allowed to send on behalf of your client.
You did this for your own domain when you first setup mailjet. See https://app.mailjet.com/support/how-to-add-a-sender-address,96.htm for what this involves, but it’s a case of asking the client to configure a DNS record.
That tells mailjet that you’re allowed to send on behalf of that domain; but you’ll also want the client to adjust their SPF and DKIM records so as the recipient of the emails knows to trust Mailjet’s servers with emails sent from your client’s domain name. Normally, recipients only trust email sent from your client’s mail server (which you have as option 1) and distrust email sent from SAAS providers.
You will (or should) have done this on mailjet for your own domain already as well. https://app.mailjet.com/docs/spf-dkim-guide
So for this, you’ll need your client to setup 3 DNS records.
If you go this way, you could setup a separate Mailjet account which they and you have access to. That way they ca see their dashboard directly and feel a sense of ownership and security around it. But you won’t be able to markup the price of it 😜
Conclusion
How important is the tracking? If you can’t lose that you need to go with option 2.
How technically savvy is the client? Are they going to be able to have those DNS records changed? Are they going to be (rightly) security conscious around giving you an account on their main mail sever.
Option 2 would be my preference. You might need to hold their hand through the DNS setup so get it configured on Mailjet (And ask about SPF in here to make sure you get it right) so you can provide them with clear instructions of the specific 3 DNS records to create/update.
Whatever approach you take make sure you’re talking to the right people at your clients side soon. Their marketing team may be keen to do this with you, but if their IT feels left out of the conversation they will be difficult to get on board when you need them to make the changes. Us IT folk can be grumpy and obstinate 😀
your web app
This is going to need some adjustment. You probably already store your Mailjet credentials in a file or environment variables; these might need to move these to a dB table so you can relate credentials with specific accounts. But we’d need more info on the web app to be able to speak more to that side of the challenge.
option 2b
just as a note instead of a real suggestion. Be aware that some email service provers allow the sending verification part to be done by sending an email to someone on that domain (eg admin#yourclient.com) and then allowing sending vía the API if the recipient clicks on the approve link on that email. But, even with that setup you still need the client to configure SPF and DKIM on their DNS, so the extra one record isn’t a big ask. AWS’s SES allows for this. This works for me; but I have different requirements around deliver ability, and a large number of non-tech users (as opposed to your one or two big clients)
you can ask your client to generate programmatic(app key/password) user for email need to use for example info#largehotel.com and some other info like (host:gmail, protocol: smtp,...) all basic info needed then in your saas retrieve all this info to create object with client info that you stored before to send email for the target (from developer prospective non network engineering )
The SPF is the most important think to do. In most cases you have to be very careful about the IP reputation, but since you are using Mailjet it's up to them to manage this part.
Be attentive to the overall quality of the email, text/image ratio... Also offers a text body version of the content and dont forget the unsubscribe link. Since you already send emails with your service, I guess it's points are already correct.

"Send to a Friend" - Risks

Let say I have a website that allows users to send articles on that website to a friend.
The way it works is that when the "send to a friend" link is clicked a form appears and it allows users to fill in the details and an email is sent to their friend.
The user can put in a "from" email address and a "to" email address into this form and a small amount of content.
When the email is received the from email address appears in the FROM and REPLY TO.
This website also sends a great deal of its own email communications to its users.
My question is:
Is there risk to allowing users (bots, attacks etc) to use this application to send emails from my SMTP, and how great is the risk?
My assumption is yes, this is not ideal.
Is it possibly worse than "not ideal"?
I do not know about bots using your form. Should it be a problem? I don't know.. I do know they program bots to be pretty clever, using your custom forms and all.
I do know that some email servers check if the FROM email address has the same IP address as the IP the mail was sent from. So imagine I put in my hotmail email address, and the mail server sees your server, it might flag the email as spam.
In the past I've an e-card websystem. It was a small joint venture with a girl I knew. She created the (cute) cards and I build her an e-card system. The website was pretty simple. Select card, enter email address, placing senders email address in the FROM and sent the email that you would have received an e-card.
Life was good...
Until I found that my entire web server IP was blacklisted at three major spam filtering mechanisms. And that 15% of all email recipients who used to receive e-cards from my site, would not receive their e-cards, because all my emails were blacklisted as spam from the get go. We have receive many many emails from angry "customers" demanding that their e-cards did not arrive. (I still find it funny how some people demanded the service, especially since it was a free service, go figure). My automatic reminder function was telling them the e-card still were not viewed, and they perhaps mistyped the email address, so that might have ticked them off :P
It was pretty annoying for my other customers as well, since they relied on sending out played newsletters and such and calling me that over 20% of the customers did not receive the newsletters.
Sending e-mails is hard. You should also check out Jeff's blog about this. So, learn from my mistake, and please put an email address associated with your email server in the FROM. This will spare you a lot of headaches ;)
yes this is definitely not ideal if this is a public website that any bot can access. but there are easy ways for you to limit spam use.
have your code limit any email
address to send ~50 emails a day and
only ~10 an hour based on your
needs. a bot would probably try to
send a million at once so limit them
on an hourly and daily basis.
store every email communication in a
database and come up with a good
program to monitor the most active
email senders. if you can verify
that an email is trusted, then let
them send as many emails as they
want
think about this site itself, it has very defined actions and reputation guidelines that limit you until you have proved you are trusted.
It may depend on whether you do any authentication to determine who's allowed to send emails. If the user has to be logged in to send articles, then you're probably fine. Bots will fail because they'll never be logged in.
The risk will increase the greater traffic you get to your site, and yes it's probably less than ideal. Unprotected, a bot will inevitably find your unprotected form, and start sending emails from your server.
There are some pretty easy solutions though, the most common probably being to implement something like Captcha
Fairly safe. I assume you do check the "From" address, if only by sending it one (standard!) mail first and asking the owner of that email address to confirm they are really humans ? This prevents most bots from finding and abusing your form. Of course, a directed attack with a human responding to your verification email will still allow spamming. But you've got a much better trail if you have received at least one reply from the alleged "From" address.
However, I don't think this will work reliably. The introduction of techniques like SPF will mean that mails from "example.com" will only be accepted if they originate from an outgoing SMTP server in the *.example.com domain. If you're faking emails with From: addresses #example.com, the receiving SMTP server will see that you are in fact not part of *.example.com and reject the email - and probably blacklist your IP range for good measure.

Best practices: Sending email on behalf of users

The company I work for provides testing services for the healthcare industry. As part of our services, we need to send email to our clients' employees. Typically, these are temp, part-time, or contract employees, and so have private email addresses (eg Hotmail, GMail, Yahoo!, etc).
Up to now, we've been sending from an internal address, but this means that replies come back to us when employees aren't paying attention or don't know to send queries to our clients. I'd like to change this, so that the person who requests that the email is sent is the person that is replied to.
We've used reply-to: in the past, but it seemed to cause additional mail to be trapped by spam filters.
I've been reading about sender: and on-behalf-of: headers, and was wondering what the current best-practice was for sending email in a scenario where we need to send email such that the reply goes to a domain we don't control.
The on-behalf-of header is the best way to do that, but you are also going to get trapped by spam filters. The best to mitigate or lessen the likelihood that you will end up in the spam filter is to implement all the industry standards around verifying your domain and mail server. As indicated in this article:
http://www.codinghorror.com/blog/2010/04/so-youd-like-to-send-some-email-through-code.html
However that is very tough to do, because you need to stay on top of SPAM standards, and abide by CAN-SPAM laws and everything else. The better bet is to use a on-demand cloud based SMTP server like this one:
https://www.postmarkapp.com
Use a company that is a domain expert in the area of sending email and has gone through all the leg work to get the highest deliverability rate. And will stay on top of the standards for you, and monitor black lists for problems.
You're probably looking for Reply-To. It's an official and widely supported header, unlike On-Behalf-Of, and it's not subject to the same spam checks as From.
If you really wanted to appear as sending on behalf of another user, the "mostly" correct way, by SMTP standards, would be to put your "real" address in Sender: and your client's address (of whom you're sending on behalf) in From:. However, From: is specifically targeted by DMARC, a very strict spam prevention protocol implemented by most major e-mail providers. They won't overlook a From: DMARC failure just because you have a valid Sender: header.
DMARC allows domain owners to specify how SPF and DKIM should be applied to the From: header. A popular policy is to reject e-mail that fails either SPF or DKIM, which means your e-mail won't even be flagged as spam: it will be downright rejected.
Sender: + From: still works, technically. It was originally created with the intention of being used by people in the same organization, such as a secretary or an assistant. This has turned into a hard constraint with the advent of spam prevention mechanisms.
You want to cheat and hack email authentication systems by trying to send emails on behalf of others. Maybe this hack can work temporarily, but in the future it will be banned by mailbox providers, as phishing attacks require more and more strict policies mailbox providers need to apply.
To avoid such hacks here is a solution I would suggest. Create a unique email address for every client and make it "mediator" for conversation between client and employees.
How it works
All email conversation must be done through your created email. You can set custom display names (e.g. John <123#yourdomain.com) to not confuse email receivers with your strange unique ids. So when A needs to write to B, it actually writes to your email, then you forward email to B, and vice versa for B to A.
This implementation have some complexity, but that will be paid in the future.

Verifying a user in "Email Submission" use case

I'm building a system that allows people to submit text and photos via email in addition to standard access on the website. I'm trying to weight the security advantages of two strategies in particular for verifying submissions from a user. Here they are as follows:
To based auth: Create a secret email address per user and present this to the user for submission. This strategy has the advantage that people can send from multiple devices that might be setup with different mail accounts
From based auth: Only accept emails from addresses that are registered in the user database. The idea being that it is impractical/difficult to impersonate registered users based on the sending address.
Can you think of other possible solutions? Which strategy of the ones proposed makes the most sense to you?
I would suggest that you not use From based authentication, at least not without some additional credentials (a passphrase, etc)
It's way too easy to forge, and certainly not difficult if you know someone's email address.
If you echo the email back to the user for confirmation, you can make things a little more difficult, but realize that your service can end up being used as a sort of spamming relay. (I could send 100 upload requests to you, with a forged FROM address, and you'd go ahead and spam the real person with 100 confirmation requests)
The better option is to check the registered email address but add the need for a code within the email subject known to the user. This way if they forge the email from address, they would still need a key to authenticate the incoming email.
I would go with "from" + confirmation, to avoid forging.
I.e. receive the email, but send a response with auth token in the subject line (or in the body) back to the "from" address. The user either will need reply, or click a link to confirm the submission.
And you post the content only after confirmation.

How do you make sure email you send programmatically is not automatically marked as spam?

This is a tricky one and I've always relied on techniques, such as permission-based emails (i.e. only sending to people you have permission to send to) and not using blatantly spamish terminology.
Of late, some of the emails I send out programmatically have started being shuffled into people's spam folder automatically and I'm wondering what I can do about it.
This is despite the fact that these particular emails are not ones that humans would mark as spam, specifically, they are emails that contain license keys that people have paid good money for, so I don't think they're going to consider them spam
I figure this is a big topic in which I am essentially an ignorant simpleton.
Use email authentication methods, such as SPF, and DKIM to prove that your emails and your domain name belong together, and to prevent spoofing of your domain name. The SPF website includes a wizard to generate the DNS information for your site.
Check your reverse DNS to make sure the IP address of your mail server points to the domain name that you use for sending mail.
Make sure that the IP-address that you're using is not on a blacklist
Make sure that the reply-to address is a valid, existing address.
Use the full, real name of the addressee in the To field, not just the email-address (e.g. "John Smith" <john#blacksmiths-international.com> ).
Monitor your abuse accounts, such as abuse#yourdomain.example and postmaster#yourdomain.example. That means - make sure that these accounts exist, read what's sent to them, and act on complaints.
Finally, make it really easy to unsubscribe. Otherwise, your users will unsubscribe by pressing the spam button, and that will affect your reputation.
That said, getting Hotmail to accept your emails remains a black art.
Sign up for an account on as many major email providers as possible (gmail/yahoo/hotmail/aol/etc). If you make changes to your emails, either major rewording, changes to the code that sends the emails, changes to your email servers, etc, make sure to send test messages to all your accounts and verify that they are not being marked as spam.
A few bullet points from a previous answer:
Most important: Does the sender address ("From") belong to a domain that runs on the server you send the E-Mail from? If not, make it so. Never use sender addresses like xxx#gmail.com. User reply-to if you need replies to arrive at a different address.
Is your server on a blacklist (e.g. check IP on spamhaus.org)? This is a possibility when you're on shared hosting when neighbours behave badly.
Are mails filtered by a spam filter? Open an account with a freemailer that has a spam folder and find out. Also, try sending mail to an address without any spam filtering at all.
Do you possibly need the fifth parameter "-f" of mail() to add a sender address? (See mail() command in the PHP manual)
If you have access to log files, check those, of course.
Do you check the "from:" address for possible bounce mails ("Returned to sender")? You can also set up a separate "errors-to" address.
You can tell your users to add your From address to their contacts when they complete their order, which, if they do so, will help a lot.
Otherwise, I would try to get a log from some of your users. Sometimes they have details about why it was flagged as spam in the headers of the message, which you could use to tweak the text.
Other things you can try:
Put your site name or address in the subject
Keep all links in the message pointing to your domain (and not email.com)
Put an address or other contact information in the email
Confirm that you have the correct email address before sending out emails. If someone gives the wrong email address on sign-up, beat them over the head about it ASAP.
Always include clear "how to unsubscribe" information in EVERY email. Do not require the user to login to unsubscribe, it should be a unique url for 1-click unsubscribe.
This will prevent people from marking your mails as spam because "unsubscribing" is too hard.
In addition to all of the other answers, if you are sending HTML emails that contain URLs as linking text, make sure that the URL matches the linking text. I know that Thunderbird automatically flags them as being a scam if not.
The wrong way:
Go to your account now: http://www.paypal.com
The right way:
Go to your account now: http://www.yourdomain.org
Or use an unrelated linking text instead of a URL:
Click here to go to your account
You may consider a third party email service who handles delivery issues:
Exact Target
Vertical Response
Constant Contact
Campaign Monitor
Emma
Return Path
IntelliContact
SilverPop
Delivering email can be like black magic sometimes. The reverse DNS is really important.
I have found it to be very helpful to carefully track NDRs. I direct all of my NDRs to a single address and I have a windows service parsing them out (Google ListNanny). I put as much information from the NDR as I can into a database, and then I run reports on it to see if I have suddenly started getting blocked by a certain domain. Also, you should avoid sending emails to addresses that were previously marked as NDR, because that's generally a good indication of spam.
If you need to send out a bunch of customer service emails at once, it's best to put a delay in between each one, because if you send too many nearly identical emails to one domain at a time, you are sure to wind up on their blacklist.
Some domains are just impossible to deliver to sometimes. Comcast.net is the worst.
Make sure your IPs aren't listed on sites like http://www.mxtoolbox.com/blacklists.aspx.
I hate to tell you, but I and others may be using white-list defaults to control our filtering of spam.
This means that all e-mail from an unknown source is automatically spam and diverted into a spam folder. (I don't let my e-mail service delete spam, because I want to always review the arrivals for false positives, something that is pretty easy to do by a quick scan of the folder.)
I even have e-mail from myself go to the spam bucket because (1) I usually don't send e-mail to myself and (2) there are spammers that fake my return address in spam sent to me.
So to get out of the spam designation, I have to consider that your mail might be legitimate (from sender and subject information) and open it first in plaintext (my default for all incoming mail, spam or not) to see if it is legitimate. My spam folder will not use any links in e-mails so I am protected against tricky image links and other misbehavior.
If I want future arrivals from the same source to go to my in box and not be diverted for spam review, I will specify that to my e-mail client. For those organizations that use bulk-mail forwarders and unique sender addresses per mail piece, that's too bad. They never get my approval and always show up in my spam folder, and if I'm busy I will never look at them.
Finally, if an e-mail is not legible in plaintext, even when sent as HTML, I am likely to just delete it unless it is something that I know is of interest to me by virtue of the source and previous valuable experiences.
As you can see, it is ultimately under an users control and there is no automated act that will convince such a system that your mail is legitimate from its structure alone. In this case, you need to play nice, don't do anything that is similar to phishing, and make it easy for users willing to trust your mail to add you to their white list.
one of my application's emails was constantly being tagged as spam. it was html with a single link, which i sent as html in the body with a text/html content type.
my most successful resolution to this problem was to compose the email so it looked like it was generated by an email client.
i changed the email to be a multipart/alternative mime document and i now generate both text/plain and text/html parts.
the email no longer is detected as junk by outlook.
Yahoo uses a method called Sender ID, which can be configured at The SPF Setup Wizard and entered in to your DNS. Also one of the important ones for Exchange, Hotmail, AOL, Yahoo, and others is to have a Reverse DNS for your domain. Those will knock out most of the issues. However you can never prevent a person intentionally blocking your or custom rules.
You need a reverse DNS entry. You need to not send the same content to the same user twice. You need to test it with some common webmail and email clients.
Personally I ran mine through a freshly installed spam assassin, a trained spam assassin, and multiple hotmail, gmail, and aol accounts.
But have you seen that spam that doesn't seem to link to or advertise anything? That's a spammer trying to affect your Bayesian filter. If he can get a high rating and then include some words that would be in his future emails it might be automatically learned as good. So you can't really guess what a user's filter is going to be set as at the time of your mailing.
Lastly, I did not sort my list by the domains, but randomized it.
I've found that using the recipients real first and last name in the body is a sure fire way of getting through a spam filter.
In the UK it's also best practice to include a real physical address for your company and its registered number.
That way it's all open and honest and they're less likely to manually mark it as spam.
I would add :
Provide real unsubscription upon click on "Unsubscribe". I've seen real newsletters providing a dummy unsubscription link that upon click shows " has been unsubscribed successfully" but I will still receive further newsletters.
The most important thing you can do is to make sure that the people you are sending email to are not likely going to hit the "Spam" button when they receive your email. So, stick to the following rules of thumb:
Make sure you have permission from the people you are sending email to. Don't ever send email to someone who did not request it from you.
Clearly identify who you are right at the top of each message, and why the person is receiving the email.
At least once a month, send out a reminder email to people on your list (if you are running a list), forcing them to opt back in to the list in order to keep receiving communications from you. Yes, this will mean your list gets shorter over time, but the up-side is that the people on your list are "bought in" and will be less likely to flag your email.
Keep your content highly relevant and useful.
Give people an easy way to opt out of further communications.
Use an email sending service like SendGrid that works hard to maintain a good IP reputation.
Avoid using short links - these are often blacklisted.
Following these rules of thumb will go a long way.
I have had the same problem in the past on many sites I have done here at work. The only guaranteed method of making sure the user gets the email is to advise the user to add you to there safe list. Any other method is really only going to be something that can help with it and isn't guaranteed.
It could very well be the case that people who sign up for your service are entering emails with typing mistakes that you do not correct. For example: chris#gmial.com -or- james#hotnail.com.
And such domains are configured to be used as spamtraps which will automatically flag your email server's IP and/or domain and hurt its reputation.
To avoid this, do a double-check for the email address that is entered upon your product subscription. Also, send a confirmation email to really ensure that this email address is 100% validated by a human being that is entering the confirmation email, before you send them the product key or accept their subscription. The verification email should require the recipient to click a link or reply in order to really confirm that the owner of the mailbox is the person who signed up.
It sounds like you are depending on some feedback to determine what is getting stuck on the receiving end. You should be checking the outbound mail yourself for obvious "spaminess".
Buy any decent spam control system, and send your outbound mail through it. If you send any decent volume of mail, you should be doing this anyhow, because of the risk of sending outbound viruses, especially if you have desktop windows users.
Proofpoint had spam + anti-virus + some reputation services in a single deployment, for example. (I used to work there, so I happen to know this off the top of my head. I'm sure other vendors in this space have similar features.) But you get the idea. If you send your mail through a basic commerical spam control setup, and it doesn't pass, it shouldn't be going out of your network.
Also, there are some companies that can assist you with increasing delivery rates of non-spam, outbound email, like Habeas.
Google has a tool and guidelines for this. You can find them on: https://postmaster.google.com/ Register and verify your domain name and Google provides an individual scoring of that IP-address and domain.
From the bulk senders guidelines:
Authentication ensures that your messages can be correctly classified. Emails that lack authentication are likely to be rejected or placed in the spam folder, given the high likelihood that they are forged messages used for phishing scams. In addition, unauthenticated emails with attachments may be outrightly rejected, for security reasons.
To ensure that Gmail can identify you:
Use a consistent IP address to send bulk mail.
Keep valid reverse DNS records for the IP address(es) from which you send mail, pointing to your domain.
Use the same address in the 'From:' header on every bulk mail you send.
We also recommend the following:
Sign messages with DKIM. We do not authenticate messages signed with keys using fewer than 1024 bits.
Publish an SPF record.
Publish a DMARC policy.
I always use:
https://www.mail-tester.com/
It gives me feedback on the technical part of sending an e-mail. Like SPF-records, DKIM, Spamassassin score and so on. Even though I know what is required, I continuously make errors and mail-tester.com makes it easy to figure out what could be wrong.
First of all, you need to ensure the required email authentication mechanisms like SPF and DKIM are in place. These two are prominent ways of proving that you were the actual sender of an email and it's not really spoofed. This reduces the chances of emails getting filtered as spam.
Second thing is, you can check the reverse DNS output of your domain name against different DNSBLs. Use below simple command on terminal:
**dig a +short (domain-name).(blacklist-domain-name)**
ie. dig a +short example.com.dsn.rfc-clueless.org
> 127.0.0.2
In the above examples, this means your domain "example.com" is listed in blacklist but due to Domain Setting Compliance(rfc-clueless.org list domain which has compliance issue )
note: I prefer multivalley and pepipost tool for checking the domain listings.
The from address/reply-to-id should be proper, always use visible unsubscribe button within your email body (this will help your users to sign out from your email-list without killing your domain reputation)
The intend of most of the programmatically generated emails is generally transactional, triggered or alert n nature- which means these are important emails which should never land into spam.
Having said that there are multiple parameters which are been considered before flagging an email as spam. While Quality of email list is the most important parameter to be considered, but I am skipping that here from the discussion because here we are talking about important emails which are sent to either ourself or to known email addresses.
Apart from list quality, the other 3 important parameters are;
Sender Reputation
Compliance with Email Standards and Authentication (SPF, DKIM, DMARC, rDNS)
Email content
Sender Reputation = Reputation of Sending IP address + Reputation of Return Path/Envelope domain + Reputation of From Domain.
There is no straight answer to what is your Sender Reputation. This is because there are multiple authorities like SenderScore, Reputation Authority and so on who maintains the reputation score for your domain. Apart from that ISPs like Gmail, Yahoo, Outlook also maintains the reputation of each domain at their end.
But, you can use free tools like GradeMyEmail to get a 360-degree view of your reputation and potential problems with your email settings or any other compliance-related issue too.
Sometimes, if you're using a new domain for sending an email, then those are also found to land in spam. You should be checking whether your domain is listed on any of the global blocklists or not. Again GradeMyEmail and MultiRBL are useful tools to identify the list of blocklists.
Once you're pretty sure with the sender reputation score, you should check whether your email sending domain complies with all email authentications and standards.
SPF
DKIM
DMARC
Reverse DNS
For this, you can again use GradeMyEmail or MXToolbox to know the potential problems with your authentication.
Your SPF, DKIM and DMARC should always PASS to ensure, your emails are complying with the standard email authentications.
Here's an example of how these authentications should look like in Gmail:
Similarly, you can use tools like Mail-Tester which scans the complete email content and tells the potential keywords which can trigger spam filters.
To allow DMARC checks for SPF to pass and also be aligned when using sendmail, make sure you are setting the envelope sender address (-f or -r parameter) to something that matches the domain in the From: header address.
With PHP:
Using PHP's built-in mail() function without setting the 5th paramater will cause DMARC SPF checks to be unaligned if not done correctly. By default, sendmail will send the email with the webserver's user as the RFC5321.MailFrom / Return Path header.
For example, say you are hosting your website domain.com on the host.com web server. If you do not set the additional parameters parameter:
mail($to,$subject,$message,$headers); // Wrong way
The email recipient will receive an email with the following mail headers:
Return-Path: <your-website-user#server.host.com>
From: <your-website-user#domain.com>
Even though this passes SPF checks, it will be unaligned (since domain.com and host.com do not match), which means that DMARC SPF check will fail as unaligned.
Instead, you must pass the envelope sender address to sendmail by including the 5th parameter in the PHP mail() function, for example:
mail($to,$subject,$message,$headers, '-r bounce_email#domain.com'); // Right way
In this case, the email recipient will receive an email with the following mail headers:
Return-Path: <bounce_email#domain.com>
From: <your-website-user#domain.com>
Since both of these headers contain addresses from domain.com, SPF will pass and also be aligned, which means that DMARC will also pass the SPF check.