I work at a university, on a project for a web driven academic management system and I'm currently facing the following problem:
Sometimes the application needs to send e-mails, most of then are sent on demand (users ask for a password recovery link, for example). Many emails for this kind of service are sent daily, and if on a peak of access they are sent massively. This has caused our email server to be included in blacklists of common email providers (like yahoo and hotmail), resulting in failures on email delivery.
What are the common causes for this kind of problem? Is it possible to avoid these blacklists? Or at least is there any good practices to follow so I can "flag" these useful emails as non-spam or safe email?
thanks for reading.
first of all, check if those messages are really sent to email addresses in your account database. maybe there is a security hole in your application that allows sending messages to arbitrary recipients. an indicator of that would be if your domain or ip is blacklisted not only at specific providers like yahoo or hotmail, but also on public blacklists like spamhaus.
("most of then are sent on demand".. makes me think.. what about the others? could they be interpreted as spam by many recipients?)
then you need to find out if your server is blocked due to
the amount of messages sent or due to the content looking "spammy".
Check your logs from the time before the blacklisting happens. Do you see many deferred messages (4xx error code), do they contain error messages that indicate too many messages from your IP?
if so, configure your MTA to throttle message delivery to those providers.
also check your mailserver setup:
correct fully qualified HELO?
matching reverse dns?
If you have DKIM , SPF and the like... are the settings correct?
finally, examine the generated messages. Do they have all required headers? Run them through spamassassin and check the result. adapt the formatting of your messages accordingly.
Related
My team supports a website for a client, and we use SendGrid to send email related to the site on their behalf.
We do not have anything to do with their own email server and I don't at present know anything about it.
So far as I can work out, SendGrid has proper authentication and is an authorised sender for their domain, and almost 98% of email is delivered successfully.
However, we have had a handful of bounces with the reason "550 relaying denied" and all of these were to addresses at our client's domain (the same one as their website and the from address of the emails.)
Most emails to their domain were delivered successfully.
Unfortunately I don't have access to the full headers of the bounce emails, only the reason.
I understand that in general this error can either be caused by
the sender not being authenticated correctly. I am very far from being an expert in this but so far as I can tell, there is nothing wrong there. Or
a DNS or similar misconfiguration on the part of the recipient's email domain. I have even less understanding about this and I have no access or responsibility for the client's email server.
My main question is, is there any way the domain being the same as the from address could be related? Being as the email is claiming to be from the same place it's sent to, is it possible for that to affect how it's handled by relays?
If not, I'd also appreciate any pointers on where to look for the issue (or what to advise the client to look at if the problem is likely to be from their end.) I have been trying to research issues with email configuration and authentication but I am very much a novice in this area.
Thanks in advance.
The domain being the same could very well be related, but normally when that happens, the receiving server refuses all mail purporting to be from itself.
Separate from DKIM & SPF, most mail servers believe they alone are responsible for the mail from their domain.com. As such, a lot of them have anti-phishing filters that reject "outside" mail that claims to be from themselves. It's like "You can't be Carrie, I'm Carrie! Go Away!"
The fact that it's only some mail is interesting. The error being relay denied may also be key, though these anti-phishing filters often use "fake" errors to not give away the game.
Do the recipients of the messages that are being rejected have some kind of internal forwarding applied? That may be the cause, in which case that bounce reason is honest.
Or they may have a more defined anti-phishing feature, only rejecting mail From or For certain addresses. You can try testing certain combinations, and see if anything is repeatable.
Ultimately however, it will come down to working with the receiving mail domain's admin, and either updating those rules, or whitelisting the SendGrid IPs that are sending the mail to them.
I am trying to send mails with mailgun. My DNS config (SPF,DKIM) seems to be ok and are being validated in mailgun service. I can send mail to several users with gmail, live and most others mail providers. However, I have a problem when I sent an email for email accounts of my university.
The message is rejected with the following alert:
"554 5.7.1 : Client host rejected: MX-CIDR"
My current DNS settings are:
TXT # "v=spf1 include:mailgun.org ~all"
MX 10 mxa.mailgun.org.
MX 10 mxb.mailgun.org.
DKIM was validated as well. I checked my domain at mxtoolbox and the dns config pass in all tests. I did not find errors related with that alert in others questions. May someone help me to fix it?
Update 1:
Just some more informations:
1) I dont send, and I have absolutely no intention to send spam. I created an educational website, used by students and instructors, and they send messages sometimes between each others. I also send mail to confirm registers, recovery password, as a lot of others websites do. I only send messages to people who was agreed with my terms of service, that includes the information about my mail policy. It is a small service, I never sent more than 2,000 messages in a month (I have 800 registered users so far)
2) I do not believe I was blacklisted, mxtools verify several blacklists databases and my IP have passed in all verifications. Also, the server is not rejecting all messages from my IP, I can send messages with my personal email with the same domain, but I use different services to handle my personal inbox with my domain and the emails send by my website. So, I guess it may be a DNS record mistake.
3) I only use mailgun (or others transactional email services like mandrill or sendgrid) because it is highly recommended (and easy). I use a small VPS and it is hard to configure my own email service (I am a programmer, I am not an expert in that kind of configuration). If exists negative factors about the use of these systems, I really like to know and learn more.
I see no evidence posted that the reason the receiving mail server is rejecting your mail is because of your SPF records.
There isn't even any evidence here that the receiving mail servers are even performing SPF checks on their incoming mail.
Can you explain why exactly you believe that this has anything to do with SPF?
Just because someone's rejecting your mail, and you happen to be messing around with your SPF records, doesn't mean that the reason for your mail being rejected is due to your SPF records.
The only ones who can tell you exactly why your email is being rejected, and what needs to be done to fix it, is the receiving mail servers' administrators, and that's who you should be asking. They are the only ones who know exactly how their mail servers are configured, and how they work. Unless it's evident from the text of the error message, and it's not, anyone else's answer will be nothing but guesswork.
And actually my guess would be that, if anything, the error message seems to suggest that they have simply blacklisted your IP address range, period, for whatever reason. I would interpret "MX-CIDR" as meaning "MX's IP address' (you can Google what "CIDR" means by yourself); i.e.: sending mail server's IP address is explicitly blacklisted from sending them mail.
Now, taking from the referenced domain's web site, I quote:
"Our software automatically manages the delivery process to give your emails the best chance of landing in the inbox."
I would think that the only type of folks who would be concerned about having "the best chance of landing in" someone inbox would be all the typical spamming parasites. I browsed through the referenced website, and I couldn't shake off a slimy feeling I get after typically wandering into a typical spam spewer.
Is this domain being used to send spam?
If so, then you probably know the answer to your question, already.
Certain SPF libraries might reject emails when trying to perform a reverse lookup on the domain that you're sending from.
They usually get this from the MX records attached to the domain and if there's a mismatch it'll fail out with a rejection (more detail here: http://www.zytrax.com/books/dns/ch9/spf.html).
It's usually only a problem if the receiving server is not necessarily configured correctly, or is being super harsh on incoming mail due to an overwhelming amount of spam.
I am build a tool that initiates an SMTP transaction with a domain to see if (a) that domain can receive emails and (b) the desired address exists on that domain. I will be batching large groups of email addresses (10,000+ at a time), but I don't want to bombard the server and get blacklisted. Are there guidelines for how often is it safe to communicate with an SMTP server?
I know about the VRFY command, but it is not implemented across the board. I plan to attempt to use the VRFY command and fall back to using,
MAIL From:<user#example.com>
RCPT To:<first.last#example.org>
QUIT
to see if the message will be deliverable. Again, are there guidelines on how often I can initiate an SMTP transaction like this on a domain?
Edit:
The purpose of this is to create a tool that my organization can use to (a) clean some bad emails from several largely inactive lists so that we do not have to pay our email delivery system to send potentially thousands of emails that will bounce, and (b) check an email when a user subscribes to a list so that we reject emails like aoghuifdgsiuvb#gmail.com.
First of all, spamming is bad. Always ask user wheter she wants to receive newsletters.
"Unsort" mail addresses by domain, leaving the "distance" between e-mail addresses with same domain as big possible.
I think it's not the programmer's decision. There should be a config value which tells a minimum amount of time between two mail sending to the same domain. You should set up a limit also for that config value, avoid setting it to zero or low value.
The only universal guideline I believe can be offered is "don't do this". If you behave like a spammer, you will be treated like a spammer. In the optimistic scenario, sites will already have controls in place, and silently throttle or block you. In less ideal scenarios, they will initiate actions against you on the (reasonable) assumption that you are collecting addresses for a spam list.
A better soluton would be to actually follow through the whole SMTP session, sending a user an email with a verification code/link. This has the advantage of showing that the user actually has control of the address in question and it keeps you from looking like a spam bot.
Volume is not as much the issue as reputation. Let the user know you're about to send them an email in your web flow. This means they're much less likely to mark it as spam.
some hosts have clear and defined guidelines as to how many emails can be sent per hour.
So i guess this would depned onyour hostng service provider, UNLESS your hosting your own mail server off course.
I'm sending newsletter using Zend Mail. I have used setReturnPath() to put all undelivered mail notifications in one place.
And what now?
How to get the list of addresses which were unreachable?
How do I read and parse the returned notifications?
How to know whether the mail returned because of non existing email or just quota exceeded?
Which headers do I need to send and check?
Related:
Variable Envelope Return Pathwiki
Handling undelivered emails in web appso
This class may be helpful. Can determine whether the mail is a bounce and return a response code with description:
http://www.phpclasses.org/package/2691-PHP-Parse-bounced-e-mail-message-reports.html
Short Answer:
you can't do that in a simple way and not in your app.
Long Answer:
You should handle that in asynchronous way and outside your php app (at least in part). First of all you must setup the return address to something like sender+recipient=recipientdomain.com#senderdomain.com as in the TimB answer. At this point all the notification sent by receiving smtp server will go to that address.
Then you need to setup the smtp daemon at senderdomain.com mail exchanger to handle that kind of bounce messages and process them in some sort of pipe.
With a pipe you can forward the returned message to an external program which parse the message and extract the needed informations (i.e. the reason why the delivery is failed)
At that point in your program (which can be a cli script in your application) you can mark the address as failing and optionally can record why.
This is a pretty difficult task, which can't be handled in a simple application. Usually I use a dedicated software for large mailing list handling such as sympa which takes care of this task for you.
Otherwise you can use an external delivery service such as Sendgrid which will do the dirty job for you and report the failing addresses with a simple API. As a bonus with this solution, they are in the whitelists for all the major providers, so your email won't be marked as spam as far as you respect some simple rules (i.e. removing bouncing addresses and use an opt-in policy for your newsletter)
Well, the second link and especially the answer by TimB explains very well the procedure.
What may not be clear is that the return path is nothing other than a regular email account, i.e. you will get the email to that address. Zend_Mail is not waiting for a response and hence there is no list of addresses.
What kind of practical issues are there concerning sending tons of e-mail from a server? Will the likelihood of that e-mail being received be just the same as if it had been sent from g-mail or a personal e-mail account if I for example just blindly call the mail() function in PHP tens of thousands of times a day?
(note: you are not helping a spammer here, this relates to a notify feature I'm thinking about for a future link sharing site)
While you may technically be able to send thousands of mails per minute, in reality you must be carefull.
Say you send out 500 emails to yahoo for example. if enough people mark your message as spam, soon, ANY email you send to yahoo will be marked as spam, or [BULK]. Many isp's routinely tar-pit or outright reject email from servers on lists such as RBL (the real-time black hole list). If your mail IP gets put on one of these lists, you can kiss sending email normally from that ip ever again goodbye. Users are very finicky and it doesn't take many complaints to get your mail ip blocked at many domains.
Also since you are sending automated messages, there are heuristics used to determine if the same message is being sent to many users on the same domain. This also increases the chance your mail will be marked as spam.
This is why clean emails from some addresses always go into the spam box. Their company may have not been careful enough when sending what could be perceived as spam. Proceed with caution.
http://wiki.apache.org/spamassassin/AvoidingFpsForSenders
http://support.microsoft.com/kb/842851
http://www.blacklistedip.com/rbl_list.php
It helps to set a 'x-mailer' and ('X-MimeOLE' if your pretending to be outlook) of a real mail client.
It also helps to send it from a server that is a mail server for the domain in the from address, with forward & reverse DNS records setup.
No issues. Once a server is correctly configured as a mail server (SMTP) for a particular domain, there is no difference if the mail it sends out came to it from Outlook, or from the mail() function in PHP - both are getting the SMTP server to do all the heavy lifting
I always make sure to set my X-Mailer headers correctly (identifying that the message was sent from within PHP) to ensure that any overzealous anti-spam services recognize it as an automatated notification as opposed to bulk/junk email. e.g.
$headers .= "X-Mailer: PHP/".phpversion();
All the configuration and limits you'll encounter are with the SMTP server, not from PHP. You can configure SMTP to rate-limit to 2 messages per second for example, this means if you queue up 1,200 messages they'll be drip fed out over the next hour rather than all at once (two is a really low number, 5-25 is more realistic).
SMTP is the backbone of email and some SMTP servers can happily handle tens of thousands of messages per minute (or more!) - the only limitation you'll likely face is bandwdith ;)
Check with your hosting provider, especially if you're on shared hosting. For example: GoDaddy limits shared hosting accounts to sending 1000 emails per day on their server (http://support.godaddy.com/groups/web-hosting/forum/topic/e-mail-sending-limit/). I'm sure other providers have their own limits (I believe the provider one of the companies I worked for used limited outgoing emails to 250 per minute or something along those lines).
Edit: In my case the solution was to contact our hosting provider. They provided info to route outgoing emails through a server they had dedicated to sending outgoing emails. Solved the problem right away.