What kind of practical issues are there concerning sending tons of e-mail from a server? Will the likelihood of that e-mail being received be just the same as if it had been sent from g-mail or a personal e-mail account if I for example just blindly call the mail() function in PHP tens of thousands of times a day?
(note: you are not helping a spammer here, this relates to a notify feature I'm thinking about for a future link sharing site)
While you may technically be able to send thousands of mails per minute, in reality you must be carefull.
Say you send out 500 emails to yahoo for example. if enough people mark your message as spam, soon, ANY email you send to yahoo will be marked as spam, or [BULK]. Many isp's routinely tar-pit or outright reject email from servers on lists such as RBL (the real-time black hole list). If your mail IP gets put on one of these lists, you can kiss sending email normally from that ip ever again goodbye. Users are very finicky and it doesn't take many complaints to get your mail ip blocked at many domains.
Also since you are sending automated messages, there are heuristics used to determine if the same message is being sent to many users on the same domain. This also increases the chance your mail will be marked as spam.
This is why clean emails from some addresses always go into the spam box. Their company may have not been careful enough when sending what could be perceived as spam. Proceed with caution.
http://wiki.apache.org/spamassassin/AvoidingFpsForSenders
http://support.microsoft.com/kb/842851
http://www.blacklistedip.com/rbl_list.php
It helps to set a 'x-mailer' and ('X-MimeOLE' if your pretending to be outlook) of a real mail client.
It also helps to send it from a server that is a mail server for the domain in the from address, with forward & reverse DNS records setup.
No issues. Once a server is correctly configured as a mail server (SMTP) for a particular domain, there is no difference if the mail it sends out came to it from Outlook, or from the mail() function in PHP - both are getting the SMTP server to do all the heavy lifting
I always make sure to set my X-Mailer headers correctly (identifying that the message was sent from within PHP) to ensure that any overzealous anti-spam services recognize it as an automatated notification as opposed to bulk/junk email. e.g.
$headers .= "X-Mailer: PHP/".phpversion();
All the configuration and limits you'll encounter are with the SMTP server, not from PHP. You can configure SMTP to rate-limit to 2 messages per second for example, this means if you queue up 1,200 messages they'll be drip fed out over the next hour rather than all at once (two is a really low number, 5-25 is more realistic).
SMTP is the backbone of email and some SMTP servers can happily handle tens of thousands of messages per minute (or more!) - the only limitation you'll likely face is bandwdith ;)
Check with your hosting provider, especially if you're on shared hosting. For example: GoDaddy limits shared hosting accounts to sending 1000 emails per day on their server (http://support.godaddy.com/groups/web-hosting/forum/topic/e-mail-sending-limit/). I'm sure other providers have their own limits (I believe the provider one of the companies I worked for used limited outgoing emails to 250 per minute or something along those lines).
Edit: In my case the solution was to contact our hosting provider. They provided info to route outgoing emails through a server they had dedicated to sending outgoing emails. Solved the problem right away.
Related
For the past 4 months we have been seeing large delays when sending emails through mandrill to gmail addresses. Sometimes it takes 15 minutes but other times it can be up to an hour. When i check the mandrill outbound section shortly after the email is sent it shows the email was delivered, but it usually takes a while before it actually shows up in my inbox. We are using this service for welcome emails and password resets so waiting long periods of time isn't acceptable.
It has been very hard to find any information on this issue. Has anyone seen this issue? Any recommendations on what i could do to fix it?
I had similar issues with delays on emails sent via Mandrill to gmail.
To fix the issue I viewed the "Sending Domains" page under "Settings" in Mandrill. I discovered the DKIM and SPF DNS records were either missing or not valid. Mandrill will provide you with new values by clicking on the "View... settings" link. After updating these settings we no longer experience the delay.
I've run into this issue a number of times. Our DNS settings were all good (DKIM and SPF confirmed my Mandrill) and after some investigation (looking at the headers of the delayed emails) the delay appeared to be entirely on Mandrill's side (once it was handed off to Gmail or Yahoo the delivery occurred within a second). When I contact Mandrill support they explained why we were seeing these delays:
In looking over the logs for your account we are seeing intermittent
delays for some of your recipients. Generally, the speed of delivery
in most cases depends largely on the receiving domain, and how quickly
they will receive and process emails. Most of the major email
providers limit how much email they'll receive in a certain period of
time, and will restrict delivery—Mandrill's sending servers are
designed to queue and back off sending if this occurs. In these cases,
the receiving mail server or ISP will return a specific kind of SMTP
response telling Mandrill's servers to 'back off' and 'try again
later,' which ultimately results in the message lingering on our mail
servers longer than expected (and since the message isn't passed off
to the receiving server at that point, and we're only getting a 'try
again' response, you won't see that information in the message headers
of the final email you receive. You'll only see that the email stayed
on our servers for a longer time period which can be confusing).
Additionally, even though we may hand the messages off to ISPs for
delivery almost immediately, it's still up to that ISP, like Gmail or
Yahoo, to actually to process that email and place it in the inbox.
Each receiving server is different though, so it may take a different
amount of time for Yahoo to process the mail than Gmail, for example.
In many cases, things like the time of day and overall email traffic
to that recipient server can affect how quickly they're able to
receive and process email.
All that said, the delays you're seeing generally aren't expected, and
while we see that messages are ultimately delivering, we are detecting
factors on our end where we may need to make some changes to help
mitigate further delays. Our delivery team is continuing to monitor
traffic to major ISPs and will make necessary adjustments as needed.
We still periodically see these delays, though they've improved is so the delays are rarely longer than 10 minutes or so, but it still can cause issues with things like password resets or confirmations that are time-sensitive. Bottom line: Mandrill is awesome for bulk mailing, but if you need instantaneous delivery you may want to rely on a different or self-hosted service.
I also had gmail showing emails sent through mandrill around 10 minutes later. And that is unacceptable to register confirmations and password resets.
I had configured my DKIM and SPF dns records and mandrill reported all green in this records.
But mail delivery to gmail was always delayed with no aparent reason.
After a while I decided do test/use my own email server to do this, instead of mandrill. Now there are no delays in gmail. I'm happy :)
After this I think I will only use mandrill for massive email delivery / marketing, where delays are not important. Time will tell.
Would like to hear other people about this subject.
In mandrillapp.com > Settings > Domains > Sending domains, verify these 3 points:
DKIM is valid,
SPF is valid,
domain is verified.
My experience has been that the Google SMTP servers are causing the delay (not Mandrill). Verify this by looking at the original email headers (in gmail, with email opened, in the top right More > Show Original) and pasting the email header into the google Message header analyzer will show you the path your email took and how long it was delayed at each server. This report will also tell you if you DKIM / SPF is invalid.
Why the delay is occurring is still a mystery to me. I suspect however that because the domain I am using to send is new, perhaps the gmail spam filters are grey listing the emails until enough users have opened emails and not clicked the spam button? I don't know.
I am trying to send mails with mailgun. My DNS config (SPF,DKIM) seems to be ok and are being validated in mailgun service. I can send mail to several users with gmail, live and most others mail providers. However, I have a problem when I sent an email for email accounts of my university.
The message is rejected with the following alert:
"554 5.7.1 : Client host rejected: MX-CIDR"
My current DNS settings are:
TXT # "v=spf1 include:mailgun.org ~all"
MX 10 mxa.mailgun.org.
MX 10 mxb.mailgun.org.
DKIM was validated as well. I checked my domain at mxtoolbox and the dns config pass in all tests. I did not find errors related with that alert in others questions. May someone help me to fix it?
Update 1:
Just some more informations:
1) I dont send, and I have absolutely no intention to send spam. I created an educational website, used by students and instructors, and they send messages sometimes between each others. I also send mail to confirm registers, recovery password, as a lot of others websites do. I only send messages to people who was agreed with my terms of service, that includes the information about my mail policy. It is a small service, I never sent more than 2,000 messages in a month (I have 800 registered users so far)
2) I do not believe I was blacklisted, mxtools verify several blacklists databases and my IP have passed in all verifications. Also, the server is not rejecting all messages from my IP, I can send messages with my personal email with the same domain, but I use different services to handle my personal inbox with my domain and the emails send by my website. So, I guess it may be a DNS record mistake.
3) I only use mailgun (or others transactional email services like mandrill or sendgrid) because it is highly recommended (and easy). I use a small VPS and it is hard to configure my own email service (I am a programmer, I am not an expert in that kind of configuration). If exists negative factors about the use of these systems, I really like to know and learn more.
I see no evidence posted that the reason the receiving mail server is rejecting your mail is because of your SPF records.
There isn't even any evidence here that the receiving mail servers are even performing SPF checks on their incoming mail.
Can you explain why exactly you believe that this has anything to do with SPF?
Just because someone's rejecting your mail, and you happen to be messing around with your SPF records, doesn't mean that the reason for your mail being rejected is due to your SPF records.
The only ones who can tell you exactly why your email is being rejected, and what needs to be done to fix it, is the receiving mail servers' administrators, and that's who you should be asking. They are the only ones who know exactly how their mail servers are configured, and how they work. Unless it's evident from the text of the error message, and it's not, anyone else's answer will be nothing but guesswork.
And actually my guess would be that, if anything, the error message seems to suggest that they have simply blacklisted your IP address range, period, for whatever reason. I would interpret "MX-CIDR" as meaning "MX's IP address' (you can Google what "CIDR" means by yourself); i.e.: sending mail server's IP address is explicitly blacklisted from sending them mail.
Now, taking from the referenced domain's web site, I quote:
"Our software automatically manages the delivery process to give your emails the best chance of landing in the inbox."
I would think that the only type of folks who would be concerned about having "the best chance of landing in" someone inbox would be all the typical spamming parasites. I browsed through the referenced website, and I couldn't shake off a slimy feeling I get after typically wandering into a typical spam spewer.
Is this domain being used to send spam?
If so, then you probably know the answer to your question, already.
Certain SPF libraries might reject emails when trying to perform a reverse lookup on the domain that you're sending from.
They usually get this from the MX records attached to the domain and if there's a mismatch it'll fail out with a rejection (more detail here: http://www.zytrax.com/books/dns/ch9/spf.html).
It's usually only a problem if the receiving server is not necessarily configured correctly, or is being super harsh on incoming mail due to an overwhelming amount of spam.
I am build a tool that initiates an SMTP transaction with a domain to see if (a) that domain can receive emails and (b) the desired address exists on that domain. I will be batching large groups of email addresses (10,000+ at a time), but I don't want to bombard the server and get blacklisted. Are there guidelines for how often is it safe to communicate with an SMTP server?
I know about the VRFY command, but it is not implemented across the board. I plan to attempt to use the VRFY command and fall back to using,
MAIL From:<user#example.com>
RCPT To:<first.last#example.org>
QUIT
to see if the message will be deliverable. Again, are there guidelines on how often I can initiate an SMTP transaction like this on a domain?
Edit:
The purpose of this is to create a tool that my organization can use to (a) clean some bad emails from several largely inactive lists so that we do not have to pay our email delivery system to send potentially thousands of emails that will bounce, and (b) check an email when a user subscribes to a list so that we reject emails like aoghuifdgsiuvb#gmail.com.
First of all, spamming is bad. Always ask user wheter she wants to receive newsletters.
"Unsort" mail addresses by domain, leaving the "distance" between e-mail addresses with same domain as big possible.
I think it's not the programmer's decision. There should be a config value which tells a minimum amount of time between two mail sending to the same domain. You should set up a limit also for that config value, avoid setting it to zero or low value.
The only universal guideline I believe can be offered is "don't do this". If you behave like a spammer, you will be treated like a spammer. In the optimistic scenario, sites will already have controls in place, and silently throttle or block you. In less ideal scenarios, they will initiate actions against you on the (reasonable) assumption that you are collecting addresses for a spam list.
A better soluton would be to actually follow through the whole SMTP session, sending a user an email with a verification code/link. This has the advantage of showing that the user actually has control of the address in question and it keeps you from looking like a spam bot.
Volume is not as much the issue as reputation. Let the user know you're about to send them an email in your web flow. This means they're much less likely to mark it as spam.
some hosts have clear and defined guidelines as to how many emails can be sent per hour.
So i guess this would depned onyour hostng service provider, UNLESS your hosting your own mail server off course.
I am trying to build a mail server using Ubuntu to send mail
I have done some research on that and find it is nearly impossible for a individual to send
the mail e.g. hotmail , gmail.
The question i am asking is not how to build a own server, it is why i can not build my own server.
To be precise:
Questions:
1) what are the requirements to send to those e.g. hotmail ,gmail server ? e.g. mx record , clear dns record . (only from server aspect , not concerning other factors such as headers or mail content), It would be easier to understand if they are listed out.
2) I read some document and it said the problem can be overcome by relayhost, what is it about and is it feasible?
3) For those ISP , what are their procedure in building the mail server? How is it different from my own small Ubuntu one?
Sorry for asking a lot of question, any help would be nice and well appreciated .
Most people use an out-of-the-box package as a mail server, rather than trying to write one that follows all of the relevant RFC specifications for SMTP, Internet Message Format, IMAP4, POP3, etc.
I'm not saying "don't write your own", just that if you do, be prepared for months and months of hard work, lots of bugs and even more frustration. It's a big project.
In terms of sending messages, you will need to follow the Simple Mail Transfer Protocol (SMTP) to send messages; and they should be sent to the correct server, as per the recipient's DNS records - see RFC 1034 and RFC 1035.
If you are correctly using SMTP to send valid messages to the right server, there's not a lot else you can do.
Your next problem is going to be reputation. This would be the same, whichever software you use to send your messages.
It's easy for a spammer to set up a new mail server and start sending messages, so it will take a while for some mail servers to trust you (particularly those that are regularly targeted, such as Hotmail, Gmail, etc).
Instead of sending messages directly to the recipient's server, you can use SMTP to send the messages to a relay server. This would usually be your own ISP's server, but it can be any willing partner. You would normally need to make advance arrangements, so that they will permit you to relay messages.
The relay server would then attempt to send the message to the recipient server. If it cannot do so, it must report the failure to the sender.
I work at a university, on a project for a web driven academic management system and I'm currently facing the following problem:
Sometimes the application needs to send e-mails, most of then are sent on demand (users ask for a password recovery link, for example). Many emails for this kind of service are sent daily, and if on a peak of access they are sent massively. This has caused our email server to be included in blacklists of common email providers (like yahoo and hotmail), resulting in failures on email delivery.
What are the common causes for this kind of problem? Is it possible to avoid these blacklists? Or at least is there any good practices to follow so I can "flag" these useful emails as non-spam or safe email?
thanks for reading.
first of all, check if those messages are really sent to email addresses in your account database. maybe there is a security hole in your application that allows sending messages to arbitrary recipients. an indicator of that would be if your domain or ip is blacklisted not only at specific providers like yahoo or hotmail, but also on public blacklists like spamhaus.
("most of then are sent on demand".. makes me think.. what about the others? could they be interpreted as spam by many recipients?)
then you need to find out if your server is blocked due to
the amount of messages sent or due to the content looking "spammy".
Check your logs from the time before the blacklisting happens. Do you see many deferred messages (4xx error code), do they contain error messages that indicate too many messages from your IP?
if so, configure your MTA to throttle message delivery to those providers.
also check your mailserver setup:
correct fully qualified HELO?
matching reverse dns?
If you have DKIM , SPF and the like... are the settings correct?
finally, examine the generated messages. Do they have all required headers? Run them through spamassassin and check the result. adapt the formatting of your messages accordingly.