I have to make a newsletter sending utility application which will collect the list of subscriber from our central database and send out the newsletter. I've considered the possibility to be blacklisted due to flooding if I just flush out all emails at once, so I decided to go on a desktop-based softwer which will email those slowly.
my question is
what is the max emails per hour that may be addressed to the same email domain (recipient/incoming server)?
or what should be delay between 2 e-mails to the same server for it doesn't consider it flooding?
whichever of the above applies more appropriate to the real-world of mail servers configuration...
thanks
I make Thread.Sleep(2000) after every 2 mail
It's really going to vary by configuration, so there's not necessarily a one-size-fits-all answer. You might want to check with your ISP - it's probably them or their upstream that you'd need to worry about.
Since you're sending a newsletter, could you add multiple recipients via BCC rather than individual messages? That should be less "abusive" to all concerned.
I've implementing sending to max 600 e-mails per recipient domain. That seems to be working fine for some time now and sounds like an OK solution.
Still, some SysAdmin insight on this would be appreciated.
Related
For a newsletter mailing, about 50,000 users, using pear, is it convenient to order the list by mail provider or leave it all randomly?
From my experiences using Exim to send large amounts of emails, performance will suffer heavily if your email queue grows too large. Depending on your hardware, once you have around 10,000 emails in the queue, you will start to see significant effects of bogosorting, where the server uses more CPU just juggling the queue than actually getting any useful work done.
One way of avoiding large queues, is of course to get the emails delivered as fast and efficient as possible. One of the many ways of achieving this, is to get Exim to deliver multiple emails over the same TCP connection. This in turn can be achieved by sorting the recipients by domain, but that is not enough! By default, Exim will try to deliver each mail it receives immediately and then each delivery will open its own connection (this gives fast deliveries for very small volumes but will drive server load through the roof for larger volumes). You need to first spool the mails to Exim, and then let a queue runner handle the actual delivery which will then automatically see all other emails in the queue that should go to the same host and will deliver them over the same connection.
Optimizing Exim for sending large amounts of emails is a very complex subject that cannot be solved with just a few magical tricks. Crucial configuration options are (but not limited to): queue_only, queue_run_max, deliver_queue_load_max, remote_max_parallel, split_spool_directory, but also fast spool disk, enough RAM, and making sure Exim starts new queue runners often enough (command-line option when starting the Exim daemon).
How this relates to PEAR escapes me, but perhaps this gives you some ideas of how to approach your problem.
I am currently developing an email server in C, and the end goal is to be able to send millions of emails to millions of people every day. Many organizations have email lists with large numbers of users that they email every week/month/etc.
The big question: how can I prevent the server and the emails from being marked as a spam? All of the SPAM-prevention stuff I've seen so far deals mostly with poor configurations, or at least does not require large numbers of emails to be send every hour. I have yet to see anything that addresses the scope of millions-of-emails-per-hour.
Here are some assumptions you can make:
EVERY single email sent is legitimate
all SPF records and MX records are accurate, up-to-date, and valid
all other common SPAM-prevention tactics are being used (reverse DNS is good, DKIM is used, return-addresses are valid, etc etc etc)
emails are one-to-one (ie, I'm not CC'ing 1000 gmail addresses; I'm sending one email to each address)
Here are some questions to get us moving in the right direction:
should I limit the number of emails sent to X emails per minute per domain? If so, how do sites like GMail and MailChimp get around this? note: there are no ISP restrictions; this is only an issue for the receiving mail server...
should I limit the number of connections to a domain at a given time? (eg, will Google think I'm a spam agent if I open 10/100/1000 simultaneous connections to gmail servers?)
how many bounce-backs (5xx errors on an address) should I accept for automatically removing that email from a subscription list? does this affect a server's spam rating?
is there anything else I should or should not do?
Final note: please remember this is a programming question, NOT a library question - I don't want to use someone else's service; we are writing our own for a reason. I'm looking for practical programming advice.
This is not a programming question, but here goes:
I strongly recommend you join your local mail operators mailing list, as well as "Spam-L" mailing list. Read the archives, and see what issues others are having.
The short answer is that destination servers can, and do, use all sorts of methods to try to prevent spam. THere are many things you will need to be aware of in order to have good deliverability, and those things change all the time.
First and most important, remember:
Free speech also includes free listening.
Nobody has to accept or transmit your mail.
Independent operators, businesses and individuals have a perfect right to refuse your mail for any reason or no reason. ISPs are limited only by their contracts with the customer and common-carrier laws, which generally give them broad discretion in what is considered spam and how they block it.
Their system, their rules. If you want your messages delivered, you must cooperate with receiving ISPs. This may mean jumping through hoops, or complying with requirements you think are stupid, or pointless.
Ensure you are not listed by SpamHaus. Most ISPs small and large use SpamHaus DNSBL service. Presence on one of SpamHaus' lists asserts their opinion that your mail meets their listing criteria. Because of SpamHaus' high reputation, most ISPs will simply block all mail you send based on their opinion.
Make sure you process unsubscribes.
Make sure you process non-delivery reports. You may not want to kill a subscription on the first NDR, as there can be intermittent network or server problems which can result in non-delivery, or even erroneous reports that an address is incorrect. But if you get several over the course of a month or two with no successful deliveries, you should kill the subscription.
Join a pay-for reputation service. These may require posting a bond which you may lose if you send Spam. SpamHaus offer one. There are others.
Get professional advice from someone like Return-Path. You will have to pay for this also.
Monitor. The hoops you have to jump through change all the time. Ensure you are aware of emerging deliverability problems.
Join feedback loops. most large ISPs offer feedback programmes where you can get feedback on how users are perceiving your mail, whether they are reporting it as spam, etc.
Ben had some good practical advice, but for others with this problem, here is what I have discovered in the past month:
Email is all about REPUTATION. You will never be able to throw together a server, ip, and/or domain name and expect to be able to send out millions upon millions of emails.
On Stack Overflow, we have a rating system (up and downvotes) to estimate the value/trust that person has with the SO community. But it takes time and effort to get points. It's the same with email - you have to start sending out small amounts of email that people actually open up and read (and would never mark as spam), and then slowly send out more and more every month until you reach the goal of millions and millions of emails.
Everytime someone "downvotes" - marks the email as spam, flags the domain, flags the ip address, deletes the email without reading it, etc - you get a hit against your reputation. You need to be continually monitoring and putting effort and best-practices into your reputation if you want to gain good standing with people.
So start small, expand in a stable and steady manner, and always keep a watchful eye out for abuse, misuses, good and bad feedback, or anything else that might affect your reputation.
It's not only possible, but very practical; you just need to give it time and effort.
I am looking for a SMTP server that we can use for outgoing mail that permits a high volume (1,000+ per hour) without getting blocked. It could be downloaded on our machine or run online, but cannot cost over $25. Isn't there a good open source solution? I haven't been able to find one yet. Thanks.
Well, there are even free SMTP servers that can send a high volume of e-mail.
The problem is not in sending them, the problem is in delivering. You see, SMTP is a best-effort store-and-forward protocol - the e-mails are passed from one SMTP server to another in hope that it gets closer to the adressee's mail server (and usually it works). However, this also means that your message can be rejected or dropped anywhere in this chain of servers - or by the recipient. Once the e-mail leaves your network, there's not much you can do about it. In other words, it's not your server blocking the e-mails, but the servers it will be passing the mails to.
This question may be a useful reference for your problem; see also this article for some tips that will help your e-mail get delivered.
For that ammount of email, probably any server will do just fine. The problem is not to be blacklisted as a spammer. Jeff Atwood has a good blog post on the subject.
I personnaly would recommend Postfix or qmail, but that just me ...
What are the risks, if any, of sending out massive amounts of emails over SMTP? Specifically, this question is regarding the risks of being labelled/blacklisted as spammers of spoofers.
Our mails are legitimate, however. Our system needs to send out reminders to our corporate users on a daily basis, which may number into the thousands, say. Our worry is that with such a setup, our domain might end up being blacklisted by the receiving organisation, thus rendering our reminder service useless.
Does anyone have any information on what might be a "safe" volume of emails to send out to avoid being blacklisted? Or can we just churn out emails with abandon?
You may be able to contract a third-party organization to take care of this for you. I know there's a lot of "direct marketing" companies that will let you use their API to send mass email (newsletters, etc). They can do the work of negotiating to get off blacklists - that's what you pay them for.
I haven't used Sendloop and don't know if it has the functionality you want, but it's probably a good example.
See: How to conduct legitimate email campaigns
In your reminder service, just follow some basic spam guidelines. Identify where the message came from, why the user got it, the link to "opt-out" or discontinue the reminders, and you'll be fine. Any blacklists you do get on will certainly remove you if you have this information in your messages.
Additionally, should you get blacklisted for some reason, have another server on a different network that you can use as a backup should your primary server get blacklisted temporarily for any reason.
Oh, and one final note - usually your entire "domain" (i.e. whatever.com) doesn't get blacklisted. Specific IP addresses or specific servers are usually what get blacklisted.
As long as you're mailing over clean IPs and domains you should be fine. You say your mailings are "legitimate" so there's no reason to worry about ISPs blocking you.
However, as you also mentioned, the volume can become a challenge. Broadly speaking, sending "thousands" of messages should be a non-issue. But... hundreds of thousands, say 250K messages a day on up, is when you start to qualify as a "high-volume" sender.
Once you start sending at this bulk level, you must run a tight ship. ISP filters will look for any clue that you're a black-hat mailer/spammer and will promptly block your deployment if anything looks off.
Make sure your list(s) are spic-and-span; all bounces, duplicates, typos and honey-pots have been scrubbed-out. Your IPs have been properly warmed-up, your DNS and domains are clean and properly registered and you remain responsive to your list recipients.
Basic common sense and following through on all the tiny, simple but crucial details goes a long way.
We run a large online community in the Netherlands. Because of that we send a lot of mail to the hotmail email addresses of our members.
Recently we have noticed that not all mail is reaching our members, because we have hit a certain limit or so it seems.
Google doesn't give a solution (yet) but we see a lot of others having the same problem.
Note, We have added (since long) SPF records for our domain, in TXT and SPF types.
What else can we do to tackle this problem?
// Ryan
To add to what bzlm said, hotmail probably isn't rejecting your mails, so much as trying to use rate-limiting to prevent spam. That said, there are a few potential solutions you could use here. You could contact hotmail and see about getting your mailserver exemption status from their rate-limiting. Depending on the size of your community, they may or may not respond to you or be willing to work with you. I suggest this only as the "diplomatic" solution.
Or, you could set up two mail servers; one for hotmail users, and one for everybody else. I know from some of the sites that I run that a lot of people register with hotmail accounts, mostly because everybody has one which they use as a "spam dump" for online services where they don't want to use their real email address. So, as you no doubt realize, the number of hotmail users in your database represents a fairly substantial percentage. Therefore, when you need to send an email, you could determine whether to send it to your normal SMTP server, or your hotmail-designated one. On the hotmail SMTP service, you'd need to add some type of waiting mechanism to sleep a certain amount of time after receiving a 421 response.
The problem with this idea is that the number of hotmail users you have, plus the delay you'll encounter in sending, means that the queue length might very well exceed the number of mails you must send. You could alleviate this problem by setting up secondary/tertiary servers, preferably on other networks... but I'm getting ahead of myself here. At any rate, I did a bit of googling around (as you probably have, too), and this isn't such an uncommon problem, but there is no obvious solution to it.
So likely, you'll either have to create some type of slightly-unorthodox network workaround, or try the "diplomatic" route and contact an organization unlikely to care about your problem. I'd suggest doing both in parallel. :)
421 means that the service is not currently available, and that the client should try again. This could be for any number of reasons, including trying to discourage you from too frequent mailing if Hotmail thinks you might be a spambot.
Why not simply let your outbound smtpd queue the mails and try again?
Why is "not all mail reaching your members"? Don't you try again if you get a 421 response?
EDIT: Do what sqook says.
The only real way to "get around" this is to become a good e-mail citizen. Make it easy for people to unsubscribe from your notifications, establish complaint feedback loops with the major mail providers, remove bouncing e-mails from your list automatically, don't send people e-mails they don't want to receive. Failing to adhere to these simple requirements makes you look like a spammer, and providers like Hotmail will treat you like one.
The mail server IP connecting to Outlook.com server has exceeded the rate limit allowed. Reason for rate limitation is related to IP/domain reputation. If you are not an email/network admin please contact your Email/Internet Service Provider for help.
https://mail.live.com/mail/troubleshooting.aspx
I advise you too wait some times