Trying to understand more about Rails and e-mail before beginning to set it all up. E-mail news notifications are already a big part of what we do and automating and improving that bit (currently MailChimp) is part of the reason for rewriting it all in Rails. From an editorial and business perspective, we want news e-mails going out to subscribers as soon as the story is published, especially for breaking stories, so as the service grows, we're looking at thousands of e-mails a day.
Are things like Sendgrid and Mandrill a substitute for trying to roll your own with Rails and Sidekiq/Resque/Delayed Job, or is it necessary to do something like Sidekiq + Sendgrid?
Assuming we use deliver_now + Sendgrid, does anyone know how long it takes before e-mails start hitting inboxes? Are we talking a couple of minutes or half an hour or an hour?
With any of the services you mention, you'll still want to use a worker queue on the client side to make sure that if you have any problems you can easily check status and retry.
I can't speak for the other services but with SendGrid you'll be looking at delivery times in the range of seconds rather than minutes.
Related
Assuming I send 10k emails daily, does it make sense to switch to an external provider such as Sendgrid / Postmark / Amazon SES? Why?
Is it because of my customers having a better chance of receiving the emails (ie. their servers have better reputation, they know better how to deal with SPF / DKIM / etc.)?
Is it because I don't need to worry about the configuration / administration of the SMTP server?
Is it because of bandwidth or server load?
Is it because of better dealing with unsubscriptions / bounces / etc?
Is it all of the above? :)
I'm asking because I have a system that works. It's not 100% reliable (still some mails getting lost on the way) and I'm sure it could be improved in other ways too - but I want to know what benefits I could expect from switching to an external provider.
PS: I'm definitely not a spammer. I'm sending some information people have asked for. :)
All of the above!
Do you want to spend your time
fussing over your SMTP server?
Do you want to always ensure that your SPF records and DKIM is correct?
Do you want to spend a lot of time building your own methods of checking for delivers, - bounces, opens, and clicks?
Do you want to spend a lot of time checking on the reputation of your IP address and making sure that it is whitelisted?
Do you want to spend a lot of time flattening your HTML and CSS for your templates?
If you answered yes to any of these, then maybe not. Otherwise, the nominal fee you would pay for an external service is well worth having a lot more time to focus on your product rather than having to spend even a minute on your emails.
I'm the Product Manager for PostageApp, an outbound transactional email provider that makes it super easy to design, send, and manage those emails, so if you have any questions about our industry or our product specifically, feel free to ask!
I'd say all things you mentioned are correct. But if you're trying to justify financially...
Postmark = $15/10K emails
So you would spend $450/mo. I think if you spend more than 5 hours a month it is a no-brainer to offload it so your time can be spent on creating value in your business instead of admin.
If finance is not a concern, then the one thing you're probably missing is the level of expertise that a company like sendgrid and postmark have. I like to think if there are whole businesses dedicated to something, I shouldn't think I can do as good of a job of it by spending a few hours here and there. I think email delivery is extremely nuanced and takes a lot of maintenance and babysitting.
In a couple of recent projects, I've written an e-mail queue as a database table that is checked every minute by a cronjob. The cron script waits a few seconds in between sends.
The reason I did this was because I read somewhere that it helps your e-mail not end up in the spam folder if it's not blasted out from the same server all at once. I don't remember where I read it.
What do you think? Should I just mail things immediately or is an e-mail queue a good idea?
What are some pros and cons to this approach?
EDIT: Added last paragraph.
Do you have control over your SMTP server? If so, then there's no point in doing an email queue PHP-side. The SMTP server will already have queueing capabilities. If you're worried about blasting a particular receiving server, you can always turn on your SMTP server's throttling (if it has it), something along these lines.
We run a large online community in the Netherlands. Because of that we send a lot of mail to the hotmail email addresses of our members.
Recently we have noticed that not all mail is reaching our members, because we have hit a certain limit or so it seems.
Google doesn't give a solution (yet) but we see a lot of others having the same problem.
Note, We have added (since long) SPF records for our domain, in TXT and SPF types.
What else can we do to tackle this problem?
// Ryan
To add to what bzlm said, hotmail probably isn't rejecting your mails, so much as trying to use rate-limiting to prevent spam. That said, there are a few potential solutions you could use here. You could contact hotmail and see about getting your mailserver exemption status from their rate-limiting. Depending on the size of your community, they may or may not respond to you or be willing to work with you. I suggest this only as the "diplomatic" solution.
Or, you could set up two mail servers; one for hotmail users, and one for everybody else. I know from some of the sites that I run that a lot of people register with hotmail accounts, mostly because everybody has one which they use as a "spam dump" for online services where they don't want to use their real email address. So, as you no doubt realize, the number of hotmail users in your database represents a fairly substantial percentage. Therefore, when you need to send an email, you could determine whether to send it to your normal SMTP server, or your hotmail-designated one. On the hotmail SMTP service, you'd need to add some type of waiting mechanism to sleep a certain amount of time after receiving a 421 response.
The problem with this idea is that the number of hotmail users you have, plus the delay you'll encounter in sending, means that the queue length might very well exceed the number of mails you must send. You could alleviate this problem by setting up secondary/tertiary servers, preferably on other networks... but I'm getting ahead of myself here. At any rate, I did a bit of googling around (as you probably have, too), and this isn't such an uncommon problem, but there is no obvious solution to it.
So likely, you'll either have to create some type of slightly-unorthodox network workaround, or try the "diplomatic" route and contact an organization unlikely to care about your problem. I'd suggest doing both in parallel. :)
421 means that the service is not currently available, and that the client should try again. This could be for any number of reasons, including trying to discourage you from too frequent mailing if Hotmail thinks you might be a spambot.
Why not simply let your outbound smtpd queue the mails and try again?
Why is "not all mail reaching your members"? Don't you try again if you get a 421 response?
EDIT: Do what sqook says.
The only real way to "get around" this is to become a good e-mail citizen. Make it easy for people to unsubscribe from your notifications, establish complaint feedback loops with the major mail providers, remove bouncing e-mails from your list automatically, don't send people e-mails they don't want to receive. Failing to adhere to these simple requirements makes you look like a spammer, and providers like Hotmail will treat you like one.
The mail server IP connecting to Outlook.com server has exceeded the rate limit allowed. Reason for rate limitation is related to IP/domain reputation. If you are not an email/network admin please contact your Email/Internet Service Provider for help.
https://mail.live.com/mail/troubleshooting.aspx
I advise you too wait some times
I have to make a newsletter sending utility application which will collect the list of subscriber from our central database and send out the newsletter. I've considered the possibility to be blacklisted due to flooding if I just flush out all emails at once, so I decided to go on a desktop-based softwer which will email those slowly.
my question is
what is the max emails per hour that may be addressed to the same email domain (recipient/incoming server)?
or what should be delay between 2 e-mails to the same server for it doesn't consider it flooding?
whichever of the above applies more appropriate to the real-world of mail servers configuration...
thanks
I make Thread.Sleep(2000) after every 2 mail
It's really going to vary by configuration, so there's not necessarily a one-size-fits-all answer. You might want to check with your ISP - it's probably them or their upstream that you'd need to worry about.
Since you're sending a newsletter, could you add multiple recipients via BCC rather than individual messages? That should be less "abusive" to all concerned.
I've implementing sending to max 600 e-mails per recipient domain. That seems to be working fine for some time now and sounds like an OK solution.
Still, some SysAdmin insight on this would be appreciated.
I'm implementing a email newsletter sender service using .NET and Windows Server technologies. Are there comprehensive guidelines which could help avoiding emails being trapped by spam filters and other mechanisms?
They should cover all aspects of (legal) bulk mail sending: SMTP configuration, DNS, HTML content, images, links within content etc. A simple example: is it better to embed images or load them from a server?
It would be great if you could provide some empirical data to show the efficiency of some measures taken.
Although I don't have a definitive answer, I think this is a very important question.
Here are few tidbits I know about it
Choose a clean hosting/smtp server. IP addresses of spamming SMTP servers are often black-listed by other ISPs.
Send a simple introductory email to every subscriber, asking them to add your sender address to their safe list.
Be very prudent in sending to only those people who are actually expecting it. You wouldn't want pattern recognizers of spam filters learning the smell of your content.
If you don't know your smtp servers in advance, its a good practice to provide configuration options in your application for controlling batch sizes and delay between batches. Some servers don't like large batches or continuous activity.
Unless you have a very specific reason to host the newsletter yourself, I think you'd be much better off using a third party service. There are lots out there, and some are very cheaply priced.
It'll save you on development work
(no point in re-inventing the
wheel).
Their system will handle all
the unsubscribe link stuff that you
need to include in email newsletters
to comply with CAN SPAM laws or
whatever.
They handle the spam
reports that you will inevitably get
if you have a list of any non-trivial size.
They keep records of who signed up,
how they signed up, and their IP
address, and can present those on
receipt of a spam report to prove
that their service wasn't sending
out spam.
You can use double-opt in
(or confirmed opt in), for extra
evidence to prove that the people
you're sending emails to actually
signed up to receive them.
If you really do need to host it yourself I'd suggest you search the web for "email deliverability". Things that are known to help include properly set up SPF records, DomainKeys/DKIM, correct DNS settings (reverse DNS especially - best to just use an online service to check your DNS settings). You can test a lot of these things by sending an email to check-auth#verifier.port25.com.
It's best to avoid using spammy words in your email - always a bit of guesswork this but you some words can trip filters.
But I'd guess that by far the most important thing is to be sending your email from a trusted server that maintains good relationships with ISPs (i.e. ensuring that ISPs don't think that the server is sending out spam). This is a big reason why it's much much easier to get a third party to handle everything for you.