I'm working on an application that will allow management to send registered users (opt-in) broadcast emails at regular intervals, or based on various other criteria. In any case, I'm curious as to whether I should send a separate email to each recipient or bcc all of them on a single message. Currently the email list would be about 1500 recipients, but it should scale all the way up to at least 25k without problems.
Thoughts? Am I getting into a range that I need to worry about being put on spam lists?
Yes, I've had spam list problem with mailing lists of that size, managing email lists for non-profits.
One wants to take extra precautions: make sure your email has SPF records, write a script to send the emails in batches, paced out over time. Definitely send them one one at a time, not as bcc, as direct mail has a better chance of arriving. Make it very easy to unsubscribe. Include people's subscribed email in the message sent -- often people have email forwarded to another account and then try to unsubscribe that account and get frustrated.
Even so, don't be surprised if you have to change your IP at some point.
You are getting into that range. This is the point where I would look to get a third party to send the email on my behalf. Let them worry about being marked as spammers, supply the bandwidth, etc.
I recently built an application with those same criteria. We do the emailing in-house, and send one email to each recipient.
Do use domain keys signing or be sure to use SPF records for your domain. We didn't do that at first, and were blacklisted by a number of different ISPs. Fortunately, it is fairly easy to get them to unblock you. Most will include an online form you can fill out or an email address you can use in the server bounce message.
Don't try to implement the actual email sending yourself. That's a huge waste of time. Either outsource the entire process to one of the many reputable vendors out there (Many organizations I deal with use Constant Contact, and it works well), or run a garden-variety mailing list server (e.g. Mailman) in-house.
Either way, take efforts to make it very easy to unsubscribe (good vendors have that covered), to authenticate that messages are from your company, and to show that your company is not spamming. Real mailing list server software supports all of these goals, by adding proper headers that identify the source very clearly and making unsubscription easy. For instance, Gmail will now offer to send unsubscribe requests in response to mailing list messages marked as 'spam', as has AOL for a long time.
Definitely set up SPF and DKIM if you can manage it.
Finally, whatever you do, make sure you keep logs of your subscriptions, so that if someone does accuse you of spamming, you can defend yourself.
The task is mostly uninteresting on a strictly technical level. You should worry about what happens when a recipient thinks that your list's content is spam and starts (a) complaining or (b) flagging the message as spam with one or more anti-spam service providers. Something like this is bound to happen with a list of the size you describe.
If you are prepared and have the time handle such cases, go for it, at least for a start. (Changing your mail server's IP address as Devin Ceartas suggests won't be of much use by the way.)
If you want to build your own thing, I have two pieces of advice:
Unsubscribing has to be easy, no more than one or two clicks. Using Mailman or any other mailing list manager that was intended for discussion mailing lists is asking for trouble.
BCCing the same message to 1500 (or 25k) recipients may take some load off your mail server, but it has one serious disadvantage: You won't be able to use VERP in order to determine if all addresses that have once been subscribed to your list are still valid. (Large mail providers tend to classify messages as spam if there are delivery attempts to many invalid addresses.)
Related
I am build a tool that initiates an SMTP transaction with a domain to see if (a) that domain can receive emails and (b) the desired address exists on that domain. I will be batching large groups of email addresses (10,000+ at a time), but I don't want to bombard the server and get blacklisted. Are there guidelines for how often is it safe to communicate with an SMTP server?
I know about the VRFY command, but it is not implemented across the board. I plan to attempt to use the VRFY command and fall back to using,
MAIL From:<user#example.com>
RCPT To:<first.last#example.org>
QUIT
to see if the message will be deliverable. Again, are there guidelines on how often I can initiate an SMTP transaction like this on a domain?
Edit:
The purpose of this is to create a tool that my organization can use to (a) clean some bad emails from several largely inactive lists so that we do not have to pay our email delivery system to send potentially thousands of emails that will bounce, and (b) check an email when a user subscribes to a list so that we reject emails like aoghuifdgsiuvb#gmail.com.
First of all, spamming is bad. Always ask user wheter she wants to receive newsletters.
"Unsort" mail addresses by domain, leaving the "distance" between e-mail addresses with same domain as big possible.
I think it's not the programmer's decision. There should be a config value which tells a minimum amount of time between two mail sending to the same domain. You should set up a limit also for that config value, avoid setting it to zero or low value.
The only universal guideline I believe can be offered is "don't do this". If you behave like a spammer, you will be treated like a spammer. In the optimistic scenario, sites will already have controls in place, and silently throttle or block you. In less ideal scenarios, they will initiate actions against you on the (reasonable) assumption that you are collecting addresses for a spam list.
A better soluton would be to actually follow through the whole SMTP session, sending a user an email with a verification code/link. This has the advantage of showing that the user actually has control of the address in question and it keeps you from looking like a spam bot.
Volume is not as much the issue as reputation. Let the user know you're about to send them an email in your web flow. This means they're much less likely to mark it as spam.
some hosts have clear and defined guidelines as to how many emails can be sent per hour.
So i guess this would depned onyour hostng service provider, UNLESS your hosting your own mail server off course.
Realize there a lot of bulk email services, but what I'm trying to do doesn't feel like bulk email; though maybe I'm just thinking about it the wrong way. Case in point, I attempted to create an account on MailChimp and it wants a subscriber list.
I'm send one, and only one email per address; for example, replying to job postings on Craigslist. I'm not sending follow-up emails, and basically the first email I send is the last to an address unless I get a reply.
Sometimes the email I send is based on a template, other times they've asked for something, and I customize the email. I've automated creating resumes and sending emails in the past, but currently the fastest way just seems to manually do it; meaning I can't imagine sending more than 500 emails in a single day.
As far as I'm able to tell if Gmail, Yahoo, Hotmail, Lycos, AOL, etc. - do not offer the service of paying per email over the send limit, since I'd be willing to pay for the service and be done with this, as it's taking up a huge amount of time and overhead to deal with.
Currently, although I've seen sites giving higher numbers, most of the providers appear to limit emails sent to 20-40 a day; possible I'm doing something wrong, but if I am, I don't understand why these limits are not easy to see within an account, so I'm either able to stay within the limits, or see the service is not a good fit for my needs; currently I just get random error-messages, locked-out, etc.
(If needed, and the setup doesn't take more an a few hours, I'm more than willing to deal with configuring scripts, CSV files, templates, etc. on my computer - as long as they're well documented and the end result addresses the issue; meaning that if I have to send a one-off email, I just copy the template, make the changes, and point to the email. The system must also support attachments, since every email has at least one attachment; best solution would be if I'm able to load an attachment once to the system, then alias it in the local script.)
Use the mail account your internet service provider provides. It is unlikely that you hit their limits, and even if that happens, you, as a paying customer, can easily negotiate a higher limit.
if I wanted to create an "email list" along the lines of google groups (and eventually integrate with a web site), what services should I look into? I haven't really dealt with email before so I'm not 100% sure how it all works. I do know I need to be careful about being blacklisted, etc. Do I use something like sendgrid/mailman? I imagine all emails go to my server when a user hits "reply all" and on my end I push that email as though it came from the user but really it came from my "mail" server? It seems a user doesn't need to "reply all" to send to the list and there is no quick way to reply to just one user (which is a good thing for this app).
So the way I imagine this working:
1) I use something like sendgrid to "listen" for emails coming to list#mywebsite.com
2) When emails come (i.e from userx) I do some filtering for spam, etc and then push them all out to "all users/subscribers"
3) All users get something that is "from user x" (but it's really from my server, on google groups it says "via googlegroups") and the "reply to" is list#mywebsite.com
I imagine this is how it all works but I'm not 100% sure this is correct or if sendgrid will allow me to do this (and if it will cost too much).
Thanks
It really depends on what your needs and skills are. For simple mailing lists there is some very established software such as ListServ or MajorDomo
If you're not running a Unix variant, or you're not confident setting up this kind of software they might not be suitable though.
You can use SendGrid to handle your incoming email, but you will have to re-assemble the message before sending it out since it will be posted to your incoming parse URL in a, well, parsed way. The good news is that SendGrid can handle the spam check for you.
SendGrid currently doesn't charge for inbound parsing, but of course that may change. So as far as your cost goes, you would only use credits for the mail you send. Keep in mind that SendGrid only provides the Parse API to silver ($80/mo) customers or customers on the free (200 emails/day) plan, but there are plans to add a lower tier package that also has access to all APIs.
You can also use SendGrid to handle unsubscribes, but if you are using another solution for list management that's probably superfluous.
Googling for "mailing list hosting" brings up a wealth of sites, some of them free at least under some conditions. Many of them will use something like Mailman / Majordomo / Listserv; certainly you can run Mailman or Majordomo on your own server if you like, as well, so your question doesn't really make sense from that point of view. Yahoo Groups / Google Groups can also happily host a list for you. Do you need the list to be visible to subscribers only? Google Groups and Yahoo Groups will host private lists, too.
A mailing list basically works like Bcc: mail. You send to the mailing list's submission address; it then replaces some of the headers (often the Sender: and perhaps Reply-To:) and then redistributes the message to all the subscribers. Behind the scenes, it might rewrite some headers to make sure bounce messages come back to the list software for further handling (for example, to automatically unsubscribe addresses which consistently return a bounce), and often, it distributes the message with some delay in order not to saturate the outgoing bandwidth and/or choke a receiving server; though some parts of this are also handled by the regular mail transport agent (MTA -- Postfix or Sendmail or what have you).
You might look at ezmlm
EZmlm-idx is a nice mailing list add-on to Qmail. When we install Qmailadmin later on, you'll see that EZmlm-idx integrates seamlessly into Qmailadmin to provide a very user friendly mailing list management interface. As an added bonus, Vpopmail (which we will install as well) will let you control what users can and cannot use mailing lists!
Quoted from qmailrocks
You do not need to worry about sendgrid. Just get a cheap vps,configure DNS and, install MTA(postfix,qmail,sendmail) imap(courier,dovecot), a modern listserve, and let it do it's magic! On the lists where I am subscribed , I have the option to reply to a post, or the whole group.
A number of applications have the handy feature of allowing users to respond to notification emails from the application. The responses are slurped back into the application.
For example, if you were building a customer support system the email would likely contain some token to link the response back to the correct service ticket.
What are some guidelines, hints and tips for implementing this type of system? What are some potential pitfalls to be aware of? Hopefully those who have implemented systems like this can share their wisdom.
Some guidelines and considerations:
The address question: The best thing to do is to use the "+" extension part of an email (myaddr**+custom**#gmail.com) address. This makes it easier to route, but most of all, easier to keep track of the address routing to your system. Other techniques might use a token in the subject
Spam: Do spam processing outside the app, and have the app filter based on a header.
Queuing failed messages: Don't, for the most part. The standard email behavior is to try for up to 3 days to deliver a message. For an application email server, all this does is create giant spool files of mail you'll most likely never process. Only queue messages if the failure reasons are out of your control (e.g., server is down).
Invalid message handling: There are a multiple of ways a message can be invalid. Some are limitations of the library (it can't parse the address, even though its an RFC valid one). Others are because of broken clients (e.g., omitting quotes around certain headers). Other's might be too large, or use an unknown encoding, be missing critical headers, have multiple values where there should only be one, violate some semantic specific to your application, etc, etc, etc. Basically, where ever the Java mail API could throw an exception is an error handling case you must determine how to appropriately handle.
Error responses: Not every error deserves a response. Some are generated because of spam, and you should avoid sending messages back to those addresses. Others are from automated systems (yourself, a vacation responder, another application mail system, etc), and if you reply, it'll send you another message, repeating the cycle.
Client-specific hacks: like above, each client has little differences that'll complicate your code. Keep this in mind anytime you traverse the structure of a message.
Senders, replies, and loops: Depending on your situation, you might receive mail from some of the following sources:
Real people, maybe from external sources
Mailing lists
Yourself, or one of your own recipient addresses
Other mail servers (bounces, failures, etc)
Entity in another system (my-ldap-group#company.com, system-monitor#localhost)
An automated system
An alias to one of the above
An alias to an alias
Now, your first instinct is probably "Only accept mail from correct sources!", but that'll cause you lots of headaches down the line because people will send the damnedest things to an application mail server. I find its better to accept everything and explicitly deny the exceptions.
Debugging: Save a copy of the headers of any message you receive. This will help out tremendously anytime you have a problem.
--Edit--
I bought the book, Building Scalable Web Sites, mentioned by rossfabricant. It -does- have a good email section. A couple of important points it has are about handling email from wireless carriers and authentication of emails.
You can set the address that the email is sent from, what will be put into the To: address if someone just presses 'Reply-to'. Make that unique, and you'll be able to tell where it came from, and to where it must be directed back to.
When it comes to putting a name beside it though '"something here" ' - put something inviting to have them just reply to the mail. I've seen one major web-app, with Email capturing that has 'do not reply', which turns people off from actually sending anything to it though.
Building Scalable Web sites has a nice section on handling email. It's written by a Flickr developer.
(source: lsl.com.au)
EDIT: I misunderstood your question.
You could configure your email server to catch-all, and generate a unique reply-to address. E.g. CST-2343434#example.com.
A polling process on the server could read the inbox and parse out the relevant part from the received email, CS-2343434 could mean Customer Support ticket ID no. 2343434.
I implemented something like this using JavaMail API.
Just a thought.
The best way to achieve this will be to write a window service that acts like a mail client [pop3 or imap]. This windows service should execute a timed action triggered by a timer, which connects to the mail server and polls the server for any unread message(s) available in the email inbox. The email ID to check for is the email ID on which the users will give their input on/to. If the windows service client finds that there exists any new mail(s) then it should download and filter the email body and push further for processing based on the user input in the email. You can host the input processing in the same windows service but it is not advisable to do so. The windows service can put the inputs in a special application directory or database from where your main appication can read the user inputs received in email and process them as needed.
You will be required to develop a high performance TCP/IP client for doing so. I advise you not to use the default .Net library due to performance issues, instead use one of the best availabel open source TCP/IP implementations for .Net like XF.Server from kodart. we have used this in our applications and achieved remarkably grear results.
Hope this helps..
Bose has a pretty great system where they embed a Queue and Ticket ID into the email itself.
My company has the traditional Case # on the subject line, but when CREATING a case, require a specific character string "New Case" "Tech Support Issue" on the subject line to get through the spam filters.
If the email doesn't match the create or update semantics, the autoresponder sends an email back to the recipient demonstrating how to properly send an email, or directs them to our forums or web support site.
It helps eliminate the spam issue, and yet is still accessible to a wide technical audience that is still heavily email dependent.
Spam is going to be a bit of a concern. However since you are initiating the conversation you can use the presence of your unique identifier (I prefer to use the subject line - "Trouble ticket: Unable to log into web...[artf123456]") to filter out spam. Be sure to check the filter on occasion since some folks mangle the subject when replying.
Email is a cesspool of bad standards and broken clients. You need to be prepared to accept almost anything as input. You will need to be very forgiving about what kinds of input are tolerated. Anything easy for you to program will likely be difficult for your users to use correctly. Consider the old mailing list programs that require you to issue commands in the subject line. Only hardcore nerds can use those effectively. And some of those trouble-ticket CRM things you mentioned have bizarre requirements, such as forcing the user to reply between two specific text markers in the text. That sort of thing is confusing to people.
You'll need to deal with email clients that send you formatted text instead of plain text. Some email clients still don't handle HTML properly (cough GMail) so your replies will also need to be designed appropriately. There are various ways in which photos might be "uploaded" via email as well, especially when mobile phones are involved. You will need to implement various hacks and heuristics to deal with these situations.
It's also entirely possible that you will get email that is valid but unusable by the email parsing library you are using. Whether or not this is important enough to roll your own will be a judgement call.
Finally, others have mentioned using specific email addresses to uniquely identify a "conversation". This is probably the easiest way to do this, as the content of the mail will often not survive a round trip to a client. Be prepared, however, to get mail to old IDs from old customers who, instead of opening a new ticket somehow, reply to an old ticket. Your application will probably need some way to push emails with an old ID into a new case, either manually or automatically. For a CRM system it's very likely that a user would reply to an old email even if you already sent him a new email with a new ID in it. As for whether you should use some.email.address+some.id#yourdomain.com or just some.id#yourdomain.com, I'd go with the latter because the plus-sign confuses some email clients. Make your IDs guids or something and have some way to validate them (such as a CRC or something) and you'll get less junk. Humans should never have to type in the GUIDs, just reply to them. The downside is spam filtering: a user's computer might view such email addresses as spam, and there wouldn't be an easy way to whitelist the addresses.
Which reminds me: sending email these days is full of pitfalls. There are many anti-spam technologies which make it extremely hard for you to send email to your customers. You will need to research all of these and you need to be careful, and do some testing, to ensure that you can reach the major email providers. A website like Campaign Monitor
can help you if you are sending email.
Our clients sometimes don't get the emails that we send out. It's a BIG loss. How do I assure that they receive the emails so that if it's not received in the other end, the program can resend it or do something about it.
None of the suggestions above will work 100% of the time. Many email clients will (rightly so) refuse to load foreign images, negating the usefulness of "web bugs". They will also refuse (or be unable to) return Outlook-style "receipts". And many mail servers either deliberately (to curb spam) or mistakenly (due to misconfiguration) won't return bounce messages. Or possibly an over-aggressive spam filter ate your message, so it arrived but was never seen by the end user. Plus there is the little matter of mail taking hours or days to reach the end user or bounce, and how do you correlate these late notifications or bounces with the mail you sent 4 days ago?
So basically, you can catch some but not all, no matter what you do. I'd say that any design that relies on being able to know with certainty whether the end user got your mail is fatally flawed.
One thing that you can do is set up a bounceback address that receives any mail that is undeliverable. Use the bounceback address as the From address -- you may want a different one for Reply-To so that replies get directed properly.
Check the bounceback mailbox daily and contact customers to get updated email addresses for the ones that fail. You may be able to automate a couple of retries to failed addresses before resorting to the manual contact in case the failure is only intermittent.
This would take some code outside your application that scans the mailbox and keeps some state information about the number of contacts, etc. and attempts the resend.
Depending on how you generate the mails, you might be able to make this process easier: generate a unique bounce address for every single email you send out. You could use bounces+1234#example.com, for example.
Many SMTP servers will allow you to use the part after the + as a parameter to an external script, etc.
The problem is that many (broken) SMTP servers don't return enough info with a bounce to identify the original message -- sometimes, when there are forwardings involved, you don't even get back the original addressee...
With the above trick you can reliably correlate outgoing messages with incoming bounces.
There is no standard way to know whether the email reached the destination. Many email clients support different types of receipts though. You can use any of those if you want.
There are some ways to know when the user actually read the email.
There are many techniques like adding an image to your email that is to be fetched from your web server. When the user reads the email, the request for the image comes to your server and you can capture the event.
The problem is that there is no way to know that the mail did not reach the destination.
I worked on a bulk email system in a previous life. Deliverability was one of our major issues. The most common cause of undelivered emails is a spam filter.
Here are the steps we took to ensure the highest delivery rates:
We used Return Path to test emails for that spam-like smell.
If you send a lot of emails, you need to make sure your SMTP server is not blacklisted.
Remind your users to add your FROM address to their "safe senders" list.
Use a system that collects bouncebacks and use them to scrub your mailing list. This will also help keep you off the blacklists.
If the emails are critical, consider sending them return-receipt-requested. This will not really guarantee anything, but it might give you some metrics on actual deliverability.
There's not really a good way to determine if the email actually arrives in their inbox, you can only confirm that you sent it. Attach a receipt that lets you know when they open it perhaps?
Microsoft Outlook provides similar functionality, however it is based on the email client. I'm not sure if other clients, like Thunderbird, support this.
However, there is nothing in the protocols that specify receipts.
One option that may work: send a link to a generate web page and monitor that page for hits. This provides its own issues however: confidentiality, etc.