SMS and Email Queues from Database - email

Question

I just wanted to discuss a situation I am facing.
I want to send eMails to the users - a lot of eMails - but if I send them at application run time the AWS SDK is slow for emails - bad user experience - atleast for my application.
So what I plan to do is enter the data (email address, content to send, 0) in the database and launch a cron job to read the table and start sending the emails - once it sends the email - it marks the database row as 1.
I read somewhere that is a wrong practice and puts overload on the database server.
Yes, I would use intelligent crons so that no 2 crons overlap or setup a cron each for even and odd numbers etc. I am also looking at 3rd Party alternatives likes http://www.iron.io/ for crons.
Could someone share their experience with a similar situation etc. I just want to use the intelligent solution and not just put a ton of resources on the database and spend hefty on transactions...

Answer 1

I had to do something similar and did as Charles Engelke suggested - I used SQS.
I eliminated the database entirely by putting the entire message contents in the SQS message. You're limited to 64k in an SQS message, so as long as thats not a problem this approach is possible.
Here is sample code to queue up the message:
package com.softwareconfidence.bsp.sending;
import com.amazonaws.services.sqs.AmazonSQS;
import com.amazonaws.services.sqs.model.SendMessageRequest;
import com.googlecode.funclate.json.Json;
import java.util.HashMap;
import java.util.Map;
public class EmailQueuer {
private final AmazonSQS sqs;
private final String sendQueueUrl;
public EmailQueuer(AmazonSQS sqs,String sendQueueUrl) {
this.sqs = sqs;
this.sendQueueUrl = sendQueueUrl;
}
public void queue() {
Map<String,String> emailModel = new HashMap<String, String>(){{
put("from","me#me.com");
put("to","you#you.com");
put("cc","her#them.com");
put("subject","Greetings");
put("body","Hello World");
}};
sqs.sendMessage(new SendMessageRequest(sendQueueUrl, Json.toJson(emailModel)));
}
}
Then in your app you need to have an executor service that polls the queue and processes messages:
new ScheduledThreadPoolExecutor(1).scheduleAtFixedRate(sendEmails(), 0, 1, MINUTES)
You will need to make sure to call shutdown() on this executor when it app is exiting. Anyway, this line will send emails every minute, where sendEmails() returns an instance of this Runnable class:
package com.softwareconfidence.bsp.standalone.sending;
import com.amazonaws.services.simpleemail.AmazonSimpleEmailService;
import com.amazonaws.services.simpleemail.model.*;
import com.amazonaws.services.sqs.AmazonSQS;
import com.amazonaws.services.sqs.model.DeleteMessageRequest;
import com.amazonaws.services.sqs.model.ReceiveMessageRequest;
import com.amazonaws.services.sqs.model.ReceiveMessageResult;
import com.amazonaws.services.sqs.model.SendMessageRequest;
import com.googlecode.funclate.json.Json;
import java.io.PrintWriter;
import java.io.StringWriter;
import java.util.List;
import java.util.Map;
public class FromSqsEmailer implements Runnable {
private final AmazonSQS sqs;
private final String sendQueueUrl;
private final String deadLetterQueueUrl;
private final AmazonSimpleEmailService emailService;
public FromSqsEmailer(AmazonSimpleEmailService emailService, String deadLetterQueueUrl, String sendQueueUrl, AmazonSQS sqs) {
this.emailService = emailService;
this.deadLetterQueueUrl = deadLetterQueueUrl;
this.sendQueueUrl = sendQueueUrl;
this.sqs = sqs;
}
public void run() {
int batchSize = 10;
int numberHandled;
do {
ReceiveMessageResult receiveMessageResult =
sqs.receiveMessage(new ReceiveMessageRequest(sendQueueUrl).withMaxNumberOfMessages(batchSize));
final List<com.amazonaws.services.sqs.model.Message> toSend = receiveMessageResult.getMessages();
for (com.amazonaws.services.sqs.model.Message message : toSend) {
SendEmailResult sendResult = sendMyEmail(Json.parse(message.getBody()));
if(sendResult != null) {
sqs.deleteMessage(new DeleteMessageRequest(sendQueueUrl, message.getReceiptHandle()));
}
}
numberHandled = toSend.size();
} while (numberHandled > 0);
}
private SendEmailResult sendMyEmail(Map<String, Object> emailModel) {
Destination to = new Destination()
.withToAddresses(get("to", emailModel))
.withCcAddresses(get("cc", emailModel));
try {
return emailService.sendEmail(new SendEmailRequest(get("from", emailModel), to, body(emailModel)));
} catch (Exception e){
StringWriter stackTrace = new StringWriter();
e.printStackTrace(new PrintWriter(stackTrace));
sqs.sendMessage(new SendMessageRequest(deadLetterQueueUrl, "while sending email " + stackTrace));
}
return null;
}
private String get(String propertyName, Map<String, Object> emailModel) {
return emailModel.get(propertyName).toString();
}
private Message body(Map<String, Object> emailModel) {
Message message = new Message().withSubject(new Content(get("subject", emailModel)));
Body body = new Body().withText(new Content(get("body", emailModel)));
message.setBody(body);
return message;
}
}
One downsize of this approach if you're using a database is that the email sending step is a HTTP call. If you have a database transaction that rollsback after this HTTP call, your business process is undone, but the email is going to be sent.
Food for thought.

Answer 2

Thanks for the detailed response Mike. I finally ended up in implementing a REST API for my application with secure Username+Password+Key access and run it from a 3rd Party Service Iron.io which gets
www.example.com/rest/messages/format/json
It iterates and sends the messages collecting status in an array - which it then posts back to
www.example.com/rest/messagesposted
I followed this approach because I had to schedule messages for over
90 days interval and queues hold messages only for say like 14 days.
What do you recon?