I've just read this excellent question about the advantages of sending an email asynchronously vs synchronously.
I fully agree with the approach that the single answer concludes, which is: store the email to send in a queue, and return the page to the user as soon as possible.
My current implementation uses a database table to store the email queue. This approach has several advantages:
It's very fast: saving a record to a table takes no time, as opposed to placing an API call (emails are sent through Amazon SES).
It's transactional: if anything goes wrong, the transaction is rolled back, and my email will never be sent for an account that wouldn't have been committed to the database.
Then I've implemented a job that runs permanently, and will check on a regular basis (currently 30s) if there are some mails to send, and process them.
I'm happy enough with this solution, apart from the fact that it's not exactly instant. People often expect to receive a mail (even more if it's a SMS) in the next few seconds.
I could obviously make the job pause no more than one second between two runs, but I don't want to waste server resources needlessly with frequent transactions. Also, because my emails are sent through an API call, I'd benefit from concurrent processing, which my current implementation does not do (it processes them sequentially).
Are there some common solutions to still process the email asynchronously, but start processing as soon as the email has been sent to the queue?
When you put an e-mail into the database, you also send a notification to your e-mail sending job to wake it up and send the new e-mail. Rather than waiting for the sending job to check, it is actively notified and deals with the new e-mail immediately.
Related
This question is not 100% technical. I've looked online and I couldn't find this being discussed.
We have a forgot username feature in our website which mails the username to the user's email address, using our own mail server.
We also put a message in the web-page asking the user to wait for 20 minutes for the mail to arrive since there can be occasional delays.
Our business tester raised an interesting point saying that 20 minutes seems an unacceptable time to ask the user to wait for. He said our technology should send emails immediately and the maximum lag should be 1 minute at worst.
Firstly, emails are usually received immediately by the recipient so there isn't a problem there.
But in our experience in using other websites, sometimes emails do take a while to arrive. I also remember reading somewhere that emails (at a network level) use a lower quality of service QoS unlike voip services. I can't seem to find it now.
Users can also experience delays in receiving emails because of issues in their own mail server.
Now, all we can do is send the mail using our mail server and ensure that the load and resources on the server is well managed.
1) Is there anything else we can do to ensure that our mails are sent quickly at all times.
2) What is the acceptable time, we can ask the user to wait until he logs a call with help-desk? I believe that there can be lags at a network/protocol level and the user's mail server which we can't do anything about.
Thanks.
All you can do is to inform user that your server "handed over" responsibility for email delivery to named SMTP server "beyond your control/responsibility". You may expect it would take a few (<5s) in most cases (>50%).
Your smtp-client may try initial delivery attempt.
On delivery success => inform the user that email delivery/delay is now beyond your control
On (initial) delivery failure => pass message to your SMTP server (with initial delivery attempt skipped).
Say I have an app that send an email to a user when something interesting in the system happens. The system would add a sendNotification job to a message queue, which when processed sends out the email.
I want to be able to add these jobs to the queue whenever something interesting happens, but have the system wait for 10 minutes to gather and group all these notifications into one email, rather than send out multiple email within that short timeframe.
How would this be implemented in a system such as RabbitMQ? As far as I can tell delay functionality isn't built in for a start, and you can't update messages once they are sent. So are there any patterns recommended for implementing this kind of thing?
With regards an OMS, what is the best method to send a confirmation email? The 2 options I have so far are;
A script on the order page sends an email once the record is written to the database.
A scheduled task on the server, send the email, polling the database every-so-often to find new entries.
Which method do systems currently use?
For e-commerce websites, it might be better to think about the best user experience.
Given that, you would want to send the email as soon as the order is received so the user knows that they have purchased the item. The sooner it gets into their inbox, the sooner they will be happy that they have made their purchase.
I agree with Digbyswift that sending the confirmation email once the record is written to the database is the least scalable. But I would argue that if your system has gotten to the point that you are taking so many orders that your system cannot keep up, you have a wonderful problem on your hands that you now probably have resources to handle.
At PostageApp, we handle the emails of a few e-commerce websites, so perhaps you would benefit from an arrangement with an email service provider to off-load this task so that all of your resources can be spent on keeping your site up and your databases running.
Here are some great alternatives if PostageApp is not your style:
Sendgrid
Postmark
Mailjet
This is a question of scalability. Sending a confirmation email once the record is written to the database is the least scalable. The more orders that are taken , the more emails are sent potentially tying up resources.
A scheduled task is certainly better as emails can be queued up and can be sent in a separate process.
A further option which you could consider is using neither and delegating the responsibility of sending emails to a 3rd party dedicated emailing service, i.e. via an API. This is much better since your hosting does not have to consider the load and you can utilise any reporting offered by the 3rd party. Plus many services offer a free quota up to a certain threshold. This will allow you OMS and business to scale appropriately.
If you apply a message based architecture; you could just publish an order created message and have any number of subscribers respond to that event. You could create a listener that sends the email in house (bespoke option) or another listener that called the API of a 3rd party emailer to send the email on your behalf (as per #Digbyswift)
What I've always liked about this approach is
You can have any number of listeners live at any one time.
You can create a new listener and change how you send the email without needing to change/redeploy the OMS application itself.
You can take the listener(s) off line and stop / delay the sending of the email without losing any notifications or affecting the OMS itself.
I'm looking for some opinions here, I'm building a web application which has the fairly standard functionality of:
Register for an account by filling out a form and submitting it.
Receive an email with a confirmation code link
Click the link to confirm the new account and log in
When you send emails from your web application, it's often (usually) the case that there will be some change to the persistence layer. For example:
A new user registers for an account on your site - the new user is created in the database and an email is sent to them with a confirmation link
A user assigns a bug or issue to someone else - the issue is updated and email notifications are sent.
How you send these emails can be critical to the success of your application. How you send them depends on how important it is that the intended recipient receives the email.
We'll look at the following four strategies in relation to the case where the mail server is down, using example 1.
TRANSACTIONAL & SYNCHRONOUS
The sending of the email fails and the user is shown an error message saying that their account could not be created. The application will appear to be slow and unresponsive as the application waits for the connection timeout. The account is not created in the database because the transaction is rolled back.
TRANSACTIONAL & ASYNCHRONOUS
The transactional definition here refers to sending the email to a JMS queue or saving it in a database table for another background process to pick up and send.
The user account is created in the database, the email is sent to a JMS queue for processing later. The transaction is successful and committed. The user is shown a message saying that their account was created and to check their email for a confirmation link. It's possible in this case that the email is never sent due to some other error, however the user is told that the email has been sent to them. There may be some delay in getting the email sent to the user if application support has to be called in to diagnose the email problem.
NON-TRANSACTIONAL & SYNCHRONOUS
The user is created in the database, but the application gets a timeout error when it tries to send the email with the confirmation link. The user is shown an error message saying that there was an error. The application is slow and unresponsive as it waits for the connection timeout
When the mail server comes back to life and the user tries to register again, they are told their account already exists but has not been confirmed and are given the option of having the email re-sent to them.
NON-TRANSACTIONAL & ASYNCHRONOUS
The only difference between this and transactional & asynchronous is that if there is an error sending the email to the JMS queue or saving it in the database, the user account is still created but the email is never sent until the user attempts to register again.
What I'd like to know is what have other people done here? Can you recommend any other solutions other than the 4 I've mentioned above? What's a reasonable way of approaching this problem? I don't want to over-engineer a system that's dealing with the (hopefully) rare situation where my mail server goes down!
The simplest thing to do is to code it synchronously, but are there any other pitfalls to this approach? I guess I'm wondering if there's a best practice, I couldn't find much out there by googling.
My 2 cents:
Once you have a user sign up, never roll back the registration if sending the E-Mail fails. For simple business reasons: They may not come back or re-register if it doesn't work out at the first try. Rather tolerate an incomplete registration and nag the user to confirm their E-Mail address as soon as possible.
In most cases when sending an E-Mail goes wrong, your app will not get immediate feedback anyway - non-existent E-Mail addresses on valid servers will send back a "undeliverable" message with some delay; if the mail gets eaten by a spam filter, you'll get no feedback at all; in other scenarios, it may take several minutes (greylisting) to several days (mail server temporarily down) for an E-Mail to get delivered. A synchronous approach waiting for the delivery of the mail is therefore doomed IMO. Even an immediate failure (because the user entered a obviously fake address) should never result in the registration getting rolled back.
What I would do is, make account creation as easy as possible, allow the user access to the account before it is confirmed, and then nag the hell out of them to confirm their E-Mail (if necessary, limit access to certain areas until confirmation). I would prevent the creation of a second account with the same E-Mail, though, to prevent clutter.
Make sure you allow changing the E-Mail address even if the previous address hasn't been confirmed yet, and enable the user to re-request the confirmation message to a different address.
This question is language independent.
I have an application that handles requests in a loop. During this loop for each request multiple actions are taken. These actions are sitting inside try / catch / log blocks.
I am now extending this to notify administrators of severe errors via email.
This is all very easy, except for one thing. We are relying on the client/s to implement their own email delivery redundancy, and I know from experience there will always be one client who just has one SMTP exchange server, and this is bound to go down from time to time.
So here is the dilemma:
Scenario 1 (don't handle the error during failed send) - when I send an email to admin and SMTP is down it will break the app (app will stop running, and additional loops will stop processing, because the error is unhandled) This means that the error reporting which was supposed to be beneficial to the app suddenly becomes the reason why 99/100 requests don't get processed because there was an issue with request 1.
Scenario 2 (handle the exception during failed send) - this means that I surround send code in try/catch/log blocks, great! the application processes all requests 99 of them, except one, but the admin now has no notification of this one error via email because when it tried to send SMTP was down, and that error was simply logged to the application log, the admin who doesn't check this log for days (even weeks) at a time now has no way to know that error took place.
So is there a win/win way to solve this problem or am I always going to be at a loss, and in the mercy of SMTP being up. Remember it is out of our scope to manage email server redundancy.
Extend scenario 2 to keep a record of which entries in the application log didn't get sent via email, and periodically poll this log for unsent entries and try to resend them - eventually the smtp service will be available again. (You might want to stop any resent errors from going back in the resend queue tho...)
I would suggest the "win-win" way would be to have a server admin who actually administrates the server, rather than one who is entirely unreachable when his mail server is down, and doesn't bother to check up on it afterwards to see if he missed any notifications.