I'm looking for some opinions here, I'm building a web application which has the fairly standard functionality of:
Register for an account by filling out a form and submitting it.
Receive an email with a confirmation code link
Click the link to confirm the new account and log in
When you send emails from your web application, it's often (usually) the case that there will be some change to the persistence layer. For example:
A new user registers for an account on your site - the new user is created in the database and an email is sent to them with a confirmation link
A user assigns a bug or issue to someone else - the issue is updated and email notifications are sent.
How you send these emails can be critical to the success of your application. How you send them depends on how important it is that the intended recipient receives the email.
We'll look at the following four strategies in relation to the case where the mail server is down, using example 1.
TRANSACTIONAL & SYNCHRONOUS
The sending of the email fails and the user is shown an error message saying that their account could not be created. The application will appear to be slow and unresponsive as the application waits for the connection timeout. The account is not created in the database because the transaction is rolled back.
TRANSACTIONAL & ASYNCHRONOUS
The transactional definition here refers to sending the email to a JMS queue or saving it in a database table for another background process to pick up and send.
The user account is created in the database, the email is sent to a JMS queue for processing later. The transaction is successful and committed. The user is shown a message saying that their account was created and to check their email for a confirmation link. It's possible in this case that the email is never sent due to some other error, however the user is told that the email has been sent to them. There may be some delay in getting the email sent to the user if application support has to be called in to diagnose the email problem.
NON-TRANSACTIONAL & SYNCHRONOUS
The user is created in the database, but the application gets a timeout error when it tries to send the email with the confirmation link. The user is shown an error message saying that there was an error. The application is slow and unresponsive as it waits for the connection timeout
When the mail server comes back to life and the user tries to register again, they are told their account already exists but has not been confirmed and are given the option of having the email re-sent to them.
NON-TRANSACTIONAL & ASYNCHRONOUS
The only difference between this and transactional & asynchronous is that if there is an error sending the email to the JMS queue or saving it in the database, the user account is still created but the email is never sent until the user attempts to register again.
What I'd like to know is what have other people done here? Can you recommend any other solutions other than the 4 I've mentioned above? What's a reasonable way of approaching this problem? I don't want to over-engineer a system that's dealing with the (hopefully) rare situation where my mail server goes down!
The simplest thing to do is to code it synchronously, but are there any other pitfalls to this approach? I guess I'm wondering if there's a best practice, I couldn't find much out there by googling.
My 2 cents:
Once you have a user sign up, never roll back the registration if sending the E-Mail fails. For simple business reasons: They may not come back or re-register if it doesn't work out at the first try. Rather tolerate an incomplete registration and nag the user to confirm their E-Mail address as soon as possible.
In most cases when sending an E-Mail goes wrong, your app will not get immediate feedback anyway - non-existent E-Mail addresses on valid servers will send back a "undeliverable" message with some delay; if the mail gets eaten by a spam filter, you'll get no feedback at all; in other scenarios, it may take several minutes (greylisting) to several days (mail server temporarily down) for an E-Mail to get delivered. A synchronous approach waiting for the delivery of the mail is therefore doomed IMO. Even an immediate failure (because the user entered a obviously fake address) should never result in the registration getting rolled back.
What I would do is, make account creation as easy as possible, allow the user access to the account before it is confirmed, and then nag the hell out of them to confirm their E-Mail (if necessary, limit access to certain areas until confirmation). I would prevent the creation of a second account with the same E-Mail, though, to prevent clutter.
Make sure you allow changing the E-Mail address even if the previous address hasn't been confirmed yet, and enable the user to re-request the confirmation message to a different address.
Related
This question is not 100% technical. I've looked online and I couldn't find this being discussed.
We have a forgot username feature in our website which mails the username to the user's email address, using our own mail server.
We also put a message in the web-page asking the user to wait for 20 minutes for the mail to arrive since there can be occasional delays.
Our business tester raised an interesting point saying that 20 minutes seems an unacceptable time to ask the user to wait for. He said our technology should send emails immediately and the maximum lag should be 1 minute at worst.
Firstly, emails are usually received immediately by the recipient so there isn't a problem there.
But in our experience in using other websites, sometimes emails do take a while to arrive. I also remember reading somewhere that emails (at a network level) use a lower quality of service QoS unlike voip services. I can't seem to find it now.
Users can also experience delays in receiving emails because of issues in their own mail server.
Now, all we can do is send the mail using our mail server and ensure that the load and resources on the server is well managed.
1) Is there anything else we can do to ensure that our mails are sent quickly at all times.
2) What is the acceptable time, we can ask the user to wait until he logs a call with help-desk? I believe that there can be lags at a network/protocol level and the user's mail server which we can't do anything about.
Thanks.
All you can do is to inform user that your server "handed over" responsibility for email delivery to named SMTP server "beyond your control/responsibility". You may expect it would take a few (<5s) in most cases (>50%).
Your smtp-client may try initial delivery attempt.
On delivery success => inform the user that email delivery/delay is now beyond your control
On (initial) delivery failure => pass message to your SMTP server (with initial delivery attempt skipped).
I've set up a Mandrill webhook which will update my app whenever an email hard-bounces or is rejected, so I don't keep that particular email address in my database. The way it works is this: a user gives me an address, I send him a confirmation, and if I don't hear from Mandrill's webhook in 30 minutes, I assume it's OK.
So I ran a some tests with non-existing addresses, and they didn't go too well. Most of them appeared as delivered for hours, long after I'd assumed they were fine.
Also, I didn't account for the delay in receiving webhook batches. One mail bounced at 2:01pm, according to the outbound activity logs, but the webhook history shows a batch being sent only at 2:52pm.
My questions is: how long should I delay my app's assumption of deliverability in order to give Mandrill enough time to detect hard-bounces / rejections and then send me webhook batches? I can live with letting some 5% bad emails going by because of delayed in processing on peak-time or other extraordinary events, but it seems like my 30 minutes isn't enough to catch anything at all...
Not the answer you're looking for, but Mandrill doesn't let you do this. The only way of checking if an email has been delivered is to poll Mandrill (with the message/info.json API). To see if the message is delivered you have to check the smtp_events and look for an event with diag starting with 250. As you've already experienced it may take a long time between delivery and when a message is accessible through the API. In my experience the normal case is around 10 minutes, but it might take many, many hours (this is the case for bounced emails as well as emails that got delivered immediately).
If it is important to you to know when an email is delivered, I would recommend you to switch to another email provider. There are plenty of different ones out there. I've personally used Amazon SES. They're cheaper than Mandrill, and you can expect a delivery notification after a second or so. Do note that Amazon SES is a bit more bare-bone than Mandrill (they don't have support for open/click-tracking, templating, dedicated IP, etc.), so it might not be the right provider for you.
Although Apple doesn't allow apps to send text messages without explicit user permission (unlike Android, which just requires you to ask for permissions like this when installing the app), I was wondering if anyone had any ideas for other ways to get around this issue. Specifically:
Is it possible to allow the user to pre-approve a specific text message, and delay sending it until the desired time?
Can an external client be used? Ex., ping a server to do the work of sending the text message for the user? This seems like it will get ugly quickly though (since provider info is required), and would require the user to have internet access, not just cell signal. However, perhaps the application can require an initialization text.
The goal here is to send a specific text message for the user when they are unable to do so themselves, so naturally, having them pre-approve the message would be impossible.
You can't send text messages directly from the device.
But I think there is no problem sending them from a webserver, however your app needs to have an internet connection to do this and I am not aware of a service, which allows you to send text messages for free, so you need to pay to provide this service to your users (I can't say if it is ok for Apple to charge your users for such a service). Another drawback of this solution is, that the sender of the text message wouldn't probably be the users number.
With regards an OMS, what is the best method to send a confirmation email? The 2 options I have so far are;
A script on the order page sends an email once the record is written to the database.
A scheduled task on the server, send the email, polling the database every-so-often to find new entries.
Which method do systems currently use?
For e-commerce websites, it might be better to think about the best user experience.
Given that, you would want to send the email as soon as the order is received so the user knows that they have purchased the item. The sooner it gets into their inbox, the sooner they will be happy that they have made their purchase.
I agree with Digbyswift that sending the confirmation email once the record is written to the database is the least scalable. But I would argue that if your system has gotten to the point that you are taking so many orders that your system cannot keep up, you have a wonderful problem on your hands that you now probably have resources to handle.
At PostageApp, we handle the emails of a few e-commerce websites, so perhaps you would benefit from an arrangement with an email service provider to off-load this task so that all of your resources can be spent on keeping your site up and your databases running.
Here are some great alternatives if PostageApp is not your style:
Sendgrid
Postmark
Mailjet
This is a question of scalability. Sending a confirmation email once the record is written to the database is the least scalable. The more orders that are taken , the more emails are sent potentially tying up resources.
A scheduled task is certainly better as emails can be queued up and can be sent in a separate process.
A further option which you could consider is using neither and delegating the responsibility of sending emails to a 3rd party dedicated emailing service, i.e. via an API. This is much better since your hosting does not have to consider the load and you can utilise any reporting offered by the 3rd party. Plus many services offer a free quota up to a certain threshold. This will allow you OMS and business to scale appropriately.
If you apply a message based architecture; you could just publish an order created message and have any number of subscribers respond to that event. You could create a listener that sends the email in house (bespoke option) or another listener that called the API of a 3rd party emailer to send the email on your behalf (as per #Digbyswift)
What I've always liked about this approach is
You can have any number of listeners live at any one time.
You can create a new listener and change how you send the email without needing to change/redeploy the OMS application itself.
You can take the listener(s) off line and stop / delay the sending of the email without losing any notifications or affecting the OMS itself.
This question is language independent.
I have an application that handles requests in a loop. During this loop for each request multiple actions are taken. These actions are sitting inside try / catch / log blocks.
I am now extending this to notify administrators of severe errors via email.
This is all very easy, except for one thing. We are relying on the client/s to implement their own email delivery redundancy, and I know from experience there will always be one client who just has one SMTP exchange server, and this is bound to go down from time to time.
So here is the dilemma:
Scenario 1 (don't handle the error during failed send) - when I send an email to admin and SMTP is down it will break the app (app will stop running, and additional loops will stop processing, because the error is unhandled) This means that the error reporting which was supposed to be beneficial to the app suddenly becomes the reason why 99/100 requests don't get processed because there was an issue with request 1.
Scenario 2 (handle the exception during failed send) - this means that I surround send code in try/catch/log blocks, great! the application processes all requests 99 of them, except one, but the admin now has no notification of this one error via email because when it tried to send SMTP was down, and that error was simply logged to the application log, the admin who doesn't check this log for days (even weeks) at a time now has no way to know that error took place.
So is there a win/win way to solve this problem or am I always going to be at a loss, and in the mercy of SMTP being up. Remember it is out of our scope to manage email server redundancy.
Extend scenario 2 to keep a record of which entries in the application log didn't get sent via email, and periodically poll this log for unsent entries and try to resend them - eventually the smtp service will be available again. (You might want to stop any resent errors from going back in the resend queue tho...)
I would suggest the "win-win" way would be to have a server admin who actually administrates the server, rather than one who is entirely unreachable when his mail server is down, and doesn't bother to check up on it afterwards to see if he missed any notifications.