We send a lot of email messages from our Siebel 7.8 application, and we'd like to determine whether they have been successfully delivered or not.
According to the Bookshelf, if the SMTP server is down, the Communications Outbound Manager retries to send the message later, so that's not a problem. However, there are still plenty of issues which could cause an email to not be delivered, such as a typo in the address, the receiver having reached its storage quota, etc.
We send our messages this way:
var ps = TheApplication().NewPropertySet();
ps.SetProperty("ActivityId", outboundEmailActivityId);
ps.SetProperty("CommProfile", commProfile);
ps.SetProperty("ProcessMode", "Local");
var bs = TheApplication().GetService("Outbound Communications Manager");
bs.InvokeMethod("SendMessage", ps, psOut);
Using ProcessMode = Local allows us to detect a few errors. For example, if we try to send a message to a non-existant account in the same domain of our SMTP server, it returns 550 Unknown user and then 503 Must have sender and recipient first. The Outbound Communications Manager raises an exception, and we capture and handle it.
However, if we send a message to a non-existant account in a different domain, our SMTP server can't know that it will fail, and therefore it returns 250 Queued, and our code completes successfully. Later (it can range from seconds to a few hours later), we will receive a "Message undeliverable" error message, but at this point we only know that an outbound message failed, we don't know which one.
Is there any way in which Siebel can handle these 'Message undeliverable' notifications automatically?
We are thinking of writing our own process for that, but it seems like a huge task: we'd have to parse the delivery failure notification, identify the failing recipient, search for all the recent messages sent to that address, and somehow, guess which one failed (based on the Message-Id if we are lucky and can read it within Siebel, or on the Subject otherwise).
The problem is that SMTP is by its nature neither a synchronous nor reliable protocol (i.e. in the sense of "engineered for guaranteed delivery"). Your Siebel app server will connect to its assigned SMTP server and ask it to accept a message for delivery and at that time there are a few high level validations that can be perform (some of which you've mentioned but which can also include policy enforcement such as checking whether your (possibly anonymous) identity is authorized for relaying messages to external domains). Once that conversation ends, there is not much else you can reliably do because again, everything from that point is asynchronous and not guaranteed for delivery (any number of intermediate relay agents can be involved, each with their own potential for outages with or without retry, each with the ability to honor or ignore requests for delivery or read receipts or to report invalid recipients, throwing your message in a junk folder or not, etc.). Certainly you can attempt to work with any bounce notifications you do happen to get to try to correlate them back to the sender but that would be outside the context of your sending code.
Related
I have an [Azure Storage] queue where I put e-mail messages to be sent. Then, there is a separate service which monitors that queue and send e-mails using some service. In this particular case I'm using SendGrid.
So, theoretically, if the sender crashes right after a successful call to SendGrid Mail Send API (https://sendgrid.com/docs/API_Reference/Web_API_v3/Mail/index.html), the message will be returned to the queue and retried later. This may result in the same e-mail being delivered more than once, which could be really annoying for some type of e-mail.
The normal way to avoid this situation would be to provide some sort of idempotency key to Send API. Then the side being called can make sure the operation is performed at most once.
After careful reading of SendGrid documentation and googling, I could not find any way to achieve what I'm looking for here (at most once semantic). Any ideas?
Without support for an idempotency key in the API itself your options are limited I think.
You could modify your email sending service to dequeue and commit before calling the Send API. That way if the service fails to send the message will not be retried as it has already been removed from the queue, it will be sent at most once.
Additionally, you could implement some limited retries on particular http responses (e.g. 429 & 5xx) from SendGrid where you are certain the message was not sent and retrying might be useful - this would maintain "at most once" whilst lowering the failure rate. Probably this should include some backoff time between each attempt.
This question is not 100% technical. I've looked online and I couldn't find this being discussed.
We have a forgot username feature in our website which mails the username to the user's email address, using our own mail server.
We also put a message in the web-page asking the user to wait for 20 minutes for the mail to arrive since there can be occasional delays.
Our business tester raised an interesting point saying that 20 minutes seems an unacceptable time to ask the user to wait for. He said our technology should send emails immediately and the maximum lag should be 1 minute at worst.
Firstly, emails are usually received immediately by the recipient so there isn't a problem there.
But in our experience in using other websites, sometimes emails do take a while to arrive. I also remember reading somewhere that emails (at a network level) use a lower quality of service QoS unlike voip services. I can't seem to find it now.
Users can also experience delays in receiving emails because of issues in their own mail server.
Now, all we can do is send the mail using our mail server and ensure that the load and resources on the server is well managed.
1) Is there anything else we can do to ensure that our mails are sent quickly at all times.
2) What is the acceptable time, we can ask the user to wait until he logs a call with help-desk? I believe that there can be lags at a network/protocol level and the user's mail server which we can't do anything about.
Thanks.
All you can do is to inform user that your server "handed over" responsibility for email delivery to named SMTP server "beyond your control/responsibility". You may expect it would take a few (<5s) in most cases (>50%).
Your smtp-client may try initial delivery attempt.
On delivery success => inform the user that email delivery/delay is now beyond your control
On (initial) delivery failure => pass message to your SMTP server (with initial delivery attempt skipped).
I'm usign amazon cloud services to host my webpage. Our web site, actually sends a lot of emails per hour. In one instant our server could be asked to send 30 mails or more.
Sometimes our clients complaint about not getting emails from the web, which is connected to our mail server to send emails. This doesn't happen if we send the email directly from our addresses to theirs, so I'm pretty much know is the web page who's causing the problem.
The thing is I don't know what is happening and neither know what to look for. I've checked memory and cpu of that server and everything seems to work fine
make sure your website sends the messages with a correct bounce address (aka envelope sender address). this does not have to be the same thing as the address in the From: header. by default, this is often something like "apache#www.example.com" - I don't know about amazon). these types of bounce addresses are bad because usually you don't receive the error message if something goes wrong. use a real email account. To check what bounce address you currently use, look at the message source of a received mail and see the Return-Path header.
check the logs of your mailserver for those missing messages. either it reports an error (in which case you should get the error to your bounce address) or it reports the message as sent to the target server (in which case you tell your clients to check THEIR maillogs since you can prove you have sent the message)
I'm looking for some opinions here, I'm building a web application which has the fairly standard functionality of:
Register for an account by filling out a form and submitting it.
Receive an email with a confirmation code link
Click the link to confirm the new account and log in
When you send emails from your web application, it's often (usually) the case that there will be some change to the persistence layer. For example:
A new user registers for an account on your site - the new user is created in the database and an email is sent to them with a confirmation link
A user assigns a bug or issue to someone else - the issue is updated and email notifications are sent.
How you send these emails can be critical to the success of your application. How you send them depends on how important it is that the intended recipient receives the email.
We'll look at the following four strategies in relation to the case where the mail server is down, using example 1.
TRANSACTIONAL & SYNCHRONOUS
The sending of the email fails and the user is shown an error message saying that their account could not be created. The application will appear to be slow and unresponsive as the application waits for the connection timeout. The account is not created in the database because the transaction is rolled back.
TRANSACTIONAL & ASYNCHRONOUS
The transactional definition here refers to sending the email to a JMS queue or saving it in a database table for another background process to pick up and send.
The user account is created in the database, the email is sent to a JMS queue for processing later. The transaction is successful and committed. The user is shown a message saying that their account was created and to check their email for a confirmation link. It's possible in this case that the email is never sent due to some other error, however the user is told that the email has been sent to them. There may be some delay in getting the email sent to the user if application support has to be called in to diagnose the email problem.
NON-TRANSACTIONAL & SYNCHRONOUS
The user is created in the database, but the application gets a timeout error when it tries to send the email with the confirmation link. The user is shown an error message saying that there was an error. The application is slow and unresponsive as it waits for the connection timeout
When the mail server comes back to life and the user tries to register again, they are told their account already exists but has not been confirmed and are given the option of having the email re-sent to them.
NON-TRANSACTIONAL & ASYNCHRONOUS
The only difference between this and transactional & asynchronous is that if there is an error sending the email to the JMS queue or saving it in the database, the user account is still created but the email is never sent until the user attempts to register again.
What I'd like to know is what have other people done here? Can you recommend any other solutions other than the 4 I've mentioned above? What's a reasonable way of approaching this problem? I don't want to over-engineer a system that's dealing with the (hopefully) rare situation where my mail server goes down!
The simplest thing to do is to code it synchronously, but are there any other pitfalls to this approach? I guess I'm wondering if there's a best practice, I couldn't find much out there by googling.
My 2 cents:
Once you have a user sign up, never roll back the registration if sending the E-Mail fails. For simple business reasons: They may not come back or re-register if it doesn't work out at the first try. Rather tolerate an incomplete registration and nag the user to confirm their E-Mail address as soon as possible.
In most cases when sending an E-Mail goes wrong, your app will not get immediate feedback anyway - non-existent E-Mail addresses on valid servers will send back a "undeliverable" message with some delay; if the mail gets eaten by a spam filter, you'll get no feedback at all; in other scenarios, it may take several minutes (greylisting) to several days (mail server temporarily down) for an E-Mail to get delivered. A synchronous approach waiting for the delivery of the mail is therefore doomed IMO. Even an immediate failure (because the user entered a obviously fake address) should never result in the registration getting rolled back.
What I would do is, make account creation as easy as possible, allow the user access to the account before it is confirmed, and then nag the hell out of them to confirm their E-Mail (if necessary, limit access to certain areas until confirmation). I would prevent the creation of a second account with the same E-Mail, though, to prevent clutter.
Make sure you allow changing the E-Mail address even if the previous address hasn't been confirmed yet, and enable the user to re-request the confirmation message to a different address.
This question is language independent.
I have an application that handles requests in a loop. During this loop for each request multiple actions are taken. These actions are sitting inside try / catch / log blocks.
I am now extending this to notify administrators of severe errors via email.
This is all very easy, except for one thing. We are relying on the client/s to implement their own email delivery redundancy, and I know from experience there will always be one client who just has one SMTP exchange server, and this is bound to go down from time to time.
So here is the dilemma:
Scenario 1 (don't handle the error during failed send) - when I send an email to admin and SMTP is down it will break the app (app will stop running, and additional loops will stop processing, because the error is unhandled) This means that the error reporting which was supposed to be beneficial to the app suddenly becomes the reason why 99/100 requests don't get processed because there was an issue with request 1.
Scenario 2 (handle the exception during failed send) - this means that I surround send code in try/catch/log blocks, great! the application processes all requests 99 of them, except one, but the admin now has no notification of this one error via email because when it tried to send SMTP was down, and that error was simply logged to the application log, the admin who doesn't check this log for days (even weeks) at a time now has no way to know that error took place.
So is there a win/win way to solve this problem or am I always going to be at a loss, and in the mercy of SMTP being up. Remember it is out of our scope to manage email server redundancy.
Extend scenario 2 to keep a record of which entries in the application log didn't get sent via email, and periodically poll this log for unsent entries and try to resend them - eventually the smtp service will be available again. (You might want to stop any resent errors from going back in the resend queue tho...)
I would suggest the "win-win" way would be to have a server admin who actually administrates the server, rather than one who is entirely unreachable when his mail server is down, and doesn't bother to check up on it afterwards to see if he missed any notifications.