In our design we have something of a paradox. We have a database of projects. Each project has a status. We have a REST api to change a project from “Ready” status to “Cleanup” status. Two things must happen.
update the status in the database
send out an email to the approvers
Currently RESTful api does 1, and if that is successful, do 2.
But sometimes the email fails to send. But since (1) is already committed, it is not possible to rollback.
I don't want to send the email prior to commit, because I want to make sure the commit is successful before sending the email.
I thought about undoing step 1, but that is very hard. The status change involves adding new records to the history table, so I need to delete them. And if another person make other changes concurrently, the undo might get messed up.
So what can I do? If (2) fails, should I return “200 OK” to the client?
Seems like the best option is to return “500 Server Error” with error message that says “The project status was changed. However, sending the email to the approvers failed. Please take appropriate action.”
Perhaps I should not try to do 1 + 2 in a single operation? But that just puts the burden on the client, which is worse!
Just some random thoughts:
You can have a notification sent status flag along with a datetime of submission. When an email is successful then it flips, if not then it stays. When changes are submitted then your code iterates through ALL unsent notifications and tries to send. No idea what backend db you are suing but I believe many have the functionality to send emails as well. You could have a scheduled Job (SQL Server Agent for MSSQL) that runs hourly and tries to send if the datetime of the submission is lapsed a certain amount or starts setting off alarms if it fails as well.
If ti is that insanely important then maybe you could integrate a third party service such as sendgrid to run as a backup sending mech. That of course would be more $$ though...
Traditionally I've always separated functions like this into a backend worker process that handles this kind of administrative tasking stuff across many different applications. Some notifications get sent out every morning. Some get sent out every 15 minutes. Some are weekly summaries. If I run into a crash and burn then I light up the event log and we are (lucky/unlucky) enough to have server monitoring tools that alert us on specified application events.
Related
After saving a data, I wait for the confirmation of the opposite server. With a different endpoint, I want to constantly check this server's approval check in the background. I want to send requests continuously until the transaction is approved and when it is approved, I want to take action. How can I make a request to a persistent endpoint?
From which sources should I get help, if there are examples, I would like to examine them.
I am creating an application which has a follow mechanism where the followed user has to accept the request of a following (similar to private accounts on instagram).
I then want the following user to find out when the other user has checked a million times (every time the following user opens the screen if I did the query in viewDidLoad). However, the problem with this, is that there will be a lot requests which will expensive to me as I will have to pay for the requests to Parse so I want to minimise these queries.
Currently, the best thing I can think of is to check once a day at midnight for example but this doesn't seem very seamless.
Is there a better way of doing this?
For starters consider how stale you are willing to allow an app's view of the world to be and cache the response that long. If a user views that screen every 30 seconds you might only want to actually check with the server 5 minutes after the last successful response (or the last response which had 0 follow requests).
You might consider switching from this sort of "pull" polling where the client decides when to ask the server if anything has changed to a "push" model where the server can inform the client when a change occurs. For example you can send a silent background push notification to a user's devices when they have a follow request, the app can then respond by performing your existing query.
You might still want polling or user triggered requests (like a "pull to refresh" gesture) as a fallback for missed notifications or devices with notifications disabled but you should be able to drastically reduce request volume.
I am developing a small REST API. As I got into analyzing all the possible failure scenarios, which I have to handle to create a reliable and stable system, I went into thinking about how to make my APIs atomic.
If we take a simple case of creating a contact through the POST API.
The server gets the POST request for the new contact.
Creates the contact in the DB.
Creates a response to send back to the client.
The server crashes before sending the response.
The client gets a timeout error (or connection refused?)
The client is bound to think that the contact creation has failed, though, in fact, the contact was in the DB.
Is this a rare case we can ignore? How do big companies deal with such an issue?
To handle this, you should make your write APIs idempotent i.e. If the same operation is executed multiple times, the result should be same as the operation was done only once.
To achieve this in your current example, you need to be able to identify a contact uniquely based on some parameter, say emailAddress. So, if the createContact is called again with the same emailAddress, check in the DB if a contact already exists with the emailAddress. If so, return the existing contact. Else, create a new contact with the emailAddress and return it.
Hope this helps.
If the request times out, the client should not make any assumption about whether it failed or succeeded.
If it is just a user making a request from a web form, then the timeout should just be exposed to the user, and they can hit the back button and check whether the operation succeeded or not, and if not they submit the request again. (This is fine as long as you always keep a consistent state. If your operation has multiple steps and fails mid way, you need to roll back.)
However if reliable messaging is important to your application you will have to use a library or build your own reliable messaging layer. This could work by having the client assign a unique ID to every request, and having another request that lets you check the result of that request ID later. Then you can do automated retries but only where necessary.
I've build a simple task management webapp: User A fills up a form, hits submit button, sends data to a server and if the data validates User B gets assigned to this task.
I'd like to notify User B by email on this new assignment. However User A can alter the task data or even delete the task and the email that already has been sent would be incorrect in this case.
One approach is to delay the notification email for couple of minutes and then upon sending update the email message if needed.
Which are the best practices for notifications sending?
I think you have a few choices:
Send out emails whenever task status changes. Don't include details; send a link to user B to let them see what the changes are.
This is a good example of Why Starbucks Does Not Use Two Phase Commit. User B will tolerate "dirty reads" because they aren't life altering.
Send out all notification emails asynchronously on a fixed schedule. Have a timed task query a database, generate all the emails, and send them at once. The task will have the chance to only send the latest one. If user A assigns a task, makes updates, then deletes, user B will only get the last meaningful one. In this case, an assign followed by a delete might result in no email being sent. Only an assign or update as last state will result in an email being sent.
I am working on an application which need to notify around 100 people at once when a specific condition is met. Now when a user who is performing the action which results in the specific condition need to wait till all 100 emails are sent which takes quite long using Gmail SMTP. The application is built on top of Cake PHP.
My question is whether there is a way application can send 100 emails without blocking the user whose action results in meeting the specific condition.
To make my question clear, think of Groupon. It sends notification to all buyers when minimum numbers of buyers are met. So when the nth person make the purchase, Google sends the notification.One way is to notify all buyers immediately after the purchase is complete (which is what we are doing n context of our application) and probably other way is to wait and send the notification using an external script/app at a pre-defined time.
In case of former, the application would block while sending emails is complete. Since PHP deosn't support multi-threading, I was wondering if there is an easy way to make this operation asynchoronous so it doesn't affect main application flow.
You could put the notification in a queue, and use a cronjob that checks and sends notifications every 5 minutes. That way your user isn't locked up while the operation happens.
I'm not 100% sure, but you might be able to use an ajax call too, which would keep the user free to carry on after the request is sent.