I am developing a small REST API. As I got into analyzing all the possible failure scenarios, which I have to handle to create a reliable and stable system, I went into thinking about how to make my APIs atomic.
If we take a simple case of creating a contact through the POST API.
The server gets the POST request for the new contact.
Creates the contact in the DB.
Creates a response to send back to the client.
The server crashes before sending the response.
The client gets a timeout error (or connection refused?)
The client is bound to think that the contact creation has failed, though, in fact, the contact was in the DB.
Is this a rare case we can ignore? How do big companies deal with such an issue?
To handle this, you should make your write APIs idempotent i.e. If the same operation is executed multiple times, the result should be same as the operation was done only once.
To achieve this in your current example, you need to be able to identify a contact uniquely based on some parameter, say emailAddress. So, if the createContact is called again with the same emailAddress, check in the DB if a contact already exists with the emailAddress. If so, return the existing contact. Else, create a new contact with the emailAddress and return it.
Hope this helps.
If the request times out, the client should not make any assumption about whether it failed or succeeded.
If it is just a user making a request from a web form, then the timeout should just be exposed to the user, and they can hit the back button and check whether the operation succeeded or not, and if not they submit the request again. (This is fine as long as you always keep a consistent state. If your operation has multiple steps and fails mid way, you need to roll back.)
However if reliable messaging is important to your application you will have to use a library or build your own reliable messaging layer. This could work by having the client assign a unique ID to every request, and having another request that lets you check the result of that request ID later. Then you can do automated retries but only where necessary.
Related
I have an application in which there are users, issuers, certificates and issues. users are the basic account, and issuers are the upgraded accounts who are able to distribute certificates to users. And distributing of certificates are called issues. The app uses postgres 14.4.
There is something called 'passive issue' in the application, which refers to issue of a certificate to a user that is not yet registered. Passive issue executes a transaction as follows:
check if user exists, if it does ignore it, if it does not, continue
create a passive issue which has awaiting register status
and then when that user actually registers, a trigger fires and updates the issue status.
The problem is that, after checking if user exists, and it does not, but right at that moment the user creates the account, and we try to passive issue, it will throw an error saying user already exists.
There are 2 possible approaches to solve this you might have thought so far:
just catch and check the error and redirect it to normal issuing path rather than passive issue within the api itself
return the error and let client retry the request by sending a request to normal issue path
The problem with above solutions:
It can not be done. Simply because this application uses blockchain, and normal issue endpoint requires the signature of the issuer. Passive issue is specifically implemented to be sent without a signature, and signature is generated on the client-side. And sending the private key to api is NOT even a possibility due to security concerns.
This is possible. This way the client will be noticed that this account registered, and it can prepare the signature and directly send a request to normal issue api. But it requires a longer and more complex implementation with retry logic etc.
And what I think would be the most neat solution in my case is such a scenario of passive issue:
check if user exists, and if it doesn't, lock the insert into the user table for that specific email ONLY
create passive issues for those accounts with await registration status (tx committed at this point and lock is released)
now even if the user tried to register in the middle, it will wait until issues are created and then user will be registered, and then trigger will fire, and since there are passive issues on this user now, it will update them.
So... Long story short: is there a way to put a lock on insertion of non-existing rows? And if there is, is it more feasible than the 2nd or any other possible solution?
Hi i am for the time being building a REST Api and i am wondering about an idea i have for secure data posting.
I have data that i have to delegate to two different endpoints in the same Post and i am wondering if there is any way to "test-post" to the endpoints before posting for real solving the problematic scenarion in which one of the systems that i'm sending to would be down for a moment.
That is if one of the systems would be down i could return a failed request message instead.
For extra explanation of my thought:
User posts data objet to my api via endpoint.
I process data and later try to send processed data to two different systems via their endpoints in same process/method.
IF systems ok then data is sent, and i return a OK.
Else One of systems not ok i roll back (that is not sending processed data to either one of them), and return for example 500 Internal server error.
It is critical that when i send the data it has to be sent to both systems in same process, or none of them in same process if one system is down.
Hope i got you guys to understand what i am after.
In our design we have something of a paradox. We have a database of projects. Each project has a status. We have a REST api to change a project from “Ready” status to “Cleanup” status. Two things must happen.
update the status in the database
send out an email to the approvers
Currently RESTful api does 1, and if that is successful, do 2.
But sometimes the email fails to send. But since (1) is already committed, it is not possible to rollback.
I don't want to send the email prior to commit, because I want to make sure the commit is successful before sending the email.
I thought about undoing step 1, but that is very hard. The status change involves adding new records to the history table, so I need to delete them. And if another person make other changes concurrently, the undo might get messed up.
So what can I do? If (2) fails, should I return “200 OK” to the client?
Seems like the best option is to return “500 Server Error” with error message that says “The project status was changed. However, sending the email to the approvers failed. Please take appropriate action.”
Perhaps I should not try to do 1 + 2 in a single operation? But that just puts the burden on the client, which is worse!
Just some random thoughts:
You can have a notification sent status flag along with a datetime of submission. When an email is successful then it flips, if not then it stays. When changes are submitted then your code iterates through ALL unsent notifications and tries to send. No idea what backend db you are suing but I believe many have the functionality to send emails as well. You could have a scheduled Job (SQL Server Agent for MSSQL) that runs hourly and tries to send if the datetime of the submission is lapsed a certain amount or starts setting off alarms if it fails as well.
If ti is that insanely important then maybe you could integrate a third party service such as sendgrid to run as a backup sending mech. That of course would be more $$ though...
Traditionally I've always separated functions like this into a backend worker process that handles this kind of administrative tasking stuff across many different applications. Some notifications get sent out every morning. Some get sent out every 15 minutes. Some are weekly summaries. If I run into a crash and burn then I light up the event log and we are (lucky/unlucky) enough to have server monitoring tools that alert us on specified application events.
I have a scenario for my app which is similar to sending friend request in Facebook.
When user A sends friend request to user B, internally a new friend request document is created. At a later time when user B also wants to send friend request to A, system would find out that a friend request document existed and so they should be friend of each other, no new friend request document would be created.
I'm trying to figure out the case when user A and user B both simultaneously sends friend request to each other which will then create 2 friend request documents and leading to undetermined behaviour...
Thanks for your suggestions.. Really appreciated!
Edit:
A few had suggested to use a request queue to solve this; however,
I'm confused about using queue because i thought it would make my rest api endpoint process requests sequentially. Wouldn't I lose all the benefit of multi-threading by using queue? I can't help but imagine how bad it would be if my service has millions of requests queued and waiting to be executed one by one just due to this issue. Has anyone seen something along similar problems seen in production?
I had similar situation with my client which has concurrent writes in the database, What I have implemented is a Queue service.
Create a request in the queue rather than writing in the database, a separate reader will
read one message from the queue at a time and check if it is valid to write it to
database, write only if there is no previous request.
You can implement your own queue or you can use service like AWS-SQS, rabbitmq, MSMQ etc.
// Specific to your case
In mongodb write operations on a single document are atomic.
mongodb has a feature of unique index.
Hence if you insert the document with an _id(or any other unique index) with person names A and B by creating unique index for both (such as "A_B" by lexicographically sorting the names) before doing insertion. You will inherently be able to insert only one instance of that document.
// General
What essentially we would like to have are transactions but since mongodb doesn't support such, as of now. There are a few tricks to achieve this:
2 phase commits :
https://docs.mongodb.org/v3.0/tutorial/perform-two-phase-commits/
Using an external source to maintain a flag, for example using memcache which supports insertion in transactional manner/Compare and Swap.
Here if you use system calls method in frontend then you should fire one request to frontend from Database when some user like, I send you request then within a sec database send you one system call and your frontend code immediate correct the button text like
"Add a friend" to "incoming request"
or else.
if you are only setting up database then just make a system call which send it to UI when friend request arrives or as you say Document created, the further process will be handled by UI Developer.
Thank you.
if you don't like the answer then I m apologize for that but don't downvote me because I M new in Stack Overflow Community.
i am writing a restful webapp using Spring 3. Part of the app is a form which when submitted triggers the sending of an email with various application log files attached. I'm not sure whether i should handle this form submission as a 'POST' or a 'PUT'.
My issue is that structurally the process would seem to be idempotent (and therefore a candidate for a PUT) - the same request submitted n times with the same data will always send an email with the same textual content, with the same files from the same file system locations attached.
However the content of the attached files is likely to be different for each execution of the request.
Is the content of these files beyond the scope of what i should be interested in when deciding on PUT or POST? Am i missing the point here completely?
Any thoughts would be much appreciated
Many thanks in advance!
I would definitely go for POST as each time you post your data a new email will be sent/created. PUT is mostly used to edit existing entities.
Can you do a GET on the url that you did the PUT on to return the same resource? If not then use POST.
It matters less what the server does after the request. What is important is that the behaviour is consistent to the client. If a client PUTs a resource, it expects to be able to GET it afterwards. If you make the client do POST then the client has no expectations, unless you return a 201 in which case it expects the Location header to contain the newly created resource.
The issue of sending multiple emails if you PUT twice is debatable. As long as the number of emails sent is not exposed to the client then you are not violating the behaviour of the uniform interface. However, someone else in the system may get confused by the fact that they are receiving multiple interfaces.