As a potential way of storing metadata about transactions, I would like to execute a query at end end of every transaction.
I have looked at adding logic inside the transact event, but there does not seem to be a way to make another request using the current transaction. Is there a way that this could be done using pg-promise? Is this an anti-pattern?
You can only query against the transaction connection while inside a transaction callback. You cannot do it by handling transact event, it does not have the transaction connection available.
Related
I have been looking for a way to design my API so it will be idempotent, meaning that some of that is to make my POST request routes idempotent, and I stumbled upon this article.
(If I have understood something not the way it is, please correct me!)
In it, there is a good explanation of the general idea. but what is lacking are some examples of the way that he implemented it by himself.
Someone asked the writer of the article, how would he guarantee atomicity? so the writer added a code example.
Essentially, in his code example there are two cases,
the flow if everything goes well:
Open a transaction on the db that holds the data that needs to change by the POST request
Inside this transaction, execute the needed change
Set the Idempotency-key key and the value, which is the response to the client, inside the Redis store
Set expire time to that key
Commit the transaction
the flow if something inside the code goes wrong:
and exception inside the flow of the function occurs.
a rollback to the transaction is performed
Notice that the transaction that is opened is for a certain DB, lets call him A.
However, it is not relevant for the redis store that he also uses, meaning that the rollback of the transaction will only affect DB A.
So it covers the case when something happends inside the code that make it impossible to complete the transaction.
But what will happend if the machine, which the code runs on, will crash, while it is in a state when it has already executed the Set expire time to that key and it is now about to run the committing of the transaction?
In that case, the key will be available in the redis store, but the transaction has not been committed.
This will result in a situation where the service is sure that the needed changes have already happen, but they didn't, the machine failed before it could finish it.
I need to design the API in such a way that if the change to the data or setting of the key and value in redis fail, that they will both roll back.
What is the solution to this problem?
How can I guarantee the atomicity of a changing the needed data in one database, and in the same time setting the key and the needed response in redis, and if any of them fails, rollback them both? (Including in a case that a machine crashes in the middle of the actions)
Please add a code example when answering! I'm using the same technologies as in the article (nodejs, redis, mongo - for the data itself)
Thanks :)
Per the code example you shared in your question, the behavior you want is to make sure there was no crash on the server between the moment where the idempotency key was set into the Redis saying this transaction already happened and the moment when the transaction is, in fact, persisted in your database.
However, when using Redis and another database together you have two independent points of failure, and two actions being executed sequentially in different moments (and even if they are executed asynchronously at the same time there is no guarantee the server won’t crash before any of them completed).
What you can do instead is include in your transaction an insert statement to a table holding relevant information on this request, including the idempotent key. As the ACID properties ensure atomicity, it guarantees either all the statements on the transaction to be executed successfully or none of them, which means your idempotency key will be available in your database if the transaction succeeded.
You can still use Redis as it’s gonna provide faster results than your database.
A code example is provided below, but it might be good to think about how relevant is the failure between insert to Redis and database to your business (could it be treated with another strategy?) to avoid over-engineering.
async function execute(idempotentKey) {
try {
// append to the query statement an insert into executions table.
// this will be persisted with the transaction
query = ```
UPDATE firsttable SET ...;
UPDATE secondtable SET ...;
INSERT INTO executions (idempotent_key, success) VALUES (:idempotent_key, true);
```;
const db = await dbConnection();
await db.beginTransaction();
await db.execute(query);
// we're setting a key on redis with a value: "false".
await redisClient.setAsync(idempotentKey, false, 'EX', process.env.KEY_EXPIRE_TIME);
/*
if server crashes exactly here, idempotent key will be on redis with false as value.
in this case, there are two possibilities: commit to database suceeded or not.
if on next request redis provides a false value, query database to verify if transaction was executed.
*/
await db.commit();
// you can now set key value to true, meaning commit suceeded and you won't need to query database to verify that.
await redis.setAsync(idempotentKey, true);
} catch (err) {
await db.rollback();
throw err;
}
}
I have a PUT request that is too long to run. I'd like to make it async, using continuations (await/promise feature).
I create a job (LongJobThatUpdatesThePassedEntity) that modifies my entity
public static void myLongPut(#required Long id, String someData) {
MyJpaModel myJpaModel = MyJpaModel.findById(id);
//straightforward modifications
updateMyJpaModel(someData);
myJpaModel.save();
//long processing modifications to entity, involving WS calls
Promise<String> delayedResult = new LongJobThatUpdatesThePassedEntity(id).now();
await(delayedResult);
render(myJpaModel.refresh());
}
How are the DB transactions managed?
is there a commit before the job's call?
the job has it's own DB transaction?
if there is an issue in the LongJobThatUpdatesThePassedEntity that rollsback, the modifications done in updateMyJpaModel are persisted?
can I do render(myJpaModel.refresh()) at the end?
will it contain the straighforward modifications and the long ones?
thank's
I can answer most of your question for Play 1.4.3, which is the version I'm currently using. I don't expect that much has changed since Play 1.2.
How are the DB transactions managed?
Play! handles the transactions for jobs and controller actions using an "invocation", which is a Play-specific concept. In short, for any invocation, each plugin gets a chance to do some setup and cleanup before and after the invoked method runs. For database access, the JPAPlugin.withinFilter method starts and closes the transaction using the JPA class's helper methods.
is there a commit before the job's call?
When you call await(Future<T>), it has the effect of closing the current transaction and starting a new one. The specific mechanism is that it throws a "Suspend" exception, which bubbles up to PlayHandler$NettyInvocation and causes the afterInvocation callbacks to be called. This causes JPAPlugin.afterInvocation to call
JPA.closeTx() which either commits or rollsback the transaction, as appropriate.
When the Job exits and the await() continuation is resumed. This is also handled as an invocation, so the transaction is started in the same way as before, using JPAPlugin.withinFilter(). However, unlike before, the controller action is not the target of the invocation, but instead ActionInvoker.invoke() calls invokeWithContinuation, which restores the saved continuation state and resumes execution by returning from await().
JPA.withTransaction looks like it has some special logic to retain the same entity manager across the continuation suspend/resume. I think without this, you wouldn't be able to call refresh().
In your code, I think there's a race condition between when await() closes the transaction and the Job starts its transaction. That is, it's possible that the Job's transaction begins before the controller commits the "before await" transaction. To avoid this, you can explicitly call JPA.closeTx() before calling Job.now().
Based on code inspection, it looks like the way Play! is implemented, it so happens that the Job will exit and the Job's transaction will be closed before the "after await()" transaction is opened. I don't know if any
documentation that says this is an intended part of the await() contract, so if this is essential for your appliaction, you can avoid using undocumented behavior by committing the transaction just before your Job.doJobWithResult() method returns.
the job has it's own DB transaction?
Yes, unless its annotated to not have a transaction.
if there is an issue in the LongJobThatUpdatesThePassedEntity that rollsback, the modifications done in updateMyJpaModel are persisted?
Based on the explanation above, each of the three transactions are independent. If one is rolled back, I don't see how it would affect the others.
I'm trying to understand how a java (client) application that communicates, through JDBC, with a pgSQL database (server) can "catch" the result produced by a query that will be fired (using a trigger) whenever a record is inserted into a table.
So, to clarify, via JDBC I install a trigger procedure prepared to execute a query whenever a record is inserted into a given database table, and from this query's execution will result an output (wrapped in a resultSet, I suppose). And my problem is that I have no idea how the client will be aware of those results, that are asynchronously produced.
I wonder if JDBC supports any "callback" mechanism able to catch the results produced by a query that is fired through a trigger procedure under the "INSERT INTO table" condition. And if there is no such "callback" mechanism, what is the best approach to achieve this result?
Thank you in advance :)
Triggers can't return a resultset.
There's no way to send such a result to the JDBC driver.
There are a few dirty hacks you can use to get results from a trigger to the client, but they're all exactly that. Things like:
DECLARE a cursor for the resultset, then send the cursor name as a NOTIFY payload, so the app can FETCH ALL FROM <cursorname>;
Create a TEMPORARY table and report the name via NOTIFY
It is more typical to append anything the trigger needs to communicate to the app to a table that exists for that purpose and have the app SELECT from it after the operation that fired the trigger ran.
In most cases if you need to do this, you're probably using a trigger where a regular function is a better fit.
Can anybody please tell me how to handle a transaction rollback in squeryl explicitly?
And also how can we add or remove columns in squeryl dynamically?
Thanx...
Just to elaborate a bit on the response from #didierd. There is one Session/Connection bound to each transaction. You can access the current Session, and thereby the Connection with code like:
Session.currentSession.connection
Or, if you're not sure if you're within a transaction
Session.currentSessionOption map {_.connection}
If you do roll back the transaction this way it will be your responsibility to start a new one or make sure there is no further use of the connection, so use with care.
You have an access to the JDBC's java.sql.Connection (connection in Session), so if you really cannot use transaction / inTransaction, you can call rollback there.
With access to the connection, you can also execute arbitrary SQL requests and so change the database schema, but be mindful that your squeryl-using code has a static, compile time known schema.
HI,
Iam trying to insert batch of records at a time when any of the record fails to insert i need to trap that record and log that to my failed record maintanance table and then the insert should continue. Kindly help on how to do this.
If using a Spring or EJB container there is a simple trick which works very well : provide a LogService witn a logWarning(String message) method. The method must be annotated/configured with the REQUIRES_NEW transaction setting.
If not then you'll have to simulate it using API calls. Open a different connection for the logging, when you enter the method begin the transaction, before leaving commit the transaction.
When not using transactions for the insert, there is actually nothing special you need to do, as by default most database run in autocommit and commit after every statement.