PG-Promise - transaction logic for two databases - pg-promise

How can I distribute transactions between two different DBs with PG-Promise? Commits or rollbacks should be applied to both DBs and if one fails, the other should revert changes
I have been using something like this but I am not sure if it works:
try {
await firstDb.tx(async (firstDbTask) => {
await secondDb.tx(async (secondDbTask) => {
// second db stuff
})
// first db stuff
});
return true;
} catch (err) {
return err;
}

Synchronizing transaction logic for multiple databases isn't trivial, but doable. And the solution is basically all about correct use of promises, not much else...
We make our multi-database transactions expose their final inner state via an external promise, so we can cross-align transaction outcome to that state:
let firstTx, secondTx;
firstTx = new Promise((resolve, reject) => {
firstDb.tx(async t => {
// do your queries here first...
// success, signal after all the queries:
resolve(/*null or whatever data*/);
// Align COMMIT-ROLLBACK logic with the other transaction:
await secondTx;
}).catch(reject);
});
secondTx = new Promise((resolve, reject) => {
secondDb.tx(async t => {
// do your queries here first...
// success, signal at the end:
resolve(/*null or whatever data*/);
// Align COMMIT-ROLLBACK logic with the other transaction:
await firstTx;
}).catch(reject);
});
await Promise.all([firstTx, secondTx]); // finish transactions logic
This approach ensures that if either of the transactions fails, both will roll back. And COMMIT can only happen either for both transactions or none.
Note however, that the solution above is a little loose in relation to the state of the transactions, i.e. after we call Promise.all there, both transactions have finished executing their logic, but the resulting COMMIT / ROLLBACK haven't finished executing yet.
In case you need full closure on both transactions, you have to follow up with a separate await for the actual transactions to end, as shown below:
let firstTx, secondTx, tx1, tx2;
firstTx = new Promise((resolve, reject) => {
tx1 = firstDb.tx(async t => {
// do your queries here first...
// success, signal after all the queries:
resolve(/*null or whatever data*/);
// Align COMMIT-ROLLBACK logic with the other transaction:
await secondTx;
}).catch(reject);
});
secondTx = new Promise((resolve, reject) => {
tx2 = secondDb.tx(async t => {
// do your queries here first...
// success, signal at the end:
resolve(/*null or whatever data*/);
// Align COMMIT-ROLLBACK logic with the other transaction:
await firstTx;
}).catch(reject);
});
await Promise.all([firstTx, secondTx]); // finish transactions logic
await Promise.all([tx1, tx2]); // finish transactions execution
// All COMMIT-s or ROLLBACK-s have been executed.
Note that the above provision only matters when both transactions succeed with a COMMIT. When either one fails, the first await will throw, so the second one won't execute, which is important because in case of failure, tx1 or tx2 may still be undefined. That's why we have two separate await-s there in the end.

Related

EF Core ChangeTracker entities snapshot and reset to that snapshot

Is it possible to take a snapshot of the current state of EF Core's ChangeTracker and then reset later to that snapshot if needed?
Let's say I want to make the following code into reality:
public async Task ExecuteTransactionAsync(DbContext context)
{
var executionStrategy = new SqlServerRetryingExecutionStrategy(context);
var retries = 0;
var snapshot = null;
await executionStrategy.ExecuteAsync(async () =>
{
var txOptions = new TransactionOptions
{
IsolationLevel = IsolationLevel.ReadCommitted
};
using var transaction = new TransactionScope(TransactionScopeOption.RequiresNew, txOptions, TransactionScopeAsyncFlowOption.Enabled)
if (retries > 0 && snapshot != null)
{
// This method in reality doesn't exist.
context.ChangeTracker.ResetToSnapshot(snapshot);
}
// This method in reality doesn't exist.
var snapshot = context.ChangeTracker.TakeSnapshot();
retries += 1;
// Let's imagine that we are doing an insert operation on the context.
context.DoInsert();
await context.SaveChangesAsync(false);
// Let's imagine that we are doing an update operation on the context.
context.DoUpdate();
// Let's say this SaveChanges fails the first time and the transaction will retry.
await context.SaveChangesAsync(false);
transaction.Complete();
}
}
Why would this be useful?
I have found it hard to work with ChangeTracker and retrying transactions. The transaction itself of course works fine, the changes will be rollbacked database side between retries. However, ChangeTracker doesn't seem to have a similar rollback functionality.
There is SaveChanges(false) option which is supposed to be used with transactions in order to preserve the state of ChangeTracker between retries. However, if I had an insert on the first run of the transaction, on the second run (first retry) another entity would be added to the ChangeTracker and then on successful SaveChanges, two entities would get inserted to database, but my expectation from the code was to have one entity inserted.
public async Task ExecuteTransactionAsync(DbContext context)
{
var executionStrategy = new SqlServerRetryingExecutionStrategy(context);
await executionStrategy.ExecuteAsync(async () =>
{
var txOptions = new TransactionOptions
{
IsolationLevel = IsolationLevel.ReadCommitted
};
using var transaction = new TransactionScope(TransactionScopeOption.RequiresNew, txOptions, TransactionScopeAsyncFlowOption.Enabled)
// On the first retry, the ChangeTracker will already have a new entity added.
// DoInsert will add another one and now I will have two new entities
// when my expectation was just one.
context.DoInsert();
await context.SaveChangesAsync(false);
context.DoUpdate();
// Let's say this SaveChanges fails the first time and the transaction will retry.
await context.SaveChangesAsync(false);
transaction.Complete();
}
}
A similar problem would occur when updating an entity with += or -= operators. If, for example, an entity had Price = 10 and .DoUpdate would do Price -= 2 - after the first retry (so after the code wrapped in the transaction would run twice) I would have Price = 6 because it was subtracted twice, when my expectation at the end of the transaction was to have Price = 8.
So it got me thinking that a total reset to some ChangeTracker state in some point of time would be extremely useful.

Sequelize transaction retry doens't work as expected

I don't understand how transaction retry works in sequelize.
I am using managed transaction, though I also tried with unmanaged with same outcome
await sequelize.transaction({ isolationLevel: Sequelize.Transaction.ISOLATION_LEVELS.REPEATABLE_READ}, async (t) => {
user = await User.findOne({
where: { id: authenticatedUser.id },
transaction: t,
lock: t.LOCK.UPDATE,
});
user.activationCodeCreatedAt = new Date();
user.activationCode = activationCode;
await user.save({transaction: t});
});
Now if I run this when the row is already locked, I am getting
DatabaseError [SequelizeDatabaseError]: could not serialize access due to concurrent update
which is normal. This is my retry configuration:
retry: {
match: [
/concurrent update/,
],
max: 5
}
I want at this point sequelize to retry this transaction. But instead I see that right after SELECT... FOR UPDATE it's calling again SELECT... FOR UPDATE. This is causing another error
DatabaseError [SequelizeDatabaseError]: current transaction is aborted, commands ignored until end of transaction block
How to use sequelizes internal retry mechanism to retry the whole transaction?
Manual retry workaround function
Since Sequelize devs simply aren't interested in patching this for some reason after many years, here's my workaround:
async function transactionWithRetry(sequelize, transactionArgs, cb) {
let done = false
while (!done) {
try {
await sequelize.transaction(transactionArgs, cb)
done = true
} catch (e) {
if (
sequelize.options.dialect === 'postgres' &&
e instanceof Sequelize.DatabaseError &&
e.original.code === '40001'
) {
await sequelize.query(`ROLLBACK`)
} else {
// Error that we don't know how to handle.
throw e;
}
}
}
}
Sample usage:
const { Transaction } = require('sequelize');
await transactionWithRetry(sequelize,
{ isolationLevel: Transaction.ISOLATION_LEVELS.SERIALIZABLE },
async t => {
const rows = await sequelize.models.MyInt.findAll({ transaction: t })
await sequelize.models.MyInt.update({ i: newI }, { where: {}, transaction: t })
}
)
The error code 40001 is documented at: https://www.postgresql.org/docs/13/errcodes-appendix.html and it's the only one I've managed to observe so far on Serialization failures: What are the conditions for encountering a serialization failure? Let me know if you find any others that should be auto looped and I'll patch them in.
Here's a full runnable test for it which seems to indicate that it is working fine: https://github.com/cirosantilli/cirosantilli.github.io/blob/dbb2ec61bdee17d42fe7e915823df37c4af2da25/sequelize/parallel_select_and_update.js
Tested on:
"pg": "8.5.1",
"pg-hstore": "2.3.3",
"sequelize": "6.5.1",
PostgreSQL 13.5, Ubuntu 21.10.
Infinite list of related requests
https://github.com/sequelize/sequelize/issues/1478 from 2014. Original issue was MySQL but thread diverged everywhere.
https://github.com/sequelize/sequelize/issues/8294 from 2017. Also asked on Stack Overflow, but got Tumbleweed badge and the question appears to have been auto deleted, can't find it on search. Mentions MySQL. Is a bit of a mess, as it also includes connection errors, which are not clear retries such as PostgreSQL serialization failures.
https://github.com/sequelize/sequelize/issues/12608 mentions Postgres
https://github.com/sequelize/sequelize/issues/13380 by the OP of this question
Meaning of current transaction is aborted, commands ignored until end of transaction block
The error is pretty explicit, but just to clarify to other PostgreSQL newbies: in PostgreSQL, when you get a failure in the middle of a transaction, Postgres just auto-errors any following queries until a ROLLBACK or COMMIT happens and ends the transaction.
The DB client code is then supposed to notice that just re-run the transaction.
These errors are therefore benign, and ideally Sequelize should not raise on them. Those errors are actually expected when using ISOLATION LEVEL SERIALIZABLE and ISOLATION LEVEL REPEATABLE READ, and prevent concurrent errors from happening.
But unfortunately sequelize does raise them just like any other errors, so it is inevitable for our workaround to have a while/try/catch loop.

MongoDB transactions

I am new to MongoDB. I am trying to learn Transactions. I am trying to add a record inside a transaction. I am throwing an error inside transaction. Here is my code
await client.connect()
console.table('.....connected');
const session = client.startSession()
console.log('...session started');
await session.withTransaction(async () => {
console.log('.....Promise started')
const db: Db = client.db('sample_mflix')
movieCollection = db.collection('movies');
movieCollection.insertOne({ abc: 11 })
new Error('error occured')
then((res) => {
console.log('.....inserted')
}).catch(err => {
console.log('...error', err);
}).finally(async () => {
console.log('...session ended')
session.endSession()
})
But even on error throwing record is being saved in database. But it should not. What shall I do make my transaction ACID.
You are not using the created sessions to do write operations. Instead of
movieCollection = db.collection('movies');
Try
movieCollection = session.getDatabase("'sample_mflix'").movies;
Then you have to start the transaction and write data
session.startTransaction();
movieCollection.insertOne({ abc: 11 });
This should now rollback the changes committed, when you call endSession() as it would trigger to abort any open transactions.
Also see: https://docs.mongodb.com/manual/reference/method/Session/ and https://docs.mongodb.com/manual/core/transactions/

How to use entity framework transaction in raw query?

I am using entity framework but doing my operations with raw queries. My operations are like following:
Check if recırd exist with integration_id
Delete record if exit
Insert new record
So I am using transaction
using (var transaction = await _context.Database.BeginTransactionAsync())
{
var isExist = await IsExist(id);
if (isExist)
{
var deleteQuery = "delete from ....";
await _context.Database.ExecuteSqlRawAsync(deleteQuery);
}
var insertQuery = "insert into ...";
await _context.Database.ExecuteSqlRawAsync(insertQuery);
}
if insert operation fails, does deleted record rollback?
UPD: https://learn.microsoft.com/en-us/ef/core/saving/transactions#controlling-transactions
transaction will auto-rollback when disposed if either commands fails
So, my code below may be overkill on the catch side, but Commit is still essential :)
======================
I believe the correct way of using transaction would be following:
using (var transaction = await _context.Database.BeginTransactionAsync())
{
try
{
var isExist = await IsExist(id);
if (isExist)
{
var deleteQuery = "delete from ....";
await _context.Database.ExecuteSqlRawAsync(deleteQuery);
}
var insertQuery = "insert into ...";
await _context.Database.ExecuteSqlRawAsync(insertQuery);
// there we tell DB to finish the transaction,
// mark all changes as permanent and release all locks
transaction.Commit();
}
catch (Exception ex)
{
// there we tell DB to discard all changes
// made by this transaction that may be discarded
transaction.Rollback();
// log error
}
}
But I never used BeginTransaction*Async* personally before.
This method doesn't start transaction on it's own. If you need to execute queries in transaction you need to first call
BeginTransaction(DatabaseFacade, IsolationLevel) or UseTransaction.
Reference
learn.microsoft.com
So in your case it will execute queries in a transaction and roll back all the queries if any of the query failed

PostgreSQL SET runtime variables with typeorm, how to ensure the session is isolated?

I would like to set run time variables for each executed query without using transactions.
for example:
SET app.current_user_id = ${userId};
How can I ensure the session will be isolated and prevent race condition on the DB?
To ensure the session will be isolated, you'll need to work with a specific connection from the pool. In postgres SESSION and CONNECTION are equivalent.
The relevant method of typeORM is createQueryRunner. There is no info about it in the docs but it is documented in the api.
Creates a query runner used for perform queries on a single database
connection. Using query runners you can control your queries to
execute using single database connection and manually control your
database transaction.
Usage example:
const foo = <T>(callback: <T>(em: EntityManager) => Promise<T>): Promise<T> => {
const connection = getConnection();
const queryRunner = connection.createQueryRunner();
return new Promise(async (resolve, reject) => {
let res: T;
try {
await queryRunner.connect();
await queryRunner.manager.query(`SET app.current_user_id = ${userId};`)
res = await callback(queryRunner.manager);
} catch (err) {
reject(err);
} finally {
await queryRunner.manager.query(`RESET app.current_user_id`)
await queryRunner.release();
resolve(res);
}
});
};
This was my answer also for How to add a request timeout in Typeorm/Typescript?