The following code create 2 documents in 2 different collections. Unless I'm misunderstanding, 2 things aren't happening properly. The saves are happening serially when I thought it should only happen after the transaction completes. And nothing rolls back upon error. I see the documents in mongo still. What am I doing wrong?
const session = await dbConn.startSession();
let issue;
await dbConn.transaction(async () => {
issue = await IssueModel.create([req.body], { session: session });
await IssueHistoryModel.create([req.body], { session: session });
console.log("test");
throw new Error("oops");
});
Related
Imagine the following scenario:
Start a session
Start a transaction for that session
run an read on Document A
a different session made an update on Document A (During execution)
write Document B based on the original read of Document A
Commit the transaction
End the session
Will the update on Document A be atomic between read and write, or is there a concurrency problem? I understand transaction does a snapshot of all write operations but not sure what happens on the reading side.
await session.withTransaction(async () => {
const coll1 = client.db('mydb1').collection('foo');
const coll2 = client.db('mydb2').collection('bar');
const docA = await coll1.findOne({ abc: 1 }, { session });
// docA is deleted by other session on this point
if (docA){
//Does this runs on an outdated condition?
await coll2.insertOne({ xyz: 999 }, { session });
}
}, transactionOptions)
update.js
import Task from "../models/Task.js";
import UserScreening from "../models/UserScreening.js";
const session = await mongoose.startSession();
session.startTransaction();
['_id1','_id2','_id3'].forEach(async (r) => {
await Task.findByIdAndUpdate(
r,
{ $push: { UserScreening: userscreening, user: existingUser } },
{ session }
);
});
userscreening = await UserScreening.update({},{}{ session });
session.commitTransaction();
here in the above line of code, i want to update multiple collections. So that I'm using session. And session is mandatory for me. Because i want all operations to be successfully completed or failed. But I don't know the proper aproach for array of _ids. SO that i'm facing error in interating. Getting error as 'MongoServerError: WriteConflict error: this operation conflicted with another operation. Please retry your operation or multi-document transaction.'
I'm testing out a trigger on MongoDB atlas which runs a Realm function for adding an object to Algolia index upon insertion to the MongoDB collection. In my case the record gets uploaded to Algolia index successfully but the function doesn't stop there and happens to exceed the time limit.
The docs mention that
Function runtime is limited to 120 seconds
and that's the reason for the function to timeout
Here is my Realm function
exports = function(changeEvent) {
const algoliasearch = require('algoliasearch');
const client = algoliasearch(context.values.get('algolia_app'),context.values.get('algolia_key'));
const index = client.initIndex("movies");
changeEvent.fullDocument.objectID = changeEvent.fullDocument._id;
delete changeEvent.fullDocument._id;
index.saveObject(changeEvent.fullDocument)
.then(({objectID}) => {
console.log('successfully inserted: ',objectID);
})
.catch(err => {
console.log(err);
});
};
Here is the result I get on the logs
Logs:
[
"successfully inserted: 61cf0a79c577393620dd8c80"
]
Error:
execution time limit exceeded
I even tried with return statements after the console.logs but still the same issue.
What I'm I doing wrong
Apparently this was fixed by MongoDB team early this March as seen by https://www.mongodb.com/community/forums/t/extremely-slow-execution-of-an-external-dependency-function/16919/27.
I tested with this code below and it worked perfect without any timeouts this time.
I made the function to be an async function. According to the logs it didn't even take 1 second to perform the indexing.
exports = async function(changeEvent) {
const algoliasearch = require('algoliasearch');
const client = algoliasearch(context.values.get('algolia_app'),context.values.get('algolia_key'));
const index = client.initIndex("movies");
changeEvent.fullDocument.objectID = changeEvent.fullDocument._id;
delete changeEvent.fullDocument._id;
try{
const result = await index.saveObject(changeEvent.fullDocument);
console.log(Date.now(),'successfully updated: ',result);
}
catch(e){
console.error(e);
}
}
Logs
I am trying to export all the documents from a collection (which is about 12 MB) using a Meteor method but it is almost always crashing the app or never returning the results.
I am considering to upload the documents to S3 then sending a download link to the client, however it seems like having an unnecessary network connections and will make the process even longer.
Is there a better way to get large sets of data from server to client?
here is the example of that code, it is very simple.
'downloadUserActions': () => {
if (Roles.userIsInRole(Meteor.userId(), ['admin'])) {
const userData = userActions.find({}).fetch();
return userData
}
}
Thanks.
You can use an approach, where you split the requests into multiple ones:
get the document count
until document count is completely fetched
get the current count of already fetched docs
fetch the next bunch of docs and skip already fetched ones
For this you need the skip option in the mongo query in order to skip the already fetched docs.
Code example
const limit = 250
Meteor.methods({
// get the max amount of docs
getCount () {
return userActions.find().count()
},
// get the next block of docs
// from: skip to: skip + limit
// example: skip = 1000, limit = 500 is
// from: 1000 to: 1500
downloadUserActions (skip) {
this.unblock()
return userActions.find({}, { skip, limit }).fetch()
}
})
Client:
// wrap the Meteor.call into a promise
const asyncCall = (name, args) => new Promise((resolve, reject) => {
Meteor.call(name, args, (err, res) => {
if (err) {
return reject(err)
}
return resolve(res)
})
})
const asyncTimeout = ms => new Promise(resolve => setTimeout(() => resolve(), ms)
const fetchAllDocs = async (destination) => {
const maxDocs = await asyncCall('getCount')
let loadedDocs = 0
while (loadedDocs < maxDocs) {
const docs = await asyncCall('downloadUserActions', loadedDocs)
docs.forEach(doc => {
// think about using upsert to fix multiple docs issues
destination.insert(doc)
})
// increase counter (skip value)
loadedDocs = destination.find().count()
// wait 10ms for next request, increase if server needs
// more time
await asyncTimeout(10)
}
return destination
}
Use it with a local Mongo Collection on the client:
await fetchAllDocs(new Mongo.Collection(null))
After the function all docs are now stored in this local collection.
Play with the limit and the timeout (miliseconds) values in order to find a sweet-spot between user-experience and server-performance.
Additional improvements
The code does not authenticate or validate requests. This is up to you!
Aölso you might think about adding a failsafe-machanism in case the while loop never completes due to some unintended errors.
Further readings
https://docs.meteor.com/api/methods.html#DDPCommon-MethodInvocation-unblock
https://docs.meteor.com/api/collections.html#Mongo-Collection
https://docs.meteor.com/api/collections.html#Mongo-Collection-find
All mongodb transaction examples I have seen so far don't allow reading results back. For example (pseudo code):
begin_transaction
collection_one.update(...)
collection_two.update(...)
commit_transaction
The problem is, what if I want to update collection_two based on the result of updating collection_one?
For example?
begin_transaction
result = collection_one.update(...)
if (result.update_count() > 0)
{
collection_two.update(...)
}
commit_transaction
I have never seen an example like the above? It seems that when use transaction, I can't get the result back.
Another example,
begin_transaction
result = collection_children.find({name: 'xxx'})
collection_team.delete({name in result})
commit_transaction
Basically, I want to perform a find on a collection, and based the find result to perform a second action on a different collection.
And I want the 2 actions together be atomic.
Here is an example of how this works as expected with Mongoose. The same example obviously is possible without Mongoose.
var author = new Author({ email: "test22#test.com", other: [{a: 1}});
var book = new Book({ title: 'ABC' })
let doFoo = async () => {
const session = await mongoose.startSession();
session.startTransaction();
try {
const opts = { session, new: true };
let _author = await author.save() // Notice we get back the record
let _book = await book.save()
// Notice we are using now the new id of the just saved record
await Author.findOneAndUpdate({ _id: _author.id }, { $set: { other: { foo: 2 } }}, opts);
await Book.findOneAndUpdate({ _id: _book.id }, { $set: { title: "ABC" }}, opts);
await session.commitTransaction();
session.endSession();
} catch (error) {
await session.abortTransaction();
session.endSession();
throw error; // Rethrow so calling function sees error
}
}
doFoo()
So in the example above we create/save two different records in their respective collections and after that based on the new records we go back and update them.