doctine-odm does not support native strategy for multi document transaction support as per this document
However, does anyone have figured out a workaround?
According to mongodb, we need to create a session and pass it to every query in the transaction to take control over full commit rollback. this implementation does the same thing but it does not work. I get error while passing session to the ->flush() method.
Error
The \"writeConcern\" option cannot be specified within a transaction. Instead, specify it when starting the transaction.
Try:
$callback = function (Session $session) { /** your code here */ }
$session = $client->startSession();
with_transaction($session, $callback, [
'readConcern' => new ReadConcern(ReadConcern::MAJORITY),
'writeConcern' => new WriteConcern(WriteConcern::MAJORITY, 1000),
'readPreference' => new ReadPreference(ReadPreference::RP_PRIMARY),
]);
Related
how can I execute varying error messages depending on where a query failed, triggering a rollback, in my transaction?
I'll be using the sample code from the documentation:
db.tx(t => {
// creating a sequence of transaction queries:
const q1 = t.none(query);
const q2 = t.one(query);
const q3 = t.one(query);
// returning a promise that determines a successful transaction:
return t.batch([q1, q2, q3]); // all of the queries are to be resolved;
})
.then(data => {
// success, COMMIT was executed
})
.catch(error => {
// failure, ROLLBACK was executed
});
Preferred output is the following:
if the transaction failed in q1:
res.json({error: true, message:"q1 failed"})
if the transaction failed in q2:
res.json({error: true, message:"q2 failed"})
if the transaction failed in q3:
res.json({error: true, message:"q2 failed"}), etc.
What I'm thinking is using a Switch statement to determine what error message to execute, although I don't have an idea on how to know what query failed in the transaction.
Thank you for your help!
P.S. I recently migrated from node-pg to pg-promise (which is why I'm a bit new with the API) due to having a hard time with transactions as recommended in my previous posts, and yeah, pg-promise made a lot of things easier the 1 day worth of refactoring code is worth it.
Since you are using method batch, you get BatchError thrown when the method fails, which has useful property data, among others:
.catch(err => {
// find index of the first failed query:
const errIdx = err.data.findIndex(e => !e.success);
// do what you want here, based on the index;
});
Note that inside such error handler, err.data[errIdx].result is the same as err.first, representing the first error that occurred.
I am working on a solution where I am using vertx 3.8.4 and vertx-mysql-client 3.9.0 for asynchronous database calls.
Here is the scenario that I have been trying to resolve, in a proper reactive manner.
I have some mastertable records which are in inactive state.
I run a query and get the list of records from the database.
This I did like this :
Future<List<Master>> locationMasters = getInactiveMasterTableRecords ();
locationMasters.onSuccess (locationMasterList -> {
if (locationMasterList.size () > 0) {
uploadTargetingDataForAllInactiveLocations(vertx, amazonS3Utility,
locationMasterList);
}
});
Now in uploadTargetingDataForAllInactiveLocations method, i have a list of items.
What I have to do is, I need to iterate over this list, for each item, I need to download a file from aws, parse the file and insert those data to db.
I understand the way to do it using CompositeFuture.
Can someone from vertx dev community help me with this or with some documentation available ?
I did not find good contents on this by googling.
I'm answering this as I was searching for something similar and I ended up spending some time before finding an answer and hopefully this might be useful to someone else in future.
I believe you want to use CompositeFuture in vertx only if you want to synchronize multiple actions. That means that you either want an action to execute in the case that either all your other actions on which your composite future is built upon succeed or at least one of the action on which your composite future is built upon succeed.
In the first case I would use CompositeFuture.all(List<Future> futures) and in the second case I would use CompositeFuture.any(List<Future> futures).
As per your question, below is a sample code where a list of item, for each item we run an asynchronous operation (namely downloadAnProcessFile()) which returns a Future and we want to execute an action doAction() in the case that all the async actions succeeded:
List<Future> futures = new ArrayList<>();
locationMasterList.forEach(elem -> {
Promise<Void> promise = Promise.promise();
futures.add(promise.future());
Future<Boolean> processStatus = downloadAndProcessFile(); // doesn't need to be boolean
processStatus.onComplete(asyncProcessStatus -> {
if (asyncProcessStatus.succeeded()){
// eventually do stuff with the result
promise.complete();
} else {
promise.fail("Error while processing file whatever");
}
});
});
CompositeFuture.all(futures).onComplete(compositeAsync -> {
if (compositeAsync.succeeded()){
doAction(); // <-- here do what you want to do when all future complete
} else {
// at least 1 future failed
}
});
This solution is probably not perfect and I suppose can be improved but this is what I found works for me. Hopefully will work for someone else.
I need to build a mongo updater process to dowload mongodb data to local IoT devices (configuration data, etc.)
My goal is to watch for some mongo collections in a fixed interval (1 minute, for example). If I have changed a collection (deletion, insertion or update) I will download the full collection to my device. The collections will have no more than a few hundred simple records, so it´s gonna not be a lot of data to download.
Is there any mechanism to find out a collection has changed since last pool ? What mongo features should be used in that case ?
To listen for changes to your MongoDB collection, set up a Mongoose Model.watch.
const PersonModel = require('./models/person')
const personEventEmitter = PersonModel.watch()
personEventEmitter.on('change', change => console.log(JSON.stringify(change)))
const person = new PersonModel({name: 'Thabo'})
person.save()
// Triggers console log on change stream
// {_id: '...', operationType: 'insert', ...}
Note: This functionality is only available on a MongoDB Replicaset
See Mongoose Model Docs for more:
If you want to listen for changes to your DB, use Connection.watch.
See Mongoose Connection Docs for more
These functions listen for Change Events from MongoDB Change Streams as of v3.6
I think best solution would be using post update middleware.
You can read more about that here
http://mongoosejs.com/docs/middleware.html
I have the same demand on an embedded that works quite autonomously, and it is always necessary to auto adjust your operating parameters without having to reboot your system.
For this I created a configuration manager class, and in its constructor I coded a "parameter monitor", which checks the database only the parameters that are flagged for it, of course if a new configuration needs to be monitored, I inform the config -manager in another part of the code to reload such an update.
As you can see the process is very simple, and of course can be improved to avoid overloading the config-manager with many updates and also prevent them from overlapping with a very small interval.
Since there are many settings to be read, I open a cursor for a query as soon as the database is connected and opened. As data streaming sends me new data, I create a proxy for it so that it can be manipulated according to the type and internal details of Config-manager. I then check if the property needs to be monitored, if so, I call an inner-function called watch that I created to handle this, and it queries the subproject of the same name to see what default time it takes to check in the database by updates, and thus registers a timeout for that task, and each check recreates the timeout with the updated time or interrupts the update if watch no longer exists.
this.connection.once('open', () => {
let cursor = Config.find({}).cursor();
cursor.on('data', (doc) => {
this.config[doc.parametro] = criarProxy(doc.parametro, doc.valor);
if (doc.watch) {
console.log(sprintf("Preparando para Monitorar %s", doc.parametro));
function watch(configManager, doc) {
console.log("Monitorando parametro: %s", doc.parametro);
if (doc.watch) setTimeout(() => {
Config.findOne({
parametro: doc.parametro
}).then((doc) => {
console.dir(doc);
if (doc) {
if (doc.valor != configManager.config[doc.parametro]) {
console.log("Parametro monitorado: %(parametro)s, foi alterado!", doc);
configManager.config[doc.parametro] = criarProxy(doc.parametro, doc.valor);
} else
console.log("Parametro monitorado %{parametro}s, não foi alterado", doc);
watch(configManager, doc);
} else
console.log("Verifique o parametro: %s")
})
},
doc.watch)
}
watch(this, doc);
}
});
cursor.on('close', () => {
if (process.env.DEBUG_DETAIL > 2) console.log("ConfigManager closed cursor data");
resolv();
});
cursor.on('end', () => {
if (process.env.DEBUG_DETAIL > 2) console.log("ConfigManager end data");
});
As you can see the code can improve a lot, if you want to give suggestions for improvements according to your environment or generics please use the gist: https://gist.github.com/carlosdelfino/929d7918e3d3a6172fdd47a59d25b150
I have never setup up a queueing system before. I decided to give it a shot. It seems the queueing system is working perfectly. However, it doesn't seem the data is being sent correctly. Here is my code.
...
$comment = new Comment(Input::all());
$comment->user_id = $user->id;
$comment->save();
if ($comment->isSaved())
{
$voters = $comment->argument->voters->unique()->toArray();
Queue::push('Queues\NewComment',
[
'comment' => $comment->load('argument', 'user')->toArray(),
'voters' => $voters
]
);
return Response::json(['success' => true, 'comment' => $comment->load('user')->toArray()]);
}
...
The class that handles this looks like this:
class NewComment {
public function fire($job, $data)
{
$comment = $data['comment'];
$voters = $data['voters'];
Log::info($data);
foreach ($voters as $voter)
{
if ($voter['id'] != $comment['user_id'])
{
$mailer = new NewCommentMailer($voter, $comment);
$mailer->send();
}
}
$job->delete();
}
}
This works beautifully on my local server using the synchronous queue driver. However, on my production server, I'm using Beanstalkd. The queue is firing like it is supposed to. However, I'm getting an error like this:
[2013-12-19 10:25:02] production.ERROR: exception 'ErrorException' with message 'Undefined index: voters' in /var/www/mywebsite/app/queues/NewComment.php:10
If I print out the $data variable passed into the NewComment queue handler, I get this:
[2013-12-19 10:28:05] production.INFO: {"comment":{"incrementing":true,"timestamps":true,"exists":true}} [] []
I have no clue why this is happening. Anyone have an idea how to fix this.
So $voters apparently isn't being put into the queue as part of the payload. I'd build the payload array outside of the Queue::push() function, log the contents, and see exactly what is being put in.
I've found if you aren't getting something out that you expect, chances are, it's not being put in like you expect either.
While you are at it, make sure that the beanstalkd system hasn't got old data stuck in it that is incorrect. You could add a timestamp into the payload to help make sure it's the latest data, and arrange to delete or bury any jobs that don't have the appropriate information - checked before you start to process them. Just looking at a count of items in the beanstalkd tubes should make it plain if there are stuck jobs.
I've not done anything with Laravel, but I have written many tasks for other Beanstalkd and SQS-backed systems, and the hard part is when the job fails, and you have to figure out what went wrong, and how to avoid just redoing the same failure over and over again.
What I ended up doing was sticking with simple numbers. I only stored the comment's ID on the queue and then did all the processing in my queue handler class. That was the easiest way to do it.
You will get data as expected in the handler by wrapping the data in an array:
array(
array('comment' => $comment->load('argument', 'user')->toArray(),
'voters' => $voters
)
)
Our application is build upon mongodb replica set.
I'd like to catch all exceptions thrown among the time frame when replica set is in process of automatic failover.
I will make application retry or wait for failover completes.
So that the failover won't influence user.
I found document describing the behavior of java driver here: https://jira.mongodb.org/browse/DOCS-581
I write a test program to find all possible exceptions, they are all MongoException but with different message:
MongoException.Network: "Read operation to server /10.11.0.121:27017 failed on database test"
MongoException: "can't find a master"
MongoException: "not talking to master and retries used up"
MongoException: "No replica set members available in [ here is replica set status ] for { "mode" : "primary"}"
Maybe more...
I'm confused and not sure if it is safe to determine by error message.
Also I don't want to catch all MongoException.
Any suggestion?
Thanks
I am now of the opinion that Mongo in Java is particularly weak in this regards. I don't think your strategy of interpreting the error codes scales well or will survive driver evolution. This is, of course, opinion.
The good news is that the Mongo driver provides a way get the status of a ReplicaSet: http://api.mongodb.org/java/2.11.1/com/mongodb/ReplicaSetStatus.html. You can use it directly to figure out whether there is a Master visible to your application. If that is all you want to know, the http://api.mongodb.org/java/2.11.1/com/mongodb/Mongo.html#getReplicaSetStatus() is all you need. Grab that kid and check for a not-null master and you are on your way.
ReplicaSetStatus rss = mongo.getReplicaSetStatus();
boolean driverInFailover = rss.getMaster() == null;
If what you really need is to figure out if the ReplSet is dead, read-only, or read-write, this gets more difficult. Here is the code that kind-of works for me. I hate it.
#Override
public ReplSetStatus getReplSetStatus() {
ReplSetStatus rss = ReplSetStatus.DOWN;
MongoClient freshClient = null;
try {
if ( mongo != null ) {
ReplicaSetStatus replicaSetStatus = mongo.getReplicaSetStatus();
if ( replicaSetStatus != null ) {
if ( replicaSetStatus.getMaster() != null ) {
rss = ReplSetStatus.ReadWrite;
} else {
/*
* When mongo.getReplicaSetStatus().getMaster() returns null, it takes a a
* fresh client to assert whether the ReplSet is read-only or completely
* down. I freaking hate this, but take it up with 10gen.
*/
freshClient = new MongoClient( mongo.getAllAddress(), mongo.getMongoClientOptions() );
replicaSetStatus = freshClient.getReplicaSetStatus();
if ( replicaSetStatus != null ) {
rss = replicaSetStatus.getMaster() != null ? ReplSetStatus.ReadWrite : ReplSetStatus.ReadOnly;
} else {
log.warn( "freshClient.getReplicaSetStatus() is null" );
}
}
} else {
log.warn( "mongo.getReplicaSetStatus() returned null" );
}
} else {
throw new IllegalStateException( "mongo is null?!?" );
}
} catch ( Throwable t ) {
log.error( "Ingore unexpected error", t );
} finally {
if ( freshClient != null ) {
freshClient.close();
}
}
log.debug( "getReplSetStatus(): {}", rss );
return rss;
}
I hate it because it doesn't follow the Mongo Java Driver convention of your application only needs a single Mongo and through this singleton you connect to the rest of the Mongo data structures (DB, Collection, etc). I have only been able to observe this working by new'ing up a second Mongo during the check so that I can rely upon the ReplicaSetStatus null check to discriminate between "ReplSet-DOWN" and "read-only".
What is really needed in this driver is some way to ask direct questions of the Mongo to see if the ReplSet can be expected at this moment to support each of the WriteConcerns or ReadPreferences. Something like...
/**
* #return true if current state of Client can support readPreference, false otherwise
*/
boolean mongo.canDoRead( ReadPreference readPreference )
/**
* #return true if current state of Client can support writeConcern; false otherwise
*/
boolean mongo.canDoWrite( WriteConcern writeConcern )
This makes sense to me because it acknowledges the fact that the ReplSet may have been great when the Mongo was created, but conditions right now mean that Read or Write operations of a specific type may fail due to changing conditions.
In any event, maybe http://api.mongodb.org/java/2.11.1/com/mongodb/ReplicaSetStatus.html gets you what you need.
When Mongo is failing over, there are no nodes in a PRIMARY state. You can just get the replica set status via the replSetGetStatus command and look for a master node. If you don't find one, you can assume that the cluster is in a failover transition state, and can retry as desired, checking the replica set status on each failed connection.
I don't know the Java driver implementation itself, but I'd do catch all MongoExceptions, then filter them on getCode() basis. If the error code does not apply to replica sets failures, then I'd rethrow the MongoException.
The problem is, to my knowledge there is no error codes reference in the documentation. Well there is a stub here, but this is fairly incomplete. The only way is to read the code of the Java driver to know what code it uses…