I am facing this issue with mongodb.
My code is something like this
for(loop) {
var cursorQuery = db.beacon_0000.aggregate([
{
$match: {
...
}
},
{
$project: {
...
}
},
{
$group: {
...
}
},
{
$sort: {
...
}
}
], {allowDiskUse: true} );
...
while(cursorQuery.hasNext()) {
var cursor = cursorQuery.next();
...
}
}
I run the above query via command and mongo shell as
$ mongo dbName file.js
After a while I get the cursor didn't exist on server error at line cursorQuery.hasNext().
In find query if I get this error, I can resolve by adding addOption(DBQuery.Option.noTimeout)
However this option does not seem to be available with aggregate
Please let me know how can I resolve or workaround this issue.
Just to provide additional update:
When say I use
var cursor = db.collection..aggregate([ ...], {allowDiskUse: true} ).addOption(DBQuery.Option.noTimeout)
I get this error
E QUERY TypeError: Object # has no method 'addOption'
However when say I use
var cursor = db.collection..find({...}, {...}).addOption(DBQuery.Option.noTimeout)
It works fine.
Checking the aggregate doc
https://docs.mongodb.com/v3.0/reference/method/db.collection.aggregate/
It says:
Returns:A cursor to the documents produced by the final stage of the aggregation pipeline operation
And then checking cursor doc
https://docs.mongodb.com/v3.0/reference/method/cursor.addOption/#cursor.addOption
There is no suggestion that aggregate cursor is different from find cursor and former does not support DBQuery.Option.noTimeout.
So is there a bug at mongodb for this. Any way to fix it or have a workaround.
Note mongodb version is 3.0
I had the same issue and solved it by setting the idle cursor timeout from default 10 minutes to 1 hour. This is configurable since mongodb 2.6.9. See:
https://jira.mongodb.org/browse/SERVER-8188
https://docs.mongodb.com/manual/reference/parameters/#param.cursorTimeoutMillis
The default cursor timeout is 600000 ms = 10 minutes. You can alter it in different ways:
on startup: mongod --setParameter cursorTimeoutMillis=<num>
or: mongos --setParameter cursorTimeoutMillis=<num>
or during operation, using the mongo shell: db.adminCommand({setParameter:1, cursorTimeoutMillis: <num>})
Mongos is not transferring the command to its mongod's of the cluster. Also the Primary does not replicate the command to its replicaSet members. Thus, you need to execute the command on every mongos and mongod where the query might run.
You have sort of answered this yourself.
Adding the option addOption(DBQuery.Option.noTimeout) will indeed fix the issue when using find because it stops the cursor from timing out and therefore it will exist when you try .hasNext()
However the cursor for aggregation does not have that option so you can't stop it from timing out unfortunately.
you can actually use : maxTimeMS
as it described in documentation:
Optional. Specifies a time limit in milliseconds for processing operations on a cursor. If you do not specify a value for maxTimeMS, operations will not time out. A value of 0 explicitly specifies the default unbounded behavior.
There is an option in mongodb documentation that you can set it no a non-negative number for the time that you want your cursor to be alive.
you can see the more detail on : documentation
Related
I work on debian 10(buster) with mongod ver. 4.2.17. I want to use TTL to remove older documents on collection. I do walk around and I'm sure that no typo on the query.
I followed dummy example offical mongo on this link, where create index on timestamp and expireAfterSeconds.
db.log_events.createIndex( { "createdAt": 1 }, { expireAfterSeconds: 30 } )
I Also added documents on collection with this
db.log_events.insert( {
"createdAt": new Date(),
"logEvent": 2,
"logMessage": "Success!"
} )
but still not work. I do read that the logs not immediately on this but after wait for some hours its still not deleted. Also on mongo logs, there's no TTL log appear. I do checking the indexes with db.log_events.getIndexes() and they do appear.
Any suggestion how to solve it?
I trying to find whats wrong with the mongo, then I tried to activate TTL thread with this reference
db.adminCommand({setParameter: 1, ttlMonitorEnabled: true});
Then I tried to using this suggestion command to check whether the thread are running or not with db.serverStatus().metrics.ttl and return as
{ "deletedDocuments": NumberLong(0), "passes": NumberLong(0) }
also using the db.adminCommand ({getParameter: 1, ttlMonitorEnabled: true}) and return was: true.
It's solved by re-install the mongo on the machine and when check the serverStatus() again it works like a charm. Still dont know whats the problem but reinstall fixing my issue.
Since version 3.6 MongoDB requires the use of cursor or explain in aggregate queries. It's a breaking change so I have to modify my earlier code.
But when I add cursor or explain to my query, the request simply enters an endless loop and MongoDB never responds. It doesn't even seem to time out.
This simple aggregation just hangs the code and never responds:
db.collection('users').aggregate([{ $match: { username: 'admin' }}],
{ cursor: {} },
(err, docs) => {
console.log('Aggregation completed.');
});
I can replace { cursor: {} } with { explain: true } and the result is the same. It works perfectly under older MongoDB versions without this one parameter.
Without cursor or explain I get this error message:
The 'cursor' option is required, except for aggregate with the explain argument
I'm not the only one who ran into this:
https://github.com/nosqlclient/nosqlclient/issues/419
OK, this was a little tricky, but finally it works. Looks like there are some major breaking changes in MongoDB's Node.js driver which nobody bothered to tell me.
1. The Node.js MongoDB driver has to be upgraded. My current version is 3.0.7.
2. The way how MongoDB connects has been changed, breaking any old code. The client connection now returns a client object, not merely the db. It has to be handled differently. There is a SO answer explaining it perfectly:
db.collection is not a function when using MongoClient v3.0
3. Aggregations now return an AggregationCursor object, not an array of data. Instead of a callback now you have to iterate through it.
var cursor = collection.aggregate([ ... ],
{ cursor: { batchSize: 1 } });
cursor.each((err, docs) => {
...
});
So it seems you have to rewrite ALL your db operations after upgrading to MongoDB 3.6. Yay, thanks for all the extra work, MongoDB Team! Guess that's where I'm done with your product.
The most common thing I do in the mongo DB shell is find objects by ID, eg:
db.collection.find({_id: ObjectId("55a3e051dc75954f0f37c2f2"})
I do this over and over and I find having to wrap the id with ObjectId over and over gets old. I wish I had a findById-like shorthand form like what mongoose provides. I feel like the shell ought to be smart enough to figure out what I mean here for example:
db.collection.find("55a3e051dc75954f0f37c2f2")
How might I do this? Or are there any alternative ways of querying by id in the mongo shell?
Fortunately, you can extend the shell quite easily, for example by adding the following method to the ~/.mongorc.js file which is executed when you start the mongo client:
DBCollection.prototype.findById = function(id) {
return db.getCollection(this._shortName).find( { "_id" : ObjectId(id) } );
}
Then you can execute something like db.collection.findById("55a3e051dc75954f0f37c2f2")
The shorthand for find({_id: ObjectId("...")}) is find(ObjectId("...")).
Due to error in client code, mongodb have created many "mr.mapreduce...." collections, how to remove them all (by mask maybe).
I run script in interactive shell:
function f() {
var names = db.getCollectionNames();
for(var i = 0; i < names.length; i++){
if(names[i].indexOf("mr.") == 0){
db[names[i]].drop();}}};
f();
It resolved my problem.
Temporary map-reduce table should be cleaned up when the connection which created them is closed:
map/reduce is invoked via a database
command. The database creates a
temporary collection to hold output of
the operation. The collection is
cleaned up when the client connection
closes, or when explicitly dropped.
Alternatively, one can specify a
permanent output collection name. map
and reduce functions are written in
JavaScript and execute on the server.
-- MongoDB docs
If not, you could delete them using them same method you would delete any other collection. It might get a bit repetitive though.
Another way to achieve the same thing is this snippet:
db.system.namespaces.find({name: /tmp.mr/}).forEach(function(z) {
try{
db.getMongo().getCollection( z.name ).drop();
} catch(err) {}
});
Pro: It won't try to collect all your namespaces into a JavaScript Array. MongoDB segfaults on too many namespaces.
Temporary map-reduce collections should be cleaned up when the connection which created them is closed. However sometimes they remains there and increase database size. You can remove them using the following script:
var database = db.getSiblingDB('yourDatabaseName');
var tmpCollections = database.getCollectionInfos(
{
name: {$regex: /tmp\.mr*/},
'options.temp': true,
});
tmpCollections.forEach(function (collection) {
database.getCollection(collection.name).drop();
});
print(`There was ${tmpCollections.length} tmp collection deleted`);
The drop.tmp.js script can be executed from command line as follow:
mongo --quiet mongodb://mongodb:27137 drop.tmp.js
db.foo.find().limit(300)
won't do it. It still prints out only 20 documents.
db.foo.find().toArray()
db.foo.find().forEach(printjson)
will both print out very expanded view of each document instead of the 1-line version for find():
DBQuery.shellBatchSize = 300
MongoDB Docs - Configure the mongo Shell - Change the mongo Shell Batch Size
From the shell you can use:
db.collection.find().toArray()
to display all documents without having to use it.
You can use it inside of the shell to iterate over the next 20 results. Just type it if you see "has more" and you will see the next 20 items.
Could always do:
db.foo.find().forEach(function(f){print(tojson(f, '', true));});
To get that compact view.
Also, I find it very useful to limit the fields returned by the find so:
db.foo.find({},{name:1}).forEach(function(f){print(tojson(f, '', true));});
which would return only the _id and name field from foo.
With newer version of mongo shell (mongosh) use following syntax:
config.set("displayBatchSize", 300)
instead of depreciated:
DBQuery.shellBatchSize = 300
Future find() or aggregate() operations will only return 300 documents per cursor iteration.
I suggest you to have a ~/.mongorc.js file so you do not have to set the default size everytime.
# execute in your terminal
touch ~/.mongorc.js
echo 'DBQuery.shellBatchSize = 100;' > ~/.mongorc.js
# add one more line to always prettyprint the ouput
echo 'DBQuery.prototype._prettyShell = true; ' >> ~/.mongorc.js
To know more about what else you can do, I suggest you to look at this article: http://mo.github.io/2017/01/22/mongo-db-tips-and-tricks.html
In the mongo shell, if the returned cursor is not assigned to a variable using the var keyword, the cursor is automatically iterated to access up to the first 20 documents that match the query. You can set the DBQuery.shellBatchSize variable to change the number of automatically iterated documents.
Reference - https://docs.mongodb.com/v3.2/reference/method/db.collection.find/
show dbs
use your database name
in my case, I'm using - use smartbank
then - show collections - just to check the document collections name.
and finally, db. your collection name.find() or find({}) -
show dbs
use smartbank
show collections
db.users.find() or db.users.find({}) or db.users.find({_id: ObjectId("60c8823cbe9c1c21604f642b")}) or db.users.find({}).limit(20)
you can specify _id:ObjectId(write the document id here) to get the single document
or you can specify limit - db.users.find({}).limit(20)