I'm using the Java mongodb driver version 3.8.0 and mongodb 3.6.3.
I created a watch on a collection with this:
MongoCursor<ChangeStreamDocument<Document>> cursor = collection.watch().maxAwaitTime(500, TimeUnit.MILLISECONDS).iterator();
The documentation here states about maxAwaitTime:
The maximum amount of time for the server to wait on new documents to satisfy a change stream query.
However, what I'm seeing is that cursor.hasNext() returns only if there is a change on the collection, not when the time passed to maxAwaitTime has elapsed.
When I turn on mongodb's verbose logging I see maxWaitTime set as expected in the getMore command.
How do I cause my watch to timeout when there are no changes? It seems as though maxAwaitTime should do this. Why doesn't it work?
MongoDB drivers implement change stream as an abstraction of a TAILABLE_AWAIT cursor, and maxAwaitTimeMS is to specify a time limit for a getMore command on on TAILABLE_AWAIT cursor.
The way it works, MongoCursor continues to send getMore command to the server in a loop until either:
A document is found
The cursor is closed
An exception occurs
Only when any of the event above happens, the cursor's method next() or hasNext() will return.
While none of the event above happens, the server's getMore command will continue to be called by the Iterator interface. The maxAwaitTime specifies how long to wait before the getMore command timed out while waiting for documents and returns even if there there are no documents found.
How do I cause my watch to timeout when there are no changes?
If your application requires a time out after maxAwaitTime, the mongo-java-driver offers the tryNext() method on MongoCursor. This method will return null after maxAwaitTime if no documents were found, and can be called repeatedly by the application. See also JAVA-2965.
Related
I want to run aggregation on my large data sets. (It's about 361K documents) and Insert them to another collection.
I getting this error:
I tried to increase Max Time but it has maximum and it's not enough for my data sets. I found https://docs.mongodb.com/manual/reference/method/cursor.noCursorTimeout/ but it seems noCursorTimeout only apply on find not aggregation.
please tell me how I can disable cursor timeout or another solution to do this.
I am no MongoDB expert but will interpret what I know.
MongoDB Aggregation Cursors don't have a mechanism to adjust Batch Size or set Cursor Timeouts.
Therefore there is no direct way to alter this and the timeout of an aggregation query solely depends on the cursorTimeoutMillis parameter of the MongoDB or mongos` instance. Its default timeout value is 10 minutes.
Your only option is to change this value by the below command.
use admin
db.runCommand({setParameter:1, cursorTimeoutMillis: 1800000})
However, I strongly advise you against using this command. That's because it's a safety mechanism built into MongoDB. It automatically deletes queries that are running idle for more than 10 minutes, so that there is a lesser load in the MongoDB server. If you change this parameter (say to 30 minutes), MongoDB will allow idle queries to be running in the background for those 30 minutes, which will not only make all the new queries slower to execute, but also increase load and memory on the MongoDB side.
You have a couple of workarounds. Reduct the amount of documents if working on MongoDB Compass or copy and run the commands on Mongo Shell (I had success so far with this method).
I am running a lookup query using aggregation pipeline.
While iterating over the cursor returned from aggregation execution, I have noticed that program hangs when Mongo sends getMore command. I believe this command is used to fetch next batch of records from DB.
Initially I tested with Batch size of 100, it brings 100 records fairly quickly, but when it tries to fetch next set of records, cursor hangs.
I tried the same with batch size of 2 and got the same result.
I am using Mongo Java Driver version 3.6 and server 3.6.2.
I also found that there was a similar issue raised for python driver in the past(https://jira.mongodb.org/browse/PYTHON-276).
Has anyone experienced this in the past ? Any suggestions would be helpful.
Let me know if any other details are required
We are troubled by eventually occurring cursor not found exceptions for some Morphia Queries asList and I've found a hint on SO, that this might be quite memory consumptive.
Now I'd like to know a bit more about the background: can sombody explain (in English), what a Cursor (in MongoDB) actually is? Why can it kept open or be not found?
The documentation defines a cursor as:
A pointer to the result set of a query. Clients can iterate through a cursor to retrieve results. By default, cursors timeout after 10 minutes of inactivity
But this is not very telling. Maybe it could be helpful to define a batch for query results, because the documentation also states:
The MongoDB server returns the query results in batches. Batch size will not exceed the maximum BSON document size. For most queries, the first batch returns 101 documents or just enough documents to exceed 1 megabyte. Subsequent batch size is 4 megabytes. [...] For queries that include a sort operation without an index, the server must load all the documents in memory to perform the sort before returning any results.
Note: in our queries in question we don't use sort statements at all, but also no limit and offset.
Here's a comparison between toArray() and cursors after a find() in the Node.js MongoDB driver. Common code:
var MongoClient = require('mongodb').MongoClient,
assert = require('assert');
MongoClient.connect('mongodb://localhost:27017/crunchbase', function (err, db) {
assert.equal(err, null);
console.log('Successfully connected to MongoDB.');
const query = { category_code: "biotech" };
// toArray() vs. cursor code goes here
});
Here's the toArray() code that goes in the section above.
db.collection('companies').find(query).toArray(function (err, docs) {
assert.equal(err, null);
assert.notEqual(docs.length, 0);
docs.forEach(doc => {
console.log(`${doc.name} is a ${doc.category_code} company.`);
});
db.close();
});
Per the documentation,
The caller is responsible for making sure that there
is enough memory to store the results.
Here's the cursor-based approach, using the cursor.forEach() method:
const cursor = db.collection('companies').find(query);
cursor.forEach(
function (doc) {
console.log(`${doc.name} is a ${doc.category_code} company.`);
},
function (err) {
assert.equal(err, null);
return db.close();
}
);
});
With the forEach() approach, instead of fetching all data in memory, we're streaming the data to our application. find() creates a cursor immediately because it doesn't actually make a request to the database until we try to use some of the documents it will provide. The point of cursor is to describe our query. The second parameter to cursor.forEach shows what to do when an error occurs.
In the initial version of the above code, it was toArray() which forced the database call. It meant we needed ALL the documents and wanted them to be in an array.
Note that MongoDB returns data in batches. The image below shows requests from cursors (from application) to MongoDB:
forEach scales better than toArray because we can process documents as they come in until we reach the end. Contrast it with toArray - where we wait for ALL the documents to be retrieved and the entire array is built. This means we're not getting any advantage from the fact that the driver and the database system are working together to batch results to your application. Batching is meant to provide efficiency in terms of memory overhead and the execution time. Take advantage of it in your application, if you can.
I am by no mean a mongodb expert but I just want to add some observations from working in a medium sized mongo system for the last year. Also thanks to #xameeramir for the excellent walkthough about how cursors work in general.
The causes of a "cursor lost" exception may be several. One that I have noticed is explained in this answer.
The cursor lives server side. It is not distributed over a replica set but exists on the instance that is primary at the time of creation. This means that if another instance takes over as primary the cursor will be lost to the client. If the old primary is still up and around it may still be there but for no use. I guess it is garbaged collected away after a while. So if your mongo replica set is unstable or you have a shaky network in front of it you are out of luck when doing any long running queries.
If the full content of what the cursor wants to return does not fit in memory on the server the query may be very slow. RAM on your servers needs to be larger than the largest query you run.
All this can partly be avoided by designing better. For a use case with large long running queries you may be better of with several smaller database collections instead of a big one.
The collection's find method returns a cursor - this points to the set of documents (called as result set) that are matched to the query filter. The result set is the actual documents that are returned by the query, but this is on the database server.
To the client program, for example the mongo shell, you get a cursor. You can think the cursor is like an API or a program to work with the result set. The cursor has many methods which can be run to perform some actions on the result set. Some of the methods affect the result set data and some provide the status or info about the result set.
As the cursor maintains information about the result set, some information can change as you use the result set data by applying other cursor methods. You use these methods and information to suit your application, i.e., how and what you want to do with the queried data.
Working on the result set using the cursor and some of its commonly used methods and features from mongo shell:
The count() method returns the count of the number of documents in the result set, initially - as the result of the query. It is always constant at any point in the life of the cursor. This is information. This information remains same even after the cursor is closed or exhausted.
As you read documents from the result set, the result set gets exhausted. Once completely exhausted you cannot read any more. The hasNext() tells if there are any documents available to be read - returns a boolean true or false. The next() returns a document if available (you first check with hasNext, and then do a next). These two methods are commonly used to iterate over the result set data. Another iteration method is the forEach().
The data is retrieved from the server in batches - which has a default size. With the first batch you read the documents and when all it's documents are read, the following next() method retrieves the next batch of documents, etc., until all documents are read from the result set. This batch size can be configured and you can also get its status.
If you apply the toArray() method on the cursor, then all the remaining documents in the result set are loaded into the memory of your client computer and are available as a JavaScript array. And, the result set data is exhausted. The following hasNext method will return false, and the next will throw an error (once you exhaust the cursor, you cannot read data from it). This method loads all the result set data into your client's memory (the array). This can be memory consuming in case of large result sets.
The itcount() returns the count of remaining documents in the result set and exhausts the cursor.
There are cursor methods like isClosed(), isExhausted(), size() which give status information about the cursor and its underlying result set as you work with your data.
Those are the basic features of cursor and result set. There are many cursor methods, and you can try and see how they work and get a better understanding.
Reference:
mongo shell's cursor
methods
Cursor behavior with Aggregate
method
(the collection's aggregate method also returns a cursor)
Example usage in mongo shell:
Assume the test collection has 200 documents (run the commands in the same sequence).
var cur = db.test.find( { } ).limit(25) creates a result set with 25
documents only.
But, cur.count() will show 200, which is the actual count of
documents by the query's filter.
hasNext() will return true.
next() will return a document.
itcount() will return 24 (and exhausts the cursor).
itcount() again will return 0.
cur.count() will still show 200.
This error also comes when you have a large set of data and are doing batch processing on that data and each batch takes more time, totalling that time be exceeded the default cursor live time.
Then you need to change that default time to tell mongo that will not expire this cursor until processing is done.
Do check No TimeOut Documentation
A cursor is an object returned by calling db.collection.find() and which enables iterating through documents (NoSQL equivalent of a SQL "row") of a MongoDB collection (NoSQL equivalent of "table").
In case your cluster is stable and no members where down or changing state, the most posible reason for not finding the cursor is this:
Default idle cursor timeout is 10min , but in the versions >= 3.6 the cursor is also associated with session which is having default session timeout 30min , so even you set the cursor to not expire with the option noCursorTimeout() you are still limited by the session timeout of 30min. To avoid your cursor to be killed by the session timeout you will need to perioducally check in your code and execute sessionRefresh command:
db.adminCommand({"refreshSessions" : [sessionId]})
to extend the session with another 30min so your cursor to not be killed if you do something with the data before fetching the next batch...
check the docs here for detail how to do it:
https://docs.mongodb.com/manual/reference/method/cursor.noCursorTimeout/
I have no knowledge of mongodb and I just want to ask if something is possible and, if possible, how it can be done. My question is how can we know when a cursor will expire. Is there any API for this purpose?
I would be grateful for any comments and recommendations.
Best regards.
From the MongoDB documentation:
By default, MongoDB will automatically close a cursor when the client has exhausted all results in the cursor. However, for capped collections you may use a Tailable Cursor that remains open after the client exhausts the results in the initial cursor.
http://docs.mongodb.org/manual/tutorial/create-tailable-cursor/
Other factors which could cause a cursor to expire are the batchSize and timeout. To sum it up factors which expire the cursor are:
result exhausation
batchSize: http://docs.mongodb.org/manual/reference/method/cursor.batchSize/
timeout: http://api.mongodb.org/java/2.6/com/mongodb/MongoOptions.html
I came across this thread looking for a solution to the following error in with pymongo: CursorNotFound: Cursor not found, cursor id. Obviously the reason is the same as this thread, but there is no solution provided for pymongo.
From the FAQ in the pymongo documentation the following links answer the question: here and here.
The quickest solution is to disable the timeout, so the server does not expire the cursor, and it's always valid.
client = MongoClient("localhost")
db = client.testdatabase
cursor = db.testcollection.find({}, no_cursor_timeout=True)
In the find() documentation you can find alternatives by changing the cursor type:
cursor_type (optional): the type of cursor to return. The valid
options are defined by CursorType:
NON_TAILABLE - the result of this find call will return a standard cursor over the result set.
TAILABLE - the result of this find call will be a tailable cursor - tailable cursors are only for use with capped collections. They are not closed when the last data is retrieved but are kept open and the
cursor location marks the final document position. If more data is
received iteration of the cursor will continue from the last document
received. For details, see the tailable cursor documentation.
TAILABLE_AWAIT - the result of this find call will be a tailable cursor with the await flag set. The server will wait for a few seconds
after returning the full result set so that it can capture and return
additional data added during the query.
EXHAUST - the result of this find call will be an exhaust cursor. MongoDB will stream batched results to the client without waiting for
the client to request each batch, reducing latency. See notes on
compatibility below.
Update: After using the no_cursor_timeout I was wondering what happeneded to the cursor when they are not executed. Here is the answer: PyMongo: What happens to cursor when no_cursor_timeout=True
Ordinarily, a cursor "dies" on the database server after a certain length of time (approximately 10 minutes) and will be closed when the client has exhausted all of the results.
Some drivers have an "immortal" option, which does the equivalent in the Java driver of setting a NoTimeout option:
dbcoll.find(...).addOption(Bytes.QUERYOPTION_NOTIMEOUT)
Set in a manner like above if you intend to keep an open cursor over this period of time.
http://api.mongodb.org/java/current/com/mongodb/Bytes.html#QUERYOPTION_NOTIMEOUT
I have a python script which needs to do some actions whenever a new object is added in a collection
is there any efficient method to poll for addition of new object in mongodb collection?
Have a look at mongodb 'tailable' cursor.
http://www.mongodb.org/display/DOCS/Tailable+Cursors
Use "find" method of your python driver with "tailable" = true,
it will keep realtime track of additions in the database just like "tail -f" of a file in linux .
Tailable is FALSE by default.
http://api.mongodb.org/python/current/api/pymongo/collection.html
find([spec=None[, fields=None[, skip=0[, limit=0[, timeout=True[, snapshot=False[, tailable=False[, sort=None[, max_scan=None[, as_class=None[, slave_okay=False[, **kwargs]]]]]]]]]]]])
tailable (optional): the result of this find call will be a tailable cursor - tailable cursors aren’t closed when the last data is retrieved but are kept open and the cursors location marks the final document’s position. if more data is received iteration of the cursor will continue from the last document received. For details, see the tailable cursor documentation.
use a separate thread to poll the data. It is less efficient but works..
the alternative solution is to use twisted and its async driver but you still need to poll
the data.