I'm using MongoDB Compass to perform an aggregation on a reasonably large dataset (~2,000,000 documents, 300MB) through an SSH tunnel.
It connects fine and I can query but stages in my pipeline appear to timeout operation exceeded time limit.
I know via the command line you can set the max time in milliseconds for a query using maxTimeMs $maxTimeMS but is it possible to apply this in MongoDB Compass?
I have also looked for connection timeout settings and can't find anything related to that in the GUI.
In Compass 1.19-beta we added an option to set maxTimeMS in the aggregation pipeline builder and in the query bar.
To anyone came here to fix this problem on other tabs.
in filter search click on options when it expands, change MAXTIMEMS to any second you want, the value must be in millisecond.
then you can use FIND, ANALYZE or EXPLAIN depends on the tab you chose.
Related
I want to run aggregation on my large data sets. (It's about 361K documents) and Insert them to another collection.
I getting this error:
I tried to increase Max Time but it has maximum and it's not enough for my data sets. I found https://docs.mongodb.com/manual/reference/method/cursor.noCursorTimeout/ but it seems noCursorTimeout only apply on find not aggregation.
please tell me how I can disable cursor timeout or another solution to do this.
I am no MongoDB expert but will interpret what I know.
MongoDB Aggregation Cursors don't have a mechanism to adjust Batch Size or set Cursor Timeouts.
Therefore there is no direct way to alter this and the timeout of an aggregation query solely depends on the cursorTimeoutMillis parameter of the MongoDB or mongos` instance. Its default timeout value is 10 minutes.
Your only option is to change this value by the below command.
use admin
db.runCommand({setParameter:1, cursorTimeoutMillis: 1800000})
However, I strongly advise you against using this command. That's because it's a safety mechanism built into MongoDB. It automatically deletes queries that are running idle for more than 10 minutes, so that there is a lesser load in the MongoDB server. If you change this parameter (say to 30 minutes), MongoDB will allow idle queries to be running in the background for those 30 minutes, which will not only make all the new queries slower to execute, but also increase load and memory on the MongoDB side.
You have a couple of workarounds. Reduct the amount of documents if working on MongoDB Compass or copy and run the commands on Mongo Shell (I had success so far with this method).
I am having trouble trying to call {allowDiskUse:true} in the mongodb compass GUI tool. I have created a view based on an aggregation of another collection. The view returns an error of
Sort exceeded memory limit of 104857600 bytes, but did not opt in to external sorting.
Hence i saw that it is required to append {allowDiskUse:true} but i am unable to find a suitable place to call. There is a possibility to use the $out stage to write to another collection but I would like to try the view first :)
ADD ON:
I have tried to run the query db.noDups.aggregate([],{allowDiskUse : true }); in command line and it works. But I would like to execute in MongoDB compass for the visualization and exporting function.
I also tried {},{},{allowDiskUse: true} in the filter condition but still no luck :(
Btw I am on MongoDB 4.2.6 Community and MongoDB compass 1.25.0
I tried appending it to the filter and it didn't work. I've looked on many different forums and kind find a solution for allowDiskUse from Compass. This seems kind of crazy that you need to add such a kludgy option to even do modest groupings on small amounts of data.
I also have been looking to see how to increase the amount of memory that Mongo can use to get around having to do this. I have Mongo installed on a server with 512GB of memory, it seems rather silly to have developers jumping through hoops like this.
I want to kill all the queries that are taking too long to execute. I have tried socketTimeOut but it is not the good way to do the same and the expected result is not achieved.
Read this for explanation http://mongodb.github.io/node-mongodb-native/2.2/reference/faq/. There is also an option of maxTimeMS, but we cannot make it general and pass it in mongoClient URI options. It is query specific. Is there any other parameter that I can set for the terminating all the queries that take more time to execute.
Any settings in mongo config can also help, but it should work for all the types of queries like insert,find,update, remove. I have also seen db.currentOp() option, but I want to know if it can be done in a better way and if db.currentOp() is the only option, how can I use it to find and terminate a query with Java Driver.
Thank you
I have gone through all the documentations, there is no way we can implement it.
The thing is maxTimeMS is implemented for each query but not on the Global URI level.
How could we make MongoDB report errors for queries that don't use indices?
We end up creating indices for every query anyway so it would be great if MongoDB would report missing indices for us. Also it would be convenient to be able to configure the restriction on a connection basis. This way indices wouldn't come into our way when working from MongoDB shell.
The notablescan ( http://docs.mongodb.org/manual/reference/parameters/#param.notablescan ) option for the MongoDB binary (mongod.exe or mongod depending on your OS) allows you to stop any query, with an emitted log error, which does not use an index at all.
This option will not stop inefficient queries though so that part will still need to be discovered manually by you.
I want to log my query times for each time when a query made.
I'm using mongodb on playframework. Simply I'm thinkinig to substract start and end time.For ex:
a=currenttime();
madequert();
querytime=currenttime()-a;
Is there any better way?
You probably want to use Mongo's DB profiler. That way you'll keep that of your code (less work to maintain it) and it will give you more options to check Mongo behaviour.