I have 2 applications that are using listCollections to get infomation. Both are going through the C client but they are using different versions. In my Mongodb output I can see they are running slighly different statements on the server :-
2019-09-22T04:24:31.707+0000 I COMMAND [conn9] command datalake.$cmd command: listCollections { listCollections: 1, $readPreference: { mode: "secondaryPreferred" }, $db: "test" } numYields:0 reslen:333 locks:{ Global: { acquireCount: { r: 2 } }, Database: { acquireCount: { r: 1 } }, Collection: { acquireCount: { r: 1 } } } protocol:op_query 0ms
2019-09-22T04:26:34.183+0000 I COMMAND [conn12] command datalake.$cmd command: listCollections { listCollections: 1.0, $readPreference: { mode: "secondaryPreferred" }, $db: "test" } numYields:0 reslen:333 locks:{ Global: { acquireCount: { r: 2 } }, Database: { acquireCount: { r: 1 } }, Collection: { acquireCount: { r: 1 } } } protocol:op_query 0ms
The one difference I can see between the two is that one is passing in listCollections: 1.0 and the other passes in listCollections: 1.
Is there any way I can convert the log output above into a db.runCommand() that I can execute to see if I get different results ?
The one difference I can see between the two is that one is passing in listCollections: 1.0 and the other passes in listCollections: 1.
The value parameter for the listCollections command is irrelevant -- it exists to create a key/value pair when constructing a BSON document to pass as a parameter to runCommand but isn't used by listCollections.
All of the following are equivalent (although a numeric value makes more semantic sense):
db.runCommand({'listCollections': 1})
db.runCommand({'listCollections': 1.0})
db.runCommand({'listCollections': 'foo'})
If you are using MongoDB 4.0+, the document format can be useful to provide additional optional fields like filter and nameOnly.
Instead of passing a document as the first parameter for runCommand() in the shell, you can also just use provide the command name to run with the default options:
db.runCommand('listCollections')
Is there any way I can convert the log output above into a db.runCommand() that I can execute to see if I get different results ?
Equivalent commands in the mongo shell to your log output would be:
use datalake
db.getMongo().setReadPref('secondaryPreferred')
db.runCommand({'listCollections': 1})
db.runCommand({'listCollections': 1.0})
The listCollections queries will be processed identically. However, since you are using the secondaryPreferred read preference it is possible that this query might be returned from different members of your replica set when your two applications are running queries concurrently. Normally this will not be an issue, but if your use case requires strong consistency you should use a primary read preference instead.
Related
I am trying to read the MongoDB log file located at /var/log/mongodb it's contents are as such:
2019-11-04T05:04:00.390-0800 I COMMAND [conn38649] command loldb.$cmd command: update { update: "SUBSCRIPTION", ordered: true, writeConcern: { w: 1 }, $db: "loldb" } numYields:0 reslen:295 locks:{ Global: { acquireCount: { r: 460, w: 460 } }, Database: { acquireCount: { w: 460 } }, Collection: { acquireCount: { w: 459 } }, oplog: { acquireCount: { w: 1 } } } protocol:op_query 568ms
2019-11-04T05:04:00.396-0800 I COMMAND [conn38657] command loldb.SUBSCRIPTION command: find { find: "SUBSCRIPTION", filter: { customerID: 75824180, policeDepartmentID: 1 }, projection: {}, $readPreference: { mode: "secondaryPreferred" }, $db: "loldb" } planSummary: COLLSCAN keysExamined:0 docsExamined:69998 cursorExhausted:1 numYields:550 nreturned:1 reslen:430 locks:{ Global: { acquireCount: { r: 1102 } }, Database: { acquireCount: { r: 551 } }, Collection: { acquireCount: { r: 551 } } } protocol:op_query 424ms
2019-11-04T05:04:00.402-0800 I COMMAND [conn38735] command loldb.SUBSCRIPTION command: find { find: "SUBSCRIPTION", filter: { customerID: 75824164 }, projection: {}, $readPreference: { mode: "secondaryPreferred" }, $db: "loldb" } planSummary: COLLSCAN keysExamined:0 docsExamined:58142 cursorExhausted:1 numYields:456 nreturned:1 reslen:417 locks:{ Global: { acquireCount: { r: 914 } }, Database: { acquireCount: { r: 457 } }, Collection: { acquireCount: { r: 457 } } } protocol:op_query 374ms
Each blockquote is a single line entry
The contents of file update each second I need to read the file and if the query time protocol:op_query 385ms is more than 300ms I need to save that entire log/line into another text file slow_queries.text.
The file from which I am reading is .log file but the content seem like JSON format (please correct me if I am wrong) preceded by timestamp and command type, is there any efficient way to read data of this format? I am just reading word by word line by line.
Also, what do I do so that the changes made to the .log file are automatically read without running the script every time?
I just tried this on my local machine, maybe needs some work for your usecase. But I added some comments, so maybe this will help you:
EDIT: I added a check for the timestamp, you would have to configure it to your needs
#!/bin/bash
# continously read from the file and pipe it into the while loop
tail -F "test.log" | \
while read LINE
do
# get timestamp from LINE and get time in seconds
timeinseconds="$(grep -P "^\S*" | date -d - +%s)"
# get current timestamp before 5 minutes
timebeforefivemin="$(date -d '-5 minutes' +%s)"
# only log if timestamp of line is smaller to time before 5 min
if [[ $(expr $timeinseconds - $timebeforefivemin) -lt 0 ]];
then
# get the time of the query from the line
querytime="$(echo "$LINE" | grep -oP '\d+ms' | grep -oP '\d+')"
#if the grep was successful and the query time is greater than 300
if [ $? = 0 ] && [ "$querytime" -gt 300 ]
then
# echo the line into the slow_queries file -> change it to the path you want
echo "$LINE" >> slow_queries.txt
fi
fi
done
I'm having trouble with the following findOneAndUpdate MongoDB query:
planSummary: IXSCAN { id: 1 } keysExamined:1 docsExamined:1 nMatched:1 nModified:1 keysInserted:1 keysDeleted:1 numYields:0 reslen:3044791
locks:{ Global: { acquireCount: { r: 1, w: 1 } }, Database: { acquireCount: { w: 1 } }, Collection: { acquireCount: { w: 1 } } }
storage:{} protocol:op_query 135ms
writeConcern: { w: 0, j: false }
As you can see it has execution time of +100 ms. The query part uses an index and takes less than 1ms (using 'Explain query'). So it's the write part that is slow.
The Mongo instance is the master of a 3 member replica set. Write concern is set to 0 and journaling is disabled.
What could be the cause of the slow write? Could it be the update of indices?
MongoDB version 4.0
Driver: Node.js native mongodb version 3.2
Edit: I think it might be the length of the result. After querying a document smaller in size, the execution time is halved.
reslen:3044791
This was the source of the bad performance. Reducing this by adding a projection option to only return a specific field improved the execution from ~90ms on average to ~7ms.
As per MongoDb documentation the MongoDB shell command:
show dbs
Print a list of all databases on the server.
and
show databases
Print a list of all available databases.
I'm confused - from that what I read and understood these are not the same effect commands - right? show databases is not the alias of the show dbs?
There could be a database listed by show dbs which is not available and not listed by show databases is that right?
If so how it is possible that a database is on the server but is not available - access right of a user? is that what's behind show databases filtering?
I don't think there is a difference between the two commands. Both of the operations call the listDatabases command with the same option.
Increasing the log level, the show dbs command logged:
2018-11-30T15:40:59.539-0800 I COMMAND [conn23] command admin.$cmd appName: "MongoDB Shell" command: listDatabases { listDatabases: 1.0, $clusterTime: { clusterTime: Timestamp(1543621253, 1), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, $db: "admin" } numYields:0 reslen:708 locks:{ Global: { acquireCount: { r: 22 } }, Database: { acquireCount: { r: 10 } } } protocol:op_msg 38ms
whereas show databases logged:
2018-11-30T15:41:01.722-0800 I COMMAND [conn23] command admin.$cmd appName: "MongoDB Shell" command: listDatabases { listDatabases: 1.0, $clusterTime: { clusterTime: Timestamp(1543621253, 1), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, $db: "admin" } numYields:0 reslen:708 locks:{ Global: { acquireCount: { r: 22 } }, Database: { acquireCount: { r: 10 } } } protocol:op_msg 5ms
For reference, this is from MongoDB 3.6.7.
Tried to initiate a replica set in MongoDB but failed.
My mongod configuration file is as follows:
dbpath=C:\data\db
logpath=C:\data\log\mongo.log
storageEngine=mmapv1
After starting mongod with the command:
mongod --config "C:\data\mongo.conf" --replSet "rs0"
I went to mongo and typed
rs.initiate()
and got the error of "no configuration file specified" (code 8). Also tried to clearly instruct mongodb using
cfg = {"_id": "rs0", "version":1, "members":[{"_id":0,"host":"127.0.0.1:27017"}]}
rs.initiate(cfg)
However, the result is still the same (code 8).
Dig deeper into the log file, I found this
replSetInitiate failed to store config document or create the oplog; UnknownError: assertion C:\data\mci\7751c6064ad5f370b9aea0db0164a05e\src\src\mongo/util/concurrency/rwlock.h:204
2017-08-26T18:36:41.760+0700 I COMMAND [conn1] command local.oplog.rs command: replSetInitiate { replSetInitiate: { _id: "rs0", version: 1.0, members: [ { _id: 0.0, host: "127.0.0.1:27017" } ] } } keyUpdates:0 writeConflicts:0 numYields:0 reslen:143 locks:{ Global: { acquireCount: { r: 1, W: 1 } }, MMAPV1Journal: { acquireCount: { w: 2 } }, Metadata: { acquireCount: { W: 6 } } } protocol:op_command 4782ms
Any hint for me please? Thank you a ton.
I have a query that is seeing some pretty long execution times. Query:
db.legs.find(
{
effectiveDate: {$lte: startDate},
discontinuedDate: {$gte: startDate}
}
).count()
and below is the output in my logs:
2016-11-21T08:58:50.470-0800 I COMMAND [conn2] command myDB.legs
command: count { count: "legs", query: { effectiveDate: { $lte: new Date(1412121600000) }, discontinuedDate: { $gte: new Date(1412121600000) } }, fields: {} }
planSummary: IXSCAN { discontinuedDate: 1 } keyUpdates:0 writeConflicts:0 numYields:82575 reslen:47 locks:{ Global: { acquireCount: { r: 165152 } }, MMAPV1Journal: { acquireCount: { r: 82576 } }, Database: { acquireCount: { r: 82576 } }, Collection: { acquireCount: { R: 82576 } } } protocol:op_command 13940ms
I have an index on {effectiveDate: 1, discontinuedDate: 1} and it is using an IXSCAN to get the data. I'm wondering if anyone can suggest any ways to speed up this query? Isn't IXSCAN the fastest operation we can hope for in this situation?
The explain output doesn’t help much, because dates in the query were compared to strings like “1/1/2015” resulting with 0 matches.
Since you have 2 range filters, index intersection doesn’t work, so basically mongo uses 1 index, fetches documents, and apply the second filter. It may still work for covered queries, but it might be a better idea to try a query without indexes at all:
db.legs.find({
effectiveDate: {$lte: startDate},
discontinuedDate: {$gte: startDate}
})
.hint({$natural:true})
.count()
Even tho it does COLLSCAN, it uses COUNT stage instead of FETCH, which may be quicker.
Store date ms in new field and apply filter on the same. Pass input date also converted to ms and apply filter. this should do faster. link to get date to ms convertion epochconverter