I’m using MongoDB Atlas’ paid tier Replicaset (Primary-Secondary-Secondary)
Every hour I create a new collection and insert about 1.5 to 2 million documents.
When I check Atlas’ cluster metrics every time I insert it, the primary is unchanged, and the query targeting of secondary is rapidly increasing.
As a result, Interferes with alerts of actual dangerous operations according to collscan and it is very noisy because the alarm of atlas occurs every hour
The alarm is using readPreference=secondary in my application, so it is difficult to disable.
I need an opinion on how this can happen.
Below is the metric information atlas metrics page that I checked.
enter image description here
Related
We have a active collection with 3000-5000 updates per second and average payload size being 500KB. We run changestream on the same but currently we only have one connector listening to all events in the collections. This works but we expect even more updates and are exploring ways to scale this approach horizontally. The current plan is to the hash the _id key and distirbute the events equally into three or more connectors.
The challenge is , since we use fulldocument=updateLookup we want to check if internally Mongo will now do a query to read the documents 3 times instead of the existing one time.
We tried testing this approach by using mongocat and setting log level to 2 but we are unable to see any Query logs on the collection while updating the documents even while having a active changestream with full document enabled. Any ideas on how we can test this out would really help
I was using MongoDB to store some time-series data at 1 Hz. To facilitate this, my central document represented an hour of data per device per user, with 3600 values pre-allocated at document creation time. Two drawbacks:
every insert is an update. I need to query for the correct record (by user, by device, by day, by hour), append the latest IoT reading to the list, and update the record.
paged queries require complex custom code. I need to query for a record count of all the data matching my search range, then manually create each page of data to be returned.
I was hoping the MongoDB Native Time-Series Collections introduced in 5.0 would give me some performance improvements and it did, but only on ingestion rate. The comparison I used is to push 108,000 records into the database as quickly as possible and average the response time, then perform paged queries and get a range for the response time for those. This is what I observed:
Mongo Custom Code Solution: 30 milliseconds inserts, 10-20 millisecond
paged query.
Mongo Native Time-Series: 138 microsecond insert, 50-90 millisecond
paged query.
The difference in insert rate was expected, see #1 above. But for paged queries I didn't expect my custom time-series kluge implementation to be significantly faster than a native implementation. I don't really see a discussion of the advantages to be expected in the Mongo 5.0 documentation. Should I expect to see improvements in this regard?
I'm fairly new to mongodb and atlas and am confused by the following alert
Query Targeting: Scanned Objects / Returned has gone above 1000
I expect there to be more data to aid in debugging such as the query or at least the collection. The query wasn't slow because the performance advisor didn't catch anything.
The only info given in the alert is
- time created
- the replica set
- a link to the shard
- the type of shard (primary/secondary)
how am I supposed to debug the supposed alerted issue?
A future alert had info on how to solve the issue - which in short is to download the mongodb logs and search for the inefficient query.
To download the logs
1. Navigate to the Cluster page
If you do not see the cluster you want on the page, ensure you have selected the > proper Project
2. Select the cluster
a. Click the ellipsis icon (...) next to the cluster containing the mongod instance whose logs you want to download.
b. Select Download Logs.
3. In the Download Logs modal, edit the following fields
Select process: Select the process for which you want logs. Valid options are mongod and mongod-audit-log.
Select server: Select the server in the cluster whose logs you want to retrieve.
Start Time: Specify the date and time in your group’s time zone defining the inclusive lower bound of log activity to return. The start time must be less than 30 days ago.
End Time: Specify the date and time in your group’s time zone defining the inclusive upper bound of log activity to return.
4. Click Download Logs
An inefficient query is explained here
The following mongod log entry shows statistics generated from an inefficient query:
planSummary: COLLSCAN keysExamined:0
docsExamined: 10000 cursorExhausted:1 numYields:234
nreturned:4 protocol:op_query 358ms<Timestamp> COMMAND <query>
This query scanned 10,000 documents and returned only 4 for a ratio of 2500, which is highly inefficient.
So I am getting an alert from Mongo - occasionally...
Query Targeting: Scanned Objects / Returned has gone above 1000
Is there a way to see the offending query specifically? I see graphs of trends over time in my dashboard - my "performance advisor" shows no "slow" queries...and the emails alert I get specifically says to check "performance advisor".
Any help
Normally, when the Scanned Object / Returned ratio is large, that means that those queries will be slow and will show up in the slow_query log. If nothing is showing up there, you can reduce the slowms setting that determines which queries will be written to the slow query log.
$explain and the $collStats aggregation operator are two other tools that are worth being aware of, but for this case I'd recommend updating your profiling level (db.setProfilingLevel) and then seeing where you're at!
If you're using Atlas, the "Profiler" tab shows the queries from the slow query log in an explorable way. If you're not on atlas, mtools has some good mongo log parsing tools.
Just recently we have been seeing same error in alerts for our mongo cluster
Query Targeting: Scanned Objects / Returned has gone above 1000
I was not able to find anything that returns this sort of huge chunk in code\logs.
It turned out to be related to our Charts page, ie it is firing when we open charts.
I suppose because of the lot's of aggregations it does, and I am not sure how to optimise those, as it's mainly UI set up
So if you have any charts on mongodb, it worth check if opening those triggers the alert.
I have a feed system using fan-out on write. I keep a list of feed ids in redis sorted set, and save the feed content in mongodb, so every time when i read 30 feeds, i have to do 30 query to mongodb, is there anyway to improve it ?
Its depend upon your setup of database. MongoDb has a vast documentation about how to increase simultaneous read and write MongoDb conncurrency
If you need so many writes in database with less latency starts using sharding Deployment Sharding.
If you need to increase number of reads in data base deploy each shards as replica set and rout your read query in secondary node Read Prefences
Also each query should covered by index Better indexing, you can check your query time by simply adding explain after a find it will show you the time and all facts
db.collection.find({a:"abcd"}).explain()
Make sure you have enough ram so that your data set fits with ram atleast your index should fit inside the ram coz each time a data fetched from disk is 10 times slower than RAM.
Check your sever status with running MongoStat it will measures your database performance , page fault , lock , query opertaion manny detail.
Also measure your hardware performance with program like iostat and make sure io wait is low and less than 1%.
Few good links to deployment of mongodb and performance tuning.
1. Production deployment of mongodb
2. Performance tuning of mongodb By 10gen
3. Using redis before mongodb to cache query and result object
4. Example of redis and mongo