I have a MongoDB collection with ~5.5M records. My attempts to index it, whether on a single field or with a compoundIndex fail as the indexing process proceeds normally but then when it reaches 100% where I presume it should stop, it goes past 100% and just continues on. I've left it running for 10 hours but it never ended.
The fields I try to index on are longs or doubles.
I'm running the latest MongoDB version on x64 Windows.
Am I right to think that this is abnormal behaviour? Any ideas what I can do?
Wed Sep 05 10:22:37 [conn1] 415000000/5576219 7442%
Wed Sep 05 10:22:48 [conn1] 417000000/5576219 7478%
Wed Sep 05 10:22:59 [conn1] 419000000/5576219 7514%
Per helpful advice from mongodb-users list:
This was likely due to running out of disk space and getting the database corrupted due to that.
What I did is I cleared up disk space, then ran "mongodump --repair" and then "mongorestore".
Related
I am working with a MongoDB database running in a replica set.
Unfortunately, I noticed that the data appears to be corrupted.
There should be over 10,000 documents in the database. However, there are several thousand records that are not being returned in queries.
The total count DOES show the correct total.
db.records.find().count()
10793
And some records are returned when querying by RecordID (a custom sequence integer).
db.records.find({"RecordID": 10049})
{ "_id" : ObjectId("5dfbdb35c1c2a400104edece")
However, when querying for a records that I know for a fact should exist, it does not return anything.
db.records.find({"RecordID": 10048})
db.records.find({"RecordID": 10047})
db.records.find({"RecordID": 10046})
The issue appears to be very sporadic, and in some cases entire ranges of records are missing. The entire range from RecordIDs 1500 to 8000 is missing.
Questions: What could be the cause of the issue? What can I do to troubleshoot this issue further and recover the corrupted data? I looked into running repairDatabase but that is for standalone instances only.
UPDATE:
More info on replication:
rs.printReplicationInfo()
configured oplog size: 5100.880859375MB
log length start to end: 14641107secs (4066.97hrs)
oplog first event time: Wed Mar 03 2021 05:21:25 GMT-0500 (EST)
oplog last event time: Thu Aug 19 2021 17:19:52 GMT-0400 (EDT)
now: Thu Aug 19 2021 17:20:01 GMT-0400 (EDT)
rs.printSecondaryReplicationInfo()
source: node2-examplehost.com:27017
syncedTo: Thu Aug 19 2021 17:16:42 GMT-0400 (EDT)
0 secs (0 hrs) behind the primary
source: node3-examplehost.com:27017
syncedTo: Thu Aug 19 2021 17:16:42 GMT-0400 (EDT)
0 secs (0 hrs) behind the primary
UPDATE 2:
We did a restore from a backup and somehow it looks like it fixed the issue.
We did a restore from a backup and somehow it looks like it fixed the issue.
Setup: replica set with 5 nodes, version 3.4.5.
Trying to switch PRIMARY with rs.stepDown(60, 30) but consistently getting the error:
rs0:PRIMARY> rs.stepDown(60, 30)
{
"ok" : 0,
"errmsg" : "No electable secondaries caught up as of 2017-07-11T00:21:11.205+0000. Please use {force: true} to force node to step down.",
"code" : 50,
"codeName" : "ExceededTimeLimit"
}
However, rs.printSlaveReplicationInfo() running in a parallel terminal confirms that all replicas are fully caught up:
rs0:PRIMARY> rs.printSlaveReplicationInfo()
source: X.X.X.X:27017
syncedTo: Tue Jul 11 2017 00:21:11 GMT+0000 (UTC)
0 secs (0 hrs) behind the primary
source: X.X.X.X:27017
syncedTo: Tue Jul 11 2017 00:21:11 GMT+0000 (UTC)
0 secs (0 hrs) behind the primary
source: X.X.X.X:27017
syncedTo: Tue Jul 11 2017 00:21:11 GMT+0000 (UTC)
0 secs (0 hrs) behind the primary
source: X.X.X.X:27017
syncedTo: Tue Jul 11 2017 00:21:11 GMT+0000 (UTC)
0 secs (0 hrs) behind the primary
Am I doing something wrong?
UPD: I've checked long running operations before and during rs.stepDown as was suggested below and it looks like this:
# Before rs.stepDown
$ watch "mongo --quiet --eval 'JSON.stringify(db.currentOp())' | jq -r '.inprog[] | \"\(.secs_running) \(.desc) \(.op)\"' | sort -rnk1"
984287 rsSync none
984287 ReplBatcher none
67 WT RecordStoreThread: local.oplog.rs none
null SyncSourceFeedback none
null NoopWriter none
0 conn615153 command
0 conn614948 update
0 conn614748 getmore
...
# During rs.stepDown
984329 rsSync none
984329 ReplBatcher none
108 WT RecordStoreThread: local.oplog.rs none
16 conn615138 command
16 conn615136 command
16 conn615085 update
16 conn615079 insert
...
Basically, long running user operations seem to happen as a result of rs.stepDown() as secs_running becomes nonzero once PRIMARY attempts to switch over and keeps growing all the way up until stepDown fails. Then everything gets back to normal.
Any ideas on why this happens and whether that's normal at all?
I have used below command to step down to secondary
db.adminCommand( { replSetStepDown: 120, secondaryCatchUpPeriodSecs: 15, force: true } )
You can find this in below mongodb official documentation
https://docs.mongodb.com/manual/reference/command/replSetStepDown/
To close the loop on this question, it was determined that the failed stepdown was due to time going backward on the host.
MongoDB 3.4.6 is more resilient to time issues on the host, and upgrading the deployment fixes the stalling issues.
Before stepping down, rs.stepDown() will attempt to terminate long running user operations that would block the primary from stepping down, such as an index build, a write operation or a map-reduce job.
Do you have some long running jobs on going? Check db. Check result of db.currentOp()
You can try to set longer stepping down time rs.stepDown(60, 360).
Quoting an answer from https://jira.mongodb.org/browse/SERVER-27015:
This is most likely due to the fact that by default the shutdown command will only succeed on a primary if the secondaries are fully caught up at the exact moment that the shutdown command is executed.
I faced a similar issue and tried the db.shutdownServer() command several times, however it worked exactly when the secondary was 0 seconds behind the primary.
ubuntu 16.04 server. My first ubuntu, so yeah, ubuntu newbie factor.
System time on server is about 17 minutes fast. I can reset it many various ways successfully at the command line. In every case, within 15 or so seconds, it will be reset to the wrong time again. systemd seems to be at the root of it. In message log:
systemd[20113]: Time has been changed
I assumed the culprit was timedatectl. Made sure to disable and check:
root#portal:~# systemctl status systemd-timesyncd.service
â systemd-timesyncd.service - Network Time Synchronization
Loaded: loaded (/lib/systemd/system/systemd-timesyncd.service; disabled; vendor preset: enabled)
Drop-In: /lib/systemd/system/systemd-timesyncd.service.d
ââdisable-with-time-daemon.conf
Active: inactive (dead)
Docs: man:systemd-timesyncd.service(8)
root#portal:~# systemctl list-units|grep 'time-sync'
root#portal:~#
This:
root#portal:~# ntpdate -s 0.north-america.pool.ntp.org
root#portal:~# journal -f
Results in this:
Dec 08 20:04:18 portal sudo[4486]: pam_unix(sudo:session): session opened
for user root by owen(uid=0)
Dec 08 19:46:44 portal systemd[2814]: Time has been changed
Dec 08 19:46:44 portal ntpdate[4502]: step time server 171.66.97.126 offset -1098.814353 sec
Dec 08 20:05:35 portal systemd[2814]: Time has been changed
Notice the timestamp of each log entry...
What, pray tell, is resetting the time and where is it getting the ntp server setting, and why is it wrong??? I thought the large drift might be part of the issue, but it doesn;t match up with the number of minutes it's off...
TIA
FTR, this turned out to be the date on the virtual server being wrong. I had checked the hardware server itself and it was accurate, but the date/time via vmware was off a weird increment. Once I corrected this, everything settled down.
I am running a conventional MongoDB Replica Set consisting of 3 members (member1 in datacenter A, member2 and member3 in datacenter B).
member1 is the current PRIMARY and I am adding members 2 and 3 via rs.add(). They are performing their initial sync and become SECONDARY very soon. Everything is fine all day long and the replication delay of both members is 0 seconds until 2 AM at nighttime.
Now: Every night at 2 AM both members shift into the RECOVERING state and stop replication at all, which leads to a replication delay of hours when I am having a look into rs.printSlaveReplicationInfo() in the morning hours. At around 2 AM there are no massive inserts or maintenance tasks known to me.
I get the following log entries on the PRIMARY:
2015-10-09T01:59:38.914+0200 [initandlisten] connection accepted from 192.168.227.209:59905 #11954 (37 connections now open)
2015-10-09T01:59:55.751+0200 [conn11111] warning: Collection dropped or state deleted during yield of CollectionScan
2015-10-09T01:59:55.869+0200 [conn11111] warning: Collection dropped or state deleted during yield of CollectionScan
2015-10-09T01:59:55.870+0200 [conn11111] getmore local.oplog.rs cursorid:1155433944036 ntoreturn:0 keyUpdates:0 numYields:1 locks(micros) r:32168 nreturned:0 reslen:20 134ms
2015-10-09T01:59:55.872+0200 [conn11111] end connection 192.168.227.209:58972 (36 connections now open)
And, which is more interesting, I get the following log entries on both SECONDARYs:
2015-10-09T01:59:55.873+0200 [rsBackgroundSync] repl: old cursor isDead, will initiate a new one
2015-10-09T01:59:55.873+0200 [rsBackgroundSync] replSet syncing to: member1:27017
2015-10-09T01:59:56.065+0200 [rsBackgroundSync] replSet error RS102 too stale to catch up, at least from member1:27017
2015-10-09T01:59:56.066+0200 [rsBackgroundSync] replSet our last optime : Oct 9 01:59:23 5617035b:17f
2015-10-09T01:59:56.066+0200 [rsBackgroundSync] replSet oldest at member1:27017 : Oct 9 01:59:23 5617035b:1af
2015-10-09T01:59:56.066+0200 [rsBackgroundSync] replSet See http://dochub.mongodb.org/core/resyncingaverystalereplicasetmember
2015-10-09T01:59:56.066+0200 [rsBackgroundSync] replSet error RS102 too stale to catch up
2015-10-09T01:59:56.066+0200 [rsBackgroundSync] replSet RECOVERING
Which is also striking - the start of the oplog "resets" itself every night at around 2 AM:
configured oplog size: 990MB
log length start to end: 19485secs (5.41hrs)
oplog first event time: Fri Oct 09 2015 02:00:33 GMT+0200 (CEST)
oplog last event time: Fri Oct 09 2015 07:25:18 GMT+0200 (CEST)
now: Fri Oct 09 2015 07:25:26 GMT+0200 (CEST)
I am not sure if this is somehow correlated to the issue. I am also wondering that such a small delay (Oct 9 01:59:23 5617035b:17f <-> Oct 9 01:59:23 5617035b:1af) lets the members become stale.
Could this also be a server (VM host) time issue or is it something completely different? (Why is the first oplog event being "resetted" every night and not "shifting" to a timestamp like NOW minus 24 hrs?)
What can I do to investigate and to avoid?
Upping the oplog size should solve this (per our comments).
Some references for others who run into this issue
Workloads that Might Require a Larger Oplog Size
Error: replSet error RS102 too stale to catch up link1 & link2
I am using Meteor with a separate MongoDB running on Windows. I ran MongoDB as a service.
My MongoDB log is full of the following:
Mon Mar 04 14:15:36 [conn19] info DFM::findAll(): extent 0:55000 was empty, skipping ahead. ns:webfm.graphdata
Mon Mar 04 14:15:38 [conn17] info DFM::findAll(): extent 0:55000 was empty, skipping ahead. ns:webfm.graphdata
Mon Mar 04 14:15:40 [conn16] info DFM::findAll(): extent 0:55000 was empty, skipping ahead. ns:webfm.graphdata
Mon Mar 04 14:15:43 [conn18] warning: can't find plugin [desc]
Mon Mar 04 14:15:43 [conn19] info DFM::findAll(): extent 0:55000 was empty, skipping ahead. ns:webfm.graphdata
Mon Mar 04 14:15:49 [conn18] info DFM::findAll(): extent 0:55000 was empty, skipping ahead. ns:webfm.graphdata
Mon Mar 04 14:16:14 [conn16] warning: can't find plugin [desc]
Mon Mar 04 14:16:20 [conn17] info DFM::findAll(): extent 0:55000 was empty, skipping ahead. ns:webfm.graphdata
Mon Mar 04 14:16:24 [conn16] warning: can't find plugin [desc]
Mon Mar 04 14:16:32 [conn20] info DFM::findAll(): extent 0:60000 was empty, skipping ahead. ns:webfm.history
Mon Mar 04 14:16:34 [conn16] warning: can't find plugin [desc]
From what I can find the "findAll(): extent 0:55000..." seemed to have to do with my application's removing the data frequently. Is that correct?
How about the can't find plugin [desc]? What's that plugin? How can I fix it?
From what I can find the "findAll(): extent 0:55000..." seemed to have to do with my application's removing the data frequently. Is that correct?
Per Nick's answer, the message extent 0:55000 was empty, skipping ahead is related to your frequent deletions. This is a warning emitted when skipping an empty data extent, with 0:55000 being the extent location. The space will normally be reused as you add more data to that collection.
If you are frequently adding and deleting this collection and disk space is of concern, you could also consider:
enabling usePowerof2Sizes flag for this collection (MongoDB 2.2 or newer) for more effective reuse of deleted space
saving temporary collections in a separate database, and dropping the database when done (rather than running frequent compacts or repairs)
How about the can't find plugin [desc]? What's that plugin? How can I fix it?
This typically means you have an invalid index definition and the index "plugin" can't be found to handle that. In this example, I expect you have an indexed field declared with a property of desc rather than -1.
You can check the index definitions with db.collection.getIndexes():
use webfm
db.graphdata.getIndexes()
The message is in fact informational and harmless. It tells you that a whole extent was empty. This can happen if you deleted a lot of data recently.
You can use the compact command to defragment and compact a collection (basically re-writing it to disk and re-creating the indexes on the collection), which would get rid of the log message. Please be aware that compact is a resource-heavy operation.
I'm not absolute sure this is a problem. You might be running mongodb with too much verbosity, so its showing excessive detail in its warnings and processes for debug reasons. This is a good thing, because if theres a problem or bug the issue can be identified.
But if you want to remove it:
Decrease the number of 'v's or remove them in your startup command.
If you use a config file change whichever of these you have to false:
vvvvv = false
vvvv = false
vvv = false
vv = false
v = false