Just after starting my MongoDB server (standalone instance, version 4.2.2) if I run db.currentOp() I see this operation:
{
"type" : "op",
"host" : "menzo:27017",
"desc" : "waitForMajority",
"active" : true,
"currentOpTime" : "2020-05-06T16:16:33.077+0200",
"opid" : 2,
"op" : "none",
"ns" : "",
"command" : {
},
"numYields" : 0,
"waitingForLatch" : {
"timestamp" : ISODate("2020-05-06T14:02:55.895Z"),
"captureName" : "WaitForMaorityService::_mutex"
},
"locks" : {
},
"waitingForLock" : false,
"lockStats" : {
},
"waitingForFlowControl" : false,
"flowControlStats" : {
}
}
It seems that this operation is always there, no matter how long it passes. In addition, it is a weird operation in some aspects:
It has a very log opid number (2)
It's op is "none"
It doesn't have the usual secs_running or microsecs_running parameters
It mentions "majority" in some literals, but I'm not running a replica set but an standalone instance
I guess it should be some kind of internal operation (maybe a kind of "waiting thread"?) but I haven't found documentation about it in the currentOp command documentation.
Do anybody knows about this operation and/or could point to documentation where it is described, please? Thanks in advance!
Wait for majority service is defined here. Looking at the history of this file, it appears to have been added as part of work on improving startup performance.
Reading the ticket description, it seems that during startup, multiple operations may need to wait for a majority commit. Previously each may have created a separate thread for waiting; with the majority wait service, there is only one thread which is waiting for the most recent required commit.
Therefore:
It's op is "none"
The thread isn't performing an operation as defined in the docs.
It has a very log opid number (2)
This is because this thread is started when the server starts.
It mentions "majority" in some literals, but I'm not running a replica set but an standalone instance
It is possible to instruct mongod to run in replica set mode and point it at a data directory created by a standalone node, and vice versa. In these cases you'd probably expect the process to preserve the information already in the database, such as transactions that are pending (or need to be aborted). Hence the startup procedure may perform operations not intuitively falling under the operation mode requested.
Related
How to configure removed members of a replica set to form new replica set?
I have a replica set with 4 mongod instances
Output of rs.config()
{
"_id" : "rs0",
"members" : [
{
"_id" : 0,
"host" : "localhost:27031"
},
{
"_id" : 1,
"host" : "localhost:27032"
},
{
"_id" : 2,
"host" : "localhost:27033"
},
{
"_id" : 3,
"host" : "localhost:27034"
}
],
"settings" : {
"replicaSetId" : ObjectId("5cf22332f5b9d21b01b9b6b2")
}
}
I removed 2 instances from the replica set
rs.remove("localhost:27033")
rs.remove("localhost:27034")
Now my requirement is to form a new replica set with these 2 removed members. What is the best way for that?
My current solution
connect to removed member
mongo --port 27033
and execute
conf = {
"_id" : "rs0",
"members" : [
{
"_id" : 2,
"host" : "localhost:27033"
},
{
"_id" : 3,
"host" : "localhost:27034"
}
],
"settings" : {
"replicaSetId" : ObjectId("5cf22332f5b9d21b01b9b6b2")
}
}
and then
rs.reconfig(conf, {force:true})
Outcome
This solution worked fine practically.
The removed members formed a replicaset, one of them became primary and other became secondary. Data was replicated among them.
And this replica set seems to be isolated from the the initial replica set from which they were removed.
Concerns
1) I had to use forced reconfiguration. Not sure about the consequences.
"errmsg" : "replSetReconfig should only be run on PRIMARY, but my state is REMOVED; use the \"force\" argument to override",
2) Is the new replica set actually new one? In the rs.config()
replicaSetId is same as old one.
"replicaSetId" : ObjectId("5cf22332f5b9d21b01b9b6b2")
I had to use same value for _id of members as in config of old replica set
"errmsg" : "New and old configurations both have members with host of localhost:27034 but in the new configuration the _id field is 1 and in the old configuration it is 3 for replica set rs0",
Is this solution good?
Is there any better solution?
Note: I need to retain data from old replica set (data which was present at the time of removal) in the new replica set.
As you have suspected, the procedure did not create a new replica set. Rather, it's a continuation of the old replica set, albeit superficially they look different.
There is actually a procedure in the MongoDB documentation to do what you want: Restore a Replica Set from MongoDB Backups. The difference being, you're not restoring from a backup. Rather, you're using one of the removed secondaries to seed a new replica set.
Hence you need to modify the first step in the procedure mentioned in the link above. The rest of the procedure would still be the same:
Restart the removed secondary as a standalone (without the --replSet parameter) and connect to it using the mongo shell.
Drop the local database in the standalone node:
use local
db.dropDatabase()
Restart the ex-secondary, this time with the --replSet parameter (with a new replica set name)
Connect to it using the mongo shell.
rs.initiate() the new set.
After this, the new set should have a different replicaSetId compared to the old set. In my quick test of the procedure above, this is the result I see:
Old set:
> rs.conf()
...
"replicaSetId": ObjectId("5cf45d72a1c6c4de948ff5d8")
...
New set
> rs.conf()
...
"replicaSetId": ObjectId("5cf45d000dda9e1025d6c65e")
...
As with any major deployment changes like this, please ensure that you have a backup, and thoroughly test the procedures before doing it on a production system.
I have a mongoDB cluster
server1:27017
server2:27017
server3:27017
For historical reason, IT team could not provide the replicaSet name for this cluster.
My question is: without knowing the replicaSet name, is the following mongoDB url legal and will missing the optional replicaSet optional parameter cause any possible problems in future?
mongodb://username:password#server1:27017,server2:27017,server3:27017
I am using Java to setup MongoDB connection using the following
String MONGO_REPLICA_SET = "mongodb://username:password#server1:27017,server2:27017,server3:27017";
MongoClientURI mongoClientURI = new MongoClientURI(MONGODB_REPLICA_SET);
mongoClient = new MongoClient(mongoClientURI);
To clarify, although it may be functional to connect to the replica set it would be preferable to specify the replicaSet option.
Depending on the MongoDB Drivers that you're using it may behaves slightly differently. For example quoting the Server Discovery and Monitoring Spec for Initial Topology Type:
In the Java driver a single seed means Single, but a list containing one seed means Unknown, so it can transition to replica-set monitoring if the seed is discovered to be a replica set member. In contrast, PyMongo requires a non-null setName in order to begin replica-set monitoring, regardless of the number of seeds.
There are variations, and it's best to check whether the connection can still handle topology discovery and failover.
For historical reason, IT team could not provide the replicaSet name for this cluster.
If you have access to the admin database, you could execute rs.status() on mongo shell to find out the name of the replica set. See also replSetGetStatus for more information.
It should be possible to find out the name of the replica set, to avoid this worry. Open a connection to any one of your nodes (e.g. direct to server1:27017), and run rs.status(); that will tell you the name of your replica set as well as lots of other useful data such as the complete set of configured nodes and their individual statuses.
In this example of the output, "rsInternalTest" is the replica set name:
{
"set" : "rsInternalTest",
"date" : ISODate("2018-05-01T11:38:32.608Z"),
"myState" : 1,
"term" : NumberLong(123),
"heartbeatIntervalMillis" : NumberLong(2000),
"optimes" : {
...
},
"members" : [
{
"_id" : 1,
"name" : "server1:27017",
"health" : 1.0,
"state" : 1,
"stateStr" : "PRIMARY",
"uptime" : 1652592,
"optime" : {
"ts" : Timestamp(1525174711, 1),
"t" : NumberLong(123)
},
"optimeDate" : ISODate("2018-05-01T11:38:31.000Z"),
"electionTime" : Timestamp(1524371004, 1),
"electionDate" : ISODate("2018-04-22T04:23:24.000Z"),
"configVersion" : 26140,
"self" : true
}
...
],
"ok" : 1.0
}
Note that you will need the login of a high-level user account, otherwise you won't have permission to run rs.status().
I'm new to mongodb. I have a local server and a remote server. After migrate the mongo database from the local server to a remote server using mongodump/mongorestore tools, I found out that the size of database became bigger on remote server.
Here is my sample :
on local server (Ubuntu 14.04.2 LTS, mongo 3.0.5):
> show dbs
Daily_data 7.9501953125GB
Monthly_data 0.453125GB
Weekly_data 1.953125GB
on remote server (CentOS 6.7, mongo 2.4.3):
> show dbs
Daily_data 9.94921875GB
Monthly_data 0.953125GB
Weekly_data 3.9521484375GB
I also checked the status of one collection to compare, the count is the same but the size (like indexSize, totalIndexSize, etc) has changed:
this is the status of collection on the local server:
> db.original_prices.stats()
{
"ns" : "Daily_data.original_prices",
"count" : 9430984,
"size" : 2263436160,
"avgObjSize" : 240,
"numExtents" : 21,
"storageSize" : 2897301504,
"lastExtentSize" : 756662272,
"paddingFactor" : 1,
"paddingFactorNote" : "paddingFactor is unused and unmaintained in 3.0. It remains hard coded to 1.0 for compatibility only.",
"userFlags" : 1,
"capped" : false,
"nindexes" : 2,
"indexDetails" : {
},
"totalIndexSize" : 627777808,
"indexSizes" : {
"_id_" : 275498496,
"symbol_1_dateTime_1" : 352279312
},
"ok" : 1
}
this is the status of collection on the remote server:
> db.original_prices.stats()
{
"ns" : "Daily_data.original_prices",
"count" : 9430984,
"size" : 1810748976,
"avgObjSize" : 192.00000508960676,
"storageSize" : 2370023424,
"numExtents" : 19,
"nindexes" : 2,
"lastExtentSize" : 622702592,
"paddingFactor" : 1,
"systemFlags" : 1,
"userFlags" : 0,
"totalIndexSize" : 639804704,
"indexSizes" : {
"_id_" : 305994976,
"symbol_1_dateTime_1" : 333809728
},
"ok" : 1
}
If mongodump/mongorestore is a good save way to migrate the mongo database?
The problem here as you seem to have already noticed is the index as is clearly shown that it is the indexSize that has grown here, and there is a perfectly logical explanation.
When running the restore the indexes are rebuilt but in a way that avoids blocking the other write operations happening in the restore operation. This is similar to the process employed in Build Indexes in the Background as described in the documentation, not exactly the same but close.
In order to get the most optimal index size it is best to first drop indexes from the target database and use the --noIndexRestore option with the mongorestore command as this will prevent index building during the data load.
Then when complete you can run a regular createIndex exluding any usage of the "background" option so the indexes are created in the foreground. The result will be that the database will be blocked from read and write during index creation, but the resulting indexes will be of a smaller size.
As for the general practice, you will note that other data sizes will in fact come out "smaller" as in the process of "rebuilding" then any slack space present in the source will not be created when the data is restored.
The data from mongodump is in a binary format and should always be used in preference to the textual format of mongoexport and related mongoimport, when of course taking data from one MongoDB instance and to use on another, since that is not the purpose of those tools.
Other alternates ae file system copies such as an LVM snapshot, which will of course restore in exactly the same state as the backup copy was made.
Factors that can affect the disk size of your collection include the underlying hardware, filesystem, and configuration. In your case, the prevailing factor seems to be a difference in the storage engine used on the local and remote servers: your local server is running Mongo 3.0 while the remote is running an older version. This is apparent based on the presence of the paddingFactorNote property, however you can confirm by running db.version() in both environments.
Between Mongo 2.4/2.6 and Mongo 3.0 there were several important changes to how collections are stored, not least the addition of the WiredTiger storage engine as an alternative to the default mmapv1 storage engine. There were also changes to how the mmapv1 engine (which you are using) pads documents during allocation to accommodate growth in document size.
The other major reason for the size differences comes from your use of mongorestore. During normal usage, mongo databases are not stored in a way that minimizes disk usage. However, mongorestore rebuilds the database/collection in a compact way, which is why for the collection you posted, the remote storageSize is smaller.
I've made sharding on one collection with 2 shards.
Mongodb version is 2.6.4.
Everything looks ok but 100% of data is on one shard.
When I do:
use config
db.locks.find( { _id : "balancer" } ).pretty()
I get:
{
"_id" : "balancer",
"state" : 0,
"who" : "ip-10-0-11-128:27018:1424099612:1804289383:Balancer:846930886",
"ts" : ObjectId("553a1223e4d292075ec2a8a6"),
"process" : "ip-10-0-11-128:27018:1424099612:1804289383",
"when" : ISODate("2015-04-24T09:51:31.498Z"),
"why" : "doing balance round"
}
So balancer is stuck in state 0. I have tried to restart it but it is still in state 0.
Also:
sh.isBalancerRunning()
> false
But:
sh.getBalancerState()
> true
Errors in my log file:
2015-04-24T10:15:47.921+0000 [Balancer] scoped connection to 10.0.11.128:20000,10.0.11.159:20000,10.0.11.240:20000 not being returned to the pool
2015-04-24T10:15:47.921+0000 [Balancer] caught exception while doing balance: error checking clock skew of cluster 10.0.11.128:20000,10.0.11.159:20000,10.0.11.240:20000 :: caused by :: 13650 clock skew of the cluster 10.0.11.128:20000,10.0.11.159:20000,10.0.11.240:20000 is too far out of bounds to allow distributed locking.
Anyone knows how to fix this?
Thanks,
Ivan
From a quick look at the code, it looks like we can tolerate about 30
seconds of skew between servers in the cluster. On linux, we
recommend people use ntp to keep skew to a minimum (ntp usually keeps
it to within a second or two). ntp is usually already installed on
most linux distros.
Mongodb user group
using the following command to do an one time ntp synx
ntpdate pool.ntp.org
I am just restoring a database which I dumped few minutes back to make some changes. Mongorestore taking around 100% CPU and much more time than expected. I thought, it may be due to indexes I created, but, the problem is same while restoring even a single collection. A collection is about 314MB in size and has about 185000 documents.
Usually, this thing does not happen. It might be due to less disk space on my system, but that too is 11GB.
Can anyone help me, what problem it could be?
Note: I'm doing things from mongo client. No driver included.
As you indicated in the comments that this was related to logging, then I would suggest taking a couple of steps:
First, use log rotation. This can be done via a command to the database or by sending the SIGUSR1 signal to the process, so it is very easy to script or enable as a cron job on a regular basis. More information here:
http://www.mongodb.org/display/DOCS/Logging#Logging-Rotatingthelogfiles
Second, verify your logging level. Starting with -v = log level 1; -vv = log leve 2 etc. You can adjust it both at start up and during runtime. For the runtime adjustment, you use the setParameter command:
// connect to the database or mongos
use admin;
// check the log level
db.runCommand({getParameter : 1, logLevel: 1})
{ "logLevel" : 0, "ok" : 1 }
// set it higher
db.runCommand({setParameter : 1, logLevel: 1})
// back to default levels
{ "was" : 0, "ok" : 1 }
db.runCommand({setParameter : 1, logLevel: 0})
{ "was" : 1, "ok" : 1 }
Finally you can also run with --quiet to cut down on some of the messaging also.