Mongod in recovering after enabling security in mongod.conf - mongodb

Question

Created 2 shards with 3 node replica set.
After enabling the security in mongod.conf file, every node was in recovering state. But if we disable/comment out in the mongod.conf file the nodes were normal.
All nodes were running on AWS.
OS: CentOS 7
Mongodb Version: 3.6.2
Background: Before enabling the security, I created a user as below.
db.createUser( { user: "cxiroot",
pwd: "root",
roles: [ { role: "root", db: "admin" }] },
{ w: "majority" , wtimeout: 5000 } );
I am able to authenticate with, db.auth("cxiroot","root")
After enabling security and restarting the service, all nodes were in recovering state.
Basically I am trying to do db.shutdownServer() by enabling security.
Log reporting:
2019-10-17T11:40:23.138-0400 I REPL_HB [replexec-1] Error in heartbeat (requestId: 1934) to 109.99.16.36:27018, response status: Unauthorized: not authorized on admin to execute command { replSetHeartbeat: "shard1rs", configVersion: 4, hbv: 1, from: "109.99.16.112:27018", fromId: 1, term: 128, $replData: 1, $clusterTime: { clusterTime: Timestamp(1571326818, 1), signature: { hash: BinData(0, 466B43AC8CDFBE9B5CBEA8AC4860925560B63296), keyId: 6745513220909301779 } }, $db: "admin" }
2019-10-17T11:40:23.150-0400 I ACCESS [conn2] Unauthorized: not authorized on admin to execute command { replSetHeartbeat: "shard1rs", configVersion: 4, hbv: 1, from: "109.99.16.137:27018", fromId: 2, term: 128, $replData: 1, $clusterTime: { clusterTime: Timestamp(1571326818, 1), signature: { hash: BinData(0, 466B43AC8CDFBE9B5CBEA8AC4860925560B63296), keyId: 6745513220909301779 } }, $db: "admin" }
mongod.conf
systemLog:
destination: file
logAppend: true
path: /mongodb/data/logs/mongod.log
storage:
dbPath: /mongodb/data/db
journal:
enabled: true
processManagement:
fork: true # fork and run in background
pidFilePath: /var/run/mongodb/mongod.pid # location of pidfile
timeZoneInfo: /usr/share/zoneinfo
net:
port: 27018
bindIp: x.x.x.x
security:
authorization: 'enabled'
replication:
replSetName: shard1rs
sharding:
clusterRole: shardsvr
What's causing this problem?.

Answer 1

Edit 2019-10-22:
OP asked to change ports to reflect MongoDB recommendations. It took a bit of digging to find any recommendations from MongoDB. For all modern versions (i.e., 3.6 or later) there is no specific port recommended. Going back to 3.0 docs, however, the default port for config servers is documented as 27019. https://docs.mongodb.com/v3.0/tutorial/deploy-shard-cluster/ . Looking at Ops Manager default deployments we see shard replica sets default to port 27018.
I assume these ports are selected to help protect developers who are writing connection strings and consuming the database from accidentally connecting to a replica set directly, and bypassing the MongoDB router MONGOS. OK, so below I have changed the ports respectively...
Here are instructions to setup 10 hosts to run a 2-shard sharded cluster on AWS. These instructions describe hardware setup only, and do not describe how to select a proper shard key to apply to data. The instructions setup the hardware but have no data. The breakdown of servers is as follows...
1 host for MONGOS
3 hosts for shard config server replica set - each host has a MONGOD installed.
3 hosts for shard0 replica set - each host has a MONGOD installed.
3 hosts for shard1 replica set - each host has a MONGOD installed.
This setup assumes a keyfile will be used for internal authentication. The keyfile is a shared secret stored in a file on all 10 hosts. When using keyfile authentication treat the keyfile like a password. It should be protected.
Generate a keyfile:
openssl rand -base64 741 > mykeyfile
for the sake of this tutorial assume the keyfile generated is ...
bgi+xXyBAHtNXmQnTjDNrSyTa+I9SGQXbBZONHRxHxKw2y/M3kGtpiJCVCyI+bDk
bXKTHnejIGXcyl7Ykc812DHEEngmqw63HfPxsUHFiDZ8FiwU/5X7W/T9lgKk9SoV
ybIL8+EBjSPvWDa9JWgVKrJFYqG0IejSyrO+js9os6n9kq5kneNOYjnovJwS9MgM
euENHfzTJ2XItcMWtcMilMoXd4Pm9VQgkW8i+Cb9hhQcwm1yA/wT7Tr04l6Pgq74
wQgp5MlYXLmlOGMhsTFGgBv4eVfKVR/r3zr2nshLowpBR6CiX098TO/+mZIGIM54
CoULxBlLBxngSXOWH86tvG05YvrjtAOaiDnHFm59fYKmT1+jfecx/NHOT4Cn1bfO
2q4cpGma3Cy2iRHuEdrm8zV1wvx94x0bLIEttiO4qelb32HLZM9MaL+lKodwhfko
A/3Bcx6+c1tTFtCE2sd5xpAngw6oMKau3nfynQgxgvnLrymCW4Hxqj3ew7F1ShYp
OAskY5/qu+ruad0VO9gqM2PHtsPrVHgkO8zBn+twtpYMOvdTE4M4vYMxA14vxkLA
FrELT/TqYmCPSO8pVS8tu12nADEkRUYRK+LqYKXsogl2FolnvYPLsiw4g9psQw4x
2CmJsEVYel1sl3cxq21Sgd+uO9nyWuNEaKBkYOOgLw67xg6xkGWBrLkg3gC840eI
JE0eOJfDLl+EkF1CUubKv8JB7bxK6kwnoTkfd2OGEHqLGQbV/hP6Gpni4gnqoNlp
65meHn2djSUGWu3wS7m5NRjCqICRTbOQs4K/ugM2hVu4e4dZV0RDt/FOF3u+6Anv
G8X5/GqWDvoIJ4WCvPyqVQoAyDG6S5DiSMmhlwCJUaXu0gFn7NuDPEtC8KAzHK75
qOmsTddIhSSs3fjmPm3wKAxyf2r5/6oBIjMq3vN8ahi+NLa+Vz+8VMRa+ajvE4ws
HPAh7P5iar4u9Uu2WE/J0WvGM88p
I generated this using the command above. It is random and only used for this tutorial.
Install MongoDB:
This example uses MongoDB Enterprise. This version requires a license purchased from MongoDB. To use the Community edition change the repo definition.
This example assumes AWS hosts are AMI2 Amazon Linux.
Run this command on all 10 hosts.
echo '[mongodb-enterprise]
name=MongoDB Enterprise Repository
baseurl=https://repo.mongodb.com/yum/amazon/2/mongodb-enterprise/4.0/$basearch/
gpgcheck=1
enabled=1
gpgkey=https://www.mongodb.org/static/pgp/server-4.0.asc' | sudo tee /etc/yum.repos.d/mongodb-enterprise.repo
sudo yum -y install mongodb-enterprise
Distribute the keyfile to all hosts:
On all 10 AWS hosts issue the following...
echo "bgi+xXyBAHtNXmQnTjDNrSyTa+I9SGQXbBZONHRxHxKw2y/M3kGtpiJCVCyI+bDk
bXKTHnejIGXcyl7Ykc812DHEEngmqw63HfPxsUHFiDZ8FiwU/5X7W/T9lgKk9SoV
ybIL8+EBjSPvWDa9JWgVKrJFYqG0IejSyrO+js9os6n9kq5kneNOYjnovJwS9MgM
euENHfzTJ2XItcMWtcMilMoXd4Pm9VQgkW8i+Cb9hhQcwm1yA/wT7Tr04l6Pgq74
wQgp5MlYXLmlOGMhsTFGgBv4eVfKVR/r3zr2nshLowpBR6CiX098TO/+mZIGIM54
CoULxBlLBxngSXOWH86tvG05YvrjtAOaiDnHFm59fYKmT1+jfecx/NHOT4Cn1bfO
2q4cpGma3Cy2iRHuEdrm8zV1wvx94x0bLIEttiO4qelb32HLZM9MaL+lKodwhfko
A/3Bcx6+c1tTFtCE2sd5xpAngw6oMKau3nfynQgxgvnLrymCW4Hxqj3ew7F1ShYp
OAskY5/qu+ruad0VO9gqM2PHtsPrVHgkO8zBn+twtpYMOvdTE4M4vYMxA14vxkLA
FrELT/TqYmCPSO8pVS8tu12nADEkRUYRK+LqYKXsogl2FolnvYPLsiw4g9psQw4x
2CmJsEVYel1sl3cxq21Sgd+uO9nyWuNEaKBkYOOgLw67xg6xkGWBrLkg3gC840eI
JE0eOJfDLl+EkF1CUubKv8JB7bxK6kwnoTkfd2OGEHqLGQbV/hP6Gpni4gnqoNlp
65meHn2djSUGWu3wS7m5NRjCqICRTbOQs4K/ugM2hVu4e4dZV0RDt/FOF3u+6Anv
G8X5/GqWDvoIJ4WCvPyqVQoAyDG6S5DiSMmhlwCJUaXu0gFn7NuDPEtC8KAzHK75
qOmsTddIhSSs3fjmPm3wKAxyf2r5/6oBIjMq3vN8ahi+NLa+Vz+8VMRa+ajvE4ws
HPAh7P5iar4u9Uu2WE/J0WvGM88p" | sudo tee /var/run/mongodb/mykeyfile
sudo chown mongod.mongod /var/run/mongodb/mykeyfile
sudo chmod 400 /var/run/mongodb/mykeyfile
Setup the Config Servers:
Select 3 AWS instances as the config servers. On these three hosts run the following...
sudo rm /etc/mongod.conf
echo "systemLog:
destination: file
logAppend: true
path: /var/log/mongodb/mongod.log
storage:
dbPath: /var/lib/mongo
journal:
enabled: true
processManagement:
fork: true # fork and run in background
pidFilePath: /var/run/mongodb/mongod.pid # location of pidfile
timeZoneInfo: /usr/share/zoneinfo
net:
port: 27019
bindIp: 0.0.0.0
security:
keyFile: /var/run/mongodb/mykeyfile
replication:
replSetName: configrs
sharding:
clusterRole: configsvr
" | sudo tee /etc/mongod.conf
sudo systemctl start mongod
On the final config server log into the mongo shell...
mongo
and initiate the replica set. This example uses host names from my AWS instances. Change the host names to match yours. Notice the name of the replica set is configrs.
rs.initiate(
{
_id: "configrs",
version: 1,
members: [
{ _id: 0, host: "ip-172-31-27-98.us-west-2.compute.internal:27019" },
{ _id: 1, host: "ip-172-31-17-202.us-west-2.compute.internal:27019" },
{ _id: 2, host: "ip-172-31-19-63.us-west-2.compute.internal:27019" }
]
}
)
Add credentials to allow root level access for administration. If the prompt says 'SECONDARY' then wait about 1 minute. Test by issuing command use admin. Continue to wait until it says 'PRIMARY'. If it never says 'PRIMARY' you have problems and cannot proceed.
use admin
db.createUser({user: "barry", pwd: "mypassword", roles: [{role: "root", db: "admin"}]})
db.auth("barry", "mypassword")
Configure Shard0:
Identify another 3 hosts for shard0.
Run the following on all 3 hosts...
sudo rm /etc/mongod.conf
echo "systemLog:
destination: file
logAppend: true
path: /var/log/mongodb/mongod.log
storage:
dbPath: /var/lib/mongo
journal:
enabled: true
processManagement:
fork: true # fork and run in background
pidFilePath: /var/run/mongodb/mongod.pid # location of pidfile
timeZoneInfo: /usr/share/zoneinfo
net:
port: 27018
bindIp: 0.0.0.0
security:
keyFile: /var/run/mongodb/mykeyfile
replication:
replSetName: shard0
sharding:
clusterRole: shardsvr
" | sudo tee /etc/mongod.conf
sudo systemctl start mongod
On the final host log into the mongo shell...
mongo
Initiate the replica set. Notice replica set name shard0. Replace host names with yours...
rs.initiate(
{
_id: "shard0",
version: 1,
members: [
{ _id: 0, host: "ip-172-31-21-228.us-west-2.compute.internal:27018" },
{ _id: 1, host: "ip-172-31-17-221.us-west-2.compute.internal:27018" },
{ _id: 2, host: "ip-172-31-17-145.us-west-2.compute.internal:27018" }
]
}
)
... and create root user...
use admin
db.createUser({user: "barry", pwd: "mypassword", roles: [{role: "root", db: "admin"}]})
db.auth("barry", "mypassword")
Configure shard1:
Select 3 remaining hosts for shard1 and apply these settings to all 3...
sudo rm /etc/mongod.conf
echo "systemLog:
destination: file
logAppend: true
path: /var/log/mongodb/mongod.log
storage:
dbPath: /var/lib/mongo
journal:
enabled: true
processManagement:
fork: true # fork and run in background
pidFilePath: /var/run/mongodb/mongod.pid # location of pidfile
timeZoneInfo: /usr/share/zoneinfo
net:
port: 27018
bindIp: 0.0.0.0
security:
keyFile: /var/run/mongodb/mykeyfile
replication:
replSetName: shard1
sharding:
clusterRole: shardsvr
" | sudo tee /etc/mongod.conf
sudo systemctl start mongod
On the final of these 3 start the mongo shell ...
mongo
... and initialize the replica set. Notice replica set name shard1. Replace host names with yours...
rs.initiate(
{
_id: "shard1",
version: 1,
members: [
{ _id: 0, host: "ip-172-31-30-65.us-west-2.compute.internal:27018" },
{ _id: 1, host: "ip-172-31-17-88.us-west-2.compute.internal:27018" },
{ _id: 2, host: "ip-172-31-23-140.us-west-2.compute.internal:27018" }
]
}
)
On the final of the 3 create a root user...
use admin
db.createUser({user: "barry", pwd: "mypassword", roles: [{role: "root", db: "admin"}]})
db.auth("barry", "mypassword")
Setup MONGOS:
On the final host reserved for the MONGOS router create a MONGOS config file. Change host names on references to config servers to match your implementation.
echo "systemLog:
destination: file
logAppend: true
path: /var/log/mongodb/mongod.log
processManagement:
fork: true
pidFilePath: /var/run/mongodb/mongos.pid # location of pidfile
timeZoneInfo: /usr/share/zoneinfo
net:
bindIp: 0.0.0.0
port: 27017
sharding:
configDB: configrs/ip-172-31-27-98.us-west-2.compute.internal:27019,ip-172-31-17-202.us-west-2.compute.internal:27019,ip-172-31-19-63.us-west-2.compute.internal:27019
security:
keyFile: /var/run/mongodb/mykeyfile
" | sudo tee /etc/mongos.conf
Create systemd script to start the MONGOS using the user 'mongod'.
echo '[Unit]
Description=MongoDB Database Server Router
Documentation=https://docs.mongodb.org/manual
After=network.target
[Service]
User=mongod
Group=mongod
Environment="OPTIONS=-f /etc/mongos.conf"
EnvironmentFile=-/etc/sysconfig/mongos
ExecStart=/usr/bin/mongos $OPTIONS
ExecStartPre=/usr/bin/mkdir -p /var/run/mongodb
ExecStartPre=/usr/bin/chown mongod:mongod /var/run/mongodb
ExecStartPre=/usr/bin/chmod 0755 /var/run/mongodb
PermissionsStartOnly=true
PIDFile=/var/run/mongodb/mongos.pid
Type=forking
# file size
LimitFSIZE=infinity
# cpu time
LimitCPU=infinity
# virtual memory size
LimitAS=infinity
# open files
LimitNOFILE=64000
# processes/threads
LimitNPROC=64000
# locked memory
LimitMEMLOCK=infinity
# total threads (user+kernel)
TasksMax=infinity
TasksAccounting=false
# Recommended limits for for mongod as specified in
# http://docs.mongodb.org/manual/reference/ulimit/#recommended-settings
[Install]
WantedBy=multi-user.target
' | sudo tee /usr/lib/systemd/system/mongos.service
Reload the systemctl daemon and start the MONGOS.
sudo systemctl daemon-reload
sudo systemctl start mongos
If you prefer to start MONGOS manually instead of using systemd scripts use the following command... This is optional.
sudo -u mongod mongos -f /etc/mongos.conf
On the host having the MONGOS log into the mongo shell and authenticate...
mongo
use admin
db.auth("barry", "mypassword")
Initialize the sharding. Only one host is needed. Mongo will discover all the other hosts in the shard. Change host names to match yours...
sh.addShard("shard0/ip-172-31-21-228.us-west-2.compute.internal:27018")
sh.addShard("shard1/ip-172-31-30-65.us-west-2.compute.internal:27018")
View the status of the sharding...
sh.status()
At this point sharding is prepared. No data is on these hosts. No databases or collections have been told to shard. No shard key has been established. These instructions are merely for setting up hardware using keyfile internal authentication. See MongoDB instructions for sharding databases
Conclusion:
If we utilize keyfile internal authentication the entire cluster requires user authorization to perform tasks. System administrators can choose to implement SCRAM-SHA username/password authentication mechanisms for users, or can use x.509 Certificate based authentication, or others. If your cluster was not using any security, applying keyfile internal authentication may cause surprise as client authentication is now required. My testing showed if I applied authorization: 'enabled' while keyfile was defined there was no change in behavior. Example:
security:
keyFile: /var/run/mongodb/mykeyfile
... behaved the same as ...
security:
keyFile: /var/run/mongodb/mykeyfile
authorization: 'enabled'