mongodb replica set error "...replSetHeartbeat requires authentication..." after running rs.initiate() - mongodb

Question

I have 2 virtual machines with mongodb running on both. I have created a user and roles and added data for testing. Everything works until I try to set up a Replica set. When I run rs.initiate() I get the "...replSetHeartbeat requires authentication... " error seen below. I can connect to the "hulk" server from "hawkeye" manually by passing --host "nodeserver-hulk:27017. Any Ideas?
system
Ubuntu Server 18.04.4 LTS
Mongod v.4.2.3
rs.initiate command
rs.initiate(
{
_id: "r1",
version: 1,
members: [
{ _id: 0, host : "nodeserver-hulk:27017"},
{ _id: 1, host : "nodeserver-hawkeye:27017"}
]
}
)
rs.initiate error
{
"ok" : 0,
"errmsg" : "replSetInitiate quorum check failed because not all proposed set members responded affirmatively: nodeserver-hawkeye:27017 failed with command replSetHeartbeat requires authentication",
"code" : 74,
"codeName" : "NodeNotFound"
}
mongod.conf file(same for both servers)
# mongod.conf
# for documentation of all options, see:
# http://docs.mongodb.org/manual/reference/configuration-options/
# Where and how to store data.
storage:
dbPath: /mnt/mongo/data
journal:
enabled: true
# engine:
# mmapv1:
# wiredTiger:
# where to write logging data.
systemLog:
destination: file
logAppend: true
path: /var/log/mongodb/mongod.log
# network interfaces
net:
port: 27017
bindIp: 0.0.0.0
# how the process runs
processManagement:
timeZoneInfo: /usr/share/zoneinfo
#security:
security:
authorization: 'enabled'
#operationProfiling:
#replication:
replication:
replSetName: 'r1'
#sharding:
## Enterprise-Only Options:
#auditLog:
#snmp:
show users command
> show users
{
"_id" : "admin.james",
"userId" : UUID("3ed97f2e-de49-4b98-84c8-566b34805863"),
"user" : "james",
"db" : "admin",
"roles" : [
{
"role" : "userAdminAnyDatabase",
"db" : "admin"
},
{
"role" : "dbOwner",
"db" : "admin"
},
{
"role" : "clusterAdmin",
"db" : "admin"
},
{
"role" : "readWriteAnyDatabase",
"db" : "admin"
}
],
"mechanisms" : [
"SCRAM-SHA-1",
"SCRAM-SHA-256"
]
}

Answer 1

You need to add a keyFile in your security config so that each node can authenticate against the others.
https://docs.mongodb.com/manual/tutorial/enforce-keyfile-access-control-in-existing-replica-set/
security:
keyFile: <path-to-keyfile>