Mongo Connector doesn't sync properly - mongodb

Here's my problem.
Our solution :
Front web connect to a cloud mongodb instances.
End-point client connect to a local mongodb instances.
People put some data throught the Front web, those data must be sync to the local instances for end-point client.
End-point client generate some data to the local's instance which must be sync to the cloud environnement.
================================
I tried to use mongo-connector to sync both way and I did sync something but everytime mongo-connector sync only half of the data randomely...
Here's my local mongodb conf :
# Where and how to store data.
storage:
dbPath: /server/mongo/data
journal:
enabled: true
# where to write logging data.
systemLog:
destination: file
logAppend: true
path: /var/log/mongodb/mongod.log
# network interfaces
net:
port: 27017
bindIp: 0.0.0.0
# how the process runs
processManagement:
timeZoneInfo: /usr/share/zoneinfo
replication:
replSetName: rs-local
Master mongodb conf:
# Where and how to store data.
storage:
dbPath: /server/mongo/data
journal:
enabled: true
# where to write logging data.
systemLog:
destination: file
logAppend: true
path: /var/log/mongodb/mongod.log
# network interfaces
net:
port: 27017
bindIp: 0.0.0.0
# how the process runs
processManagement:
timeZoneInfo: /usr/share/zoneinfo
#operationProfiling:
replication:
replSetName: master-replica-01
This is the command I use for mongo-connector :
/usr/local/bin/mongo-connector -m localhost:27017 -t X.X.X.X:27017 -d mongo_doc_manager -n "db.role,db.site,db.split,db.organism,db.users,db.namespace1.namespace2.namespace3.*" --admin-username mongosu --password verysecretpassword--oplog-ts /server/mongo/data/oplog.timestamp --no-dump --enable-syslog
Everytime I dump the database then launch the mongo-connector I got some of my data but not all of it...
I thought it was an oplog problem so set the oplog to 16Gb.
Many help ask :)
Elenui

Related

Mongo auth fails to login when using mongod.conf

So this is the weirdest thing.
I have two centOS 7 servers running mongo. I now wanted to enforce authentication so I added the security.authorization: enabled to the mongod.conf file.
I already have a user on database "buzzztv".
So when I ran mongod --conf /etc/mongod.conf on the first server everything went fine.
Then I did the exact same thing on the second server and whenever I try to connect with one of the users I get the following error:
connecting to: mongodb://127.0.0.1:27017/?authSource=buzzztv&compressors=disabled&gssapiServiceName=mongodb
2020-02-20T13:02:35.166+0000 E QUERY [js] Error: Authentication failed. :
connect#src/mongo/shell/mongo.js:341:17
#(connect):2:6
2020-02-20T13:02:35.168+0000 F - [main] exception: connect failed
2020-02-20T13:02:35.168+0000 E - [main] exiting with code 1
Now if I run mongod --fork --logpath /var/log/mongodb/mongod.log --auth the login works perfectly fine.
So obiously I could just run this command, but I want to use the mongod.conf.
Here is my mongod.conf file, I checked and it is a perfect copy of the file from the server in which it does work.
Any ideas?
# mongod.conf
# for documentation of all options, see:
# http://docs.mongodb.org/manual/reference/configuration-options/
# where to write logging data.
systemLog:
destination: file
logAppend: true
path: /var/log/mongodb/mongod.log
# Where and how to store data.
storage:
dbPath: /var/lib/mongo
journal:
enabled: true
# engine:
# wiredTiger:
# how the process runs
processManagement:
fork: true # fork and run in background
pidFilePath: /var/run/mongodb/mongod.pid # location of pidfile
timeZoneInfo: /usr/share/zoneinfo
# network interfaces
net:
port: 27017
bindIp: 0.0.0.0 # 127.0.0.1 # Enter 0.0.0.0,:: to bind to all IPv4 and IPv6 addresses or, alternatively, use the net.bindIpAll setting.
security:
authorization: enabled
#operationProfiling:
#replication:
#sharding:
## Enterprise-Only Options
#auditLog:
#snmp:
So after several hurtful hours of looking into it, I neede to change the
storage:
dbPath: /var/lib/mongo
journal:
enabled: true
Apparently something was wrong with my /var/lib/mongo so I backed up the data, and created a new folder /var/lib/mongodb
Then edited the mongod.conf file to:
storage:
dbPath: /var/lib/mongodb
journal:
enabled: true
Then it all worked fine. Now I'll just re-create the users and re-insert all the data and I'm good to go.
Hope this saves someone the wasteful hours I've lost

Could not connect to replica set in MongoDB with pymongo

I am trying to connect to replica set in MongoDB with the following command:
client = MongoClient(
"PublicIP:27017,PublicIP:27017,PublicIP:27017,PublicIP:27017,PublicIP:27017",
replicaSet="rs0", readPreference='nearest',
localThresholdMS=500)
db = client.sampledb
# checks the connection to RS in Mongo
try:
db.command("serverStatus")
except Exception as e:
print(e)
else:
print("\n Connection established to the Replica Set!)
I have setup 5 nodes in AWS (1 master, 4 slaves) and the /etc/mongod.conf file in all nodes looks like this:
systemLog:
destination: file
logAppend: true
path: /var/log/mongodb/mongod.log
# Where and how to store data.
storage:
dbPath: /var/lib/mongo
journal:
enabled: true
# engine:
# mmapv1:
# wiredTiger:
# how the process runs
processManagement:
fork: true # fork and run in background
pidFilePath: /var/run/mongodb/mongod.pid # location of pidfile
# network interfaces
net:
port: 27017
#bindIp: 127.0.0.1 # Listen to local interface only, comment to listen on all interfaces.
security:
authorization: "disabled"
#operationProfiling:
replication:
replSetName: rs0
When I try to connect from my local machine I get a timed out like this:
ip:27017: timed out,ip:27017: timed out,ip:27017: timed out,ip:27017:
timed out,ip:27017: timed out
Does anyone know why?
Starting 3.6 bindIp defaults to localhost only. So, assuming you are running 3.6 or later your configuration is not accessible from outside, since you commented out bindIp.
This post might be helpful to understand some of the nuances: https://www.mongodb.com/blog/post/enabling-ip-security-for-mongodb-36-on-ubuntu
Obviously, don't forget to make sure AWS Security Group on your instances allows Ingress connection on port 27017.

mongodb: pre-bake conf file to start replica set

I am trying to set up a MongoDB replica set via ansible.
Therefore I am templating the following file (and it works)
storage:
dbPath: /var/lib/mongodb
journal:
enabled: true
systemLog:
destination: file
logAppend: true
path: /var/log/mongodb/mongod.log
net:
port: 27017
bindIp: 127.0.0.1,{{ inventory_hostname }}
processManagement:
timeZoneInfo: /usr/share/zoneinfo
replication:
replSetName: "{{ mongodb_replicaset_name }}"
What are the additions/modifications that I should make, so that one of the hosts becomes primary and the other two secondaries, without me havint to resort to the mongo cli?
According to the documentation (https://docs.mongodb.com/manual/tutorial/deploy-replica-set/),
Run rs.initiate() on just one and only one mongod instance for the replica set.
It appears you must connect via mongo CLI to at least one member of the replica set and initiate it. From there the voting system should kick in automatically and one of the RS becomes primary.

Cannot connect remotely to mongodb installed on AWS instance

I have installed and started mongodb on my RHEL AWS instance. Now I want to connect to the mongodb server from my local machine but am not able to do so.
I have made changes to the bind_ip parameter in the mongodb.conf file and also have added rules to the security group for the AWS instance to allow connections on port 27017 for any inbound request. Am I missing anything ??
MongoDB Version - 3.4.14
AWS Instance - RHEL
Thanks in advance !!!
Adding the mongod.conf file
systemLog:
destination: file
logAppend: true
path: /var/log/mongodb/mongod.log
# Where and how to store data.
storage:
dbPath: /home/ec2-user/AutoMaTics/setup/dbData
journal:
enabled: true
# engine:
# mmapv1:
# wiredTiger:
# how the process runs
processManagement:
fork: true # fork and run in background
pidFilePath: /var/run/mongodb/mongod.pid # location of pidfile
# network interfaces
net:
port: 27017
# bindIpAll: true
# bindIp: 127.0.0.1,13.127.31.137 # Listen to local interface only, comment to listen on all interfaces.
#security:
#operationProfiling:
#replication:
#sharding:
## Enterprise-Only Options
#auditLog:
#snmp:

Mongodb restrict to given IP to localhost

I am trying to restrict the mongodb to local ip, it is not working.
Using MongoDb 3.2 version.
Below is the configuration file for the service, which is located in "C:\Program Files\MongoDB\Server\3.2\mongod.cfg"
systemLog:
destination: file
path: c:\data\mongo.log
quiet: false
logAppend: true
storage:
dbPath: C:\data\db
directoryPerDB: true
wiredTiger:
engineConfig:
cacheSizeGB: 2
directoryForIndexes: true
net:
port: 27017
bindIp: [127.0.0.0, 10.64.87.231]
replication:
oplogSizeMB: 500
replSetName: "mongoDbRepSet"
setParameter:
enableLocalhostAuthBypass: false
processManagement:
windowsService:
serviceName: "MongoDB"
displayName: "MongoDB 3.2 Standard"
description: "This is the service for MongoDB 3.2 Standard version"
I am using below command
mongod --config "C:\Program Files\MongoDB\Server\3.2\mongod.cfg" --install
I am restarting the mongo service after installing it with the configuraiton.
Strange thing is, when I open in RoboMongo and try to connect using 127.0.0.0 and 10.64.87.231 it is not working.
Also noticed that, it is connecting with ip addresses 127.0.0.1 to 127.0.0.255 (so many loopback ips?)
I also checked by assigning enableLocalhostAuthBypass: true but still not working.
What is wrong happening no idea. Please help me to restrict mongo service to give ip address.