I have a mongo instance running with oplogMinRetentionHours set to 24 hours and max oplog size set to 50G. But despite this config settings oplog entries seem to be withhold indefinitely since oplog has entries past 24 hours and oplog size has reached 1.4 TB and .34 TB on disk
db.runCommand( { serverStatus: 1 } ).oplogTruncation.oplogMinRetentionHours
24 hrs
db.getReplicationInfo()
{
"logSizeMB" : 51200,
"usedMB" : 1464142.51,
"timeDiff" : 3601538,
"timeDiffHours" : 1000.43,
"tFirst" : "Fri Mar 19 2021 14:15:49 GMT+0000 (Greenwich Mean Time)",
"tLast" : "Fri Apr 30 2021 06:41:27 GMT+0000 (Greenwich Mean Time)",
"now" : "Fri Apr 30 2021 06:41:28 GMT+0000 (Greenwich Mean Time)"
}
MongoDB server version: 4.4.0
OS: Windows Server 2016 DataCenter 64bit
what I have noticed is event with super user with root role is not able to access replset.oplogTruncateAfterPoint, not sure if this is by design
mongod.log
{"t":{"$date":"2021-04-30T06:35:51.308+00:00"},"s":"I", "c":"ACCESS",
"id":20436, "ctx":"conn8","msg":"Checking authorization
failed","attr":{"error":{"code":13,"codeName":"Unauthorized","errmsg":"not
authorized on local to execute command { aggregate:
"replset.oplogTruncateAfterPoint", pipeline: [ { $indexStats: {} }
], cursor: { batchSize: 1000.0 }, $clusterTime: { clusterTime:
Timestamp(1619764547, 1), signature: { hash: BinData(0,
180A28389B6BBA22ACEB5D3517029CFF8D31D3D8), keyId: 6935907196995633156
} }, $db: "local" }"}}}
Not sure why mongo would not delete older entries from oplog?
Mongodb oplog truncation seems to be triggered with inserts. So as and when insert happens oplog gets truncated.
Related
It's my first time to use monstache.
In fact I've migrated my infrastructure from on premise to the cloud and I'm now using mongoDB Atlas, and AWS opensearch.
I've installed monstache on an aws ec2 instance and well configured it. Everything seems working and monstache is connected to Elasticsearch and MongoDB, but it's indexing documents that have been migratred into mongoDB atlas in Elasticsearch. It keeps waiting for events on my collection/index like this
[ec2-user#ip-172-31-1-200 ~]$ journalctl -u monstache.service -f
-- Logs begin at Wed 2022-11-09 10:22:04 UTC. --
Jan 26 08:54:00 ip-172-31-1-200.eu-west-3.compute.internal systemd[1]: Starting monstache sync service...
Jan 26 08:54:00 ip-172-31-1-200.eu-west-3.compute.internal monstache[27813]: INFO 2023/01/26 08:54:00 Started monstache version 6.1.0
Jan 26 08:54:00 ip-172-31-1-200.eu-west-3.compute.internal monstache[27813]: INFO 2023/01/26 08:54:00 Successfully connected to MongoDB version 4.4.18
Jan 26 08:54:01 ip-172-31-1-200.eu-west-3.compute.internal monstache[27813]: INFO 2023/01/26 08:54:01 Successfully connected to Elasticsearch version 7.10.2
Jan 26 08:54:01 ip-172-31-1-200.eu-west-3.compute.internal systemd[1]: Started monstache sync service.
Jan 26 08:54:01 ip-172-31-1-200.eu-west-3.compute.internal monstache[27813]: INFO 2023/01/26 08:54:01 Joined cluster HA
Jan 26 08:54:01 ip-172-31-1-200.eu-west-3.compute.internal monstache[27813]: INFO 2023/01/26 08:54:01 Starting work for cluster HA
Jan 26 08:54:01 ip-172-31-1-200.eu-west-3.compute.internal monstache[27813]: INFO 2023/01/26 08:54:01 Listening for events
Jan 26 08:54:01 ip-172-31-1-200.eu-west-3.compute.internal monstache[27813]: INFO 2023/01/26 08:54:01 Watching changes on collection wtlive.myuser
Jan 26 08:54:01 ip-172-31-1-200.eu-west-3.compute.internal monstache[27813]: INFO 2023/01/26 08:54:01 Resuming from timestamp {T:1674723241 I:1}
Should I absolutely initiate a write on the mongoDB collection for monstache to start syncing? Why doesn't it start syncing current data from mongoDB?
My elasticsearch still shows 0 document count while the collection is full of document in mongoDB.
[ec2-user#ip-172-31-0-5 ~]$ curl --insecure -u es-appuser https://vpc-wtlive-domain-staging-om2cbdeex4qk6trkdrcb3dg4vm.eu-west-3.es.amazonaws.com/_cat/indices?v
Enter host password for user 'es-appuser':
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
green open wtlive.myuser YzqLx9_uTZ2qFVjFF2CMag 1 1 0 0 416b 208b
green open .opendistro_security Jb1fLqGjRd2vvluoX-ZgKw 1 1 9 4 129kb 64.4kb
green open .kibana_1 v6WdqQDvSN2L16EZTXxuHQ 1 1 30 2 70.4kb 33.4kb
green open .kibana_252235597_esappuser_1 OY1bbDGvTqK8oEgwopzbhQ 1 1 1 0 10.1kb 5kb
[ec2-user#ip-172-31-0-5 ~]$
Here is my monstache configuration :
[ec2-user#ip-172-31-1-200 ~]$ cat monstache/wtlive_pipeline.toml
enable-http-server = true
http-server-addr = ":8888"
#direct-read-namespaces = ["wtlive.myuser"]
change-stream-namespaces = ["wtlive.myuser"]
namespace-regex = '^wtlive.myuser$'
cluster-name="HA"
resume = true
replay = false
resume-write-unsafe = false
exit-after-direct-reads = false
elasticsearch-user = "es-appuser"
elasticsearch-password = "9V#xxxxxx"
elasticsearch-urls = ["https://vpc-wtlive-domain-staging-om2cbdeek6trkdrcb3dg4vm.eu-west-3.es.amazonaws.com"]
mongo-url = "mongodb://admin:VYn7ZD4CHDh8#wtlive-dedicated-shard-00-00.ynxpn.mongodb.net:27017,wtlive-dedicated-shard-00-01.ynxpn.mongodb.net:27017,wtlive-dedicated-shard-00-02.ynxpn.mongodb.net:27017/?tls=true&replicaSet=atlas-lmkye1-shard-0&authSource=admin&retryWrites=true&w=majority&tlsCAFile=/home/ec2-user/mongodb-ca.pem"
#[logs]
#info = "/home/ec2-user/logs/monstache/info.log"
#error = "/home/ec2-user/logs/monstache/error.log"
#warn = "/home/ec2-user/logs/monstache/warn.log"
#[[mapping]]
#namespace = "wtlive.myuser"
#index = "wtlive.myuser"
[[pipeline]]
namespace = "wtlive.myuser"
script = """
module.exports = function(ns, changeStream) {
if (changeStream) {
return [
{
$project: {
_id: 1,
operationType : 1,
clusterTime : 1,
documentKey : 1,
to : 1,
updateDescription : 1,
txnNumber : 1,
lsid : 1,
"fullDocument._id": 1,
"fullDocument.created": 1,
"fullDocument.lastVisit": 1,
"fullDocument.verified": 1,
"fullDocument.device.locale": "$fullDocument.device.locale",
"fullDocument.device.country": "$fullDocument.device.country",
"fullDocument.device.tz": "$fullDocument.device.tz",
"fullDocument.device.latLonCountry": "$fullDocument.device.latLonCountry",
"fullDocument.details.firstname": "$fullDocument._details.firstname",
"fullDocument.details.gender": "$fullDocument._details.gender",
"fullDocument.details.category": "$fullDocument._details.category",
"fullDocument.details.dob": "$fullDocument._details.dob",
"fullDocument.details.lookingFor": "$fullDocument._details.lookingFor",
"fullDocument.details.height": "$fullDocument._details.height",
"fullDocument.details.weight": "$fullDocument._details.weight",
"fullDocument.details.cigarette": "$fullDocument._details.cigarette",
"fullDocument.details.categorizedBy": "$fullDocument._details.categorizedBy",
"fullDocument.details.origin": "$fullDocument._details.origin",
"fullDocument.details.city": "$fullDocument._details.city",
"fullDocument.details.country": "$fullDocument._details.country",
"fullDocument.lifeSkills.educationLevel": "$fullDocument._lifeSkills.educationLevel",
"fullDocument.lifeSkills.pets": "$fullDocument._lifeSkills.pets",
"fullDocument.lifeSkills.religion": "$fullDocument._lifeSkills.religion",
"fullDocument.loveLife.children": "$fullDocument._loveLife.children",
"fullDocument.loveLife.relationType": "$fullDocument._loveLife.relationType",
"fullDocument.searchCriteria": "$fullDocument._searchCriteria",
"fullDocument.blocked" : 1,
"fullDocument.capping" : 1,
"fullDocument.fillingScore" : 1,
"fullDocument.viewed" : 1,
"fullDocument.likes" : 1,
"fullDocument.matches" : 1,
"fullDocument.blacklisted" : 1,
"fullDocument.uploadsList._id" : 1,
"fullDocument.uploadsList.status" : 1,
"fullDocument.uploadsList.url" : 1,
"fullDocument.uploadsList.position" : 1,
"fullDocument.uploadsList.imageSet" : 1,
"fullDocument.location" : 1,
"fullDocument.searchZone" : 1,
"fullDocument.locationPoint" : "$fullDocument.location.coordinates",
"fullDocument.selfieDateUpload" : 1,
"ns": 1
}
}
]
} else {
return [
{
$project: {
_id: 1,
"_id": 1,
"created": 1,
"lastVisit": 1,
"verified": 1,
"device.locale": "$device.locale",
"device.country": "$device.country",
"device.tz": "$device.tz",
"device.latLonCountry": "$device.latLonCountry",
"details.firstname": "$_details.firstname",
"details.gender": "$_details.gender",
"details.category": "$_details.category",
"details.dob": "$_details.dob",
"details.lookingFor": "$_details.lookingFor",
"details.height": "$_details.height",
"details.weight": "$_details.weight",
"details.cigarette": "$_details.cigarette",
"details.categorizedBy": "$_details.categorizedBy",
"details.origin": "$_details.origin",
"details.city": "$_details.city",
"details.country": "$_details.country",
"lifeSkills.educationLevel": "$_lifeSkills.educationLevel",
"lifeSkills.pets": "$_lifeSkills.pets",
"lifeSkills.religion": "$_lifeSkills.religion",
"loveLife.children": "$_loveLife.children",
"loveLife.relationType": "$_loveLife.relationType",
"searchCriteria": "$_searchCriteria",
"blocked" : 1,
"capping" : 1,
"fillingScore" : 1,
"viewed" : 1,
"likes" : 1,
"matches" : 1,
"blacklisted" : 1,
"uploadsList._id" : 1,
"uploadsList.status" : 1,
"uploadsList.url" : 1,
"uploadsList.position" : 1,
"uploadsList.imageSet" : 1,
"location" : 1,
"searchZone" : 1,
"selfieDateUpload" : 1,
"locationPoint" : "$location.coordinates"
}
}
]
}
}
"""
What could be the issue? And what action should I take from here please?
By uncommenting the #direct-read-namespaces = ["wtlive.myuser"] line, monstache can now do the initial sync, and everything is going well.
I'll comment out aigain and restart monstache service after the initial sync, to avoid re-syncing from scratch.
I try to get started with TTL indices in mongo, and wanted to get a simple demo setup running, but I just cannot get it to work and I am not sure what I do wrong.
Mongod version:
$ mongod --version
db version v4.4.5
Build Info: {
"version": "4.4.5",
"gitVersion": "ff5cb77101b052fa02da43b8538093486cf9b3f7",
"openSSLVersion": "OpenSSL 1.1.1m 14 Dec 2021",
"modules": [],
"allocator": "tcmalloc",
"environment": {
"distmod": "debian10",
"distarch": "x86_64",
"target_arch": "x86_64"
}
}
I start with a completely fresh, default configured mongod instance:
$ mkdir /tmp/db && mongod --dbpath /tmp/db
Then I run the following commands
use test
db.ttldemo.insertOne({
created_at: Date(Date.now() + 60_000)
})
db.ttldemo.createIndex({created_at: 1}, {expireAfterSeconds: 1})
db.ttldemo.find()
I would expect the document to vanish after some time, but even after running find() after waiting for a few minutes, the document is still present.
Is there anything I am missing here?
db.adminCommand({getParameter:1, ttlMonitorSleepSecs: 1}) yields 60 to me
I tried in mongo 4.2.17, your insertion of documents yields a document like:
{
"_id" : ObjectId("61f17ea34844b5f0505d80ea"),
"created_at" : "Wed Jan 26 2022 19:02:27 GMT+0200 (GTB Standard Time)"
}
when the
db.getCollection('ttldemo').insertOne({created_at: new Date()})
yields a document like:
{
"_id" : ObjectId("61f17f404844b5f0505d80ec"),
"created_at" : ISODate("2022-01-26T17:05:04.023Z")
}
Emphasis here on the created_at field, where this on the first case is a string, in the second case is an ISODate.
as from the documentation , this index works on dates only:
To create a TTL index, use the createIndex() method on a field whose value is either a date or an array that contains date values, and specify the expireAfterSeconds option with the desired TTL value in seconds.
The solution here is to construct the date as follows:
db.getCollection('ttldemo').insertOne({created_at: new Date(new Date().getTime()+60000)})
this will create a new ISODate based on the current date + 60 seconds.
Is there any options for only logging the errors in MongoDB log files?
With the current configuration, it seems that every request to Mongo server is logged in log files:
Wed Sep 17 08:08:07.030 [conn117] insert my_database.myCol ninserted:1 keyUpdates:0 locks(micros) w:243505 285ms
Wed Sep 17 08:08:54.447 [conn101] command anotherDatabase.$cmd command: { findandmodify: "myCol", query: { ... }, new: 0, remove: 0, upsert: 0, fields: {...}, update: {...} } update: {...} ntoreturn:1 idhack:1 nupdated:1 fastmod:1 keyUpdates:0 locks(micros) w:124172 reslen:248 124ms
Wed Sep 17 08:10:24.370 [conn95] command my_database.$cmd command: { count: "cms.myCol", query: { ... }, fields: null } ntoreturn:1 keyUpdates:0 locks(micros) r:197368 reslen:48 197ms
...
The current configuration is:
# mongodb.conf
dbpath=/var/lib/mongodb
logpath=/var/log/mongodb/mongodb.log
logappend=true
How can be the configuration updated to only log errors?
Running Mongo shell version: 2.4.10:
$ mongo --version
MongoDB shell version: 2.4.10
Appending quiet=true will reduce a lot of output.
Perhaps it is impossible to avoid any output information except error on current stage.
Appending slowms=threshold to configuration file can reduce normal log output further.
threshold is a integer value (milliseconds). It means if one operation duration doesn't exceed this value, normal log information won't output. The default value is 100.
Also, you can change this value by another way if the instance is running.
var slowms = theValueYouWant;
var level = db.getProfilingStatus().was;
db.setProfilingLevel(level, slowms);
Our mongodb server deployed with 2 shards, each has 1 master server and 2 slave servers.
The four slave servers run mongo config as proxy, and two of the slave servers run arbiters.
But the mongodb coundn't be used now.
I can connect to 192.168.0.1:8000(mongos) and exec queries like 'use database' or 'show dbs', but i cann't exec queries in a choosed database such as 'db.foo.count()', 'db.foo.findOne()'
Here is the error log:
mongos> db.dev.count()
Fri Aug 16 12:55:36 uncaught exception: count failed: {
"assertion" : "DBClientBase::findN: transport error: 10.81.4.72:7100 query: { setShardVersion: \"\", init: true, configdb: \"10.81.4.72:7300,10.42.50.26:7300,10.81.51.235:7300\", serverID: ObjectId('520db0a51fa00999772612b9'), authoritative: true }",
"assertionCode" : 10276,
"errmsg" : "db assertion failure",
"ok" : 0
}
Fri Aug 16 11:23:29 [conn8431] DBClientCursor::init call() failed
Fri Aug 16 11:23:29 [conn8430] Socket recv() errno:104 Connection reset by peer 10.81.4.72:7100
Fri Aug 16 11:23:29 [conn8430] SocketException: remote: 10.81.4.72:7100 error: 9001 socket exception [1] server [10.81.4.72:7100]
Fri Aug 16 11:23:29 [conn8430] DBClientCursor::init call() failed
Fri Aug 16 11:23:29 [conn8430] DBException in process: could not initialize cursor across all shards because : DBClientBase::findN: transport error: 10.81.4.72:7100 query: { setShardVersion: "", init: true, configdb: "10.81.4.72:7300,10.42.50.26:7300,10.81.51.235:7300", serverID: ObjectId('520d99c972581e6a124d0561'), authoritative: true } # s01/10.36.31.36:7100,10.42.50.24:7100,10.81.4.72:7100
i can only start on mongos, queries wouldn't be exec if more than 1 mongos run at the same time, error log:
mongos> db.dev.count() Fri Aug 16 15:12:29 uncaught exception: count failed: { "assertion" : "DBClientBase::findN: transport error: 10.81.4.72:7100 query: { setShardVersion: \"\", init: true, configdb: \"10.81.4.72:7300,10.42.50.26:7300,10.81.51.235:7300\", serverID: ObjectId('520dd04967557902f73a9fba'), authoritative: true }", "assertionCode" : 10276, "errmsg" : "db assertion failure", "ok" : 0 }
Could you please clarify if your set-up was working before, if you are just setting it up now?
To repair your MongoDB, you might want to follow this link:
http://docs.mongodb.org/manual/tutorial/recover-data-following-unexpected-shutdown/
References
MongoDB Documentation : Deploying a Shard-Cluster
MongoDB Documentation : Add Shards to an existing cluster
Older, outdated(!) info:
YouTube Video on Setting-up Sharding for MongoDB
Corresponding Blog on blog.serverdensity.com
I have tried to export a json file from MongoDB with mongoexport in the following way:
$ mongoexport --db db --collection ds --dbpath ~/db --out ds.json
exported 0 records
Sat Apr 20 23:13:18 dbexit:
Sat Apr 20 23:13:18 [tools] shutdown: going to close listening sockets...
Sat Apr 20 23:13:18 [tools] shutdown: going to flush diaglog...
Sat Apr 20 23:13:18 [tools] shutdown: going to close sockets...
Sat Apr 20 23:13:18 [tools] shutdown: waiting for fs preallocator...
Sat Apr 20 23:13:18 [tools] shutdown: closing all files...
Sat Apr 20 23:13:18 [tools] closeAllFiles() finished
Sat Apr 20 23:13:18 [tools] shutdown: removing fs lock...
Sat Apr 20 23:13:18 dbexit: really exiting now
I do not understand why the created json file empty is, because the database actually contains the following data:
$ mongo
MongoDB shell version: 2.2.3
connecting to: test
> use ds
switched to db ds
> db.ds.find().pretty()
{
"_id" : "1_522311",
"chr" : 1,
"kg" : {
"yri" : {
"major" : "D",
"minor" : "A",
"maf" : 0.33036
},
"ceu" : {
"major" : "C",
"minor" : "A",
"maf" : 0.05263
}
},
"pos" : 522311
}
{
"_id" : "1_223336",
"chr" : 1,
"kg" : {
"yri" : {
"major" : "G",
"minor" : "C",
"maf" : 0.473214
},
"ceu" : {
"major" : "C",
"minor" : "G",
"maf" : 0.017544
},
"jptchb" : {
"major" : "C",
"minor" : "G",
"maf" : 0.220339
}
},
"pos" : 223336
}
What did I do wrong?
Thank you in advance.
It appears that you have a database called ds:
> use ds
switched to db ds
use ds switches the current database to the ds database (db from the shell is just an alias for the current database).
Then, you have a collection called ds as well:
> db.ds.find().pretty()
So, that means you have a ds database with a ds collection (ds.ds).
You should then use the export like this with the --db option set to ds (assuming the path to the database is correct):
mongoexport --db ds --collection ds --dbpath ~/db --out ds.json
3.0+ Update: --dbpath is unavailable.
I know this answer may don't satisfied the question, but I hope it will help people struggling with mongoexport does not write any records to json output file
My problem was that I was using quotes. For example:
$mongoexport --db 'my-database' --collection 'my-collection' --out ds.json
but the correct query is (without quotes):
$mongoexport --db my-database --collection my-collection --out ds.json
I discover this when I did a $mongodump and it creates a folder with quotes. This was very estrange to me, but I understood that mongoexport interprets the quotes as part of the name. so when I removed it, it workout fine.