Is there a way to prevent mongod from pre-allocating these 100 MB files in journal folder?
WiredTigerPreplog.0000000001
WiredTigerPreplog.0000000002
I want journaling to be enabled.
Find below some notes from MongoDB Documentation for Journaling
WiredTiger will pre-allocate journal files.
Preallocation Lag
MongoDB may preallocate journal files if the mongod process determines
that it is more efficient to preallocate journal files than create new
journal files as needed.
The preallocation can be avoided using only for MMapv1 storage engine using --noprealloc option when starting the mongod. But this is not applicable for wiredtiger storage engine.
few more references
Meaning of WiredTigerPreplog files
I often need to run test instances of mongod, and these preallocated WiredTiger journal (log) files just waste 200MB each time.
They can disabled by adding this parameter to your mongod command line:
--wiredTigerEngineConfigString 'log=(prealloc=false)'
Or this to your mongod.conf file:
storage:
wiredTiger:
engineConfig:
configString: log=(prealloc=false)
Of course this should never be done in production, and only when testing things that are unrelated to journalling. Journal file preallocation is a deliberate performance feature, which is almost always a win in the real world (and so is why it defaults to true).
Related
whats the command in mongo to check current Journal space used or whats the limit set
using Mongo version 3.2.8 and storage engine wiredTiger
What is the Journal space used by Mongo?
MongoDB uses a journal file size limit of 100 MB, WiredTiger creates a new journal file approximately every 100 MB of data. When WiredTiger creates a new journal file, WiredTiger syncs the previous journal file.
Source Journaling
How can I reduce the Journal file size?
Yes - there is a way to minimize the default size of the journal
files, subject to a couple of caveats. From the MongoDB configuration
documentation:
To reduce the impact of the journaling on disk usage, you can leave journal enabled, and set smallfiles to true to reduce the size of the
data and journal files.
Here is the smallfiles config information:
Set to true to modify MongoDB to use a smaller default data file size. Specifically, smallfiles reduces the initial size for data files
and limits them to 512 megabytes. The smallfiles setting also reduces
the size of each journal files from 1 gigabyte to 128 megabytes.
Use the smallfiles setting if you have a large number of databases that each hold a small quantity of data. The smallfiles setting can
lead mongod to create many files, which may affect performance for
larger databases.
Source Run a MongoDB configuration server without 3GB of journal files, answer by platforms
What's the best/fastest/safest way to recover deleted files from ext4 ?
Specs:
The disk is 1TB SSHD (hibrid HDD + SSD), also the partition is encrypted with LUKS Encryption (version 1)
Mongodb is using WiredTiger as a storage engine.
Also if I manage a partial recovery of files, could I do a partial recovery of mongo's collections?
Step 1: File recovery
Fast Recovery of files using extundelete:
sudo umount /path/to/disk &&
sudo extundelete /path/to/disk --restore-directory /path/to/dir -o /restored/path/
/path/to/disk represents the disk path, e.g. /dev/sdd , /dev/mapping/label
/path/to/dir represents the path that you want recovered relative to disk mounting point, e.g. if /dev/ssd would be mounted at /mnt/label/ the full path would be /mnt/label/path/to/dir and the relative path is /path/to/dir
pros of recovery with extundelete:
it's lightweight
can work if the disk is mounted or encrypted
pretty fast, it gave answers if recovery is possible in seconds and it writes the recovered files with over 100 MB/s
cons for data recovery in general
no guarantee for success
won't work if new data was written in the deleted sectors (so unmount the disk as soon as possible and make an image of the broken disk before any recovery)
Step 2 : repair mongodb if missing data
Backup before this step, mongod --repair could delete good data
Untested, but from my understanding mongod --repair should help repair the database if incomplete otherwise you can continue recovery for WiredTiger with :
Recovering a WiredTiger collection from a corrupt mongodb installation
I try to start mongod.exe but I have and I get the following error:
C:\MongoDB\Server\30\bin>mongod.exe
2015-12-16T19:12:17.108+0100 I CONTROL 2015-12-16T19:12:17.110+0100 W CONTROL 32-bit servers don't have journaling enabled by default.
Please use --journal if you want durability.
2015-12-16T19:12:17.110+0100 I CONTROL
2015-12-16T19:12:17.120+0100 I CONTROL Hotfix KB2731284 or later update is not installed, will zero-out data files
2015-12-16T19:12:17.132+0100 I STORAGE [initandlisten] **************
2015-12-16T19:12:17.132+0100 I STORAGE [initandlisten] Error: journal files are present in journal directory, yet starting without journaling enabled.
2015-12-16T19:12:17.133+0100 I STORAGE [initandlisten] It is recommended that you start with journaling enabled so that recovery may occur.
2015-12-16T19:12:17.133+0100 I STORAGE [initandlisten] **************
2015-12-16T19:12:17.135+0100 I STORAGE [initandlisten] exception in initAndListen: 13597 can't start without --journal enabled when journal/ files are present, terminating
2015-12-16T19:12:17.135+0100 I CONTROL [initandlisten] dbexit: rc: 100
I also tried to run it with --repair but then I get the same error.
Finally, I tried to delete the mongod.lock file but I still get the error.
How should I fix the unclean shutdown?
The solution to this problem is mongod --repair. This command automatically shuts down the all processes and repairs Mongodb issues. You can find more details in the official documentation.
Ok, to get some confusion right here. Journal files are not there to annoy you. They hold data not yet applied to the datafiles, but already received and acknowledged by the server. The mongod process finished a request after applying the data to the journal, but before applying them to the data files.
This behavior is configured by the chosen write concern.
Bottom line: special measurements were taken to make the data in the journal durable, you should not ignore that.
So you should create a configuration file containing this (among other things, if one already exists):
storage:
journal:
enabled: true
Please follow the documentation on running MongoDB on windows to the letter. Adjust the configuration file with options according to your needs.
If you are absolutely, positively sure that you do not need journaling, you can start mongodb with the --journal command line option just once, shut the instance down after the journal was successfully applied and remove the journal files then. Expect any write with a write concern involving the journal to fail, however.
Note 1 You are using the 32-bit version of MongoDB, which is only suitable for testing. Note that the 32-bit version only supports up to 2Gb of data.
Note 2 MongoDB is VERY well documented. You really should read the manual from top to bottom – it get's you started fast enough with providing a lot of information on the internals.
start cmd shell as admin and call start_mongo. This should fix it
Same error.
It' permission issue. If you get this error on Windows platform you should do all operations with administrator privilegies.
On Linux run
mongod --repair
but you should run it with sudo or under root. If under root you should change ownership of the files in data DB directory. Do it carefully or another error will appear.
Try removing the lock file, then running with --repair.
Here's what the Mongo Docs say about recovering data / restarting after an unexpected shutdown.
I am running mongoDB on a local appliance machine, with journaling enabled.
Is it always guaranteed that mongo will recover itself automatically even on power outage(meaning that the database was not closed properly) when journaling is enabled?
On what scenarios MongoDB will be corrupted even if journaling is enabled(besides filesystem corruption)?
Yes, it is guaranteed (assuming no filesystem corruption):
With journaling enabled, MongoDB creates a journal subdirectory within the directory defined by dbPath, which is /data/db by default. The journal directory holds journal files, which contain write-ahead redo logs. The directory also holds a last-sequence-number file. A clean shutdown removes all the files in the journal directory. A dirty shutdown (crash) leaves files in the journal directory; these are used to automatically recover the database to a consistent state when the mongod process is restarted.
(Journaling /core/journaling in the manual)
This is a big point for journaling in the first place and one of the primary reason journaling is used. Note data will still likely be lost (from the last 100ms or so) but the DB will be in a consistent state.
Background:
I installed below 2 rpm packages on CentOS 6.3(Final) 64bits with Ext4 file system -
mongo-10gen-2.2.1-mongodb_1.x86_64.rpm
mongo-10gen-server-2.2.1-mongodb_1.x86_64.rpm
Symptom:
When mongod starts up at first time, there are no preallocated journal files created in specified folder. As aside note, the journal option is enable by default.
Questions:
Is it normal case or not? I review manual on MongoDB. It claims
If no journal files exist, when mongod starts, it MUST preallocates new journal files
I posted mongod relevant mongo.conf below -
configurations:
logpath=/drbd0/mongo/log/mongod.log
logappend=true
fork = true
port = 27017
dbpath=/drbd0/mongo/data
pidfilepath = /var/run/mongodb/mongod.pid
bind_ip=admin1_ss_nc
replSet=rstest
Thanks in advance.
MongoDB will only pre-allocate the journal files if it believes that it'll be faster to pre-allocate files of a given size (three files 128 MB each if running with --smallfiles and three files 1 GB each if not running with --smallfiles) than to allocate them on-demand.