I'm working on setting up point in time backups for a PSQL server that I have running, and I'm following a tutorial for an earlier version. I'm trying to figure out what the specific directory is for the DB cluster in PSQL-12 so that I can clear out that directory and test what I've setup. In the video, he runs a recursive remove on the PSQL-10 directory /var/lib/postgresql/10/main, and is still able to start the PSQL-10 service again when he's finished the restoration.
When I attempted it, I ran the recursive remove on the directory /var/lib/pgsql/12/data/ because the command SHOW data_directory; told me that is where my server's cluster data is stored. Removing all the data, however, messes up the postgresql-12.service, so I can't start it back up when I've completed the recovery.
This is displayed when I restore the backup and run systemctl start postgresql-12.service:
Process: 26672 ExecStart=/usr/pgsql-12/bin/postmaster -D ${PGDATA} (code=exited, status=1/FAILURE)
Dec 31 11:07:29 localhost.localdomain systemd[1]: Failed to start PostgreSQL 12 data....
I've tried making a backup of the working /data/ directory and doing a diff -qr to see what files differ between the working backup and the point in time backup, but coping those files from the working directory to the PIT directory doesn't seem to fix the issue, and I'm still unable to start the postgresql-12.service. It seems, however, that I am able to start the service back up successfully if I just do a mass copy of the working directory to /var/lib/postgresql/10/main.
Can someone please point me in the right direction? I've done plenty of research trying to find the working cluster directory so I can just erase table information and work on a PIT recovery without messing up the core application prereqs (such as the service), but I can't seem to find the information I'm looking for. Any assistance would be greatly appreciated! Additionally, if there's a way to spot this directory more quickly in the future, either by a command or looking at the files within, I would love to know so I can implement this procedure on different PSQL versions. Thank you!
Related
I accidentally deleted a volume of docker mongo-data:/data/db , i have a copy of that folder , now the problem is when i run docker-compose up mongodb container doesn't start and gives an error of mongo_1 exited with code 14 below more details of the error and the mongo-data folder , can you someone help me please
in docker-compose.yml
volumes:
- ./mongo-data:/data/db
Restore from backup files
A step-by-step process to repair the corrupted files from a failed mongodb in a docker container:
! Before you start, make copy of the files. !
Make sure you know which version of the image was running in the container
Spawn new container with to run the repair process as follows
docker run -it -v <data folder>:/data/db <image-name>:<image-version> mongod --repair
Once the files are repaired, you can start the containers from the docker-compose
If the repair fails, it usually means that the files are corrupted beyond repair. There is still a chance to repair it with exporting the data as described here.
How to secure proper backup files
The database is constantly working with the files, so the files are constantly changed on the disks. In addition, the database will keep some of the changes in the internal memory buffers before they are flushed to the filesystem. Although the database engines are doing very good job to assure the the database can recover from abrupt failure by using the 2-stage commit process (first update the transaction-log than the datafile), when the files are copied there could be a corruption that will prevent the database from recovery.
Reason for such corruption is that the copy process is not aware of the database written process progress, and this creates a racing condition. With very simple words, while the database is in middle of writing, the copy process will create a copy of the file(s) that is half-updated, hence it will be corrupted.
When the database writer is in middle of writing to the files, we call them hot files. hot files are term from the OS perspective, and MongoDB also uses a term hot backup which is a term from MongoDB perspective. Hot backup means that the backup was taken when the database was running.
To take a proper snapshot (assuring the files are cold) you need to follow the procedure explained here. In short, the command db.fsyncLock() that is issued during this process will inform the database engine to flush all buffers and stop writing to the files. This will make the files cold, however the database remains hot, hence the difference between the terms hot files and hot backup. Once the copy is done, the database is informed to start writing to the filesystem by issuing db.fsyncUnlock()
Note the process is more complex and can change with different version of the databse. Here I give a simplification of it, in order to illustrate the point about the problems with the file snapshot. To secure proper and consistent backup, always follow the documented procedure for the database version that you use.
Suggested backup method
Preferred backup should always be the data dump method, since this assures that you can restore even in case of upgraded/downgraded database engines. MongoDB provides very useful tool called mongodump that can be used to create database backups by dumping the data, instead by copy of the files.
For more details on how to use the backup tools, as well as for the other methods of backup read the MongoDB Backup Methods chapter of the MondoDB documentation.
At my work I was running a complex query. I cancelled it and went home yesterday. This morning in the back the query was impossible to be terminated, also with the 'terminate backend' functionality. A colleague of mine restarted the host machine where postgres is installed. After the machine restart, the postgres database sever would not start up.
In my log files I see the error:
'pg_ctl: this data directory appears to be running a pre-existing postmaster'
I am not sure how to handle this problem. I could try to fix it or try to extrapolate the data from the save files. What is the most logical step to take and do you know how to fix this?
Earlier it gave this error message :
2016-01-28 15:52:33 GMT FATAL: lock file "postmaster.pid" already exists
2016-01-28 15:52:33 GMT HINT: Is another postmaster (PID 2100) running in data directory "C:/PostgreSQL/9.1/data"?
UPDATE... I located the file postmaster.pid and deleted it. Now I am restarting the computer and hoping it will start.
UPDATE... It works now. I rebooted the computer and postgres just instantly started. Happy as a child but at the same time not fully satisfied because of the following forum: https://superuser.com/questions/553045/fatal-lock-file-postmaster-pid-already-exists . Here it is stated to NEVER delete the postmaster.pid because of possible data corruption. So because of that I will backup all databases I have in postgres now.
So if anyone can share some more light on my ICT adventure of today I would be very satisfied. That is why I will not state that this question is answered, since I have no idea what went wrong and perhaps will run into it again someday.
The explanation is pretty straightforward. PostgreSQL writes the process ID to a file called postmaster.pid — the presence of the file is supposed to indicate that the server is running. When the PostgreSQL shuts down cleanly, it removes the postmaster.pid file.
However, when your colleague restarted the host machine, the PostgreSQL server got killed without having had a chance to remove the postmaster.pid file. Therefore, when you tried to start PostgreSQL, the presence of the file made it look complain that the server was already running.
This answer provides more complete advice. In general, you should never delete postmaster.pid for no good reason, because it's supposed to help prevent two servers from running at once on the same data files. However, if you are certain that the process indicated by the postmaster.pid file is already dead, then by all means just delete the stale PID file manually.
In windows Delete all running postgres processes and start the service
Our DB of +- 400Gb is stopping on our one server.
From the logs:
2015-07-07T09:09:51.072+0200 I STORAGE [conn10] _getOpenFile() invalid file index requested 8388701
2015-07-07T09:09:51.072+0200 I - [conn10] Invariant failure false src/mongo/db/storage/mmap_v1/mmap_v1_extent_manager.cpp 201
2015-07-07T09:09:51.082+0200 I CONTROL [conn10]
Any idea in what are I should start looking? Storage issue?
I am just answering this question in case some people make the same non-technical mistake again:
I tried to scp all the files in the /data/db directory to the server. As the files are many (dbname.1 to dbname.55, about 100GB), it was interrupted in the middle (last successful file dbname.22), and I restarted and uploaded dbname.23 to dbname.55. And when I run queries in mongo client, it worked for some cases, and failed for some others showing the error message the same as in the question. I thought it might be some file broken in the file transferring, but the md5 check was all right. Only after I spent a long time finishing all the md5 check I found the reason.
It turned out to be that scp uploads dbname.21 to dbname.29 after it uploads dbname.2, so dbname.3 to dbname.9 was never uploaded to the server. I am going to upload them, and this should solve the problem.
I ran into a variant of this today as well. Mysteriously one of my data files disappeared (or didn't make it in a migration from another server). None of the repair/recovery procedures would work, failing on the same error you reference. Luckily I have a separate mongod that has a collection with the same name, so as a cheap hack I copied the (admittedly wrong) data file to the other server, and while I knew I wouldn't get any data back, the repair tools (such as mongod --repair) were then able to work their magic, but as expected, they recovered some data from the bad file I copied in, so I had to weed out some docs. Luckily it was the "mycollection.1" file, which is only 128MB.
I don't think this applies in your case since index of the missing data file your log is talking about is ridiculously high. Your log is essentially saying it can't find /data/dbname/mycollection.8388701. You said your data-set is only 400GB, so an index that high just doesn't make sense. You should have only roughly 200 data files since most of them are 2GB each by default. What is the result of db.stats() (specifically the fileSize attribute)?
This mongolab blog entry helped me understand the data file structure.
My advice for where you should start looking:
run the db.stats() command to get an idea of how big your data on
disk actually is.
Does it make sense for your server to be looking for a data file with a crazy high index? If not, the issue isn't really with storage, but with the extents and the metadata of your collection/database.
Do your repair tools work? If you have at least enough free disk space as the size of your data set (on disk), try the mongod --repair, or db.repairDatabase() tools to start a repair. I'm assuming it won't work since my repair attempts crashed with the same invalid file index requested error.
Try copying a "bad" file like I did that roughly matches what the missing file would look like (keeping in mind how the file sizes of the data files aren't all the same, do your best to match it up and try a repair). If this works, your data files will be cleaned up (but it does take a lot of disk space).
Hope that helps point you in the right direction.
In my case this happened in a development setting with MongoDB 3.6.20 on macOS 10.14.6. Another program restarted the mac and close any open terminals, including the terminal that ran the mongod process. After the OS restart, I could not restart the mongod because the Invariant failure. The error also mentioned a bad lockfile.
I was able to solve the issue with the following steps, yet I am not exactly sure which did the job:
remove corrupted lock file: rm -rf data/db/mongod.lock
direct outcome: mongod still failed due to Invariant failure but at least no mention about the lockfile anymore.
run mongod --repair
direct outcome: repair still failed due to Invariant failure. Error output mentions SocketException: Address already in use.
restart the machine again to free the socket.
direct outcome: mongod starts and runs without problems. Yay.
The first successful mongod run after the issue gave the following output:
[ftdc] Unclean full-time diagnostic data capture shutdown detected, found interim file, some metrics may have been lost.
Thus, it runs smoothly again. Maybe I was fortunate. I hope the same approach helps some of you.
I have tried to start mongod.exe from my 2008 R2 server and im getting this error:
I CONTROL Hotfix kb2731284 or a later update is not installed, will zero-out files.
I didnt find any update or what is I CONTROL, someone have encounter with the problem?
Thank you.
Well, I just faced the same issue. I installed the fix, but saw nothing but same error. So that's how I got over it: just create a folder structure in your C catalog like this one: C:\data\db. So it worked for me perfectly. I use Windows 7 x64.
I had to run the command in following format & it worked for me:
C:\mongodb\bin\mongod.exe --dbpath d:\test\mongodb\data
Ref: Original documentation link
In my case the problem was that I had a space in my data path: E:\Program Files\MongoDB\Data\
I ran dir /x within E:\ to get the short path for Program Files and used the short path instead: mongod.exe --dbpath E:\PROGRA~1\MongoDB\Data\
It worked. Shocking that spaces in paths are still an issue!
The above error comes when mongodb not found the directory like "C:\data\db" which is by default when you install mongodb on your machine.
So just go to c drive create a directory like "data\db" and now run the command "mongodb.exe".
If you want to relocate the database location to any other drive say D drive in that case create a directory like "D:\mongodb\data" and run the command like "mongodb.exe --dbpath d:\mongodb\data".
I got the similar error
I CONTROL Hotfix kb2731284 or a later update is installed, will zero-out files.
Notice the message says later update is installed not later update is not installed in my case. But It was not starting the server.
Creating a different folder structure for the db Drive\<mongo-db-folder>\<yourdb>and providing it with --dbpath flag while running the mongod worked for me. The short name for Program Files PROGRA~1 didn't work for me. Maybe this has something to do with the rights managment.
This known issue is in Windows 7. In Windows 8 this problem haven't appeared yet.
Firstly I would recommend to change your mongo installation path to path with folders names without gaps. For example, default mongo installation path is C:\program files\bla-bla... to avoid a wasting of time for looking for solutions of this issue, change your path to C:\mongo.. and make sure that you haven't names with gaps in your path.
After that install this hot-fix.
Another thing I suggest you to do: Create folder data\db in disc C.
You could also change your mongo work path each time you use mongo shell, but it's overwork by my modest opinion.
I've done these steps and my mongo has started to work.
Getting started on mongodb was not that difficult. Well I have been facing similar error on mongodb to start.
I have getting error
Services
Windows could not start the MongoDB on Local Computer. For more information, review the System Event Log. If this is a non-Microsoft service, contact the service vendor, and refer to service-specific error code 100.
OK
very simple way to overcome.
Solution is the Mongodb service has not been started
to do so. Open CMD in admin mode and type NET START "Mongodb" > This will start the service.
Inspite of this if services is not getting started or mongo.exe is not starting. It is coz your mongodb has locked the file. Mongodb is shared file system and it gets locked. You need to for mongo.lock file in your created dir. Delete the file and start the service. You should be able to start now.
Cheers
By just creating a folder structure like below in my "C:\" drive , i resolved the issue :=>
c:\data\db
Sometime it works without creating the above mentioned folder structure, i.e. MONGODB itself creates the same (as it worked for me in Windows 8 laptop) but sometimes it needs to be explicitly done (as i need to for my windows 7 installed desktop)
Also you can browse through the complete installation and connection process for MONGODB.
Similiar to above, but I had a typo in my mongodb.cfg file which meant that Mongo couldn't find the data directory on startup. Fixed the typo and Mongo starts. Given the previous response I suspect that the failure to start is more of an issue over finding the data directory than the Hotfix message
If you had got the below error message::
2015-05-29T05:05:54.207+0530 I CONTROL Hotfix KB2731284 or later update is not
installed, will zero-out data files
2015-05-29T05:05:54.219+0530 I STORAGE [initandlisten] exception in initAndLis
en: 29 Data directory D:\data\db\ not found., terminating
2015-05-29T05:05:54.219+0530 I CONTROL [initandlisten] dbexit: rc: 100
then just create a directory "D:\data\db\""
then restart the mongod
it should work fine
you need to have mongod.exe running in one cmd window. then use another cmd window, write mongo and hit enter
You need to create the C:\Data\Db directory if you don't specify an existing directory when you run mongod.exe.
This directory is the default one.
http://docs.mongodb.org/manual/tutorial/install-mongodb-on-windows/
You can do fix [initandlisten] exception in initAndListen: 29 Data directory /data/db not found., terminating by following given two steps.
You can create the D:\data\db\ directory and restart the MongoDB server.
You can create the data folder as per your choice and run the server with the path of your data folder and restart the MongoDB server
i.e. D:\MongoDB\Server\3.0\bin>mongod.exe --dbpath D:\MongoDB\Server\data
I just deleted mongod.lock in my db directory, and now it works fine.
1 : Download and install the fix from here :
http://hotfixv4.microsoft.com/Windows%207/Windows%20Server2008%20R2%20SP1/sp2/Fix405791/7600/free/451413_intl_x64_zip.exe
them reboot windows
2 : this you should make a folder called "data" and within it another one called "db"
c:/data/db ;
I would start with the second step , it's most likely to solve The problem .
As for what "I CONTROL" means, the "I" is for "Information" (we have W for warning and E for error) and CONTROL is the component writing that log/message (we have also NETWORK, WRITE, QUERY and COMMAND components).
So you could get also I COMMAND ..., I QUERY ..., I WRITE ... :)
Well, that is a common problem when you shut down your machine and mongodb server is running, it is pretty clear that MongoDb has created a lock in your db folder.
Just go to c:\data\db and remove MONGOD.LOCK
Run Mongo and it will work again.
I am installing postgres 8.4 on an ubuntu lucid server (no, at the moment we are using the "lucid" LTS version on that server so an upgrade is not possible yet (although we are going to start testing the system on precise quite soon now))
I have set up an own partition for the /var/lib/postgresql/8.4/main directory with a ext4 file system. (Those of you who are really into postgres installs knows what is happening now...) Since ext4 puts a lost+found directory in the root of all file system, postgres will not use that directory as its data-directory since it is initially not empty...
initdb: directory "/var/lib/postgresql/8.4/main" exists but is not empty
If you want to create a new database system, either remove or empty
the directory "/var/lib/postgresql/8.4/main" or run initdb
with an argument other than "/var/lib/postgresql/8.4/main".
The easiest way to proceed would be to remove the lost+found and recreate it after initdb has done its job. - could that cause any problems? Does the lost+found have any special attributes or anything that makes it impossible to recreate, and also, it is needed at any other time than if checkdisk finds something it needs to put there?
Another way would be to unmount the .../main/ file system, init the database, temporary mount the .../main/ filesystem somewhere else, move things over there and mount it in place. Seems to be a bit more work than the "easiest way".
Or is it some way to make initdb ignore that the directory is not empty? (couldn't see any command line switches for that)
May a lost+found directory within postgres main directory cause any problems?
At the moment I am running the system on a virtual machine for testing, so it really doesn't matter if I mess up things, but before making this an official way of installing a mission-critical system, it would be nice to have some thoughts on this.
lost+found has preallocated blocks that make it easier for fsck to move data into it when the partition is short of free blocks. To create it, better use the mklost+found command rather than mkdir.
If you don't recreate it, fsck will do it anyway when it's needed.
But if it comes to the point where fsck finds corruption within PGDATA, I'd think about going for a backup rather than counting on lost+found to retrieve anything.