I was editing a long temporary file using the IPython's magis %edit -p function hwne my computer crashed from hardware reasons. I have /tmp mounted to ram, so the file cannot be retrieved from there. Can I recover the content of the temporary file? I spent a few hours writing that.
There is a way but its not very easy. You can search for deleted i nodes and you can try to recover them. But the best way is to be careful and don't work in your /tmp directory.
Here is a good explanation:
Unix undelete / recover deleted files
Related
I am getting the error like following while accessing a Postgres database
ERROR: could not access status of transaction 69675
DETAIL: Could not open file "pg_clog/0000": No such file or directory.
I didn't do anything with the pg_clog folder but the 0000 file is not there.
Is there any way to recover that file or in any way to fix this issue?
Any help would be appreciated.
You are experiencing database corruption, and you should restore from a backup. You should try to figure out what happened to the database so you can prevent it in the future.
Is your storage reliable?
Are you using dangerous settings like fsync = off?
Were there any crashes recently?
Are you really running 9.1? If yes, you shouldn't do that, as it is out of support.
Are there any files in the pg_clog directory? There should be.
Did you have an out-of-space problem recently that may have led someone to remove files from a “log” directory?
As stated in the previous response, you're better off restoring from backup, however, I discovered the metadata for those transaction files are not stored in the same location as the data when we restored the data on a server where we were doing some testing with full vacuum and needed to restore the database to an earlier state before the vacuum. In the event where your data integrity isn't as critical like a test database you can get away with creating empty files for the missing transaction logs like this:
dd if=/dev/zero of=/path/to/db/pg_clog/xxxx bs=256k count=1
chown postgres.postgres /path/to/db/pg_clog/xxxx
chmod go-rwX /path/to/db/pg_clog/xxxx
There may be multiple missing files, but if it's just a few files this is an alternative to consider.
If I want to backup database by copying raw files. What files do I need to copy ? only db-name.ns, db-name.0, db-name.1.... or whole folder (local.ns.., journal). I'm running replica set. I understand procedure for locking hidden secondary node and then copying files to new location. But I'm wondering do I need to copy whole folder or just some files.
Thx
Simple answer: All of them. As obvious as it might sound. And here is why:
If you don't copy a namespaces file, your database will most likely not work.
When not copying all datafiles, some of your data is missing and your indices will point to void locations. The database in question might work (minus the data stored in the missing data file), but I would not bet on that – and since the data was important enough to create a backup in the first place, you don't want this to happen, do you?
Config, admin and local databases are vitally necessary for their respective features – and since you used the feature, you probably want to use it after a restore, too.
How do I backup all files?
The best solution save for MMS backup I have found so far is to create LVM snapshots of the filesystem the MongoDB data resides on. In order for tis to work, the journal needs to be included. Usually, you don't need a dedicated backup node for this approach. It is a bit complicated to set up, though.
Preparing LVM backups
Let's assume you have your data in the default data directory /data/db and you have not changed any paths. Then you would mount a logical volume to /data/db and use this to hold the data. Assuming that you don't have anything like this, here is a step by step guide:
Create a logical volume big enough to hold your data. I will call that one /dev/VolGroup/LogVol1 from now on. Make sure that you only use about 80% of the available disk space in the volume group for creating the logical volume.
Create a filesystem on the logical volume. I prefer XFS, so we create an xfs filesystem on /dev/VolGroup/LogVol1:
mkfs.xfs /dev/VolGroup/LogVol1
Mount the newly created filesystem on /mnt
mount /dev/VolGroup/LogVol1 /mnt
Shut down mongod:
killall mongod
(Note that the upstart scripts sometimes have problems shutting down mongod, and this command gracefully stops mongod anyway).
Copy the datafiles from /data/dbto /mntby issuing
cp -a /data/db/* /mnt
Adjust your /etc/fstab so that the logical volume gets mounted on reboot:
# The noatime parameter increases io speed of mongod significantly
/dev/VolGroup/LogVol1 /data/db xfs defaults,noatime 0 1
Umount the logical volume from it's current outpoint and remount it on the correct one:
cd && umount /mnt/ && mount /data/db
Restart mongod
Creating a backup
Creating a backup now becomes as easy as
Create a snapshot:
lvcreate -l100%FREE -s -n mongo_backup /dev/VolGroup/LogVol1
Mount the snapshot:
mount /dev/VolGroup/mongo_backup /mnt
Copy it somewhere. The reason we need to do this is that the snapshot can only be held up until the changes to the data files do not exceed the space in the volume group you did not allocate during preparation. For example, if you have a 100GB disk and you allocated 80GB for /dev/VolGroup/LogVol1, the snapshot size would be 20GB. While the changes on the filesystem from the point you took the snapshot are less than 20GB, everything runs fine. After that, the filesystem will refuse to take any changes. So you aren't in a hurry, but you should definitely move the data to an offsite location, an FTP server or whatever you deem appropriate. Note that compressing the datafiles can take quite long and you might run out of "change space" before finishing that. Personally, I like to have a slower HDD as a temporary place to store the backup, doing all other operations on the HDD. So my copy command looks like
cp -a /mnt/* /home/mongobackup/backups
when the HDD is mounted on /home/mongobackup.
Destroy the snapshot:
umount /mnt && lvremove /dev/VolGroup/mongo_backup
The space allocated for the snapshot is released and the restrictions to the amount of changes to the filesystem are removed.
Whole db-Data folder + where ever you have your logs and journalling
The best solution to backup data on MongoDB would be to use Mongo monitoring Service(MMS). All other solutions including copying files manually, mongodump, mongoexport are way behind MMS.
I am installing postgres 8.4 on an ubuntu lucid server (no, at the moment we are using the "lucid" LTS version on that server so an upgrade is not possible yet (although we are going to start testing the system on precise quite soon now))
I have set up an own partition for the /var/lib/postgresql/8.4/main directory with a ext4 file system. (Those of you who are really into postgres installs knows what is happening now...) Since ext4 puts a lost+found directory in the root of all file system, postgres will not use that directory as its data-directory since it is initially not empty...
initdb: directory "/var/lib/postgresql/8.4/main" exists but is not empty
If you want to create a new database system, either remove or empty
the directory "/var/lib/postgresql/8.4/main" or run initdb
with an argument other than "/var/lib/postgresql/8.4/main".
The easiest way to proceed would be to remove the lost+found and recreate it after initdb has done its job. - could that cause any problems? Does the lost+found have any special attributes or anything that makes it impossible to recreate, and also, it is needed at any other time than if checkdisk finds something it needs to put there?
Another way would be to unmount the .../main/ file system, init the database, temporary mount the .../main/ filesystem somewhere else, move things over there and mount it in place. Seems to be a bit more work than the "easiest way".
Or is it some way to make initdb ignore that the directory is not empty? (couldn't see any command line switches for that)
May a lost+found directory within postgres main directory cause any problems?
At the moment I am running the system on a virtual machine for testing, so it really doesn't matter if I mess up things, but before making this an official way of installing a mission-critical system, it would be nice to have some thoughts on this.
lost+found has preallocated blocks that make it easier for fsck to move data into it when the partition is short of free blocks. To create it, better use the mklost+found command rather than mkdir.
If you don't recreate it, fsck will do it anyway when it's needed.
But if it comes to the point where fsck finds corruption within PGDATA, I'd think about going for a backup rather than counting on lost+found to retrieve anything.
I have looked at the postgres documentation and the synopsis below is given:
pg_resetxlog [-f] [-n] [-ooid ] [-x xid ] [-e xid_epoch ] [-m mxid ] [-O mxoff ] [-l timelineid,fileid,seg ] datadir
But at no point in the documentation do they explain what the datadir is.
Is it the %postgres-path%/9.0/data or could it be %postgres-path%/9.0/data/pgxlog ?
Also, if I want to change my xlog directory, can I simply move the items in my current pg_xlog directory and run the command to point to another directory? (Assume my current pg_xlog directory is in /data1/postgres/data/pg_xlog AND the directory I want it the logs to go to is: /data2/pg_xlog)
Would the following command achieve what I've just described?
mv /data1/postgres/data/pg_xlog /data2/pg_xlog
pg_resetxlog /data2
pg_resetxlog is a tool of last resort for getting your database running again after:
You deleted files you shouldn't have from pg_xlog;
You restored a file system level backup that omitted the pg_xlog directory due to a backup system configuration mistake (this happens more than you'd think, people think "it has log in the name so it must be unimportant; I'll leave it out of the backups").
File-system corruption due to a hardware fault or hard drive failure damaged your data directory; or potentially even
a PostgreSQL bug or operating system bug damaged the write-ahead logs (exceedingly rare).
As the manual says:
pg_resetxlog clears the write-ahead log (WAL) [...]. This
function is sometimes needed if these files have become corrupted. It
should be used only as a last resort, when the server will not start
due to such corruption.
Do not run pg_resetxlog unless you know exactly what you are doing and why. If you are unsure, ask on the pgsql-general mailing list or on https://dba.stackexchange.com/.
pg_resetxlog may corrupt your database, as the documentation warns. If you have to use it, you should REINDEX, dump your database(s), re-initdb, and reload your databases. Do not just continue using the damaged cluster. As per the documentation:
After running this command, it should be possible to start the server,
but bear in mind that the database might contain inconsistent data due
to partially-committed transactions. You should immediately dump your
data, run initdb, and reload. After reload, check for inconsistencies
and repair as needed.
If you simply want to move your write-ahead log directory to another location, you should:
Stop PostgreSQL
Move pg_xlog
Add a symbolic link from the old location to the new location
Start PostgreSQL
Or, as the documentation says:
It is advantageous if the log is located on a different disk from the
main database files. This can be achieved by moving the pg_xlog
directory to another location (while the server is shut down, of
course) and creating a symbolic link from the original location in the
main data directory to the new location.
If PostgreSQL fails to start, you've done something wrong. Do not use pg_resetxlog to "fix" it. Undo your changes and work out what you did wrong.
Move the contents of your pg_xlog directory to the desired location like '/home/foo/pg_xlog'
mv pg_xlog/* /home/foo/pg_xlog
Delete the pg_xlog directory
rm -rf pg_xlog
Create a soft-link of pg_xlog
ln -s /home/foo/pg_xlog pg_xlog
Verify the link
ls -lrt pg_xlog
Note: pg_resetxlog is not the right tool to move the pg_xlog please read
http://www.postgresql.org/docs/9.2/static/app-pgresetxlog.html
The data directory corresponds to the data_directory entry in the postgresql.conf file, or the PGDATA environment variable, and it can also be queried live in SQL with the SHOW data_directory statement. It does not point to the pg_xlog directory, but one level above.
To change the location of the WAL files, the PG server must be shut down, the pg_xlog directory and its contents moved to the new location, a symbolic link should be created from the old location to the new location, and the server restarted. pg_resetxlog should not be used for this, as it may suppress the latest transactions (this tool is typically used in crash recovery situations when all else fails).
You should never manually touch the WAL files, that is perfectly clear.
If there is dangling files in the pg_xlog directory, that is, there are is file which ends with .done* in the sub-folder archive_status which need to be cleaned up manually, that can be accomplished with the sql command
CHECKPOINT;
which forces a transaction checkpoint which includes cleaning up the WAL segment files.
See documentation for 9.3 but exists in all current versions of Postgresql.
I need to back up my database but I don't have enough disk space to dump it. Can I just use duplicity to perform incremental backups on the data directory? Would that corrupt the backup somehow? I don't mind a few of the latest rows missing, but I would like my backup to not be destroyed.
Does anyone know what the case is?
Thanks!
See this page. I believe duplicity uses rsync type mechanism, so you cannot simply grab the directory and go - see the page of that page about rsync. If you need to do a file system level backup, while online, then you'll need some sort of atomicity like snapshots.
Most likely, the backup simply wouldn't work.
Postgres has lots of backup options though, like PITR. I suggest a read through the fine manual.
No, you can't backup the data directory while the database service is running. You could backup the WAL-segments for a point in time recovery when you want to restore. You have to make sure you test your recovery as well, it's a litte more complicated then pgrestore of an ordinary dump.