due to a bad foreign key, we had some non written inserts in to database, but i have the wal files generated when this actions, just a few, so im wondering if is possible to decrypt them and read the original query values. i cant replay anithing because database owner just can give me those wal files.
thanks in advance.
I have seen the pg_waldump but no example of how to use and wanted to know if there is something just to make this bin readable
I would have something at output like "insert blabla into blabla"
finally with the plugin for vscode hex editor, i have been able to read SOME values inside each wal file. The problem is data is not so ordered, like i have a huge list of emails, then a huge list of messages, so i cant be sure to which email correspond wich message, and also there were boolean values that i cant find, despite of message. but yes, the data is here. in bad way but bit readable
No, that is not possible, since WAL contains low-level binary information that cannot readily be translated into SQL statements. Moreover, it is unclear what "non-written inserts" are – if they were not written, they were probably not logged in the WAL either.
Without a base backup, WAL files are pretty useless.
Related
So, I am (well... I was) running PostgreSQL within a container (Ubuntu 14.04LTS with all the recent updates, back-end storage is "dir" because of convince).
To cut the long story short, the container folder got deleted. Following the use of extundelete and ext4magic, I have managed to extract some of the database physical files (it appears as if most of the files are there... but not 100% sure if and what is missing).
I have two copies of the database files. One from 9.5.3 (which appears to be more complete) and one from 9.6 (I upgraded the container very recently to 9.6, however it appears to be missing datafiles).
All I am after is to attempt and extract the SQL code the relates to the user defined functions. Is anyone aware of an approach that I could try?
P.S.: Last backup is a bit dated (due to bad practices really) so it would be last resort if the task of extracting the needed information is "reasonable" and "successful".
Regards,
G
Update - 20/4/2017
I was hoping for a "quick fix" by somehow extracting the function body text off the recovered data files... however, nothing's free in this life :)
Starting from the old-ish backup along with the recovered logs, we managed to cover a lot of ground into bringing the DB back to life.
Lessons learned:
1. Do implement a good backup/restore strategy
2. Do not store backups on the same physical machine
3. Hardware failure can be disruptive... Human error can be disastrous!
If you can reconstruct enough of a data directory to start postgres in single user mode you might be able to dump pg_proc. But this seems unlikely.
Otherwise, if you're really lucky you'll be able to find the relation for pg_proc and its corresponding pg_toast relation. The latter will often contain compressed text, so searches for parts of variables you know appear in function bodies may not help you out.
Anything stored inline in pg_proc will be short functions, significantly less than 8k long. Everything else will be in the toast relation.
To decode that you have to unpack the pages to get the toast hunks, then reassemble them and uncompress them (if compressed).
If I had to do this, I would probably create a table with the exact same schema as pg_proc in a new postgres instance of the same version. I would then find the relfilenode(s) for pg_catalog.pg_proc and its toast table using the relfilenode map file (if it survived) or by pattern matching and guesswork. I would replace the empty relation files for the new table I created with the recovered ones, restart postgres, and if I was right, I'd be able to select from the tables.
Not easy.
I suggest reading up on postgres's storage format as you'll need to understand it.
You may consider https://www.postgresql.org/support/professional_support/ . (Disclaimer, I work for one of the listed companies).
P.S.: Last backup is a bit dated (due to bad practices really) so it would be last resort if the task of extracting the needed information is "reasonable" and "successful".
Backups are your first resort here.
If the 9.5 files are complete and undamaged (or enough so to dump the schema) then simply copying them in place, checking permissions and starting the server will get you going. Don't trust the data though, you'll need to check it all.
Although it is possible to partially recover given damaged files, it's a long complicated process and the fact that you are asking on Stack Overflow probably means it's not for you.
In my postgesql database, unfortunately I truncate this table mail_group, and the table is delete from the database, how to I get back this table.
Kindly help me, waiting for reply.
Thanks
Anyone else in the same situation: immediately stop your database with pg_ctl stop -m immediate (the immediate is important, you need to simulate a crash and prevent a checkpoint) then do not restart it.. If you had concurrent transactions still in progress you might be really lucky and PostgreSQL might not have unlinked the backing files for the table yet, so it could maybe be recoverable.
You very likely can't get the data back, you deleted it. Restore from a backup.
A normal DELETE in PostgreSQL marks the rows as deleted but does not actually erase the data immediately, so it can often be recovered if you promptly stop the database and you don't write anything else to the table.
This is not the case for TRUNCATE. TRUNCATE deletes the underlying files that represent the database table from the file system.
Recovering the data, if possible at all, would require forensic analysis of your hard drive. If the data is truly important then power the computer off now and take a disk image of the hard drive. Expect recover work to cost multiple thousand dollars, if it is possible at all, since you will need someone who knows both (a) file system internals and (b) PostgreSQL internals. The only person I can think of who I know has the skills to possibly be able to do this would probably cost about €5000 to €10000 for the time required for this sort of work. (It isn't me).
If you didn't have backups you have just learned a very expensive lesson.
If someone else is reading this and DELETEd rows, please immediately follow the instructions in corruption since the first recovery steps are the same. This will not help if you ran TRUNCATE.
Is there any way to recover or re-create pg_filenode.map file that was accidentally deleted? Or is there any solution on how to fix this issue without affecting the database? Any suggestions to fix this issue is highly appreciated! The postgres version that we have is 9.0 running in Redhat Linux 5. Thanks!
STOP TRYING TO FIX ANYTHING RIGHT NOW. Everything you do risks making it worse.
Treat this as critical database corruption. Read and act on this wiki article.
Only once you have followed its advice should you even consider attempting repair or recovery.
Since you may have some hope of recovering the deleted file if it hasn't been overwritten yet, you should also STOP THE ENTIRE SERVER MACHINE or unmount the file system PostgreSQL is on and disk image it.
If this data is important to you I advise you to contact professional support. This will cost you, but is probably your best chance of getting your data back after a severe administrator mistake like this. See PostgreSQL professional support. (Disclaimer: I work for one of the listed companies as shown in my SO profile).
It's possible you could reconstruct pg_filenode.map by hand using information about the table structure and contents extracted from the on-disk tables. Probably a big job, though.
First, if this is urgent and valuable, I strongly recommend contacting professional support initially. However, if you can work on a disk image, if it is not time critical, etc. here are important points to note and how to proceed (we recently had to recover a bad pg_filenode.map. Moreover you are better off working on a disk image of a disk image.
What follows is what I learned from having to recover a damaged file due to an incomplete write on the containing directory. It is current to PostgreSQL 10, but that could change at any time
Before you begin
Data recovery is risky business. Always note what recovery means to your organization, what data loss is tolerable, what downtime is tolerable etc before you begin. Work on a copy of a copy if you can. If anything doesn't seem right, circle back, evaluate what went wrong and make sure you understand why before proceeding.
What this file is and what it does
The standard file node map for PostgreSQL is stored in the pg_class relation which is referenced by object id inside the Pg catalogs. Unfortunately you need a way to bootstrap the mappings of the system tables so you can look up this sort of informatuion.
In most deployments this file will never be written. It can be copied from a new initdb on the same version of Postgres with the same options passed to initdb aside from data directory. However this is not guaranteed.
Several things can change this mapping. If you do a vacuum full or similar on system catalogs, this can change the mapping from the default and then copying in a fresh file from an initdb will not help.
Some Things to Try
The first thing to try (on a copy of a copy!) is to replace the file with one from a fresh initdb onto another filesystem from the same server (this could be a thumb drive or whatever). This may work. It may not work.
If that fails, then it would be possible perhaps to use pg_filedump and custom scripting/C programming to create a new file based on efforts to look through the data of each relation file in the data directory. This would be significant work as Craig notes above.
If you get it to work
Take a fresh pg_dump of your database and restore it into a fresh initdb. This way you know everything is consistent and complete.
A stupid newbie question: I want to make a PostgreSQL (9.2.2 with PostGIS 2.0.1; on 32-bit Windows XP) database with rasters saved outside the database (I will need the rasters to be accessed from outside the database and they won't be uploaded/migrated frequently, so consistence is not an issue). My problem is: I don't know how to make the links to the rasters (from database with metadata), and I didn't find anything comprehensible enough.
I have found something about data wrappers, but they seem to be intended for data with table structure, not files like rasters. DATALINK seems better, but I'm afraid it's the same case, plus I'm not sure I understood how to use it. In some of the discussions I've found a mention of symbollic links, but these seem to be something Unix-based, and probably only vaguely related.
I'm sure it must be simple, but I didn't manage to solve it myself.
Databases provide no possibilities to link outside objects.
I can think of at least 2 approaches:
Save a full path to your files in some metadata table as one of the attributes or type text. Don't use it for joining tables in queries though, having artifitial key of internal numeric type (like integer or bigint) is a better choice for performance reasons;
Name your raster files according to their numeric keys in the database. This approach has a drawback — without database you will not be able to obtain any usefull info about your files.
Further paths depends on the complexity of your system and choosen optimization techniques.
How can I write a program that can recover files in FAT32?
This is pretty complex, but FAT32 is very good documented:
I wrote a tool for direct FAT32 access once using only those ressources:
http://en.wikipedia.org/wiki/File_Allocation_Table
http://support.microsoft.com/kb/154997/
http://www.microsoft.com/whdc/system/platform/firmware/fatgen.mspx
But I've never actually tried to recover files. If you will successfully recover a file depends on several factors:
The file must still "exist" physically on the hard disk
You must know where the file starts
You must know what you are looking for (Headers..)
It depends on what happened to the files you're trying to recover. The data may still be on the partition, or it could be overwritten by now. There are a lot of pre-written solutions. A simple google search should give you a plethora of software that can try to recover the data, but it's not 100% sure to get them back. If you really want to recover them yourself, you'll need to write something the read the raw partition and ignore missing file markers.
here is a program (written by Thomas Tempelman. This guy is great.) that might help you out. You can make a copy of the partition, ignoring corrupt bits, then operate on the copy so you don't mess anything up, and you may also be able to recover the data directly with it.
I think you are referring to data carving, that is, reading the physical device and reconstructing previously unlinked files based on some knowledge (e.g. when you find two letters, PK, it's highly probable than a zip archive is following, same for JFIF for JPEG).
In this case, I suggest you to study the source code of PhotoRec a great (in my opinion, the best) Open Source tool for data carving.