You've heard the story, disk fails, no recent db backup, recovered files in disarray...
I've got a pg 9.1 database with a particular table I want to bring up to date. In the postgres data/base/444444 directory are all the raw files with table and index data. One particular table I can identify and it's files are as follows:
[relfindnode]
[relfindnode]_vm
[relfindnode]_fsm
where [relfindnode] is the number corresponding to the table I want to reconstruct.
In the current out-of-date database the main [relfindnode] file is 16MB.
In my recovered files, I have found the corresponding [relfindnode] file and the _vm and _fsm files. The main [relfindnode] file is 20MB, so I'm hopeful that this contains more upto-date data.
However, when I swap the files over and restart my machine (OS X) and I inspect the table, it has approximately the same number of records in it (not exactly the same).
Question: is it possible just to swap out these files and expect it to work? How else can I extract the data from the 20MB table file? I've read the other threads here regarding constructing from raw data files.
Thanks.
Related
I'm trying to load some large data sets from CSV into a Postgres 11 database (Windows) to do some testing. Fist problem I ran into was that with very large CSV I got this error: "ERROR: could not stat file "'D:/temp/data.csv' Unknown error". So after searching, I found a workaround to load the data from a zip file. So I setup 7-zip and was able to load some data with a command like this:
psql -U postgres -h localhost -d MyTestDb -c "copy my_table(id,name) FROM PROGRAM 'C:/7z e -so d:/temp/data.zip' DELIMITER ',' CSV"
Using this method, I was able to load a bunch of files of varying sizes, one with 100 million records that was 700MB zipped. But then I have one more large file with 100 million records that's around 1GB zipped, that one for some reason is giving me grief. Basically, the psql process just keeps running and never stops. I can see based on data files growing that it generates data up to a certain point, but then at some point it stops growing. I'm seeing 6 files in a data folder named 17955, 17955.1, 17955.2, etc. up to 17955.5. The Date Modified date on those files continues to be updated, but they're not growing in size and my psql program just sits there. If I shut down the process, I lose all the data since I assume it rolls it back when the process does not run to completion.
I looked at the logs in the data/log folder, there doesn't seem to be anything meaningful there. I can't say I'm very used to Postgres, I've used SQL Server the most, so looking for tips on where to look, or what extra logging to turn on, or anything else that could help figure out why this process is stalling.
Figured it out thanks to #jjanes comment above (sadly he/she didn't add an answer). I was adding 100 million records to a table with a foreign key to another table with 100 million records. I removed the foreign key, added the records, and then re-added the foreign key, that did the trick. I guess checking the foreign keys is just too much with a bulk insert this size.
We are creating dump file of multiple database at every one hour but the issue is storage and time of restoration. We have multiple location create dump file and using vpn to get the file. file become large day by day. Please suggest me any solution
If I am correct, SQLite stores a database per file, and a file can't store more than one databases.
How does PostgreSQL store a database in terms of file(s)? Does it also store a database per file, and a file can't store more than one databases?
(SQLite uses more than one file for the rollback journal or when in WAL mode.)
The PostgreSQL database file layout is documented in its documentation:
Each table and index is stored in a separate file. For ordinary relations, these files are named after the table or index's filenode number, which can be found in pg_class.relfilenode. […] in addition to the main file (a/k/a main fork), each table and index has a free space map …, which stores information about free space available in the relation. The free space map is stored in a file named with the filenode number plus the suffix _fsm. Tables also have a visibility map, stored in a fork with the suffix _vm, to track which pages are known to have no dead tuples. […]
Mongodb Database generate files automatically after certain period as follow
Doc.0
Doc.1
Doc.2
Doc.3
Doc.4
but Doc.ns file never regenerate like above file
I'm not sure exactly what, if anything, you are specifying as a problem. This is expected behavior. MongoDB allocates new data files as the data grows. The .ns file, which stores namespace information, does not grow like data files, and shouldn't need to.
I have written a procedure which will get data from some tables and update a master table. each table contains records in the range 50k - 100k. when ever i execute this procedure, the archive log files is getting generated and the size of those log files is too much for my disk to handle.. it is generating more than 30GB of log files for each execution.
What are the possible ways to handle this.