I am new in postgresql, and I have a big dataset which is a postgresql backup. I have problem to import this dataset to my PostgreSQL.
Actually, this is a "pgdata" format, consists of some files and folders. One of these folders (base folder) has all the main files, (2000 files, each of which is 1 GB). But all of these files are in the "file" format, with no extension!!
I would be so grateful if you could give me some advice on this issue and help me to restore this backup.
Best,
From your description I guess that you have a physical backup, a copy of the data files like pg_basebackup creates.
If there is a backup_label file in the backup, and all the required WAL files are in the pg_xlog (or pg_wal) directory, then all you have to do is start the server on the data directory (pg_ctl start -D <directory here>) and wait until recovery has completed.
Then you can use pg_dump and pg_restore to extract the data from this new PostgreSQL cluster and import it into the destination.
Related
I have taken full backup of postgresql database which consists of 100 databases. The backup format is .sql (eg pg_dumpall.exe -U postgres > D:\Backup\fullbkp.sql) now one of my database got crashed and I want to extract this file to get that database backup only for restoration.
I have searched a lot but couldn't find any way to decompress so that I can get that particular database from full backup file.
Please suggest !!!!
Regards
Sadam
Such a backup is not compressed. Also, it contains a backup of all databases in th cluster, and there is no easy way to extract a single database.
Create a new PostgreSQL cluster with initdb, restore the dump there using psql, then use pg_dump to extract the single database you need.
Note: this question is with regards to PostgreSQL version 13.
On my Ubuntu server, I have a cluster called main which has 2 databases inside it (the first one being for a fruits company and the second one for a car company).
Here are my postgresql.conf file settings:
wal_level = replica
archive_mode = on
archive_command = 'pxz --compress --keep --force -6 --to-stdout --quiet %p > /datadrive/postgresql/13/wal_archives/%f.xz'
This creates .xz files in /datadrive/postgresql/13/wal_archives/ as expected.
For example: a file name may look like this:
0000000100000460000000A4.xz
Now my question's regarding this archiving process are as follows:
Is this particular .xz file an archive of all the databases in the postgresql cluster? i.e. does this particular xz file contain an archive for both the fruits and the car databases or does it only contain an archive for only one of them?
What is an archive file? Is it just a single WAL file or is it an archive point + a WAL file?
I have read the official documentation found here and here and also looked at a large number of stackoverflow and database stack exchange questions and have not managed to gain a good understanding of the archive concept.
Such a file is called a "WAL segment". WAL is short for "write ahead log" and is the transaction log, which contains the information required to replay data modifications for the whole database cluster. So it contains data for all databases in the cluster.
WAL is an endless append-only stream, which is split into segments of a fixed size. A WAL archive is nothing more than a faithful copy of a WAL segment.
WAL archives are used together with a base backup to perform point-in-time-recovery. Other uses for WAL files are crash recovery and replication, but these don't require archived WAL segments.
I am not a DBA but i am using postgresql for production server and i am using postgresql 10 database. I am using Bigsql and i started replication of my production server to other server and on replication server everything is working but on my production server their is no space left. And after du command on my production server i am getting that pg_wal folder have 17 gb file and each file is of 16 mb size.
After some google search i change my postgresql.conf file as:
wal_level = logical
archive_mode = on
archive_command = 'cp -i %p /etc/bigsql/data/pg10/pg_wal/archive_status/%f'
i install postgresql 10 from Bigsql and did above changes.
After changes the dir /pg_wal/archive_status had 16 gb of log. So my question is that should i delete them manually or i have to wait for system delete them automatically.
And is that if i write archive_mode to on should that wal file getting removed automatically??
Thanks for your precious time.
This depends on how you do your backups and whether you'd ever need to restore the database to some point in time.
Only a full offline filesystem backup (offline meaning with database turned off) or an on-line logical backup with pg_dumpall will not need those files for a restore.
You'd need those files to restore a filesystem backup created while the database is running. Without them the backup will fail to restore. Though there exist backup solutions that copy needed WAL files automatically (like Barman).
You'd also need those files if your replica database will ever fall behind the master for some reason. Or you'd need to restore the database to some past point-in-time.
But these files compress pretty well - should be less than 10% size after compression - you can write your archive_command to compress them automatically instead of just copying.
And you should delete them eventually from the archive. I'd recommend to not delete them until they're at least a month old and also at least 2 full successful backups are done after creating them.
I'm using pgAdmin to restore PostgreSQL database. To restore the database I need to delete, drop and remake it. How to restore the database without deleting and remaking it?
This cannot be done in pgAdmin or with any database tools. Regular backup files cannot be restored without deleting the data first because they consist of normal COPY statements which will fail if you have rows in the database (primary keys collide etc).
For a simple way to get back to an earlier snapshot in a testing environment take a look at PostgreSQL documentation - 24.2. File System Level Backup:
For backup:
Shut down your database
copy all the files from your data directory
For restore:
Shut down your database
replace your data directory with the backup directory
Note:
the size of the data might be significantly larger than with a regular backup especially if you have a lot of indexes
this is a server wide backup so you can't do this on individual databases
don't attempt to use it on a different version of PostgreSQL
this really deletes the data too - by replacing it with the backup
Also with regular backups you don't have to do a DROP TABLE if you do a data-only restore with pg_restore --data-only for example. You still have to delete the data though.
We have a Postgresql database that is over 732 GB when backed as a file system backup. When we do a pg_dump we can get it down to 585 GB. If I combined the pg_dump with the PITR method will this give me the best backup with smallest backup data file size? My plan was to run the pg_start_backup, then the pg_dump, then the pg_stop_backup. I know the documentation states to run a file system backup but I want a smaller backup data set. I would then copy off WAL files and then backup them up at night.
To truly get the smallest file, you'll have to try compressing your pg_dump -Fc dump file with one of many compression tools and settings. Using gzip or xz with maximum possible compression would be a start. This will of course require an excellent CPU and lots of CPU time.