I'm trying to dump a huge database and compress the dump in order to not have to wait hours till it's done.
I dump the database the following way:
pg_dump -Fc -U -v | gzip > db$(date +%d-%m-%y_%H-%M).tar.gz
This leaves me with a compressed tar file.
I know want to unzip it in order to have a .tar file only:
tar -xvf xxx.tar.gz
This leaves me with an error message saying This does not look like a tar archive file
My goal is to then import it via psql.
I do not see what I am doing wrong – according to the Postgres documentation on dumps, I can use -Fc to dump in any wanted format? Thank you
It is not a tar archive, even if you named it like that, so you cannot use tar to unpack it.
To restore, you could use
zcat xxx.tar.gz | pg_restore -d target_database -U dbuser
By the way, it is pretty pointless to compress a custom format dump, since it is already compressed. You could choose better compression during pg_dump with the -Z 9 option.
Related
I have a dump file (size around 5 GB) which is taken via this command:
pg_dump -U postgres -p 5440 MYPRODDB > MYPRODDB_2022.dmp
The database consists multiple schemas (let's say Schema A,B,C and D) but i need to restore only one schema (schema A).
How can i achieve that? The command below didn't work and gave error:
pg_restore -U postgres -d MYPRODDB -n A -p 5440 < MYPRODDB_2022.dmp
pgrestore: error: input file appears to be a text format dump. please
use psql.
You cannot do that with a plain format dump. That's one of the reasons why you always use a different format unless you need an SQL script.
If you want to stick with a plain text dump:
pg_dump -U postgres -p 5440 -n A MYPRODDB > MYPRODDB_2022.dmp
psql -U postgres -d MYPRODDB -p 5440 -f MYPRODDB_2022.dmp
Though dumping back over the same database as above will throw errors unless you use --clean or its short form -c to create commands to drop existing objects before restoring them:
-c
--clean
Output commands to clean (drop) database objects prior to outputting the commands for creating them. (Unless --if-exists is also specified, restore might generate some harmless error messages, if any objects were not present in the destination database.)
This option is ignored when emitting an archive (non-text) output file. For the archive formats, you can specify the option when you call pg_restore.
Probably also a good idea to throw in --if-exists:
--if-exists
Use conditional commands (i.e., add an IF EXISTS clause) when cleaning database objects. This option is not valid unless --clean is also specified.
So I regularly backup and restore databases and schema's using pgadmin4. I would like to do it with a batch file using commands as pg_dump and pg_restore. I however always fail to succeed in this and could use some help. The way I try to dump one schema (with data) is the following:
pg_dump -U postgres -F c -d database -n schema > mw2
Then I try to restore it with pg_restore:
pg_restore -U postgres -d otherdatabase -n schema mw2
First I tried to use .dump in stead of tar but the result stays the same; which is an empty schema in my database.
Or the following error message:
pg_restore: [archiver] input file does not appear to be a valid archive
https://www.postgresql.org/docs/current/static/app-pgrestore.html
--format=format Specify format of the archive. It is not necessary to specify the format, since pg_restore will determine the format
automatically. If specified, it can be one of the following:
custom, directory and tar
https://www.postgresql.org/docs/current/static/app-pgdump.html
--format=format
Selects the format of the output. format can be one of the following:
p plain Output a plain-text SQL script file (the default).
others are custom, directory and tar
in short - you used defualt plain format, which is meant for using with psql, not pg_restore. So either specify different format with pg_dump or use your file as
psql -f mw2.tar
According to a website I can download their sample file dvdrental.zip, but
The database file is in zipformat ( dvdrental.zip) so you need to extract > it to dvdrental.tar
First of all, what is a tar? I thought it had to be tar.gz to be compressed? I don't even know how to create a "tar" by itself. I tried:
tar -zcvf dvdrental.tar.gz dvdrental
and
tar -cf dvdrental.tar dvdrental
I try to import with pgAdmin 4 and I get either:
pg_restore: [archiver] input file does not appear to be a valid archive
or
pg_restore: [tar archiver] could not find header for file "toc.dat" in tar archive
respectively. Now, don't ask me why a popular tutorial site created a file in the wrong format. But, can you tell me how to repackage this file so I can use it as a sample DB?
Using Mac OS 10.12.4. Postgres 9.6. And PgAdmin 4 (not sure if it's in beta? It crashed and does all kinds of nonsensical window movement and highlighting)
I have extracted .zip archive first. Then opened pgAdmin and followed the guide "Load the DVD Rental database using the pgAdmin"
https://www.postgresqltutorial.com/load-postgresql-sample-database/
Pay attention to changing 'Format' field from 'Custom or Tar' to 'Directory'. Then you should be able to restore DB.
If you look into the .tar archive you will find the restore.sql where at the top:
-- File paths need to be edited. Search for $$PATH$$ and
-- replace it with the path to the directory containing
-- the extracted data files.
So to create sample DB you could to extract .tar content somewhere and use single command:
sed -e 's/\$\$PATH\$\$/\/path\/to\/extracted\/files/g' restore.sql | psql
Or
sed -e 's/\$\$PATH\$\$/\/path\/to\/extracted\/files/g' restore.sql > r.sql
and try to execute the r.sql content using PgAdmin.
get sample dataset from the link you cited and save somewhere.
Assuming postgres is installed and running do the following:
Run createdb dvdrental
Run pg_restore -d dvdrental ./dvdrental where "./dvdrental" is the path to the downloaded and unzipped file.
For create sample DB in postgres you following this steps:
1.- Create directory and enter it:
mkdir -p /tmp/dvdrental && cd /tmp/dvdrental
2.- Download zip file dvdrental.zip:
wget https://www.postgresqltutorial.com/wp-content/uploads/2019/05/dvdrental.zip
3.- Uncompress file .zip and later .tar:
unzip dvdrental.zip
tar -xvf dvdrental.tar
4.- Replace in execution time $$PATH variable and review it with grep command:
sed -e 's/\$\$PATH\$\$/\/tmp\/dvdrental/g' restore.sql | grep --color dvdrental
5.- Import DB sample for specific host (localhost), port (5433), user (db) and database name (postgres):
sed -e 's/\$\$PATH\$\$/\/tmp\/dvdrental/g' restore.sql | psql -h localhost -p 5433 -U db -d postgres
Finally, I show import successful with program pgAdmin III
I'm working with Heroku's new PGBackups functionality and I have used the public-url command to download a backup. The file isn't in a format that I'm used to. I was expecting a pg_dump file that I could open with Vim or another text editor. Does anyone know what the files that result from public-url are?
The pg_dump -Fc "custom" format
c
custom
Output a custom-format archive suitable for input into pg_restore. Together with the directory output format, this is the most flexible output format in that it allows manual selection and reordering of archived items during restore. This format is also compressed by default.
It sends you a file that you can use with the pg_restore command.
For example, I can download the file with:
curl -o /tmp/latest.dump <database_url>
And then restore with:
pg_restore --verbose --clean --no-acl --no-owner -h localhost -d <destination_database_name> /tmp/latest.dump
I used this command to backup 200GB database (postgres 9.1, win7 x64):
pg_dump -Z 1 db_name > backup
It created 16GB file, which is fine I think because previous backups which works (and were packed by ext. tools) had similar size. Now, when I'm trying to restore into PG9.2 using pg_restore, I'm getting the error:
input file does not appear to be a valid archive
With pg_restore -Ft:
[tar archiver] corrupt tar header found in ▼ (expected 13500752, com
puted 78268) file position 512
Gzip also shows it's corrupted. When I open the backup file in Total Commander, the inner file has only 1.8GB.
When I was looking for a solution, dump should be done with -Cf parameter probably.
Which format has the file right now? Is it only tar or gzip (winrar shows gzip)?
Is there any way to restore this properly or is it corrupted somehow (no error when dumped)? Could it be due to file size limitations of tar or gzip?
What you have as output in "backup" is just zipped plain sql.
You could check it by prompting:
gzip -l backup
Unfortunately pg_retore do not provide possibility to restore PLAIN SQL,
so you just need to decompress the file and use psql -f <FILE> command:
zcat backup > backup.sql
psql -f backup.sql
It is not possible to make dump with pg_dump -Fc from postgres 9.1 as proposed by "Frank Heikens",
because dump formats are not compatible between primary versions, like 9.0 -> 9.1 -> 9.2
and "pg_restore" will give you an error on 9.2
Mostly this error mean that your restore action used invalid format
From manual of pg_dump ( pg_dump --help )
-F, --format=c|d|t|p output file format (custom, directory, tar,
plain text (default))
This mean that if you create dump with pg_dump without option --format / -F that your dump will be created in plain text format
NOTE: Plain text format cannot be restored with pg_restore tool. Use psql < dump.sql instead.
Examples:
# plain text export/import
pg_dump -Fp -d postgres://<db_user>:<db_password>#<db_host>:<db_port>/<db_name> > dump.sql
psql -d postgres://<target_db_user>:<target_db_password>#<target_db_host>:<target_db_port>/<target_db_name> -f dump.sql
# custom format
pg_dump -Fc -d postgres://<db_user>:<db_password>#<db_host>:<db_port>/<db_name> > dump.sql.custom
pg_restore -Ft postgres://<target_db_user>:<target_db_password>#<target_db_host>:<target_db_port>/<target_db_name> dump.sql.custom
# tar format
pg_dump -Ft -d postgres://<db_user>:<db_password>#<db_host>:<db_port>/<db_name> > dump.sql.tar
pg_restore -Ft postgres://<target_db_user>:<target_db_password>#<target_db_host>:<target_db_port>/<target_db_name> dump.sql.tar
Error from subject also can occur when restoring format not match backup.
For example created dump will be in custom format but for restore specified tar
Your dump is plain SQL, it's not a tar format, like you try to use in pg_restore. Use --format=custom or -Fc when you want a compressed format and use this setting in pg_restore as well. Check the manual.
This is an old thread though I had the exact same issue and managed to fix the somewhat corrupted dump with fixgz:
Short answer: run fixgz http://www.gzip.org/fixgz.zip on compressed dump.
fixgz.exe bad.gz fixed.gz
Long answer:
So if you used pg_dump with --compresss or -Z without specifying custom format option (-Fc) what you actually get is a compressed file in ASCII mode instead of BINARY mode.
Quoting from http://www.gzip.org/#faq1
If you have transferred a file in ASCII mode and you no longer have
access to the original, you can try the program fixgz to remove the
extra CR (carriage return) bytes inserted by the transfer. A Windows
9x/NT/2000/ME/XP binary is here. But there is absolutely no guarantee
that this will actually fix your file. Conclusion: never transfer
binary files in ASCII mode.
I got this problem when restoring using PGAdmin III. The problem doesn't occur with PGAdmin 4.