Using PostgreSQL v13.x, Windows 2019 Server
I'm using the following command either from a Powershell or command window
> pg_dump -h 127.0.0.1 -U postgres -W --format custom --file my_db_dump.sql my_db
(p/w prompted and entered)
The dump is created successfully. However it's full of extraneous non-printable control-sequence characters, e.g., ^A^N^#^X^#^#^#...., you name it. Post processing, e.g., dos2unix, :set ff=unix, :%!col -xb doesn't eliminate the characters. Is there a switch in pg_dump to control this? I didn't see it in the pg_dump documentation.
You have specified the custom format which is not plain SQL. It has structure and some compression. If you want plain SQL that is also available as a format.
Related
I make a backup with the command
docker exec -t arcane-aio pg_dump arcane-aio -c -U admin > arcane_aio_db.sql
I restore the backup with the command
cat arcane_aio_db.sql | docker exec -i arcane-aio psql -U admin -d arcane-aio
All is good, but all Russian symbols are replaced by "?".
The string value before the restore is Привет, hi!.
The string value after the restore of the backup is ??????, hi!.
I checked the encoding of the backup, the database before the backup, the database after the restore, and they are the same (en_US.utf8). Could it be that this encoding don't support the Russian language?
We are using Windows.
After a change of the system encoding from Cyrillic to UTF-8,
the values in the data dump become correct.
But after the restore, we still see "?" instead of Russian symbols in the database.
the cat command uses your shell character encoding.
Did you try running simply the first part:
cat arcane_aio_db.sql
I bet it also shows the ???.
You need to set the charset to the same encoding on both sides. You probably have UTF-8 on one side and some russian language on the other.
The pipe, that writes to file is binary and doesn't care about the encoding, but cat does.
You can check your encoding with
echo $LANG
make sure it is UTF-8 on both sides and that should fix your issue.
** EDIT
a work-around is to do the backup and restore within the container:
#get into the container
docker exec -it arcane-aio /bin/bash
# in the container run:
pg_dump arcane-aio -c -U admin > arcane_aio_db.sql
# try restore:
cat arcane_aio_db.sql | psql -U admin -d arcane-aio
if that works, then it's an encoding issue between your docker container and local machine.
You can do the dump / restore within the container and copy the file in/out with docker cp
On another thought, the SQL you 'cat' may contain quotes or $ or # or other characters that are problematic sent directly into a TTY.
So you may want to try this instead, to make sure the whole thing is quoted:
eval "echo \"$(cat arcane_aio_db.sql)\"" | docker exec -t arcane-aio psql -U admin -d arcane-aio
Since a pg_dump includes instructions to set the client_encoding correctly, the data in your target database will be correct. That is, unless the database encoding is SQL_ASCII, in which case you are lost anyway if you need Cyrillic characters.
The problem must be with your client software or your terminal encoding.
To ascertain that the characters are correct in the database, connect with psql and cast the string to bytea so that you can see the bytes:
SELECT charcol, CAST(charcol TO bytea)
FROM tab WHERE ...;
So I regularly backup and restore databases and schema's using pgadmin4. I would like to do it with a batch file using commands as pg_dump and pg_restore. I however always fail to succeed in this and could use some help. The way I try to dump one schema (with data) is the following:
pg_dump -U postgres -F c -d database -n schema > mw2
Then I try to restore it with pg_restore:
pg_restore -U postgres -d otherdatabase -n schema mw2
First I tried to use .dump in stead of tar but the result stays the same; which is an empty schema in my database.
Or the following error message:
pg_restore: [archiver] input file does not appear to be a valid archive
https://www.postgresql.org/docs/current/static/app-pgrestore.html
--format=format Specify format of the archive. It is not necessary to specify the format, since pg_restore will determine the format
automatically. If specified, it can be one of the following:
custom, directory and tar
https://www.postgresql.org/docs/current/static/app-pgdump.html
--format=format
Selects the format of the output. format can be one of the following:
p plain Output a plain-text SQL script file (the default).
others are custom, directory and tar
in short - you used defualt plain format, which is meant for using with psql, not pg_restore. So either specify different format with pg_dump or use your file as
psql -f mw2.tar
I'm working with Heroku's new PGBackups functionality and I have used the public-url command to download a backup. The file isn't in a format that I'm used to. I was expecting a pg_dump file that I could open with Vim or another text editor. Does anyone know what the files that result from public-url are?
The pg_dump -Fc "custom" format
c
custom
Output a custom-format archive suitable for input into pg_restore. Together with the directory output format, this is the most flexible output format in that it allows manual selection and reordering of archived items during restore. This format is also compressed by default.
It sends you a file that you can use with the pg_restore command.
For example, I can download the file with:
curl -o /tmp/latest.dump <database_url>
And then restore with:
pg_restore --verbose --clean --no-acl --no-owner -h localhost -d <destination_database_name> /tmp/latest.dump
I have a UTF8 database and a UTF8 script to fill tables with data. However I want to run this script with psql -d instance -U user -f fillTables.sql. As my system has a Windows CP1252 encoding it looks like psql uses this to parse the file. I found this documentation and saw these backslash commands, but don't get it working
psql \encoding UTF8 -d instance -U user -f fillTables.sql
It looks like these are meant for starting psql and entering commands inside the psql console, right? How can I set different encoding for a batch processing of different files?
I got it working with export PGCLIENTENCODING=UTF8 (in cygwin, there is another syntax for windows), but would accept other answers if they can achieve the same with an option of psql.
I used this command to backup 200GB database (postgres 9.1, win7 x64):
pg_dump -Z 1 db_name > backup
It created 16GB file, which is fine I think because previous backups which works (and were packed by ext. tools) had similar size. Now, when I'm trying to restore into PG9.2 using pg_restore, I'm getting the error:
input file does not appear to be a valid archive
With pg_restore -Ft:
[tar archiver] corrupt tar header found in ▼ (expected 13500752, com
puted 78268) file position 512
Gzip also shows it's corrupted. When I open the backup file in Total Commander, the inner file has only 1.8GB.
When I was looking for a solution, dump should be done with -Cf parameter probably.
Which format has the file right now? Is it only tar or gzip (winrar shows gzip)?
Is there any way to restore this properly or is it corrupted somehow (no error when dumped)? Could it be due to file size limitations of tar or gzip?
What you have as output in "backup" is just zipped plain sql.
You could check it by prompting:
gzip -l backup
Unfortunately pg_retore do not provide possibility to restore PLAIN SQL,
so you just need to decompress the file and use psql -f <FILE> command:
zcat backup > backup.sql
psql -f backup.sql
It is not possible to make dump with pg_dump -Fc from postgres 9.1 as proposed by "Frank Heikens",
because dump formats are not compatible between primary versions, like 9.0 -> 9.1 -> 9.2
and "pg_restore" will give you an error on 9.2
Mostly this error mean that your restore action used invalid format
From manual of pg_dump ( pg_dump --help )
-F, --format=c|d|t|p output file format (custom, directory, tar,
plain text (default))
This mean that if you create dump with pg_dump without option --format / -F that your dump will be created in plain text format
NOTE: Plain text format cannot be restored with pg_restore tool. Use psql < dump.sql instead.
Examples:
# plain text export/import
pg_dump -Fp -d postgres://<db_user>:<db_password>#<db_host>:<db_port>/<db_name> > dump.sql
psql -d postgres://<target_db_user>:<target_db_password>#<target_db_host>:<target_db_port>/<target_db_name> -f dump.sql
# custom format
pg_dump -Fc -d postgres://<db_user>:<db_password>#<db_host>:<db_port>/<db_name> > dump.sql.custom
pg_restore -Ft postgres://<target_db_user>:<target_db_password>#<target_db_host>:<target_db_port>/<target_db_name> dump.sql.custom
# tar format
pg_dump -Ft -d postgres://<db_user>:<db_password>#<db_host>:<db_port>/<db_name> > dump.sql.tar
pg_restore -Ft postgres://<target_db_user>:<target_db_password>#<target_db_host>:<target_db_port>/<target_db_name> dump.sql.tar
Error from subject also can occur when restoring format not match backup.
For example created dump will be in custom format but for restore specified tar
Your dump is plain SQL, it's not a tar format, like you try to use in pg_restore. Use --format=custom or -Fc when you want a compressed format and use this setting in pg_restore as well. Check the manual.
This is an old thread though I had the exact same issue and managed to fix the somewhat corrupted dump with fixgz:
Short answer: run fixgz http://www.gzip.org/fixgz.zip on compressed dump.
fixgz.exe bad.gz fixed.gz
Long answer:
So if you used pg_dump with --compresss or -Z without specifying custom format option (-Fc) what you actually get is a compressed file in ASCII mode instead of BINARY mode.
Quoting from http://www.gzip.org/#faq1
If you have transferred a file in ASCII mode and you no longer have
access to the original, you can try the program fixgz to remove the
extra CR (carriage return) bytes inserted by the transfer. A Windows
9x/NT/2000/ME/XP binary is here. But there is absolutely no guarantee
that this will actually fix your file. Conclusion: never transfer
binary files in ASCII mode.
I got this problem when restoring using PGAdmin III. The problem doesn't occur with PGAdmin 4.