I've generated a PostgreSQL script that I want to use to restore a database. When I go to my backup server to try to restore, I get the error: syntax error at or near "\".
It's getting stuck on the following characters \.
These appear like this:
COPY admin.roles (role_id, role_name, is_role_auto) from stdin;
\.
What's wrong with this statement? Is there config I missed? I'm on PostgreSQL 11.4 on Windows, the backup was taken with pg_dump, and I restore it using pgAdmin.
You cannot use pgAdmin to restore a "plain format" dump taken with pg_dump. It doesn't understand the psql syntax where COPY and its data are interleaved.
You will have to use psql to restore the dump:
psql -h server -p 5432 -U user -d database -f dumpfile.sql
It's really hard to know the specific error without seeing your backup and restore commands in their entirety, but if it helps, here is the boilerplate I use when I want to copy a table from production to a backup server:
$BIN/pg_dump -h production_server -p 5432 \
--dbname=postgres \
--superuser=postgres \
--verbose \
--format=c \
--compress=9 \
--table=admin.roles > backup.sql
$BIN/pg_restore \
--host=backup_server \
--port=5432 \
--username=postgres \
--dbname=postgres \
--clean \
--format=custom \
backup.sql
The format=c (or --format=custom) makes the content completely unreadable, but on a plus side it also avoids any weird errors with delimiters and the like, and it also perfectly copies complex data structures like arrays and BLOBs.
Related
I am doing pg_dump using -
pg_dump -U <username> -h <host> <database> > backup.sql
pg_dump is working fine.
I am trying to do pg_restore doing -
pg_restore -U <username> -h <host> -d <databse> backup.sql
Then it is showing pg_restore: error: input file does not appear to be a valid archive
I have checked many StackOverflow answers about this, but I could not figure out anything. Please help me. Thanks in advance.
Update : As per comments we can not use pg_restore for .sql files. Actually I have an restriction that I must have to restore the database using pg_restore command. Can you please give the pg_dump command using which I can restore that using pg_restore?
You created a plain format dump, which is an SQL file. You have to restore plain-format dumps with psql:
psql -U <username> -h <host> -d <databse> -f backup.sql
pg_restore is used to restore dumps in all other formats. You get dumps in other formats by using the appropriate -F option with pg_dump: for example, -F c produces a custom format dump.
If you want to restore a plain format dump with a client other than psql, you have to create it with the option --inserts.
My question is similar to this I even vote up it, but unfortunately in my case I have text dump and I can not restore using pg_restore =(
zcat /home/kes/work/projects/bennet/db/bennet.sql.gz | \
pg_restore -h 127.0.0.1 -p 5432 -U bennet --no-owner --role=bennet -d bennet
pg_restore: error: input file appears to be a text format dump. Please use psql.
But psql has no role and no-owner options
Is there a way to restore text dump on different owner?
No. You have two options:
Manually edit the SQL script. This is cumbersom and error-prone.
Restore the complete dump to a new, empty scratch database using psql and dump that with --role and --no-owner as you need.
I have take backup by pg_dumpall > test.out
and test.out successfully generated, hence backup completed.
I have used command psql -f test.out postgres for restore
But got following errors with restoring backup:
databases already exists
relation "products" already exists
duplicate key value violates unique constraint "products_pkey"
I actually want to replace the data in the existing db with backup. How to do that?
The problem is that the database you're trying to restore already exists.
You can run a DROP DATABASE database_name command that will delete your existing database and then you can run your test.out file.
Or you can run pgdumpall --clean > test.out and then run the resulting file. The clean flag will make the resulting files have the DROP DATABASE command in them.
Do you use the bellow command ?
psql -h localhost -U [login role] database_name -f /home/database.backup
I think a flow like this might help, because we don't want drop the database each time we call the backup file.
First, we need to create a backup file using the --format=custom [-Fc] to restore it using pg_restore. We can use a connection string postgresql://<user>:<pass>#localhost:5432/<dbname> and replace <user>, <pass>, and <dbname> with your information.
pg_dump -v -Fc \
postgresql://<user>:<pass>#localhost:5432/<dbname> \
> db-20211122-163508.sql
To restore we will call it using --clean [-c] and --create [-C] to drop the database before restoring. Replace <user>, <host>, <port>, and <dbname> with your information.
pg_restore -vcC \
-U <user> \
-h <host> \
-p <port> \
-d <dbname> \
< db-20211122-163508.sql
This way you don't need to use clean when you create the backup file.
I use the postgres today
and got a problem
I dump the database that way
pg_dump zeus_development -U test > zeus_development.dump.out
what if I wnat to restore to another database zeus_production
How could I do?
Simple, first create your database using template0 as your template database:
createdb -U test -T template0 zeus_production
Then, restore your dump on this database:
psql -U test zeus_production -f /path/to/zeus_development.dump.out
When restoring, always use template0 explicit, as it is always an empty and unmodifiable database. If you don't use an explicit template, PostgreSQL will assume template1, and if it has some objects, like a table or function that your dumped database already has, you will get some errors while restoring.
Nonetheless, even if you were restoring on a database with the same name (zeus_development) you should create (or recreate) it the same way. Unless you used -C option while dumping (or -C of pg_restore if using a binary dump), which I don't recommend, because will give you less flexibility (like restoring on a different database name).
The PostgresSQL documentation has influenced me to use the custom format. I've been using it for years and it seems to have various advantages but your mileage may vary. That said, here is what worked for me:
pg_restore --no-owner --dbname postgres --create ~/Desktop/pg_dump
psql --dbname postgres -c 'ALTER DATABASE foodog_production RENAME TO foodog_development'
There was no foodog_development nor foodog_production databases existing before the sequence.
This restores the database from the dump (~/Desktop/pg_dump) which will create it with the name it was dumped as. The rename names the DB to whatever you want.
The --no-owner may not be needed if your user name is the same on both machines. In my case, the dump was done as user1 and the restore done as user2. The new objects need to be owned by user2 and --no-owner achieves this.
Isn't it easier to simply do the following?
createdb -U test -T zeus_development zeus_production
This has an answer on dba.stackexchange, which I reproduce here:
Let's define a few variables to make the rest easier to copy/paste
old_db=my_old_database
new_db=new_database_name
db_dump_file=backups/my_old_database.dump
user=postgres
The following assumes that your backup was created with the "custom" format like this:
pg_dump -U $user -F custom $old_db > "$db_dump_file"
To restore $db_dump_file to a new database name $new_db :
dropdb -U $user --if-exists $new_db
createdb -U $user -T template0 $new_db
pg_restore -U $user -d $new_db "$db_dump_file"
Here's a hacky way of doing it, that only works if you can afford the space and time to use regular .sql format, and if you can safely sed out your database name and user.
$ pg_dump -U my_production_user -h localhost my_production > my_prod_dump.sql
$ sed -i 's/my_production_user/my_staging_user/g' my_prod_dump.sql
$ sed -i 's/my_production/my_staging/g' my_prod_dump.sql
$ mv my_prod_dump.sql my_staging_dump.sql
$ sudo su postgres -c psql
psql> drop database my_staging;
psql> create database my_staging owner my_staging_user;
psql> \c my_staging;
psql> \i my_staging_dump.sql
If your dump does not include the name, the restore will use the DB defined in DESTINATION. Both SOURCE and DESTINATION are Connection URLs.
Dump without --create
pg_dump \
--clean --if-exists \
--file ${dump_path} \
--format=directory \
--jobs 5 \
--no-acl \
--no-owner \
${SOURCE}
Restore without --create
pg_restore \
--clean --if-exists \
--dbname=${DESTINATION} \
--format=directory \
--jobs=5 \
--no-acl \
--no-owner \
$dump_path
I'm trying to port a database from MySQL to PostgreSQL. I've rebuilt the schema in Postgres, so all I need to do is get the data across, without recreating the tables.
I could do this with code that iterates all the records and inserts them one at a time, but I tried that and it's waaayyyy to slow for our database size, so I'm trying to use mysqldump and a pipe into psql instead (once per table, which I may parallelize once I get it working).
I've had to jump through various hoops to get this far, turning on and off various flags to get a dump that is vaguely sane. Again, this only dumps the INSERT INTO, since I've already prepared the empty schema to get the data into:
/usr/bin/env \
PGPASSWORD=mypassword \
mysqldump \
-h mysql-server \
-u mysql-username \
--password=mysql-password \
mysql-database-name \
table-name \
--compatible=postgresql \
--compact \
-e -c -t \
--default-character-set=utf8 \
| sed "s/\\\\\\'/\\'\\'/g" \
| psql \
-h postgresql-server \
--username=postgresql-username \
postgresql-database-name
Everything except that ugly sed command is manageable. I'm doing that sed to try and convert MySQL's approach to quoting single-quotes inside of strings ('O\'Connor') o PostgreSQL's quoting requirements ('O''Connor'). It works, until there are strings like this in the dump: 'String ending with a backslash \\'... and yes, it seems there is some user input in our database that has this format, which is perfectly valid, but doesn't pass my sed command. I could add a lookbehind to the sed command, but I feel like I'm crawling into a rabbit hole. Is there a way to either:
a) Tell mysqldump to quote single quotes by doubling them up
b) Tell psql to expect backslashes to be interpreted as quoting escapes?
I have another issue with BINARY and bytea differences, but I've worked around that with a base64 encoding/decoding phase.
EDIT | Looks like I can do (b) with set backslash_quote = on; set standard_conforming_strings = off;, though I'm not sure how to inject that into the start of the piped output.
Dump the tables to TSV using mysqldump's --tab option and then import using psql's COPY method.
The file psqlrc and ~/.psqlrc may contain SQL commands to be executed when the client starts. You can put these three lines, or any other settings you would like in that file.
SET standard_conforming_strings = 'off';
SET backslash_quote = 'on';
SET escape_string_warning = 'off';
These settings for psql combined with the following mysqldump command will successfully migrate data only from mysql 5.1 to postgresql 9.1 with UTF-8 text (Chinese in my case). This method may be the only reasonable way to migrate a large database if creating an intermediate file would be too large or too time consuming. This requires you manually migrate the schema, since the two database's data types are vastly different. Plan on typing out some DDL to get it right.
mysqldump \
--host=<hostname> \
--user=<username> \
--password=<password> \
--default-character-set=utf8 \
--compatible=postgresql \
--complete-insert \
--extended-insert \
--no-create-info \
--skip-quote-names \
--skip-comments \
--skip-lock-tables \
--skip-add-locks \
--verbose \
<database> <table> | psql -n -d <database>
Try this:
sed -e "s/\\\\'/\\\\\\'/g" -e "s/\([^\\]\)\\\\'/\1\\'\\'/g"
Yeah, "Leaning Toothpick Syndrome", I know.