I'm having trouble performing the restore from a dump. The scenario is as follows: I am migrating an environment from GCP to AWS, and at the moment I am working on the migration of the bank.
A partner dumped db that is in GCP and placed the file on AWS S3 (I don't know the command he used to perform the dump).
I created an EC2 in the AWS environment and copied the dump from S3 to EC2 (the file is 13 GB). I also created the RDS to host the new db with all the correct security group settings.
Here comes the problem, I connect to the RDS from the server (EC2) without problems, but when doing the restore using pg_restore I get the following error message: pg_restore: too many command line arguments (first is "dbclient. dump ").
The complete command I used was this:
pg_restore -h client-aurora-cluster-hmg-legado-instance-1.c23ltjbbz7ms.us-east-1.rds.amazonaws.com -U postgres -d db_hmg_legado dbclient.dump -W
OK, I changed the approach. I tried with psql instead of pg_restore and then the command was like this:
psql -h client-aurora-cluster-hmg-legado-instance-1.c23ltjbbz7ms.us-east-1.rds.amazonaws.com -U postgres -d db_hmg_legado dbclient.dump -W
Only this time it worked !!!!
But I received some error messages while performing the restore. Which I put below:
psql: dbclient.dump: 23: ERROR: schema "dw" already exists
CREATE EXTENSION
psql: dbclient.dump: 37: ERROR: must be owner of extension hstore
CREATE EXTENSION
psql: dbclient.dump: 51: ERROR: must be owner of extension intarray
CREATE EXTENSION
psql: dbclient.dump: 65: ERROR: must be owner of extension pg_trgm
CREATE EXTENSION
psql: dbclient.dump: 79: ERROR: must be owner of extension unaccent
But the restore takes a long time and is partially finished.
In general I wanted to understand why pg_restore didn't work. Has anyone ever experienced this?
And about these owner errors does anyone know how to resolve this using psql?
As documented in the manual the file to be restored is the last parameter and it is specified without a "switch". But you are using -W after the dump file. Move the -W parameter somewhere before that (although it's usually not necessary to begin with)
So you need something like this:
pg_restore -W -h ... -U postgres -d db_hmg_legado dbclient.dump
However, if the restore worked when using psql then the dump file is a "plain text" dump which can't be restored using pg_restore to begin with.
Concerning the errors:
You should restore the dump into an empty database that doesn't contain any schemas except the default ones.
You need a superuser for CREATE EXTENSION, which you don't have in a hosted database. So pre-install these extensions with the techniques that Amazon provides, then restore the dump and ignore the errors.
Related
Due to this change by heroku, I am not able to restore my Heroku Postgres backup.
Following errors are thrown while restoring:
pg_restore: error: could not execute query: ERROR: extension "pg_stat_statements" must be installed in schema "heroku_ext"
Command was: CREATE EXTENSION IF NOT EXISTS "pg_stat_statements" WITH SCHEMA "public";
pg_restore: error: could not execute query: ERROR: extension "pg_stat_statements" does not exist
Command was: COMMENT ON EXTENSION "pg_stat_statements" IS 'track planning and execution statistics of all SQL statements executed';
Command was: CREATE EXTENSION IF NOT EXISTS "postgis" WITH SCHEMA "public";
Any many such errors.
I contacted Heroku support, and they have provided the following step to try fix the issue manually.
Here are the steps in detail:
Download a backup of your database.
Convert the dump file to a .sql file with: pg_restore -f
Modify the CREATE EXTENSION commands to use CREATE EXTENSION IF NOT EXISTS
extension_name WITH SCHEMA heroku_ext. You can do this by using sed or a text editor
of your choice.
Restore the backup using pg_restore to Heroku Postgres from your app. The extensions
in your production database will now be owned by the correct schema; future
migrations, backups, and restores should proceed without needing to repeat this
process.
I have modified the .sql file as per step3, but I am not able to figure hot how to convert this .sql file to a .dump file to be restored on the Heroku app.
I have tried heroku pg:psql --app <app> <sql_file> but it does not work and throws errors.
Can someone help me in importing this .sql file to Heroku?
Steps followed to solve the issue:
Download the DB dump file from Heroku.
convert the dump to a .sql file using pg_restore.
Reset the Heroku DB using pg:reset
log in to the DB using pg:psql and install the extensions in the "heroku_ext" schema
Now execute the .sql file on the DB using pg:psql and check the errors
modify the .sql file lines that are causing the errors. Most of the errors can be solved by replacing "public" with "heroku_ext" at that lines.
After modifying the sql, again follow steps 3 to step 7 until all errors are fixed.
How to restore a Heroku database dump in a non-Heroku environment
Download the latest db dump from heroku
heroku pg:backups:download --app <app>
Drop local db if exists and create a new one
psql <db_url> -c "DROP DATABASE IF EXISTS <db_name>"
psql <db_url> -c "CREATE DATABASE <db_name> WITH ENCODING 'UTF8' TEMPLATE template0"
Here comes the important part
Create the heroku_ext schema inside the newly created database
psql <db_url><db_name> -c "CREATE SCHEMA IF NOT EXISTS heroku_ext AUTHORIZATION <db_user>"
Create the extention in the heroku_ext schema
psql <db_url><db_name> -c "CREATE extension IF NOT EXISTS <extension> WITH schema heroku_ext"
Grant the necessary permissions, eg:
psql <db_url><db_name> -c "GRANT ALL ON SCHEMA heroku_ext TO public"
psql <db_url><db_name> -c "GRANT USAGE ON SCHEMA heroku_ext TO public"
Add schema to search_path
psql <db_url><db_name> -c "SET search_path TO heroku_ext,public"
Grant permissions to the schema to the relevant user
psql <db_url><db_name> -c "GRANT ALL PRIVILEGES ON ALL SEQUENCES IN SCHEMA heroku_ext TO <db_user>"
Finally...
Restore the database dump
pg_restore --no-acl --no-owner -h <db_host> -U <db_user> -d <db_name> ./latest.dump
I had the same issue and found the answer in this question.
https://stackoverflow.com/a/73773443/17517157
Just add --extensions 'postgis' to your restore command.
You can ignore the error and manually create the extension in the required schema. If you want to avoid the error message, drop the extension in the database before dumping it. That should not be a problem, because nothing else in your database should depend on that extension.
I'm having difficulty restoring a DB to an AWS RDS Postgresql instance. Context is that i am backing up from one RDS instance and restoring to another RDS insurance. They both have the same version of Postgresql 9.6.5.
I was able to take a dump using the following command:
./pg_dump.exe -U dbuser -W -h prod-pgsql-rds.3ft5coqxjdnq.eu-west-2.rds.amazonaws.com -d devdb > c:\tmp\backup.sql
From the resulting .sql file, I then attempted a restore to another RDS instance which is also using Postgresql 9.6.5 using below command:
./pg_restore.exe -U dbuser -d testdevdb -h dev-pgsql-rds.cym8coqx52lq.eu-west-2.rds.amazonaws.com "c:\tmp\backup.sql"
*I also tried the -f switch in the above restore command instead of the " " quotes before/after the file name
But when I try to restore it to a newly created database I get the following error:
pg_restore: [archiver] input file does not appear to be a valid archive
Can anyone help? FYI, I am using PGAdmin 4 via Windows PowerShell. I have to edit some of the values in the strings above due to data sensitivity.
pg_restore is only used for the other, non-plain-text output formats that pg_dump can output. For .sql dumps, you just use psql. See the docs on restoring from backups.
In a Unix env, you'd do psql [yourflags] < /tmp/backup.sql, but I'm unfamiliar with powershell and don't know if it supports < for input redirection; hopefully either it's present or you know the equivalent PowerShell syntax.
So I couldn't get psql or pg_restore to work so opted to import the .SQL file into via the SQL query tool in PGAmdin. This through up some errors so had to make several changes to the .SQL file and perform below:
Commented out a couple of lines that were causing errors
Elevated permissions for the user and made him the owner of for the Schema and DB properties by right-clicking on these via PGAdmin
The .sql file was making several references to the user from the source RDS DB so had to do a find and replace with a user account created for the destination RDS DB. Alternatively, I could have just created a new user on the destination DB with the same username and password as the source DB and then make him the owner in ref to step 2.
I'm trying to restore my Postgres database to an RDS Postgres Instance in the AWS using pg restore from an EC2 instance.
I am using the following command to restore the database:
pg_restore -v -h host --no-owner -d PostgresDB postgres.dump
Now the problem is that originally, I didn't specify the --no-owner option, and since the owner of the local database that has been backed up, and the owner of the RDS Instance aren't the same. This threw an error, which is why I read that specifying this option helps solve the issue.
However, now I get a
pg_restore: [archiver (db)] connection to database "XY" failed:
FATAL: password authentication failed for user "ec2-user"
error message, although the password is right. Now I read that this does happens with Postgresql, but I can't find a way to solve this on EC2. According to this thread, I need to change the format of one of the configuration files. But this my postgres is an AWS Instance, how can I achieve this? I browsed my EC2 server instance and didn't find any pgpass.conf file (according to 'which' the file doesn't exist on the server). How can I solve this?
What username did you create for Postgres? Surely it isn't ec2-user. You need to be specifying the username and password with the --username and --password options. Here is the documentation.
In reference to this question:
PostgreSQL Job Scheduling: Creating Schedule Backup using Bat File
It made me successful to have a back-up for my database.
Filename: Database_backup.backup
However, using PgAdmin III using Restore selection, i wasn't able to restore it, it shows Error:
C:\Program Files\PostgreSQL\9.4\bin\pg_restore.exe --host localhost
--port 5432 --username "postgres" --dbname "db_name" --no-password --list "C:\Users\Name\Documents\Backup_20160805.backup"
pg_restore: [archiver] input file appears to be a text format dump.
Please use psql.
What I am missing here?
Is it in Backup?
Again, I need your guidance here.
Thanks so much.
pg_dump, which is called by pgAdmin III to perform backups, can create them in four formats:
plain: SQL commands
custom: compressed proprietary binary format
directory: one backup file per table
tar: like "directory", but as a tar archive
There is a "format" dropdown in pgAdmin III that lets you select the format.
To restore a plain format dump, you'll have to execute the SQL script with pgAdmin III or psql.
For the other three formats, you use pg_restore, which is internally called by pgAdmin III's "restore" functionality.
So you took a plain format dump, which causes the reported error with pg_restore. Execute it as a SQL script instead!
I have a database hosted using Amazon's RDS service and I am attempting to write a web service that will update said database. The problem I am having is that it will not let me use the COPY command as I get this error: "ERROR: must be superuser to COPY to or from a file". I am using the only user I have made for the database and I am fairly certain it has superuser access. I can, however, use PGAdmin's import tool to import the data which, when looking at the log, uses almost the exact same command as I do. The only difference is instead of the file path it has stdin. How can I fix this error?
You're using:
COPY tablename FROM 'filename';
this won't work - RDS has no idea what 'filename' is.
You must use the psql command's \copy, which copies from the local client, or PgAdmin-III's "import data" option.
The RDS manual covers this in more detail.
Workaround without using psql
1) Import data locally to a temporary table using simple copy command
2) Right click the table in the pgAdmin III object browser and select "Backup..."
3) Set the format to Plain and save the file as a .sql file
4) Click on the Dump Options #1 Tab and check Data
5) Click on the Dump Options #2 Tab and check Use Column Inserts and Use Insert Commands
6) Click Backup button
7) Now you can open the sql file and run it in your RDS server
Alternatively you can use the below command to generate the sql file
pg_dump --host localhost --port 5432 --username "postgres" --no-password --format plain --section data --inserts --column-inserts --file "C:\test\test.sql" --table "public.envelopes" "testdb"