I have a database hosted using Amazon's RDS service and I am attempting to write a web service that will update said database. The problem I am having is that it will not let me use the COPY command as I get this error: "ERROR: must be superuser to COPY to or from a file". I am using the only user I have made for the database and I am fairly certain it has superuser access. I can, however, use PGAdmin's import tool to import the data which, when looking at the log, uses almost the exact same command as I do. The only difference is instead of the file path it has stdin. How can I fix this error?
You're using:
COPY tablename FROM 'filename';
this won't work - RDS has no idea what 'filename' is.
You must use the psql command's \copy, which copies from the local client, or PgAdmin-III's "import data" option.
The RDS manual covers this in more detail.
Workaround without using psql
1) Import data locally to a temporary table using simple copy command
2) Right click the table in the pgAdmin III object browser and select "Backup..."
3) Set the format to Plain and save the file as a .sql file
4) Click on the Dump Options #1 Tab and check Data
5) Click on the Dump Options #2 Tab and check Use Column Inserts and Use Insert Commands
6) Click Backup button
7) Now you can open the sql file and run it in your RDS server
Alternatively you can use the below command to generate the sql file
pg_dump --host localhost --port 5432 --username "postgres" --no-password --format plain --section data --inserts --column-inserts --file "C:\test\test.sql" --table "public.envelopes" "testdb"
Related
Is there a simple way to create a copy of a database or schema in PostgreSQL 8.1?
I'm testing some software which does a lot of updates to a particular schema within a database, and I'd like to make a copy of it so I can run some comparisons against the original.
If it's on the same server, you just use the CREATE DATABASE command with the TEMPLATE parameter. For example:
CREATE DATABASE newdb WITH TEMPLATE olddb;
pg_dump with the --schema-only option.
If you have to copy the schema from the local database to a remote database, you may use one of the following two options.
Option A
Copy the schema from the local database to a dump file.
pg_dump -U postgres -Cs database > dump_file
Copy the dump file from the local server to the remote server.
scp localuser#localhost:dump_file remoteuser#remotehost:dump_file
Connect to the remote server.
ssh remoteuser#remotehost
Copy the schema from the dump file to the remote database.
psql -U postgres database < dump_file
Option B
Copy the schema directly from the local database to the remote database without using an intermediate file.
pg_dump -h localhost -U postgres -Cs database | psql -h remotehost -U postgres database
This blog post might prove helpful for you if you want to learn more about options for copying the database using pg_dump.
This can be done by running the following command:
CREATE DATABASE [Database to create] WITH TEMPLATE [Database to copy] OWNER [Your username];
Once filled in with your database names and your username, this will create a copy of the specified database. This will work as long as there are no other active connections to the database you wish to copy. If there are other active connections you can temporarily terminate the connections by using this command first:
SELECT pg_terminate_backend(pg_stat_activity.pid)
FROM pg_stat_activity
WHERE pg_stat_activity.datname = '[Database to copy]'
AND pid <> pg_backend_pid();
A good article that I wrote for Chartio's Data School which goes a bit more in depth on how to do this can be found here:
https://dataschool.com/learn/how-to-create-a-copy-of-a-database-in-postgresql-using-psql
I'm having trouble performing the restore from a dump. The scenario is as follows: I am migrating an environment from GCP to AWS, and at the moment I am working on the migration of the bank.
A partner dumped db that is in GCP and placed the file on AWS S3 (I don't know the command he used to perform the dump).
I created an EC2 in the AWS environment and copied the dump from S3 to EC2 (the file is 13 GB). I also created the RDS to host the new db with all the correct security group settings.
Here comes the problem, I connect to the RDS from the server (EC2) without problems, but when doing the restore using pg_restore I get the following error message: pg_restore: too many command line arguments (first is "dbclient. dump ").
The complete command I used was this:
pg_restore -h client-aurora-cluster-hmg-legado-instance-1.c23ltjbbz7ms.us-east-1.rds.amazonaws.com -U postgres -d db_hmg_legado dbclient.dump -W
OK, I changed the approach. I tried with psql instead of pg_restore and then the command was like this:
psql -h client-aurora-cluster-hmg-legado-instance-1.c23ltjbbz7ms.us-east-1.rds.amazonaws.com -U postgres -d db_hmg_legado dbclient.dump -W
Only this time it worked !!!!
But I received some error messages while performing the restore. Which I put below:
psql: dbclient.dump: 23: ERROR: schema "dw" already exists
CREATE EXTENSION
psql: dbclient.dump: 37: ERROR: must be owner of extension hstore
CREATE EXTENSION
psql: dbclient.dump: 51: ERROR: must be owner of extension intarray
CREATE EXTENSION
psql: dbclient.dump: 65: ERROR: must be owner of extension pg_trgm
CREATE EXTENSION
psql: dbclient.dump: 79: ERROR: must be owner of extension unaccent
But the restore takes a long time and is partially finished.
In general I wanted to understand why pg_restore didn't work. Has anyone ever experienced this?
And about these owner errors does anyone know how to resolve this using psql?
As documented in the manual the file to be restored is the last parameter and it is specified without a "switch". But you are using -W after the dump file. Move the -W parameter somewhere before that (although it's usually not necessary to begin with)
So you need something like this:
pg_restore -W -h ... -U postgres -d db_hmg_legado dbclient.dump
However, if the restore worked when using psql then the dump file is a "plain text" dump which can't be restored using pg_restore to begin with.
Concerning the errors:
You should restore the dump into an empty database that doesn't contain any schemas except the default ones.
You need a superuser for CREATE EXTENSION, which you don't have in a hosted database. So pre-install these extensions with the techniques that Amazon provides, then restore the dump and ignore the errors.
I'm having difficulty restoring a DB to an AWS RDS Postgresql instance. Context is that i am backing up from one RDS instance and restoring to another RDS insurance. They both have the same version of Postgresql 9.6.5.
I was able to take a dump using the following command:
./pg_dump.exe -U dbuser -W -h prod-pgsql-rds.3ft5coqxjdnq.eu-west-2.rds.amazonaws.com -d devdb > c:\tmp\backup.sql
From the resulting .sql file, I then attempted a restore to another RDS instance which is also using Postgresql 9.6.5 using below command:
./pg_restore.exe -U dbuser -d testdevdb -h dev-pgsql-rds.cym8coqx52lq.eu-west-2.rds.amazonaws.com "c:\tmp\backup.sql"
*I also tried the -f switch in the above restore command instead of the " " quotes before/after the file name
But when I try to restore it to a newly created database I get the following error:
pg_restore: [archiver] input file does not appear to be a valid archive
Can anyone help? FYI, I am using PGAdmin 4 via Windows PowerShell. I have to edit some of the values in the strings above due to data sensitivity.
pg_restore is only used for the other, non-plain-text output formats that pg_dump can output. For .sql dumps, you just use psql. See the docs on restoring from backups.
In a Unix env, you'd do psql [yourflags] < /tmp/backup.sql, but I'm unfamiliar with powershell and don't know if it supports < for input redirection; hopefully either it's present or you know the equivalent PowerShell syntax.
So I couldn't get psql or pg_restore to work so opted to import the .SQL file into via the SQL query tool in PGAmdin. This through up some errors so had to make several changes to the .SQL file and perform below:
Commented out a couple of lines that were causing errors
Elevated permissions for the user and made him the owner of for the Schema and DB properties by right-clicking on these via PGAdmin
The .sql file was making several references to the user from the source RDS DB so had to do a find and replace with a user account created for the destination RDS DB. Alternatively, I could have just created a new user on the destination DB with the same username and password as the source DB and then make him the owner in ref to step 2.
I’m using Postgres 9.5 on Mac Sierra. I want to export some table data from my local machine and import that into a Postgres 9.5 database on a Linux machine. Note I don’t want to blow away the data on the Linux machine, only add the my local machine table rows to the rows that already exist on the tables in the Linux environment (and ignore duplicates). So on my local machine, I ran
pg_dump -U myusername --format custom --section data --inserts --file "t1.backup" --table "table1" --table "table2" --table "addresses" "mydb"
However on my remote machine, when I try and import the file, I get the error
myuser#remote-machine:/tmp$ psql db_production < t1.backup
The input is a PostgreSQL custom-format dump.
Use the pg_restore command-line client to restore this dump to a database.
But I don’t want to use pg_restore because I don’t want to erase the existing table data, I simply want to add to it. How can I achieve this?
Use --format plain instead of custom one. The latter is designed to work exclusively with pg_restore. Plain format also allows you to take a look into dumped data with text editor and verify if that's what you want.
However my quick test shows that it's also possible to append data with pg_restore and custom format data-only dump:
pg_restore -d db_production --data-only t1.backup
In reference to this question:
PostgreSQL Job Scheduling: Creating Schedule Backup using Bat File
It made me successful to have a back-up for my database.
Filename: Database_backup.backup
However, using PgAdmin III using Restore selection, i wasn't able to restore it, it shows Error:
C:\Program Files\PostgreSQL\9.4\bin\pg_restore.exe --host localhost
--port 5432 --username "postgres" --dbname "db_name" --no-password --list "C:\Users\Name\Documents\Backup_20160805.backup"
pg_restore: [archiver] input file appears to be a text format dump.
Please use psql.
What I am missing here?
Is it in Backup?
Again, I need your guidance here.
Thanks so much.
pg_dump, which is called by pgAdmin III to perform backups, can create them in four formats:
plain: SQL commands
custom: compressed proprietary binary format
directory: one backup file per table
tar: like "directory", but as a tar archive
There is a "format" dropdown in pgAdmin III that lets you select the format.
To restore a plain format dump, you'll have to execute the SQL script with pgAdmin III or psql.
For the other three formats, you use pg_restore, which is internally called by pgAdmin III's "restore" functionality.
So you took a plain format dump, which causes the reported error with pg_restore. Execute it as a SQL script instead!