Setup --
I have a postgresql db running on AWS RDS. We access it from an EC2 instance using pgAdmin 3.
Current Scenario --
AWS takes a daily backup of the entire RDS instance as RDS snapshots which is stored for 15 days.
I take a manual backup of one important schema (all the data in schema included) using pgAdmin and then push this backup file to s3.
Question --
How do I automate this backup of a single schema (including data) in postgresql and store it on S3 (any application or script). Is there a way to automate it in pgadmin just for one schema and not the entire database.
You need to use a command line tool pg_dump:
pg_dump \
--host=myhostname.qwertyuiop.sc-moon-1.rds.amazonaws.com \
--username=myusername --dbname=mydbname \
--format=custom --file=mydbname.pg_dump
It needs to be the same major version as the database.
Related
In the Gcloud SQL Postgres database, I am currently using gcloud export and import command to create database backup in a different project. which takes a long time.
As gcloud allows us to create read replica but it creates with in the same project.
my question is?
how can i create a replica of cloud sql postgres instance in another project postgres instacne?
I want to create an automated job that can copy the entire database to a different one, both are in AWS RDS Postgres, how can I do that?
Thanks.
You can use Database create/restore snapshot.
Here is the example for command line:
aws rds create-db-snapshot \
--db-instance-identifier mydbinstance \
--db-snapshot-identifier mydbsnapshot
aws rds restore-db-instance-from-db-snapshot \
--db-instance-identifier mynewdbinstance \
--db-snapshot-identifier mydbsnapshot
The same APIs such as CreateDBSnapshot are available for multiple languages via AWS SDK.
I have had success in the past running a script that dumps data from one Postgres server and pipes it into another server. It was basically like this pseudo-code:
psql target-database -c "truncate foo"
pg_dump source-database --data-only --table=foo | psql target-database
The pg_dump command outputs normal SQL commands that can be piped into a receiving psql command, which then inserts the data.
To understand how this works, run pg_dump on one table and then take a look at the output. You'll need to tweak the command to get exactly what you want (eg using --no-owner to avoid sending access configurations).
Is there a way to automatically generate the database creation string for an empty database with settings from an existing database?
I created an AWS RDS postgresql database instance using the console in my browser. I selected all kind of setting in my browser and launched the instance. Now I want to replicate this database (empty) with a command for programmatic use. I would like to delete the existing RDS instance so a raw command would be preferred (no reference to another RDS) but something like:
aws create-db-instance --db-name database02 --allocated-storage 200 --engine postgres etc.
You'll need to create a snapshot of the source Database, instance or cluster and then create a new database based on that snapshot. You can delete the source database after the snapshot is complete.
aws rds create-db-snapshot
--db-snapshot-identifier mydbsnapshot
--db-instance-identifier mydb
aws rds restore-db-instance-from-db-snapshot
--db-instance-identifier mynewdbinstance
--db-snapshot-identifier mydbsnapshot
http://docs.aws.amazon.com/cli/latest/reference/rds/create-db-snapshot.html
http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_RestoreFromSnapshot.html
http://docs.aws.amazon.com/cli/latest/reference/rds/restore-db-instance-from-db-snapshot.html
I had so much trouble doing this - I thought I would make a Q/A on StackOverflow to explain the process.
The question is about copying an RDS postgres database for development usage - especially for testing database migrations scripts, etc. That's why the focus on a "single schema" within a "single database".
In my case, I want to create a test database that's as isolated as possible, while remaining within a single RDS instance (because spinning up entire RDS instances takes anywhere from 5 - 15 minutes and because I'm cheap).
Here is an answer using only the command line.
Pre-requisites:
you must have Postgres client tools installed (don't need the actual server)
client version must be same or higher than your postgres server version
network access to the RDS instance
credentials for accessing the relevant database accounts
Example context:
I have an RDS instance at rds.example.com which has a master user named rds_master.
I have an "application user" named db_dev_user, a database named dev_db that contains the schema app_schema.
note that "user" and "role" in postgres are synonymous
Note: this guide was written in 2017, for postgres version 9.6.
If you find that some steps are no longer working on a recent version of postgres - please do post any fixes to this post as comments or alternative answers.
pg_dump prints out the schema and data of the original database and will work even while there are active connections to the database. Of course, performance for those connections is likely to be affected, but the resultant copy of the DB is transactional.
pg_dump --host=rds.example.com --port=5432 \
--format=custom \
--username=db_dev_user --dbname=dev_db \
> pgdumped
The createuser command creates the user that your test application/processes should connect with (for better isolation), note that the created user is not a superuser and it cannot create databases or roles.
createuser --host=rds.example.com --port=5432 \
--username=rds_master \
--no-createdb --no-createrole --no-superuser \
--login --pwprompt \
db_test_user
Without this next grant command the following createdb will fail:
psql --host=rds.example.com --port=5432 \
--username=rds_master --dbname=postgres \
--command="grant db_test_user TO rds_master"
createdb does what it says on the tin; note that the db_test_user role "owns" the DB.
createdb --host=rds.example.com --port=5432 \
--username=rds_master --owner=db_test_user test_db
The create schema command is next. The db_test_user cannot create the schema, but it must be authorized for the schema or the pg_restore would fail because it would end up trying to restore into the pg_catalog schema (so note that user=rds_master, but dbname=test_db).
psql --host=rds.example.com --port=5432 \
--username=rds_master --dbname=test_db \
--command="create schema app_schema authorization db_test_user"
Finally, we issue the pg_restore command, to actually create the schema objects (tables, etc.) and load the data into them:
pg_restore --host=rds.example.com --port=5432 \
--verbose --exit-on-error --single-transaction \
--username=db_test_user --schema=app_schema \
--dbname=test_db --no-owner \
./pgdumped
exit-on-error - because otherwise finding out what went wrong involves too much scrolling and scanning (also it's implied by single-transaction anyway)
single-transaction - avoids having to drop or recreate the DB if things go pear-shaped
schema - only do the schema we care about (can also supply this to the original pg_dump command)
dbname - to ensure use of the DB we created
no-owner - we're connecting as db_test_user anyway, so everything should be owned by the right user
For production, you'd be better off just taking an RDS snapshot of your instance and restoring that, which will create an entirely new RDS instance.
On a mostly empty database - it takes a few minutes to create the snapshot and another 5 minutes or so to create the new RDS instance (that's part of why it's a pain during development).
You will be charged for the new RDS instance only while it is running. Staying within the free tier is one of the reasons I wanted to create this DB with the same instance for development purposes, plus not having to deal with a second DNS name; and that effect is multiplied as you start to have multiple small development environments.
Running a second RDS instance is the better option for production because you nearly completely eliminate any risk to your original DB. Also, when you're dealing with real amounts of data - snapshot/DB creation times will be dwarfed by the amount of time spent reading/writing the data. For large amounts of data, it's likely the Amazon RDS snapshot creation/restore process is going to have far better parallelisation than a set of scripts running on a single server somewhere. Additionally, the RDS console gives you visilibility into the progress of the restore - which becomes invaluable as the dataset grows larger and more people become involved.
I dumped my production db from an Amazon RDS postgresql instance and on occasion, I restore production to our staging and development databases.
Currently the staging and development databases reside on an RDS instance and the import works great. I am currently attempting to restore the database to a postgres installation that isn't an RDS instance and I continuously get the error invalid command \N before that I get ERROR: relation "locations" does not exist. I have been trying everything to get this to work. I have recreated the database several times ensuring all of the settings match what I can see of the RDS instance, and am having no luck.
I am attempting to use psql -h {host} -U {user} -d {db} < production.sql