I am in the process of configuring the backup gem (http://backup.github.io/backup/v4/) to run on my EC2 instance, copy the PostgreSQL database in RDS, and store the backup in a new S3 bucket.
The backup gem runs the pg_dump command, however AWS doesn't allow for the same version of Postgres to be installed on both EC2 and RDS, resulting in the following error:
pg_dump: server version: 9.4.7; pg_dump version: 9.2.13
pg_dump: aborting because of server version mismatch
This is because the EC2 instance has version:
$ pg_dump --version
pg_dump (PostgreSQL) 9.2.13
And the RDS instance has version:
9.4.7-R1 (with the only other version option of 9.5.2-R1)
On EC2, running yum list postgres* only offers Available Packages up to PostgreSQL 9.3.
So it seems like I am unable to either downgrade RDS or upgrade EC2 to a matching version.
Here is my Backup gem model config if it helps: https://gist.github.com/anonymous/35f6f9e81846f53693fb03662c2192ad
Before too many people start reminding me that RDS has built-in backups, I am aware. My use-case: instead of only having full database fallbacks, I would also like the ability to roll back individual users' data to different time periods without affecting the whole database. I planned on keeping these manual backups and eventually writing a script to pull previous user specific data from them.
My friend recommended another option: If a user wants to roll back, I could spin up a new RDS from the automated snapshots, clone my EC2 instance, connect them to each other, collect the user specific data from that snapshot, and then merge those changes back into the main EC2 instance.
Set up PostgreSQL’s YUM repository on your EC2 instance:
https://yum.postgresql.org/
and install a newer PostgreSQL client version.
Related
I'm trying to upgrade an RDS database cluster engine from Aurora PostgreSQL 9.6.19 before its end of life, I made copy and tried to upgrade to 9.6.21 and 10.16 but everytime the same problem happens:
Database cluster is in a state that cannot be upgraded: Postgres cluster is in a state where pg_upgrade can not be completed successfully.
The status of the database is Available so maybe it refers to something else but I don't know what and how to fix it, I've tried looking for answers to no avail.
Has anyone fixed this?
The pg_upgrade_internal log file will usually contain details on any failures/errors.
You can take a look on these logs using the command line:
aws rds describe-db-log-files --db-instance-identifier my-db-instance
Or via console, or RDS API.
For more information take a look on these links: Upgrading the PostgreSQL DB engine for Amazon RDS, Viewing and listing database log files
I have a kubernetes space where the PostgreSQL DB is on a different POD than the executor where my Typescrtipt code is being executed, which means that I do not have access to pg_dump and psql. Is there a typescript npm library which can backup and restore the DB programmatically and remotely or a special client which I can install in project so that I can use? I could not find such an mpm library or client on the internet.
Another thing that I can think of is some "universal" SQL script, which makes the backup and another one for the restore, if there is such a thing at all.
I'm currently in the process of switching my cloud server from Heroku to Digital Ocean. However is there a way to migrate the database from the heroku server to the digital ocean one? I use postgresql for my database
I hope you already got a solution, but in case you didn’t, I’ll provide a simple guide on how I did it. I am going to assume that you have already created a postgres database on digitalocean. Also you need navigate to your project directory and log in to heroku using the heroku cli. And, you need to have postgresql installed or a psql client. Installing postgresql would do it as it comes with psql.
Step 1: Create a backup and download the backup from heroku postgres
heroku pg:backups:capture --app <app_name>
heroku pg:backups:download --app <app_name>
The first command will create a backup of your database and the second command will download it to your current directory, its a .dump file. If you would like to read more, here is an article.
Step 2: Connect to your remote (digital ocean’s) database using psql
Before you can do this, you need to go and add your machine you are connecting from to the list of database’s list of trusted sources, If you don’t, you’ll get a Connection Timed Out error. That’s because the database’s firewall doesn’t allow you to connect to the database from your local machine or resource (for security reasons).
Step 3: Import the Database
pg_restore -d "postgresql://<database_username>:<database_password>#<host>:<port>/<database>?sslmode=require" --jobs 4 -c "/path/to/dump_file.dump"
This will import your database from your dump file. Just substitute the variables will your connection parameters that you get from your dashboard. If you would like to read more, here is another article for this step.
Another thing to make clear is, sometimes, you will see some harmless error messages when running this command, but it will push through anyway. To learn more about pg_restore read this article.
And that’s it, your database has been migrated. Now, can you confirm it worked?, well, as for me, I used pgAdmin to connect to the remote database and I saw the tables and data as expected.
Hope this helps anyone with the same problem :)
I have deployed my SpringBoot app to Heroku. Now I would like to copy my local PostgreSQL to Heroku.
I have found some information on devcenter.heroku.com.
However I don't understand enough about the using of file db.changelog-master.yaml.
Could anyone give me details about the simplest solutions to copy the database?
Create a valid dump of your local postgres database and host it somewhere publicly available. Now you will be able to restore this entire dataset (schema and records) with pg:backups:restore as shown here. The sole caveat here is that the target database must be completely empty for this to work. You can empty a Heroku postgres database with heroku pg:reset.
If you cannot take the approach listed above then you can run pg_restore directly from your local instance, provided your local version of Postgres is >= the target version of Postgres. This also applies to creating the dumpfile and is a requirement because pg utilities are not guaranteed to be forward compatible. Documentation for pg_restore is here.
I want to migrate my application from Heroku to my own ubuntu server. What is the correct syntax to do this using taps/heroku db:pull?
If you are moving data between a local and cloud database both of which are Postgres you should follow the instructions on Heroku's Devcenter:
https://devcenter.heroku.com/articles/migrating-data-between-plans
You would simply do heroku db:pull to pull the database from Heroku to your local db which you would then backup and restore to your own server. If it's a large database you might want to look into Heroku pgbackups as an alternative as it will provide you with a downloadable backup.
Note: If you want to migrate between databases, ie postgres => mysql then you would have to use the db:pull approach.