we are using Heroku as our Cloud platform service
Now I have 3 apps 1 as production and the other two as staging, development.
I made some changed in staging app and I need to copy the database (Postgres) from production to staging.
I found some guidelines in Heroku documentation but it's not clear.
can any one share his experience with me?
the database reach 25 GB Now
As always, it depends. I see two ways:
heroku backups / restore: just use the last backup you have, or create a new one, then restore it to the staging server.
forking the database: on the staging app, delete the postgres addon, and recreate it as a fork (aka copy) of the production server. The --fast option makes it even faster when the data doesn't have to be 100% up-to-date.
Since 25 GB is already over the recommended size for normal Heroku backups (20 GB), I would recommend going for option 2.
(though we are using Heroku backups for a 200GB database without any issues).
Related
Is there a way to backup all of the databases on a hyperscaler managed postgres server as of a certain time in order to maintain data consistency between the databases with either pg_dumpall, pg_dump or something else?
Background:
With the utilization of micro-services, an application may have many databases associated to it on a single hyperscaler managed postgres server. The hyperscalers do perform a functional snapshot backup; however, when a hyperscaler managed postgres server is accidentally deleted, the postgres backups are lost as well. These hyperscalers provide locks to prevent accidental deletes of a postgres server and mention that their support teams can be contacted to restore a deleted server, however, we still had a postgres server get deleted. We were able to recover by contacting the hyperscalers support team but would like to have a second way of backing up a hyperscaler managed postgres server.
I realize that the micro-services should be able to auto-recover to a data consistent point but the reality is that many of the micro-services have not been designed nor written to that requirement. I really do not want to get into the aspect of micro-service design and want to retain this to be a DBA backup question.
pg_dumpall will not take a consistent backup across all databases. Each database backup will be consistent, but the snapshots for the backups of the different databases will be taken at different times.
If you need a consistent backup across several databases in a single cluster, use an online file system backup with pg_basebackup.
You can use pg_basebackup which most of the PostgreSQL DBA’s would end up using on a daily basis be it via scripts or manually. This creates the base backup of the database which can help in recovering in multiple situations. This takes an online backup of the database and hence is very useful when being used in production
you can review this for more details
Using this article: Heroku dev environments
I successfully made two separate Heroku apps, one test, and one prod. This each have their own remotes on my local development box. But now I don't know how to separate the postgres tables of my now distinct development and prod applications.
This is the command I use to create a postgres table with my Heroku app:
heroku addons:add heroku-postgresql
But then I lost access to my original postgres DB. I would like this setup so I don't inadvertently hose my prod DB from my development branch.
TL;DR: How do i keep distinct postgres databases with a single Heroku application that has multiple environments?
EDIT #1: I found that I can call:
heroku addons:add heroku-postgresql:dev
multiple times to create several dev databases. What is the best practice for pointing my dev vs. prod apps to each database without having to hard code my database links like this:
//this function connects to the Heroku postgres db
function pg_connection_string(){
return "dbname=dcs1k5588jbfad host=ec2-54-243-224-162.compute-1.amazonaws.com port=5432 user=###user_name### password=######### sslmode=require";
}
Is there a Heroku-fung-shui of swapping database pointers?
You can set the environment variable or heroku config variable DATABASE_URL to anything you wish. This is what Heroku applications expect to be the default database for that application.
If you're using a production Database you can Fork from your production app to a test app to get a copy of the data as well: https://devcenter.heroku.com/articles/heroku-postgres-fork
we are using Heroku Postgres with Ruby on Rails 3.2.
A few days before, we deleted important data by mistake using 'heroku run db:load' with misconfigured data.yml, that is, drop tables and the recreate tables with almost no data.
Backup data is only available 2 weeeks before, so we lost data of 2 weeks.
So We need to recover not by PG Backup/pg_dump but by postgresql's system data files.
I think, the only way to recover data is to restore data from xlog or archive file, but of course we don't have permission to be Super User/Replication Role to copy postgres database on heroku (or Amazon EC2) to local server.
Is there anyone who confronted such a case and resolved the problem?
Your only option is the backups provided by the PgBackups service (if you had that running). If not, Heroku support might have more options available.
At a minimum, you will have some data loss, but you can guarantee you won't do it again ;)
I'd like to have my master Postgres DB, which is is hosted on Heroku, replicate down to a slave DB on my laptop. Is this possible?
Heroku's documentation talks about both master and slave hosted within Heroku:
https://devcenter.heroku.com/articles/heroku-postgres-follower-databases
Someone else asked whether it's possible to have the master outside Heroku and a slave inside Heroku (it's not):
Follow external database from Heroku
I haven't seen an answer for the reverse -- having the master in Heroku and the slave outside.
Why do I want this? To speed up development. With my app running locally and the DB in the cloud, the round-trip is long so data access is slow. Most data access is read-only. If I could have a local slave, it would speed things up significantly.
Related: what if my laptop is disconnected for a while? Would that cause problems for the master?
You cannot make a follower (slave) outside of the Heroku network – followers need superuser access to create, which Heroku Postgres doesn't provide you, so you are limited to running a follower on Heroku.
If you want to pull down a copy locally for use/inspection, you can do so with pgbackups: https://devcenter.heroku.com/articles/heroku-postgres-import-export
I'd highly recommend the program Parity for this.
It copies down the last Heroku backup to your local machine with a nice command line interface:
development restore production
I'd rather just pull the production database's contents from Heroku every now and then.
$ heroku db:pull
You can speed that up with a rake task.
# lib/tasks/deployment.rake
namespace :production do
desc 'Pull the production DB to the local env'
task :pull_db do
puts 'Pulling PRODUCTION db to local...'
system 'heroku db:pull --remote MY_REMOTE_NAME --confirm MY_APP_NAME'
puts 'Pulled production db to local'
end
end
You can call rake production:pull_db to overwrite your local development database.
Our database on Heroku has grown to size which requies three pgbackups URLs to download. I know how to download this database backup to local computer, merge the parts together and do restore, but what to do if I want to copy database from one Heroku app to other?
If you want to copy from one application from another you should be able to fork your database across applications. You can do this with:
heroku addons:add heroku-postgresql:crane --fork HEROKU_POSTGRESQL_SOURCE_OF_OLD_DB_AS_URL --app newapphere