How can I export pgbackups dump (database backup) to s3 bucket.
Some of the pgbackup export/restore commands are as follows.
$ heroku pgbackups:capture
$ curl -o latest.dump `heroku pgbackups:url`
To restore database from s3 we can do something like this.
heroku pgbackups:restore DATABASE 'https://s3.amazonaws.com/me/items/3H0q/mydb.dump'
How can we export it from Heroku command line or from python script?
Just configuring the heroku db backup s3 buildpack. I know it is late but maybe for future insterested.
Related
I am trying to copy my local Postgres database to Heroku.
So I use pg:backups:restore from a publicly accessible URL.
As mentioned in the documentation, I have downloaded my Postgres dump to Amazon S3 and gave the public access. The link to file is: https://s3.eu-west-2.amazonaws.com/kirillch/localizedb.sql.
So now I run a command:
heroku pg:backups:restore 'https://s3.eu-west-2.amazonaws.com/kirillch/localizedb.sql' DATABASE_URL -a my_app_name
and get an error:
Backup https://s3.eu-west-2.amazonaws.com/kirillch/localizedb.sql
not found for my_app
What could be done to fix the issue?
For windows users;
heroku pg:backups:restore "https://s3.amazonaws.com/me/items/3H0q/mydb.dump" DATABASE_URL
According to the official docs, be sure to use double-quotes around the remote url if you're using windows. see official docs
So I'm trying to add my local database info to heroku database.
Firstly I tried to get public url:
justas#justas-Lenovo-M490s ~/dev/myrubyblog $ heroku pg:backups public-url
The following URL will expire at 2016-10-06 10:04:52 +0000:
"https://xfrtu.s3.amazonaws.com/c9d6480e-da41-4783-b54e-027ff267d19e/2016-10-05T20%3A06%3A08Z/67d4ad22-8cc7-4b86-aee9-7c2e648bb5aa?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAJCSJJKP7D4MYHIDQ%2F20161006%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20161006T090452Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Signature=d5807003573a49d241caf0e6805e4b770e6b25c861a7ccca0772f282167a4bed"
and I can freely download it using browser. But if I try to import it:
justas#justas-Lenovo-M490s ~/dev/myrubyblog $ heroku pg:backups restore DATABASE "URL described above"
! Unknown database: URL described above. Valid options are: DATABASE_URL
I also tried using DATABASE_URL but I get same results.
Then secondly I tried to import like this:
justas#justas-Lenovo-M490s ~/dev/myrubyblog $ heroku pg:backups capture
Use Ctrl-C at any time to stop monitoring progress; the backup
will continue running. Use heroku pg:backups info to check progress.
Stop a running backup with heroku pg:backups cancel.
DATABASE ---backup---> b002
Backup completed
and if I list all backups, I can clearly see that it was created:
justas#justas-Lenovo-M490s ~/dev/myrubyblog $ heroku pg:backups
=== Backups
ID Backup Time Status Size Database
---- ------------------------- ----------------------------------- ------ --------
b002 2016-10-06 09:09:15 +0000 Completed 2016-10-06 09:09:17 +0000 7.45kB DATABASE
b001 2016-10-05 20:06:08 +0000 Completed 2016-10-05 20:06:09 +0000 7.45kB DATABASE
=== Restores
No restores found. Use `heroku pg:backups restore` to restore a backup
=== Copies
No copies found. Use `heroku pg:copy` to copy a database to another
but it wont let me restore/import it:
justas#justas-Lenovo-M490s ~/dev/myrubyblog $ heroku pg:backups restore DATABASE b002
! Unknown database: b002. Valid options are: DATABASE_URL
same results with using DATABASE_URL instead of DATABASE. I realy need your help. Hope I gave you enough information.
To import .sql file in heroku
heroku pg:psql --app YOUR_APP_NAME_HERE < updates.sql
To import from another heroku app database.
heroku pg:backups restore `heroku pg:backups public-url -a YOUR_PRODUCTION_APP_NAME` YOUR_STAGING_DATABASE_NAME --app YOUR_STAGING_APP_NAME --confirm YOUR_STAGING_APP_NAME
Similar question: How can I import a .sql file into my Heroku postgres database?
I deployed my app on a Ubuntu server using mup deploy (https://github.com/arunoda/meteor-up) with the option "setupMongo": true in the mup.json file.
Everything works fine, and I would like to save the mongoDB database daily to FTP or S3, or to set a mongoDB replica to another server (to avoid copying the whole database every time, but it seems more complicated).
If deployed with mup, you are in luck.
You can find the steps here: https://github.com/xpressabhi/mup-data-backup
Here are the steps again:
MongoDB Data Backup deployed via mup
These commands run well only if meteor deployed with mup tool. Mup creates docker for mongodb hence taking backup becomes easy with these commands.
Backup
Take backup of running app data from docker then copy to local folder out of docker.
docker exec -it mongodb mongodump --archive=/root/mongodump.gz --gzip
docker cp mongodb:/root/mongodump.gz mongodump_$(date +%Y-%m-%d_%H-%M-%S).gz
Copy backup to server
Move data to another server/local machine or a backup location
scp /path/to/dumpfile root#serverip:/path/to/backup
Delete old data from meteor deployment
Get into mongo console running in docker then drop current database before getting new data.
docker exec -it mongodb mongo appName
db.runCommand( { dropDatabase: 1 } )
Restore data to meteor docker
docker cp /path/to/dumpfile mongodb:/root/mongodump.gz
docker exec -it mongodb mongorestore --archive=/root/mongodump.gz --gzip
The best way is to mongodump it.
Assuming its running on the mup instance itself since it only listens to 127.0.0.1 you would have to ssh in and use mongodump.
If you simply run it:
mongodump
It will create a directory dump containing your backup.
If you want to do this remotely you would have to edit /etc/mongodb.conf to ensure it binds globally, you will have to create users though since it will be publicly accessible. Then set auth to true.
You could then mongodump from your own machine (you can download the mongodump binary from mongodb.org):
./mongodump --host <your server ip address> --username <username> --password <password>
This answer is inspired by:
sheharyar.me/blog/regular-mongo-backups-using-cron
It uses a script to: mongodump -> tar -> wput (ftp)
First, create a bash script:
#!/bin/bash
MONGO_DATABASE="your_db_name"
APP_NAME="your_app_name"
MONGO_HOST="127.0.0.1"
MONGO_PORT="27017"
TIMESTAMP=`date +%F-%H%M`
MONGODUMP_PATH="/usr/bin/mongodump"
BACKUPS_DIR="/home/username/backups/$APP_NAME"
BACKUP_NAME="$APP_NAME-$TIMESTAMP"
# mongo admin --eval "printjson(db.fsyncLock())"
# $MONGODUMP_PATH -h $MONGO_HOST:$MONGO_PORT -d $MONGO_DATABASE
$MONGODUMP_PATH -d $MONGO_DATABASE
# mongo admin --eval "printjson(db.fsyncUnlock())"
mkdir -p $BACKUPS_DIR
mv dump $BACKUP_NAME
tar -zcvf $BACKUPS_DIR/$BACKUP_NAME.tgz $BACKUP_NAME
rm -rf $BACKUP_NAME
wput $BACKUP_NAME.tgz ftp://login:password#ftp.domain.com/backups/
Save it as mongo_backup.sh and run:
chmod +x mongo_backup.sh
bash mongo_backup.sh
sudo su
crontab -e
And enter this new line:
00 00 * * * /bin/bash /home/username/scripts/mongo_backup.sh
That's it.
I am trying to restore a heroku postgres data base from a dump of a local data base using the following command:
heroku pgbackups:restore HEROKU_POSTGRESQL_MAROON_URL 'https://s3-eu-west-1.amazonaws.com/my-app-data/test.dump' --app my-app
I see this:
HEROKU_POSTGRESQL_MAROON_URL (DATABASE_URL) <---restore--- test.dump
! WARNING: Destructive Action
! This command will affect the app: smc-staging
! To proceed, type "my-app" or re-run this command with --confirm my-app
and after entering my-app, I get this:
←[0KRetrieving... done
! An error occurred and your restore did not finish.
! The backup url is invalid. Use `pgbackups:url` to generate a new temporary URL.
I am running windows 7, the backup was taken using pg_dumps, and uploaded to S3, and heroku --version gets me:
heroku/toolbelt/3.3.0 (i386-mingw32) ruby/1.9.3
please help!
Can you paste your URL into a browser and get the file you expect? I suspect the file is not publicly available. Instead, I would suggest something like this to restore the latest staging backup:
heroku pgbackups:restore DATABASE -a example-staging `heroku pgbackups:url -a example`
or this to restore a specific older backup:
heroku pgbackups:restore DATABASE -a example-staging `heroku pgbackups:url a114 -a example`
I'm using heroku and I want to download the database from my app(heroku) so I can make some changes in it, I've installed pgbackups, but using heroku pgbackups:url downloads a .dump file
How can I download a postgresql file or translate that .dump into a postgresql file?
If you're using Heroku's pgbackups (which you probably should be using):
$ heroku pg:backups capture
$ curl -o latest.dump `heroku pg:backups public-url`
"Translate" it into a postgres db with
$ pg_restore --verbose --clean --no-acl --no-owner -h localhost -U myuser -d mydb latest.dump
See https://devcenter.heroku.com/articles/heroku-postgres-import-export
There's a command for this in the CLI - heroku db:pull which will do this for you. db:pull can be a bit slow mind you so you may be better to use the next option.
If you are using complex postgress data types (hstore, arrays etc) then you need to use the pgtransfer plugin https://github.com/ddollar/heroku-pg-transfer which will basically does a backup on Heroku and a restores it locally.
UPDATE: db:pull and db:push have been deprecated and should be replaced with pg:pull and pg:push - read more at https://devcenter.heroku.com/articles/heroku-postgresql#pg-push-and-pg-pull
I found the first method suggested in the documentation pull/push even easier. No password or username needed.
pg:pull
pg:pull can be used to pull remote data from a Heroku Postgres
database to a database on your local machine. The command looks like
this:
$ heroku pg:pull HEROKU_POSTGRESQL_MAGENTA mylocaldb --app sushi
This command will create a new local database named “mylocaldb” and
then pull data from database at DATABASE_URL from the app “sushi”. In
order to prevent accidental data overwrites and loss, the local
database must not exist. You will be prompted to drop an already
existing local database before proceeding.
At first I had an error: /bin/sh: createdb: command not found; which I solved following this SO post.
An alternative described also in the documentation (I did not try it yet) is:
To export the data from your Heroku Postgres database, create a new
backup and download it.
$ heroku pg:backups:capture
$ heroku pg:backups:download
Source: Importing and Exporting Heroku Postgres Databases with PG Backups
To export the data from Heroku Postgres database, just follow below steps
Login to heroku
Go to APP->settings->reveal config variable
Copy DATABASE_URL
run pg_dump --DATABASE_URL_COPIED_IN_STEP_3 > database_dump_file
Note this will provide postgresql file or for dump file you can download directly from postgres addon interface.
I think the easiest way to download and replicate the database on local server:
**PGUSER**=LOCAL_USER_NAME PGPASSWORD=LOCAL_PASSWORD heroku pg:pull --app APP_NAME HEROKU_POSTGRESQL_DB_NAME LOCAL_DB_NAME
Go through this document for more info:
https://devcenter.heroku.com/articles/heroku-postgresql#pg-push-and-pg-pull
This is the script that I like to use.
namespace :heroku do
desc "Import most recent database dump"
task :import_from_prod => :environment do
puts 'heroku run pg:backups capture --app APPNAME'
restore_backup 'APPNAME'
end
def path_to_heroku
['/usr/local/heroku/bin/heroku', '/usr/local/bin/heroku'].detect {|path| File.exists?(path)}
end
def heroku(command, site)
`GEM_HOME='' BUNDLE_GEMFILE='' GEM_PATH='' RUBYOPT='' #{path_to_heroku} #{command} -a #{site}`
end
def restore_backup(site = 'APPNAME')
dump_file = "#{Rails.root}/tmp/postgres.dump"
unless File.exists?(dump_file)
pgbackups_url = heroku('pg:backups public-url -q', site).chomp
puts "curl -o #{dump_file} #{pgbackups_url}"
system "curl -o #{dump_file} '#{pgbackups_url}'"
end
database_config = YAML.load(File.open("#{Rails.root}/config/database.yml")).with_indifferent_access
dev_db = database_config[Rails.env]
system "pg_restore -d #{dev_db[:database]} -c #{dump_file}".gsub(/\s+/,' ')
puts
puts "'rm #{dump_file}' to redownload postgres dump."
puts "Done!"
end
end