I have a local psql database dump that needs to be uploaded to heroku. I followed the steps in the Update section from this link. Everything worked fine until the last part, the actual uploading step:
heroku pgbackups:restore --app myAppName DATABASE
'https://www.dropbox.com/myAppPSQLDumpLink/myAppName_local.dump' --confirm myAppName
This was what was showing up in the console:
HEROKU_POSTGRESQL_SILVER_URL (DATABASE_URL) <---restore--- myAppName_local.dump
Retrieving... done
! An error occurred and your restore did not finish.
And this was the error from the logs (courtesy of Toby Hede's question):
2013-01-09T15:39:09+00:00 app[pgbackups]: Invalid dump format: /tmp/GgUz5yU4bF/project_mgr_development_local.dump: HTML document text
I tried searching for this error, but could not find an answer. Does anyone know what needs to be done to solve the problem? The actual dump for my local psql database was performed this way:
pg_dump -Fc --no-acl --no-owner -U myUserName > myAppName_local.dump
Thank you!
Looks like the link to the dump on Dropbox is redirecting or pointing to an HTML page (HTML document text in the error). Visit the link and make sure you are directly getting the dump. Or download the dump in your browser, right-click on it and Copy Download Link. That link should work with pgbackups:restore.
Dropbox provides an explanation for directly downloading files (https://www.dropbox.com/en/help/201)
This can be helpful for using the dropbox links in pg_backups.
In short it says to have the download link with option "dl=1" rather than "dl=0". But this didn't work for me. Even copying the the address of downloaded file didn't work for me.
If you face the above problems, try moving the file into the public folder and copy the link from there. This worked for me.
According to Importing and Exporting Heroku Postgres Databases with PG Backups, you can restore dump at terminal with:
$ curl -o latest.dump `heroku pgbackups:url --app heroku_appname`
$ pg_restore --verbose --clean --no-acl --no-owner -h localhost -U myuser -d mydb latest.dump
Got the same error, but different reason, so different solution. Maybe it helps someone.
If you've stored the dump file in a server with HTTPS, and you mistakenly used HTTP for the database URL, the forwarding will be interpreted as an HTML document.
So change
heroku pgbackups:restore --app myAppName DATABASE 'http://www.example.com/my.dump' --confirm myAppName
to
heroku pgbackups:restore --app myAppName DATABASE 'https://www.example.com/my.dump' --confirm myAppName
Related
Problem
I am following this article from Heroku regarding a database migration (import). I created the database dump like in the example with PGPASSWORD=mypassword pg_dump -Fc --no-acl --no-owner -h localhost -U myuser mydb > mydb.dump and then tried to upload it with this command after creating a public accessible file via dropbox:
heroku pg:backups:restore 'https://www.dropbox.com/s/2hjbz9573oodmpp/my_new_db.dump?dl=1' DATABASE_URL -a my-app
This produces this error:
▸ An error occurred and the backup did not finish.
▸
▸ pg_restore: error: could not read from input file: end of file
▸ pg_restore finished with errors
▸ waiting for download to complete
▸ download finished with errors
▸ please check the source URL and ensure it is publicly accessible
Thoughts
It seems like the problem that the url is not publicy accessible. But I tried to open the url in the browser and it directly opens a download prompt. So is there anything I am not seeing? Is the only option using a signed url from AWS?
Try removing the ?dl=1 from the url
Been trying to pull my remote db into local env with heroku pg:pull, prior without problems, but at this point the command is returning this:
/bin/sh: pg_dump: command not found
▸ pg_dump errored with 127
pg_restore: [custom archiver] could not read from input file: end of file
looked for an answer here and updated the bash profile as some topics mention:
export PATH=/Applications/Postgres.app/Contents/Versions/10/bin:$PATH
rebooted and restarted terminal, but with no avail. anyone knows what's going on?
Ypsql tries to interpret SQL queries while you are adding it a compressed dump (heroku use this).
When You can not read the dump, pg_restore -O latest.dump gives you valid SQL you may pipe to psql
the easiest here :
pg_restore -O -d app_development latest.dump
Imporatant:
Try Use -O you may not use the random username of your remote heroku postgres db.
Check this link for more help here stackoverflow.com/a/11391586/6392696
Update your Heroku CLI; the issue where Postgres.app paths weren't being handled correctly was just fixed. https://github.com/heroku/cli/pull/1074
I am starting my first meteor app, using MongoDB for my database.
I have copied my data in from a JSON file into mongodb collection and when hosting locally, the data appears as expected; the collection is in the meteor mongo database and I can interpret at will
When I deploy the app to xx.meteor.com, the meteor collection I need (named 'assets') does not get posted to the mongodb on the server. I can check this by using meteor mongo onemore.meteor.com. Other collections are posted though.
How do I deploy my mongoDB colletion ('assets') along with the app using meteor deploy?
Is this issue due to the way the mongodatabase was orginally imported? I used this method: https://github.com/awatson1978/meteor-cookbook/blob/master/cookbook/database-management.md
I copied the collection ('assets) from a staging database using db.copyDatabase('staging3','meteor','localhost')
Using mongodump and mongorestore also works:
Dump data from existing mongodb (mongodb url: mongodb://USER:PASSWORD#DBHOST/DBNAME)
mongodump -h DBHOST -d DBNAME -u USER -p PASSWORD
This will create a "dump" directory, with all the data going to dump/DBNAME.
Get the mongodb url for the deployed meteor app (i.e. www.mymeteorapp.com)
meteor mongo --url METEOR_APP_URL
This will return the following:
mongodb://#USERNAME#:#PASSWORD###HOSTNAMEANDPORT#/#YOURAPPLICATION#
Note: the PASSWORD expires every min.
Upload the db dump data to the meteor app (using an example meteor db url)
mongorestore -u #USERNAME# -p #PASSWORD# -h #HOSTNAMEANDPORT# -d www_mymeteorapp_com dump/DBNAME/
All the data should get transferred!
This answer is basically a modified version of Davidd8's answer here, but because that was never accepted I reposted it here.
None of the above worked for me. Here is what did the trick for me:
1) How do I get my localhost dump of my meteor db?
tip 1: make sure mongo is running (type mongod to run it)
Which port was my meteor running on?
$ mongodump --host localhost:3002 (nope)
$ mongodump --host localhost:3000 (nope)
$ mongodump --host localhost:3001 (yes!) this worked
Now where the heck did it put my dump?
$ ls (shows me the dump is in my current directory and meteor is inside the dump directory)
I need to somehow get that meteor data onto my live site on meteor.com
Now the next problem is that meteor.com will give me the information I need but I only have 1 minute before their information expires.
This information is what I need to connect to my remote meteor site and import my local meteor data.
I type this in the terminal:
$ meteor mongo --url iamcool.meteor.com
Then it spits back something like this:
mongodb://client-11f3014w:f1c52f68-8506-b682-c880-b5db03a9510e#production-db-a2.meteor.io:27017/iamcool_meteor_com
I used info from above that told me what data to extract from the above code and where to put it in the mongorestore code
mongorestore -u #USERNAME# -p #PASSWORD# -h #HOSTNAMEANDPORT# -db www_mymeteorapp_com dump/DBNAME/
This is where you have to be a super fast typer. I recommend opening a blank file in your IDE (Sublime Text or Atom or whatever you use) and create this template:
mongorestore -u XXX -p XXX -h production-db-a2.meteor.io:27017 -db iamcool_meteor_com dump/meteor
So I took the data meteor.com gave me and I plugged it in below:
mongorestore -u client-4bf9be36 -p d1cdef86-6c71-3b11-ef4c-221fbdcf07d0 -h production-db-a2.meteor.io:27017 -db iamcool_meteor_com dump/meteor
And then the errors came. And boy did they come.
Sometimes I got this error:
too many positional arguments
Sometimes I got this error:
Failed: error connecting to db server: auth fails
But this error tip gave me the info that saved the day:
try 'mongorestore --help' for more information
I did and I found out that the flag to pass for database wasn't -db but rather -d
I made the following change and typed it fast using my trusty template (see above)
mongorestore -u client-4bf9be36 -p d1cdef86-6c71-3b11-ef4c-221fbdcf07d0 -h production-db-a2.meteor.io:27017 -d iamcool_meteor_com dump/meteor
And then it worked.
It took me two hours to figure out. Hope it saves you the time I lost.
I am trying to restore a heroku postgres data base from a dump of a local data base using the following command:
heroku pgbackups:restore HEROKU_POSTGRESQL_MAROON_URL 'https://s3-eu-west-1.amazonaws.com/my-app-data/test.dump' --app my-app
I see this:
HEROKU_POSTGRESQL_MAROON_URL (DATABASE_URL) <---restore--- test.dump
! WARNING: Destructive Action
! This command will affect the app: smc-staging
! To proceed, type "my-app" or re-run this command with --confirm my-app
and after entering my-app, I get this:
←[0KRetrieving... done
! An error occurred and your restore did not finish.
! The backup url is invalid. Use `pgbackups:url` to generate a new temporary URL.
I am running windows 7, the backup was taken using pg_dumps, and uploaded to S3, and heroku --version gets me:
heroku/toolbelt/3.3.0 (i386-mingw32) ruby/1.9.3
please help!
Can you paste your URL into a browser and get the file you expect? I suspect the file is not publicly available. Instead, I would suggest something like this to restore the latest staging backup:
heroku pgbackups:restore DATABASE -a example-staging `heroku pgbackups:url -a example`
or this to restore a specific older backup:
heroku pgbackups:restore DATABASE -a example-staging `heroku pgbackups:url a114 -a example`
I'm using heroku and I want to download the database from my app(heroku) so I can make some changes in it, I've installed pgbackups, but using heroku pgbackups:url downloads a .dump file
How can I download a postgresql file or translate that .dump into a postgresql file?
If you're using Heroku's pgbackups (which you probably should be using):
$ heroku pg:backups capture
$ curl -o latest.dump `heroku pg:backups public-url`
"Translate" it into a postgres db with
$ pg_restore --verbose --clean --no-acl --no-owner -h localhost -U myuser -d mydb latest.dump
See https://devcenter.heroku.com/articles/heroku-postgres-import-export
There's a command for this in the CLI - heroku db:pull which will do this for you. db:pull can be a bit slow mind you so you may be better to use the next option.
If you are using complex postgress data types (hstore, arrays etc) then you need to use the pgtransfer plugin https://github.com/ddollar/heroku-pg-transfer which will basically does a backup on Heroku and a restores it locally.
UPDATE: db:pull and db:push have been deprecated and should be replaced with pg:pull and pg:push - read more at https://devcenter.heroku.com/articles/heroku-postgresql#pg-push-and-pg-pull
I found the first method suggested in the documentation pull/push even easier. No password or username needed.
pg:pull
pg:pull can be used to pull remote data from a Heroku Postgres
database to a database on your local machine. The command looks like
this:
$ heroku pg:pull HEROKU_POSTGRESQL_MAGENTA mylocaldb --app sushi
This command will create a new local database named “mylocaldb” and
then pull data from database at DATABASE_URL from the app “sushi”. In
order to prevent accidental data overwrites and loss, the local
database must not exist. You will be prompted to drop an already
existing local database before proceeding.
At first I had an error: /bin/sh: createdb: command not found; which I solved following this SO post.
An alternative described also in the documentation (I did not try it yet) is:
To export the data from your Heroku Postgres database, create a new
backup and download it.
$ heroku pg:backups:capture
$ heroku pg:backups:download
Source: Importing and Exporting Heroku Postgres Databases with PG Backups
To export the data from Heroku Postgres database, just follow below steps
Login to heroku
Go to APP->settings->reveal config variable
Copy DATABASE_URL
run pg_dump --DATABASE_URL_COPIED_IN_STEP_3 > database_dump_file
Note this will provide postgresql file or for dump file you can download directly from postgres addon interface.
I think the easiest way to download and replicate the database on local server:
**PGUSER**=LOCAL_USER_NAME PGPASSWORD=LOCAL_PASSWORD heroku pg:pull --app APP_NAME HEROKU_POSTGRESQL_DB_NAME LOCAL_DB_NAME
Go through this document for more info:
https://devcenter.heroku.com/articles/heroku-postgresql#pg-push-and-pg-pull
This is the script that I like to use.
namespace :heroku do
desc "Import most recent database dump"
task :import_from_prod => :environment do
puts 'heroku run pg:backups capture --app APPNAME'
restore_backup 'APPNAME'
end
def path_to_heroku
['/usr/local/heroku/bin/heroku', '/usr/local/bin/heroku'].detect {|path| File.exists?(path)}
end
def heroku(command, site)
`GEM_HOME='' BUNDLE_GEMFILE='' GEM_PATH='' RUBYOPT='' #{path_to_heroku} #{command} -a #{site}`
end
def restore_backup(site = 'APPNAME')
dump_file = "#{Rails.root}/tmp/postgres.dump"
unless File.exists?(dump_file)
pgbackups_url = heroku('pg:backups public-url -q', site).chomp
puts "curl -o #{dump_file} #{pgbackups_url}"
system "curl -o #{dump_file} '#{pgbackups_url}'"
end
database_config = YAML.load(File.open("#{Rails.root}/config/database.yml")).with_indifferent_access
dev_db = database_config[Rails.env]
system "pg_restore -d #{dev_db[:database]} -c #{dump_file}".gsub(/\s+/,' ')
puts
puts "'rm #{dump_file}' to redownload postgres dump."
puts "Done!"
end
end