I am having troubles restoring my backups to the heroku database via an external dump file. I run this command:
heroku pg:backups:restore 'https://s3.us-east-2.amazonaws.com/some-bucket/some-dump.dump' DATABASE_URL
And I get this error in return:
Backup 'https://s3.us-east-2.amazonaws.com/some-bucket/some-dump.dump' not found for some-heroku-app
I can access the file from the browser, but it seems that pg:backups can't access the file. I've also tried different methods of hosting the file (including a private FTP server and Dropbox).I've searched for other people with the same problems, but none of the questions seemed to be answered
Related
I'm currently in the process of switching my cloud server from Heroku to Digital Ocean. However is there a way to migrate the database from the heroku server to the digital ocean one? I use postgresql for my database
I hope you already got a solution, but in case you didn’t, I’ll provide a simple guide on how I did it. I am going to assume that you have already created a postgres database on digitalocean. Also you need navigate to your project directory and log in to heroku using the heroku cli. And, you need to have postgresql installed or a psql client. Installing postgresql would do it as it comes with psql.
Step 1: Create a backup and download the backup from heroku postgres
heroku pg:backups:capture --app <app_name>
heroku pg:backups:download --app <app_name>
The first command will create a backup of your database and the second command will download it to your current directory, its a .dump file. If you would like to read more, here is an article.
Step 2: Connect to your remote (digital ocean’s) database using psql
Before you can do this, you need to go and add your machine you are connecting from to the list of database’s list of trusted sources, If you don’t, you’ll get a Connection Timed Out error. That’s because the database’s firewall doesn’t allow you to connect to the database from your local machine or resource (for security reasons).
Step 3: Import the Database
pg_restore -d "postgresql://<database_username>:<database_password>#<host>:<port>/<database>?sslmode=require" --jobs 4 -c "/path/to/dump_file.dump"
This will import your database from your dump file. Just substitute the variables will your connection parameters that you get from your dashboard. If you would like to read more, here is another article for this step.
Another thing to make clear is, sometimes, you will see some harmless error messages when running this command, but it will push through anyway. To learn more about pg_restore read this article.
And that’s it, your database has been migrated. Now, can you confirm it worked?, well, as for me, I used pgAdmin to connect to the remote database and I saw the tables and data as expected.
Hope this helps anyone with the same problem :)
I developed a FastAPI app in a Virtual Environment using an SQLite database but deployed it on Heroku with a PostgresSQL database on Heroku as suggested in the tutorial. Although it worked on my PC, adding PostegresSQL as an addon & replacing the value of SQLALCHEMY_DATABASE_URL in the database.py broke everything. Note that I've properly frozen the dependencies on the requirements.txt file. Yet I can't figure out what went wrong.
For further clarification, I've pushed my code to GitHub & it can be accessed at this repository - Self_calculation.
If you are using Postgres Addon on Heroku, probably your solution is simple.
Use os.environ to get the connection parameters , don't try to connect directly, it's Heroku's recommended solution from Heroku Postgres
import os
DATABASE_URL = os.environ.get('DATABASE_URL')
I just need to run this:
1)heroku git:remote -a my_heroku_app_name
2)heroku logs --tail
after that I could see my problem.
In my case I forgot to change postgres url in alembic.ini file
I'm trying to import a local Postgresql database to Heroku and I'm following these steps https://devcenter.heroku.com/articles/heroku-postgres-import-export#import-to-heroku-postgres.
I have successfully:
created a dump
uploaded it to an S3 Bucket
created from AWS CLI a signed link
ran the command heroku pg:backups:restore '<SIGNED URL>' DATABASE_URL (adding -a with my app name).
The process to restore a backup starts correctly but then exits with this code:
! An error occurred and the backup did not finish.
!
! Could not initialize transfer
!
! Run heroku pg:backups:info r011 for more details.
Opening the log shows:
Database: BACKUP
Finished at: 2020-01-09 18:49:30 +0000
Status: Failed
Type: Manual
Backup Size: 0.00B (0% compression)
=== Backup Logs
2020-01-09 18:49:30 +0000 Could not initialize transfer
I've tried:
re-uploading the file to the bucket,
generating a new signed link,
putting the app in maintenance mode,
I've created a user in my IAM management service with full S3 access and saved the credentials in the app environment as from https://devcenter.heroku.com/articles/s3
Not sure where to go from here but would appreciate any help. (I'm on the hobby plan therefore I can't ask Heroku's support for help)
Edit: I also tried:
deleting and recreating the S3 Bucket
installing version 1 of the AWS CLI to see if by chance the structure of a presigned link had changed
Edit 2: Since I could not find a solution I've opted to migrate the hosting entirely on AWS for the moment
Make sure that your credentials on your machine that are stored in ~/.aws/ the default value is set to the credentials you created for your heroku configs. Then also make sure the signed url is created with those credentials and configs. I had to set my default credentials to the credentials I put in my heroku configs. Then I also had to set my default region in ~/.aws/config to match the bucket location. Should work after that.
Here are some instructions if you are on mac or linux.
Sorry Windows people. I would assume it is something similar.
Create new access id and key in IAM on AWS
Set heroku configs to use those credentials heroku config:set AWS_ACCESS_KEY_ID=xxx AWS_SECRET_ACCESS_KEY=yyy
Optional (You may have to set the bucket name in heroku config too)
On your machine set your credentials you just created to the default in ~/.aws/credentials
On your machine set your default region that corresponds to your bucket in ~/.aws/config
Create signed URL aws s3 presign s3://your-bucket-address/your-object
Run restore heroku pg:backups:restore '<SIGNED URL>' DATABASE_URL
Had the exact same error and made these 2 adjustments. In the S3 console click on the file you want to use for the backup. You should see the name fo your file followed by 4 tabs. In the General information tab, do the following:
Click on Make public to make the file available for download.
Get the URL for that object where it says URL of object
(should be something like https://mybucket.s3.amazonaws.com/my.file, you can test if it works by pasting that url in a new Chrome tab and hitting that url. That should trigger the download of your file)
Once the previous check is working you can proceed to
heroku pg:backups restore 'https://mybucket.s3.amazonaws.com/my.file' DATABASE_URL
I ran into the same issue and discovered the issue was that I had my bucket's region set as us-east rather than us-east-1.
Have gone through different solutions available on stackoverflowand also on different forums. But none addresses the precise problem.
As per the documentation: https://devcenter.heroku.com/articles/heroku-postgres-import-export
I have the dump file created from my local database, with this command:
pg_dump -Fc --no-acl --no-owner -h localhost -U postgres dss_iaya>dss_iaya_db_dump1.dump
Then as per documentation, uploaded to a server with public access URL: https://firebasestorage.googleapis.com/v0/b/iaya-664f3.appspot.com/o/dss_iaya_db_dump1.dump?alt=media&token=06167d04-1e98-4e4b-b0e0-9d83a86dd167
Now when I try to restore on Heroku as per its documentation syntax heroku pg:backups:restore [BACKUP] [DATABASE] --app APP using following command, it returns error message when restoring.
heroku pg:backups:restore --app heroku-postgres-*** 'https://firebasestorage.***/dss_iaya_db_dump1.dump?alt=media&token=***' 'postgres://quesu***:I***#ec2-54-***.eu-west-1.compute.amazonaws.com:5432/d3n***k0'
I have used *** for security purpose only, as can not mention full credentials. But I believe one can understand the whole syntax.
When I restore same .dump file on a newly created local database it works without any issues and creates/restores the whole database with tables and data.
Just found the solution, actually two things were wrong in my case.
One, the uploaded .dump file was not well readable/usable by the heroku.
Two, the heroku postgresql DB complete URL was not required to be provided.
So, the right way that worked for me was that the uploaded file should be accessible without any token and also without any virtual/indirect path, etc. The URL to the file should point to the file directly. In my questioned problem, I was using firebase to host my DB file temporarily to do the heroku operation. And firebase was not giving direct URL to the uploaded physical file.
heroku pg:backups:restore --app heroku-postgres-f3*** 'https://www.h***.com/dss_iaya_db_dump2.dump' DATABASE_URL
After typing this command, I was asked to retype the heroku app name just to confirm the operation. Once done, everything worked like a charm.
In logs you see dump size: 0 Bytes. Also you see aborting and 403,
so if you check your file: https://llfirebasestorage.googleapis.com/vo/b/iaya-664f3.appspot.com/0/dss_iaya_db_dump1.dump?a1t=media
you get:
404. That’s an error.
The requested URL /vo/b/iaya-664f3.appspot.com/0/dss_iaya_db_dump1.dump was not found on
this server. That’s all we know.
I'm done with the pgbackups process.
First, I installed the pgbackups addon (HEROKU)
via command: heroku addons:add pgbackups
Second, I login using heroku auth:login command
Third, I capture the backup by running the command: heroku pgbackups:capture --app myapp
Fourth, I get the url of the backup by
heroku pgbackups:url b002
So I accessed the url that I saw after running the command: heroku pgbackups: url b002
in this format: https://s3.amazonaws.com/hkpgbackups/app2955630#heroku.com/b002.dump?AWSAccessKeyId=.......
something like that.
After that, I downloaded the file in .dump format
So now, I have the dump file and dont know what to do where to specifically use it...
Can somebody give me an advise to how to use the .dump file?
Please help me. I'm in trouble right now. I've to get a copy of our existing database from heroku.
Your .dump file should be a pg_dump file so you'd need to set up PostgreSQL somewhere and then use pg_restore to load the .dump file into your local database.