I am attempting to migrate my heroku database to a database I have hosted on linode (linux vm).
I am able to get heroku to recognize its own database as well as the correct username/database name of the remote db. However it is attempting to pick my localhost pg db not the remote one.
This is Heroku's description of the pg:pull command:
$ heroku help pg:pull
See more details with DEBUG=*
pull Heroku database into local or remote database
USAGE
$ heroku pg:pull SOURCE TARGET
OPTIONS
-a, --app=app (required) app to run command against
-r, --remote=remote git remote of app to use
--exclude-table-data=exclude-table-data tables for which data should be excluded (use ';' to split multiple names)
DESCRIPTION
Pull from SOURCE into TARGET.
TARGET must be one of:
* a database name (i.e. on a local PostgreSQL server) => TARGET must not exist and will be created
* a fully qualified URL to a local PostgreSQL server => TARGET must not exist and will be created
* a fully qualified URL to a remote PostgreSQL server => TARGET must exist and be empty
To delete a local database run `dropdb TARGET`
To create an empty remote database, run `createdb` with connection command-line options (run `createdb --help` for details).
Examples:
# pull Heroku DB named postgresql-swimmingly-100 into local DB mylocaldb that must not exist
$ heroku pg:pull postgresql-swimmingly-100 mylocaldb --app sushi
# pull Heroku DB named postgresql-swimmingly-100 into empty remote DB at postgres://myhost/mydb
$ heroku pg:pull postgresql-swimmingly-100 postgres://myhost/mydb --app sushi
I thought alright not bad, so I tried:
heroku pg:credentials:url
Which gives you info like:
Connection information for default credential.
Connection info string:
"dbname=[db name here] host=[aws ip here] port=5432 user=[db username] password=[db password here] sslmode=require"
Connection URL:
postgres://[db username]:[db password here]#[db ip here]:5432/[db name here]
I thought wonderful I can use that connection url above for the heroku db and follow its syntax for my own linode pg db.
heroku pg:pull postgres://[heroku db username]:[heroku db password here]#[heroku db ip here]:5432/[heroku db name here] postgres://[linode db username here]:[linode db password here]#[linode ip here]/[linode db name here] --app [my-app-name-here]
Then I get the issue with it doesn't even accept its own db uri is valid:
Unknown database:
[Stuff I put in for their db url here, like the full string]
Valid options are: DATABASE_URL
Alright this is a little upsetting because I know that the credentials above work... I have connected to my heroku database from my rails app using those credentials. Whatever maybe they just want something else for it so I look at:
$ heroku pg:info
task: runHook init
plugin: heroku
root: /usr/lib/heroku
See more details with DEBUG=*
=== DATABASE_URL
Plan: Hobby-dev
Status: Available
Connections: 1/20
PG Version: 11.6
Created: 2019-06-03 16:06 UTC
Data Size: 9.8 MB
Tables: 15
Rows: 450/10000 (In compliance)
Fork/Follow: Unsupported
Rollback: Unsupported
Continuous Protection: Off
Add-on: postgresql-[random name here]
So I go alright maybe they want that "Add-on:" name instead:
$ heroku pg:pull postgresql-[random name here] postgres://[linode db username here]:[linode db password here]#[linode ip here]/[linode db name here] --app [my-app-name-here]
This gets me really close but I am now stumped on how to get it to look at the correct ip address/host, this is what I got:
heroku-cli: Pulling postgresql-[random name here] ---> postgres://[linode db username here]:[linode db password here]#[linode ip here]:5432/[linode db name here]
psql: FATAL: no pg_hba.conf entry for host [my **LOCAL** ip address], user [linode db username here], database [linode db name here], SSL on
▸ Remote database is not empty. Please create a new database or use heroku pg:reset
I have also tried:
$ heroku pg:pull postgresql-[random name here] postgres://[linode ip here]:5432/[linode db name here] --app [my-app-name-here]
Thinking it was somehow falling back to my local one when it couldn't figure out the remote. Same error but the username is my local computers username and still my ip not the remote's.
This is so close but why did it pick my local ip address? It got the linode username/db name correct but it is trying to connect it locally which is obviously not what I wanted.
Any suggestions would be very appreciated.... Feel like I followed their commands as well as I possibly could.
On linode I have my settings changed so it is open on 0.0.0.0:5432 not just localhost. So it should be able to accept outside connections.
EDIT:
I have looked at these resources for help but found them woefully lacking in any specific details which are the crucial point to actually do this:
https://devcenter.heroku.com/articles/heroku-postgresql#pg-push-and-pg-pull
https://blog.heroku.com/push_and_pull_databases_to_and_from_heroku
EDIT:
I got the dump file..... but this means heroku's commands don't work. Yes I can do it with the dump file. However I'm upset saying to us this api command can work and it can't... anyone prove me wrong. I'll praise you if it can.
* a database name (i.e. on a local PostgreSQL server) => TARGET must not exist and will be created
* a fully qualified URL to a local PostgreSQL server => TARGET must not exist and will be created
* a fully qualified URL to a remote PostgreSQL server => TARGET must exist and be empty
To create an empty remote database, run `createdb` with connection command-line options (run `createdb --help` for details).
# pull Heroku DB named postgresql-swimmingly-100 into empty remote DB at postgres://myhost/mydb
$ heroku pg:pull postgresql-swimmingly-100 postgres://myhost/mydb --app sushi
heroku-cli: Pulling postgresql-[random name here] ---> postgres://[linode db username here]:[linode db password here]#[linode ip here]:5432/[linode db name here]
psql: FATAL: no pg_hba.conf entry for host [my **LOCAL** ip address], user [linode db username here], database [linode db name here], SSL on
▸ Remote database is not empty. Please create a new database or use heroku pg:reset
It is weird that it is listing your local ip address. However it is recognizing it as a remote database and not local.
A common problem for missing pg_hba.conf is that SSL is not enabled but it says there SSL on. Maybe that error is misleading. It later says Remote database is not empty so it is attempting to pull into but can't due to TARGET must exist and be empty
Have you ensured that the remote database server was empty and have you run createdb just in case?
createdb -h [host] -p [port] -U [dbuser] [dbname]
Related
Is it possible to run pgbench on heroku?
I see here it's a CLI argument (not a postgres command)
As of nearly 10 years ago, it wasn't a thing. But is it now? https://github.com/heroku/heroku-pg-extras/issues/7
When you connect to your Heroku database using heroku pg:psql you actually use a psql running locally on your machine to connect to the remote server.
Similarly, you should be able to use a local pgbench even though there isn't a Heroku CLI wrapper for it. Retrieve your database connection information using heroku config:get DATABASE_URL or heroku pg:credentials, then run the command with the common connection options:
pgbench -h HOST -p 5432 -U USER DATABASE_NAME
Since it is a standard URI, you can pull the hostname, username, password, and port out of Heroku's database URL like so:
postgres://user:password#host:port/database
I have a live app running with a database on Heroku but the database needs an update so my plan is to pull the database, update it, test it on my development server, then push it to production. With the first step I am struggling with pulling the heroku db to overwrite my local db. I am using the following command:
PGUSER=userforlocaldb PGPASSWORD=passwordforlocaldb heroku pg:pull HEROKU_POSTGRESQL_herokudbcolor localdbname --app herokuappname
I get this error message:
! createdb: database creation failed: ERROR: permission denied to create database
!
! Unable to create new local database. Ensure your local Postgres is working and try again.
I have also tried filling the PGUSER and PGPASSWORD values with my heroku db but that did not work.
What am I missing here?
EDIT: I changed the password for superuser postgres and re-ran the command. I get this error: Peer authentication failed for user "postgres"
I am able to reproduce your issue, but I am also able to correct it with Gabriel's suggestion to change the access method for postgres from peer to md5. You have to remember to restart Postgres whenever you change pg_hba.conf. On Ubuntu, this should do it:
sudo service postgresql restart
Another solution:
Alternatively, if user permissions are not a huge concern for you (e.g., you have no other users on your system), such as in a VM, you can give yourself PostgreSQL-superuser privileges:
sudo -u postgres createuser -s $YOUR_USERNAME [-P]
This will then allow you to create a new db (localdbname) as well as load the plpgsql extension which seems to be necessary in this case. (It appears to be insufficient to have the CREATEDB role because of plpgsql.)
Now you should be able to run your original command without PGUSER or PGPASSWORD:
heroku pg:pull HEROKU_POSTGRESQL_herokudbcolor localdbname --app herokuappname
EDITED ANSWER:
If you are still being prompted for a password when running the heroku pg:pull command, you will want to set a password for the new Postgres user, which may be the same as your OS login user (echo \whoami``). You can do this by including the -P flag in the createuser command above, which will prompt you for a password for the new user, or by logging in as a Postgres superuser (e.g., postgres) and setting the password in psql.
ADDENDUM/CAVEAT:
It may not be a good idea to combine the two solutions because you may have issues running sudo -u postgres ... if the postgres user is authenticated by md5 but no password is set, which is usually the case on Linux installations by default. Personally I find it more convenient to issue commands as myself (whoami) as a Postgres superuser than as the postgres superuser, once the new user/role is created.
Either way, to run the heroku pg:pull command, I think you're going to need to set a password the user and also have superuser privileges.
Im new to Postgres and to Heroku. I am trying to pull the database from Heroku but I'm missing something simple. I did:
heroku pg:pull HEROKU_POSTGRESQL_IVORY_URL localdb
And I got the error:
createdb: database creation failed: ERROR: permission denied to create database
Then I tried the same with "sudo". and I got:
createdb: could not connect to database template1: FATAL: role "root" does not exist
So, it must be I'm missing some simple commands I can't find. Im on Linux, I have Postgres installed and working.
createdb is a wrapper around the SQL statement CREATE DATABASE and as such it needs to connect to the database.
By default all Postgres commandline tools try to connect to the database using the current operating system user. As the error message indicates there is not user named root in the database. So you need to pass the name of the Postgres superuser in order for createdb to be able to connect. This user is usually named postgres.
Another option is to switch the Linux user to to postgres if such a Linux user exists.
I don't know Heroku and I don't know how you started createdb, but the parameter to pass a username is -U (for all Postgres command line programs). So you'd need
createdb -U postgres name_of_new_database
Try to use:
sudo su - postgres
Then createdb using:
createdb name_of_db;
When it comes to heroku pg:pull specifically, you can specify the PostgreSQL user and password by specifying the PGUSER and PGPASSWORD environment variables specifically:
PGUSER=postgres PGPASSWORD=password heroku pg:pull HEROKU_POSTGRESQL_MAGENTA mylocaldb --app sushi
If you want to specify the host, for example 127.0.0.1, this is how:
heroku pg:pull HEROKU_POSTGRESQL_MAGENTA postgres://127.0.0.1/mylocaldb --app sushi
See documentation for heroku pg:pull
In some cases you need to execute createdb as the postgres user.
Run createdb (in-line) as postgres user:
sudo -u postgres createdb name_of_database
The command suggested by Ezrqn Kemboi, sudo su - postgres, switches you to the user postgres at which point you would need to exit out of that user to return to whomever you were before hand. This isn't great if you're running this in a script or some form of automation.
The command I suggested will not "switch" to the user and instead will run the command "in-place". My suggestion is more suitable for running in scripts.
Pick whichever you feel is better for your use case.
I have a postgresql demon on my virtual Linux box, and just installed pgAdmin III on the Windows machine. Now I'd like to configure the first connection.
What I did so far:
I edited /etc/postgresql/9.1/main/pg_hba.conf and added a line to make the demon accept connections from my Windows machine:
host all all 192.168.123.45/32 md5
I set a shell passphrase for the system user postgres (sudo passwd postgres)
Edit: this seems to be irrelevant
I copied /usr/share/postgresql/9.1/pg_service.conf.sample to /etc/postgresql-common/pg_service.conf and uncommented the sample service, giving me
[postgres]
dbname=postgres
user=postgres
Edit: this seems to be irrelevant
I put a line into /etc/environment to point to the pgservice.conf file:
PGSERVICEFILE=/etc/postgresql-common/pg_service.conf
(which works; when I duplicated the PuTTY session, it had this variable)
I restarted the database server after each change:
sudo /etc/init.d/postgresql restart
Finally, I restartet the Linux box,
The server seems to have restarted successfully:
[ ok ] Restarting PostgreSQL 9.1 database server: main.
However, when I try to connect to the service, pgAdmin III gives me an error message:
Error connecting to the server: definition of service "postgres" not found.
dmesg | grep postgres gives me a single line:
[ 18.054965] postgres (2242): /proc/2242/oom_adj is deprecated, please use /proc/2242/oom_score_adj instead.
I have no clue whether postgresql uses my pq_service.conf at all, or whatever else is the problem; any help is appreciated ...
Edit:
My original question, "How can I know whether postgresql uses my pg_service.conf file?", seems to be answered - it simply doesn't. I still can't connect; but now the question doesn't match the error messages anymore.
I removed the "Service" entry from my "New Server Registration" data. Now I get another error - something like "password authentication for user >>postgres<< failed". I'm quite sure the password is correct - I just set it ... and I tested it by commenting out my ssh key from the authorized_keys file.
I'd happily connect with my ssh key, but this seems to be difficult as well. With PuTTY, my key is taken from Pageant, and I'm logged in without any problem; pgAdmin talks about "SSH tunnelling", but I normally don't need tunnels for this local machine ...
I tried to create a tunnel anyway.
PuTTY session:
Source port is 5432
destination is my-vbox.host.name:5432
In pgAdmin, "SSH Tunnel":
Username postgres
using Identity file
Tunnel host localhost → error:
"SSH error: Could not connect to socket with error code 10051"
Tunnel host localhost:5432 → error:
"SSH error: Unable to resolve host: localhost:5432"
Tunnel host 127.0.0.1:5432 → error:
"SSH error: Unable to resolve host: 127.0.0.1:5432"
How the heck is this supposed to be configured?!
I finally managed to connect as db user tobias (I never had problems to connect locally, with psql).
# sudo -u postgres psql postgres
psql (9.1.9)
Type "help" for help.
postgres=# \du
List of roles
Role name | Attributes | Member of
-----------+------------------------------------------------+-----------
postgres | Superuser, Create role, Create DB, Replication | {}
tobias | Superuser | {}
postgres=# ALTER USER tobias WITH PASSWORD 's3cr3t';
ALTER ROLE
After doing this, I could connect as tobias, using the password just set.
The "Maintenance DB" happens to be template1; I didn't try to change this.
For further reference - I activated log_connections in /etc/postgresql/9.1/main/postgresql.conf and watched the log:
sudo tail -f /var/log/postgresql/postgresql-9.1-main.log
I want to create a PostgreSQL database on a Heroku server.
My database.yml is
production:
adapter: postgresql
encoding: utf8
database: ddb
username: postgres
port: 5432
password: admin
host: localhost
When I run heroku rake db:create it gives me the error:
mydatabase already exists
(in /disk1/home/slugs/181380_8d7032f_f439-4fe4f5a6-f181-4150-a968-fadcf45f0af5/mnt)
I have tried it for various different database names, but get the same message.
How can I resolve this error?
heroku autogenerates database.yml on the deploy. Reads like: it doesn't matter what you put in your database.yml, which to me is nice, as I can include it in git without worrying about production db passwords.
You may choose another database by setting the ENV['DATABASE_URL'] (use heroku config:add DATABASE_URL=....)
Check heroku documentation
https://devcenter.heroku.com/articles/ruby-support#build-behavior