Just strange, I am doing back database maintenance, I tried to to a heroku pg:pull <DB_URL> local_db --app <appname> and it says:
pg:pull is not a heroku command.
I tried to create a backup with heroku pg:backups:capture <DB_URL> and i get this:
pg:backups:capture is not a heroku command.
Just trying heroku pg:backups shows:
pg:backups is not a heroku command.
Its strange because even in the --help section, looks like all the useful commands are gone:
heroku pg --help
Usage: heroku pg [DATABASE]
-a, --app APP # app to run command against
-r, --remote REMOTE # git remote of app to run command against
Commands for pg, type "heroku help pg:COMMAND" for more details:
heroku pg [DATABASE] #
heroku pg:info [DATABASE] #
heroku pg:wait [DATABASE] #
I looked around online and didn't see anything about these pg command deprications. I checked my heroku CLI version: heroku-cli/5.3.0-5329fb8 (darwin-amd64) go1.6.2
Strange becuase I've been using these commands on this same computer many times recently and suddenly they're gone.
Have these pg been depricated (I doubt)? If not does anyone know what I am doing wrong? If so, anyone know how to do push/pull/capture in the new framework?
The answer was that my CLI version was out-of-date. I followed these instructions on my Mac:
rm -rf /usr/local/heroku
rm -rf ~/.local/share/heroku ~/.config/heroku ~/.cache/heroku
And then re-installed using homebrew:
brew update
brew install heroku
Apparently the new version of the CLI is supposed to keep itself up-to-date.
Related
I had planned an upgrade of artifactory from 6.7.5 to 6.8.1. As part of the upgrade I checked jfrog's repo on github and it looks like they have a new recommended nginx and postgres version.
The current docker-compose is using postgres 9.5 and the new default version if 9.6. Simply pulling down postgres 9.6 however does not do an inplace upgrade.
FATAL: database files are incompatible with server DETAIL: The data
directory was initialized by PostgreSQL version 9.5, which is not
compatible with this version 9.6.11.
The upgrade instructions do not mention anything about how to do the upgrade.
The examples provided in github (https://github.com/jfrog/artifactory-docker-examples) are just examples.
Using them in production could cause issues and backwards compatibility is not guaranteed.
To get over the PostgreSQL matter when upgrading, I would suggest:
$ docker-compose -f yml-file-name.yml stop
edit the yml-file-name.yml and change the docker.bintray.io/postgres:9.6.11 to docker.bintray.io/postgres:9.5.2
$ docker-compose -f yml-file-name.yml up -d
Artifactory should be upgraded after following this, however it will keep using the previous version of the PostgreSQL DB
I have been able to upgrade database using following approach:
Dump all database to an SQL script using old database image; store it in a volume for future import:
# Override PostgreSQL image used to export using old binaries
printf "version: '2.1'\nservices:\n postgresql:\n image: docker.bintray.io/postgres:9.5.2\n" > image_override.yml
started_container=$(docker-compose -f artifactory-pro.yml -f image_override.yml run -d -v sql_dump_volume:/tmp/dump --no-deps postgresql)
# Dump database to a text file in a volume (to make it available for import)
docker exec "${started_container}" bash -c "until pg_isready -q; do sleep 1; done"
docker exec "${started_container}" bash -c "pg_dumpall --clean --if-exists --username=\${POSTGRES_USER} > /tmp/dump/dump.sql"
docker stop "${started_container}"
docker rm --force "${started_container}"
Back up old database directory and prepare a new one:
mv -fv /data/postgresql /data/postgresql.old
mkdir -p /data/postgresql
chown --reference=/data/postgresql.old /data/postgresql
chmod --reference=/data/postgresql.old /data/postgresql
Run a new database image with mounting dump script from step 1. It processes SQL scripts upon startup when setting up a new database, provided it's started as postgres something. We just don't need to leave the server running afterwards, so I provided --version to make entrypoint execute, import the data and quit:
docker-compose -f artifactory-pro.yml run --rm --no-deps -e POSTGRES_DB=postgres -e POSTGRES_USER=root -v sql_dump_volume:/docker-entrypoint-initdb.d postgresql postgres --version
After all this is done, I was able to start Artifactory normally with docker-compose -f artifactory-pro.yml up -d and it started up normally, applying rest of schema and file upgrade procedure as usual.
I have also prepared a script that basically does the above steps along with some additional checks and cleanup. Feel free to use if you find it useful.
I currently have a production PostgreSQL database running on a separate Heroku dyno.
I'd like to deploy a simple shell script on a different dyno (for a number of reasons), that among other things, will reset the database every 2 weeks.
I've tried using both the heroku and psql commands to connect to my database in my script, but unfortunately I get the following errors:
$ heroku pg:psql DATABASE_URL
bash: heroku: command not found
Which makes sense because the Heroku toolbelt isn't installed on a heroku dyno.
And then:
$ psql -h DATABASE_HOST -p
Error: You must install at least one postgresql-client-<version> package
Unfortunately, you can't do a simple apt-get install postgresql-client because Heroku prevents you from being able to install packages. (You'll get the following error):
~ $ apt-get install postgresql-client
W: Not using locking for read only lock file /var/lib/dpkg/lock
E: Unable to write to /var/cache/apt/
E: The package lists or status file could not be parsed or opened.
Is there any workaround for this? I know I could write a python script and use a pip module to connect to the database, but I'd rather this be a shell script to build off an existing one.
Thanks.
I need to transfer db from app_1 to app_2
I created backup on app_1
Then ran:
heroku pg:backups restore HEROKU_POSTGRESQL_COLOR --app app_2 heroku pgbackups:url --app app_1
HEROKU_POSTGRESQL_COLOR = database URL for app_2
Then I get:
! `pg:backups` is not a heroku command.
! Perhaps you meant `pgbackups`.
! See `heroku help` for a list of available commands.
So I ran:
heroku pgbackups:restore HEROKU_POSTGRESQL_COLOR --app app_2 heroku pgbackups:url --app app_1
Then I get the following:
! WARNING: Destructive Action
! This command will affect the app: app_2
! To proceed, type "app_2" or re-run this command with --confirm app_2
So I confirmed with:
> app_2
! Please add the pgbackups addon first via:
! heroku addons:add pgbackups
So then I ran: heroku addons:add pgbackups --app app_2
Adding pgbackups on app_2... failed
! Add-on plan not found.
Is there a way around this issue? any help would be greatly appreciated!
* Solution *
I ended up emailing Heroku, they advised that I need to heroku update; heroku plugins:update but heroku update is only available to heroku toolbelt only and I had the gem installed.
Solution:
Install Heroku toolbelt here
Then uninstall the gem:
gem uninstall heroku --all
run the following to get the version and it should output heroku-toolbelt, instead of the gem, more info here
$ heroku --version
heroku-toolbelt/2.39.0 (x86_64-darwin10.8.0) ruby/1.9.3
To copy the databases over:
heroku pg:backups restore `heroku pgbackups:url --app app_1` HEROKU_POSTGRESQL_COLOR --app app_2
But even better—you can copy directly from one database to another without needing the backup:
Assuming app_2 database url is: HEROKU_POSTGRESQL_GOLD
heroku pg:copy app_1::DATABASE_URL GOLD -a app_2
That will copy the main database from app_1 to the GOLd database on app_2
its only 1 command to copy database from app to app now you don't have to backup:
heroku pg:copy app_name_to_copy_from::database_color_to_copy_from database_color_to_copy_to --app app_name_to_copy_to
check it here
If you look at heroku docs it says
PG Backups as an add-on has been deprecated. The commands exist as part of the Heroku Postgres namespace in the CLI. The new functionality is live and available for use.
So you can use the pgbackups functionality directly without having to add any add-ons
To create a backup you can run
heroku pg:backups capture --app app_name
if you have multiple databases then you can specify database url like this
heroku pg:backups capture HEROKU_POSTGRESQL_PINK
To restore from a backup on another app you can run
heroku pg:backups restore b001 DATABASE_URL --app app_name
You can transfer database by
heroku pg:copy DATABASE_URL HEROKU_POSTGRESQL_PINK_URL --app app_name
You can also upload your database to a public url and then use that url to import database on another app by
heroku pg:backups public-url b001 --app app_name
and then import it by
heroku pg:backups restore 'https://s3.amazonaws.com/me/items/3H0q/mydb.dump' DATABASE -a app_name
If you are moving from one app to another and want to use same database for another app then you can follow these steps:
Login to your heroku account
Select your old app and go to settings tab
Reveal config vars for your old app
Copy DATABASE_URL
Go back and select your new app
Replace new apps DATABASE_URL with the old apps value
I needed something slightly different so sharing it here:
heroku pg:copy name_of_app_being_copied::DATABASE_URL DATABASE_URL --app name_of_app_being_copied_to
Note: Both references to DATABASE_URL do not need to be changed. It may seem odd, but the first instance references the database url for the app being copied and the second one references the database url of the app being copied to.
heroku pg:copy app1_name::HEROKU_POSTGRESQL_ONYX_URL HEROKU_POSTGRESQL_AQUA_URL --app app2_name
Where the second db url is on app2_name
There are simple ways to do this, and there is a fast way to do it
The simple ways generally involve using a backup/restore methodology (including pg:copy, which is a backup that streams the data directly to a pg_restore process), but these are slow in creating the new database because you are restoring a logical definition of tables, loading the data, and creating indexes on the data.
That's a lot of work, and for my 30GB standard-2 databases it can literally take hours.
The fast way to do it is to provision a follower of the database to be copied (e.g. production) on the application you want the data on (e.g. test). On the same 30GB databases that take hours to do a restore, this last took about 15 minutes.
The methodology I use is:
# Get the name of the source database addon (e.g. postgresql-clean-12345)
heroku pg:info -a production-app
# Create a follower on the destination app (choose your own plan)
# You create the follower on the new app because otherwise it is
# perpetually associated with the source as it's "billing app"
heroku addons:create heroku-postgresql:standard-2 --follow postgresql-clean-12345 -a test-app
heroku pg:wait -a test-app
# Quiesce the destination app
heroku scale web=0 worker=0 -a test-app
heroku maintenance:on -a test-app
# Get the colour of the new database (e.g. HEROKU_POSTGRESQL_GRAY_URL)
heroku pg:info -a test-app
# Unfollow the source database.
# If you want to upgrade the database, do that now instead of the
# unfollow.
heroku pg:unfollow HEROKU_POSTGRESQL_GRAY_URL -a test-app
# Promote the new database on the destination app
heroku pg:promote HEROKU_POSTGRESQL_GRAY_URL -a test-app
# Get the colour of the old database, if any(e.g. HEROKU_POSTGRESQL_MAROON_URL)
heroku pg:info -a test-app
# Destroy the old database (if any)
heroku addons:destroy HEROKU_POSTGRESQL_MAROON_URL -a test-app
# Bring the test app back up
heroku scale web=1 worker=1 -a test-app
heroku maintenance:off -a test-app
Why is this faster?
I believe that when creating a follower, Heroku creates the new database by either copying data files or restoring from a physical (file) backup, then replaying the logs to bring it up to date.
I found the simpler solution to reuse/share the same resource (postgres database in this case - or any others that allow sharing/reuse) with more than one app on heroku is doing the following:
Go to the older (source) app dashboard on Heroku
Select the "Resources" tab
Locate the resource (postgres database, in our case here)
Click on the icon next to the plan name at the right most part of the row listing the resource
Select the "Attach to another app" option and select the newer (target) app name from the list that shows up
Sample of the extended menu mentioned # step #4 above!
That's all it takes to share the resource between the apps as it automatically updates all the related configuration settings on the target app. This comes handy since none of the add-on related configuration variables are directly editable, at least from the dashboard (have not checked through the CLI). Hope this helps anyone looking for similar thing.
As per the website the addon is depreciated. So that could be the reason for the failure message.
Backups as an add-on has been deprecated.
Since your aim is to move the db from one app to another, why don't you try the instructions mentioned in the link below.
https://devcenter.heroku.com/articles/heroku-postgres-backups#direct-database-to-database-copies
heroku addons:create heroku-postgresql:standard-0 --fork postgresql-translucent-12345 --app target-app
Where postgresql-translucent-12345 is the addon name of the database you want to fork from. Note that the color url does not work across different apps but the addon name does.
You can also add the "--fast" option:
heroku addons:create heroku-postgresql:standard-0 --fork postgresql-translucent-12345 --fast --app target-app
Which creates a fork of the database that could be up to 30 hours old. Even without the fast option, I copied a 40GB database in 10 minutes-ish.
https://devcenter.heroku.com/articles/heroku-postgres-fork
Then you have to, of course, promote the database. See David Aldridge's answer -and my comments- above for some further details about the process.
I had a related issue. You can save a backup to your local machine, then upload it to some hosting, like amazon s3, and import from given url.
This question and following answer may help you: Can't import to heroku postgres database from dump
You may find useful pgAdmin III (http://pgadmin.org/), a free db management tool specifically designed to do these types of tasks. You can edit/view/import/export from/to your Heroku db directly. Let me know if you need help in setup. (It's like MySQL Workbench, but for PostgreSQL).
I've been trying to pull a database from Heroku. For some reason when I try
heroku pg:pull HEROKU_POSTGRESQL_RAINBOW_URL newlocaldb --app myapp
it returns
! pg:pull is not a heroku command. ! Perhaps you meant
db:pull, pg:kill or pg:psql. ! See heroku help for a list
of available commands.
On the other hand, the command
heroku db:pull HEROKU_POSTGRESQL_RAINBOW_URL newlocaldb --app myapp
is recognized, despite that db:pull was replaced by pg:pull according to e.g. this.
When I try heroku help, it shows an entry for db:pull but not for pg:pull.
So, it would be nice to use the command pg:pull. But it doesn't work for me just now. (I believe it did fairly recently.) What could be the problem?
I'm using heroku and I want to download the database from my app(heroku) so I can make some changes in it, I've installed pgbackups, but using heroku pgbackups:url downloads a .dump file
How can I download a postgresql file or translate that .dump into a postgresql file?
If you're using Heroku's pgbackups (which you probably should be using):
$ heroku pg:backups capture
$ curl -o latest.dump `heroku pg:backups public-url`
"Translate" it into a postgres db with
$ pg_restore --verbose --clean --no-acl --no-owner -h localhost -U myuser -d mydb latest.dump
See https://devcenter.heroku.com/articles/heroku-postgres-import-export
There's a command for this in the CLI - heroku db:pull which will do this for you. db:pull can be a bit slow mind you so you may be better to use the next option.
If you are using complex postgress data types (hstore, arrays etc) then you need to use the pgtransfer plugin https://github.com/ddollar/heroku-pg-transfer which will basically does a backup on Heroku and a restores it locally.
UPDATE: db:pull and db:push have been deprecated and should be replaced with pg:pull and pg:push - read more at https://devcenter.heroku.com/articles/heroku-postgresql#pg-push-and-pg-pull
I found the first method suggested in the documentation pull/push even easier. No password or username needed.
pg:pull
pg:pull can be used to pull remote data from a Heroku Postgres
database to a database on your local machine. The command looks like
this:
$ heroku pg:pull HEROKU_POSTGRESQL_MAGENTA mylocaldb --app sushi
This command will create a new local database named “mylocaldb” and
then pull data from database at DATABASE_URL from the app “sushi”. In
order to prevent accidental data overwrites and loss, the local
database must not exist. You will be prompted to drop an already
existing local database before proceeding.
At first I had an error: /bin/sh: createdb: command not found; which I solved following this SO post.
An alternative described also in the documentation (I did not try it yet) is:
To export the data from your Heroku Postgres database, create a new
backup and download it.
$ heroku pg:backups:capture
$ heroku pg:backups:download
Source: Importing and Exporting Heroku Postgres Databases with PG Backups
To export the data from Heroku Postgres database, just follow below steps
Login to heroku
Go to APP->settings->reveal config variable
Copy DATABASE_URL
run pg_dump --DATABASE_URL_COPIED_IN_STEP_3 > database_dump_file
Note this will provide postgresql file or for dump file you can download directly from postgres addon interface.
I think the easiest way to download and replicate the database on local server:
**PGUSER**=LOCAL_USER_NAME PGPASSWORD=LOCAL_PASSWORD heroku pg:pull --app APP_NAME HEROKU_POSTGRESQL_DB_NAME LOCAL_DB_NAME
Go through this document for more info:
https://devcenter.heroku.com/articles/heroku-postgresql#pg-push-and-pg-pull
This is the script that I like to use.
namespace :heroku do
desc "Import most recent database dump"
task :import_from_prod => :environment do
puts 'heroku run pg:backups capture --app APPNAME'
restore_backup 'APPNAME'
end
def path_to_heroku
['/usr/local/heroku/bin/heroku', '/usr/local/bin/heroku'].detect {|path| File.exists?(path)}
end
def heroku(command, site)
`GEM_HOME='' BUNDLE_GEMFILE='' GEM_PATH='' RUBYOPT='' #{path_to_heroku} #{command} -a #{site}`
end
def restore_backup(site = 'APPNAME')
dump_file = "#{Rails.root}/tmp/postgres.dump"
unless File.exists?(dump_file)
pgbackups_url = heroku('pg:backups public-url -q', site).chomp
puts "curl -o #{dump_file} #{pgbackups_url}"
system "curl -o #{dump_file} '#{pgbackups_url}'"
end
database_config = YAML.load(File.open("#{Rails.root}/config/database.yml")).with_indifferent_access
dev_db = database_config[Rails.env]
system "pg_restore -d #{dev_db[:database]} -c #{dump_file}".gsub(/\s+/,' ')
puts
puts "'rm #{dump_file}' to redownload postgres dump."
puts "Done!"
end
end