I am creating a local Label Studio server to host images to annotate in our office. I would like the database back end to be postgresql and not sqlite and be located in a particular directory, not the default and not the same as the 'data-dir'. I have got a test server working across the network with various machines annotating images on the server, but the backend was sqlite for this test.
Everything I've tried to get a postgresql backend db has failed for various reasons. Some commands result in a sqlite db (occasionally with the name 'postgresql') located in my required directory; others create postgres/pyscopg2 errors but I think they're up a garden path.
The host machine is running Ubuntu 20.04 LTS. And serves another postgresql db over the network using other APIs. Postgresql version running is 12.9.
I have created a conda environment and pip installed Label Studio as the documentation suggested.
Here's what I've tried:
Start the conda environment. Follow instructions to assign environment variables from https://labelstud.io/guide/storedata.html#PostgreSQL-database which at time of writing is:
DJANGO_DB=default
POSTGRE_NAME=postgres
POSTGRE_USER=postgres
POSTGRE_PASSWORD=
POSTGRE_PORT=5432
POSTGRE_HOST=db
Then a few variations on the start command (I didn't include the backslashes, just put here for readability/comparability):
label-studio start --init \
-db postgresql \
--database /path/to/label-studio/databases/newdb \
--data-dir /path/to/label-studio/media_dirs/test_proj
result: db is where expected, but:
file newdb
gives "newdb: SQLite 3.x database, last written using SQLite version 3038002"
label-studio start --init \
--database /path/to/label-studio/databases/newdb \
-db postgresql \
--data-dir /path/to/label-studio/media_dirs/test_proj
result: a db at specified path named 'postgresql' and still an sqlite db. This seems to mirror the mistake mentioned at: https://github.com/heartexlabs/label-studio/issues/1660
I have also tried the above two commands with the '--init' argument omitted with same results.
Then I tried adding something on the front of the command suggested at the same link above:
DJANGO_DB=default label-studio start \
--database /path/to/label-studio/databases/newdb \
--data-dir /path/to/label-studio/media_dirs/test_proj
result: psycopg2.OperationalError: FATAL: password authentication failed for user "postgres"
FATAL: password authentication failed for user "postgres"
DJANGO_DB=default POSTGRE_PASSWORD= label-studio start \
--database /path/to/label-studio/databases/newdb \
--data-dir /path/to/label-studio/media_dirs/test_proj
result: psycopg2.OperationalError: fe_sendauth: no password supplied
Any help and resolution would be highly appreciated.
Also, I can't tag this with 'label-studio' because I'm not quite at the required reputation to create a new tag, so if anyone who can feels like doing so, pleaseandthankyou!
Your last option was closer than all the others. Have you tried to run LS using this:
DJANGO_DB=default POSTGRE_NAME=<postgres_name> POSTGRE_USER=<postgres_user> POSTGRE_PASSWORD=<password> POSTGRE_PORT=<db_port> POSTGRE_HOST=<db_host> label-studio
Sure, you have to run postgres service by yourself, configure it properly, create the DB <postgres_name>, the user <postgres_user> and set the password <password>, grant access rights to this user. Also don't forget to specify <db_host> (localhost?), <db_port> (5432?)
I need to transfer db from app_1 to app_2
I created backup on app_1
Then ran:
heroku pg:backups restore HEROKU_POSTGRESQL_COLOR --app app_2 heroku pgbackups:url --app app_1
HEROKU_POSTGRESQL_COLOR = database URL for app_2
Then I get:
! `pg:backups` is not a heroku command.
! Perhaps you meant `pgbackups`.
! See `heroku help` for a list of available commands.
So I ran:
heroku pgbackups:restore HEROKU_POSTGRESQL_COLOR --app app_2 heroku pgbackups:url --app app_1
Then I get the following:
! WARNING: Destructive Action
! This command will affect the app: app_2
! To proceed, type "app_2" or re-run this command with --confirm app_2
So I confirmed with:
> app_2
! Please add the pgbackups addon first via:
! heroku addons:add pgbackups
So then I ran: heroku addons:add pgbackups --app app_2
Adding pgbackups on app_2... failed
! Add-on plan not found.
Is there a way around this issue? any help would be greatly appreciated!
* Solution *
I ended up emailing Heroku, they advised that I need to heroku update; heroku plugins:update but heroku update is only available to heroku toolbelt only and I had the gem installed.
Solution:
Install Heroku toolbelt here
Then uninstall the gem:
gem uninstall heroku --all
run the following to get the version and it should output heroku-toolbelt, instead of the gem, more info here
$ heroku --version
heroku-toolbelt/2.39.0 (x86_64-darwin10.8.0) ruby/1.9.3
To copy the databases over:
heroku pg:backups restore `heroku pgbackups:url --app app_1` HEROKU_POSTGRESQL_COLOR --app app_2
But even better—you can copy directly from one database to another without needing the backup:
Assuming app_2 database url is: HEROKU_POSTGRESQL_GOLD
heroku pg:copy app_1::DATABASE_URL GOLD -a app_2
That will copy the main database from app_1 to the GOLd database on app_2
its only 1 command to copy database from app to app now you don't have to backup:
heroku pg:copy app_name_to_copy_from::database_color_to_copy_from database_color_to_copy_to --app app_name_to_copy_to
check it here
If you look at heroku docs it says
PG Backups as an add-on has been deprecated. The commands exist as part of the Heroku Postgres namespace in the CLI. The new functionality is live and available for use.
So you can use the pgbackups functionality directly without having to add any add-ons
To create a backup you can run
heroku pg:backups capture --app app_name
if you have multiple databases then you can specify database url like this
heroku pg:backups capture HEROKU_POSTGRESQL_PINK
To restore from a backup on another app you can run
heroku pg:backups restore b001 DATABASE_URL --app app_name
You can transfer database by
heroku pg:copy DATABASE_URL HEROKU_POSTGRESQL_PINK_URL --app app_name
You can also upload your database to a public url and then use that url to import database on another app by
heroku pg:backups public-url b001 --app app_name
and then import it by
heroku pg:backups restore 'https://s3.amazonaws.com/me/items/3H0q/mydb.dump' DATABASE -a app_name
If you are moving from one app to another and want to use same database for another app then you can follow these steps:
Login to your heroku account
Select your old app and go to settings tab
Reveal config vars for your old app
Copy DATABASE_URL
Go back and select your new app
Replace new apps DATABASE_URL with the old apps value
I needed something slightly different so sharing it here:
heroku pg:copy name_of_app_being_copied::DATABASE_URL DATABASE_URL --app name_of_app_being_copied_to
Note: Both references to DATABASE_URL do not need to be changed. It may seem odd, but the first instance references the database url for the app being copied and the second one references the database url of the app being copied to.
heroku pg:copy app1_name::HEROKU_POSTGRESQL_ONYX_URL HEROKU_POSTGRESQL_AQUA_URL --app app2_name
Where the second db url is on app2_name
There are simple ways to do this, and there is a fast way to do it
The simple ways generally involve using a backup/restore methodology (including pg:copy, which is a backup that streams the data directly to a pg_restore process), but these are slow in creating the new database because you are restoring a logical definition of tables, loading the data, and creating indexes on the data.
That's a lot of work, and for my 30GB standard-2 databases it can literally take hours.
The fast way to do it is to provision a follower of the database to be copied (e.g. production) on the application you want the data on (e.g. test). On the same 30GB databases that take hours to do a restore, this last took about 15 minutes.
The methodology I use is:
# Get the name of the source database addon (e.g. postgresql-clean-12345)
heroku pg:info -a production-app
# Create a follower on the destination app (choose your own plan)
# You create the follower on the new app because otherwise it is
# perpetually associated with the source as it's "billing app"
heroku addons:create heroku-postgresql:standard-2 --follow postgresql-clean-12345 -a test-app
heroku pg:wait -a test-app
# Quiesce the destination app
heroku scale web=0 worker=0 -a test-app
heroku maintenance:on -a test-app
# Get the colour of the new database (e.g. HEROKU_POSTGRESQL_GRAY_URL)
heroku pg:info -a test-app
# Unfollow the source database.
# If you want to upgrade the database, do that now instead of the
# unfollow.
heroku pg:unfollow HEROKU_POSTGRESQL_GRAY_URL -a test-app
# Promote the new database on the destination app
heroku pg:promote HEROKU_POSTGRESQL_GRAY_URL -a test-app
# Get the colour of the old database, if any(e.g. HEROKU_POSTGRESQL_MAROON_URL)
heroku pg:info -a test-app
# Destroy the old database (if any)
heroku addons:destroy HEROKU_POSTGRESQL_MAROON_URL -a test-app
# Bring the test app back up
heroku scale web=1 worker=1 -a test-app
heroku maintenance:off -a test-app
Why is this faster?
I believe that when creating a follower, Heroku creates the new database by either copying data files or restoring from a physical (file) backup, then replaying the logs to bring it up to date.
I found the simpler solution to reuse/share the same resource (postgres database in this case - or any others that allow sharing/reuse) with more than one app on heroku is doing the following:
Go to the older (source) app dashboard on Heroku
Select the "Resources" tab
Locate the resource (postgres database, in our case here)
Click on the icon next to the plan name at the right most part of the row listing the resource
Select the "Attach to another app" option and select the newer (target) app name from the list that shows up
Sample of the extended menu mentioned # step #4 above!
That's all it takes to share the resource between the apps as it automatically updates all the related configuration settings on the target app. This comes handy since none of the add-on related configuration variables are directly editable, at least from the dashboard (have not checked through the CLI). Hope this helps anyone looking for similar thing.
As per the website the addon is depreciated. So that could be the reason for the failure message.
Backups as an add-on has been deprecated.
Since your aim is to move the db from one app to another, why don't you try the instructions mentioned in the link below.
https://devcenter.heroku.com/articles/heroku-postgres-backups#direct-database-to-database-copies
heroku addons:create heroku-postgresql:standard-0 --fork postgresql-translucent-12345 --app target-app
Where postgresql-translucent-12345 is the addon name of the database you want to fork from. Note that the color url does not work across different apps but the addon name does.
You can also add the "--fast" option:
heroku addons:create heroku-postgresql:standard-0 --fork postgresql-translucent-12345 --fast --app target-app
Which creates a fork of the database that could be up to 30 hours old. Even without the fast option, I copied a 40GB database in 10 minutes-ish.
https://devcenter.heroku.com/articles/heroku-postgres-fork
Then you have to, of course, promote the database. See David Aldridge's answer -and my comments- above for some further details about the process.
I had a related issue. You can save a backup to your local machine, then upload it to some hosting, like amazon s3, and import from given url.
This question and following answer may help you: Can't import to heroku postgres database from dump
You may find useful pgAdmin III (http://pgadmin.org/), a free db management tool specifically designed to do these types of tasks. You can edit/view/import/export from/to your Heroku db directly. Let me know if you need help in setup. (It's like MySQL Workbench, but for PostgreSQL).
Is there a way to get console access to Dokku's PostgreSQL plugin? On Heroku I'd do heroku pg:psql. Is this possible in a Dokku environment and if so how?
There is in fact a way to do this directly with the dokku-pg-plugin.
The command postgresql:restore <db> < dump_file.sql connects to the specified database and restores it with the provided dump file. If you simply omit the dump file part (< dump_file.sql), a psql console session opens.
Since postgresql:restore <db> is semantically not the best way to open a console session, I have opened a pull request to add the command postgresql:console <db>.
So, until my PR is merged, the options for opening a psql console for a database are either:
doing it manually with psql -h 172.17.42.1 -p <port> -U root db with the <port> and password taken from the output of dokku postgresql:info <db>,
using the semantically incorrect command dokku postgresql:restore <db>, or
use my forked and patched version of the plugin which adds the command postgresql:console <db>.
Edit:
The owner of the dokku-pg-plugin has merged my pull request. If you're using this plugin and are looking for a way to access your PostgreSQL console with it, you might want to update it to the latest version. Once you have done that, you can use the command postgresql:console <db> to open a psql session to the specified database.
This worked for me for my Rails app that I'm running on Dokku:
dokku run <app-name> rails db
That brought up the console for the PostgreSQL container I created (via dokku postgresql:create <db>). I couldn't figure out another way to get at the PostgreSQL instance in that container, short of attempting to directly connect to the DB, with the connection info/credentials listed when you do this:
dokku postgresql:info <db>
I haven't tried that, though I suspect it would work.
I'm trying to push a local postgresql database to my heroku project. Unfortunately, it doesn't work.
Postgresql works perfectly on my computer and the psql command works as well (Path is set correctly to the bin folder)!
But when I try to push my databse with
heroku pg:push mylocaldb HEROKU_POSTGRESQL_MAGENTA --app sushi
(with my db name as mylocaldb and my real app name), it doesn't work at all.
Problem: The local psql command could not be located
Why? As said before, psql works fine. Even a reinstall of postgresql doesn't help.
The pull statement (see: doesn't work either because postgres always wants to authenticate as my windows user (I don't know how to change the user name when using the heroku pg:pull command, this didn't help: https://devcenter.heroku.com/articles/heroku-postgresql#pg-push-and-pg-pull)
Thanks for your help!
Have you tried calling the psql command with its full path?
For example: /opt/PostgreSQL/9.1/bin/psql
Or if that doesn't help, you can try this
I can't manage to share two PostgreSQL heroku dbases between two heroku apps.
Doesn't work
What I've tried so far (I'm using the Heroku CLI):
$ heroku config:set DATABASE_URL=`heroku config:get DATABASE_URL -a my-heroku-app-original`
Setting config vars and restarting my-heroku-app... done, v20
DATABASE_URL: postgres://...correct url...
$ heroku config
=== msite-poniai Config Vars
DATABASE_URL: postgres://...correct url...
When I try to access psql, depending on whether I've added a new db or not, I get these two answer (I both cases heroku doesn't see the DATABASE_URL I've just provided.
$ heroku pg:psql
! Your app has no databases.
$ heroku pg:psql
! Unknown database. Valid options are: HEROKU_POSTGRESQL_NEWLYADDED_URL
Works
If I provide psql with the exact location of the db (in this case the original app that has the db as an add-on) it does manage to connect.
$ heroku pg:psql HEROKU_POSTGRESQL_ORIGINAL_URL --app my-heroku-app-original
psql (9.1.9)
I've read
I've closely followed advice given in Share database between 2 apps in Heroku and heroku-postgresql#create-new-db.
Simply copying a config var from one app to another does not attach the referenced database to the app. Were the host or password to change on the first, your second app would be broken. There is a highly alpha plugin you can try https://github.com/heroku/heroku-attachable-resources but it does some weird things like no longer give default names to databases, and overwrites DATABASE_URL if you don't pass in --config and sometimes doesn't let you remove databases.