Unable to run pg_dump from Supabase - postgresql

I intend to backup my postgres database at Supabase
$ pg_dump -h db.PROJECT_REF.supabase.co -U postgres --clean --schema-only > supabase_backup.sql
I ran the command
GRANT ALL ON ALL SEQUENCES IN SCHEMA auth TO postgres;
grant all on auth.identities to postgres, dashboard_user;
Yet, I still get
pg_dump: error: query failed: ERROR: permission denied for table schema_migrations
pg_dump: error: query was: LOCK TABLE realtime.schema_migrations IN ACCESS SHARE MODE

EDIT Oct-2022:
There's a new/improved script to perform database migration:
Before you begin:
Install Postgres so you can run psql and pg_dump.
Create a new Supabase project.
Store the old project's database URL as $OLD_DB_URL and the new project's as $NEW_DB_URL.
Migrating the database:
Enable Database Webhooks in your new project if you enabled them in your old project.
In your new project, enable all extensions that were enabled in your old project.
Run the following command from your terminal:
#!/usr/bin/env bash
#Edit here:
OLD_DB_URL=db.old_project_ref.supabase.co
NEW_DB_URL=db.new_project_ref.supabase.co
OLD_DB_PASS=secret_password_here
NEW_DB_PASS=secret_new_password_here
#Script:
# Default case for Linux sed, just use "-i"
sedi=(-i)
case "$(uname)" in
# For macOS, use two parameters
Darwin*) sedi=(-i "")
esac
PGPASSWORD="$OLD_DB_PASS" pg_dump -d postgres -U postgres \
--clean \
--if-exists \
--quote-all-identifiers \
--exclude-table-data 'storage.objects' \
--exclude-schema 'extensions|graphql|graphql_public|net|pgbouncer|pgsodium|pgsodium_masks|realtime|supabase_functions|pg_toast|pg_catalog|pg_*|information_schema' \
--schema '*' \
-h "$OLD_DB_URL" > dump.sql
sed "${sedi[#]}" -e 's/^DROP SCHEMA IF EXISTS "auth";$/-- DROP SCHEMA IF EXISTS "auth";/' dump.sql
sed "${sedi[#]}" -e's/^DROP SCHEMA IF EXISTS "storage";$/-- DROP SCHEMA IF EXISTS "storage";/' dump.sql
sed "${sedi[#]}" -e 's/^CREATE SCHEMA "auth";$/-- CREATE SCHEMA "auth";/' dump.sql
sed "${sedi[#]}" -e 's/^CREATE SCHEMA "storage";$/-- CREATE SCHEMA "storage";/' dump.sql
sed "${sedi[#]}" -e 's/^ALTER DEFAULT PRIVILEGES FOR ROLE "supabase_admin"/-- ALTER DEFAULT PRIVILEGES FOR ROLE "supabase_admin"/' dump.sql
PGPASSWORD="$NEW_DB_PASS" psql -d postgres -U postgres \
--variable ON_ERROR_STOP=1 \
--file dump.sql \
-h "$NEW_DB_URL" -p 6543
OLD ANSWER (obsolete):
I believed you may have missed the part to alter the role in the migration guide. I've copied the instructions below:
Before you begin
Make sure Postgres is installed so you can run psql and pg_dump.
Create a new Supabase project.
If you enabled Function Hooks on your old project, enable it on your new project.
Store the old project's database URL as $OLD_DB_URL and the new project's as $NEW_DB_URL.
Migrate the database
Run ALTER ROLE postgres SUPERUSER in the old project's SQL editor
Run pg_dump --clean --if-exists --quote-all-identifiers -h $OLD_DB_URL -U postgres > dump.sql from your terminal
Run ALTER ROLE postgres NOSUPERUSER in the old project's SQL editor
Run ALTER ROLE postgres SUPERUSER in the new project's SQL editor
Run psql -h $NEW_DB_URL -U postgres -f dump.sql from your terminal
Run TRUNCATE storage.objects in the new project's SQL editor
Run ALTER ROLE postgres NOSUPERUSER in the new project's SQL editor

Related

Cannot restore Postgresql databases got "database already exists" error

I have take backup by pg_dumpall > test.out
and test.out successfully generated, hence backup completed.
I have used command psql -f test.out postgres for restore
But got following errors with restoring backup:
databases already exists
relation "products" already exists
duplicate key value violates unique constraint "products_pkey"
I actually want to replace the data in the existing db with backup. How to do that?
The problem is that the database you're trying to restore already exists.
You can run a DROP DATABASE database_name command that will delete your existing database and then you can run your test.out file.
Or you can run pgdumpall --clean > test.out and then run the resulting file. The clean flag will make the resulting files have the DROP DATABASE command in them.
Do you use the bellow command ?
psql -h localhost -U [login role] database_name -f /home/database.backup
I think a flow like this might help, because we don't want drop the database each time we call the backup file.
First, we need to create a backup file using the --format=custom [-Fc] to restore it using pg_restore. We can use a connection string postgresql://<user>:<pass>#localhost:5432/<dbname> and replace <user>, <pass>, and <dbname> with your information.
pg_dump -v -Fc \
postgresql://<user>:<pass>#localhost:5432/<dbname> \
> db-20211122-163508.sql
To restore we will call it using --clean [-c] and --create [-C] to drop the database before restoring. Replace <user>, <host>, <port>, and <dbname> with your information.
pg_restore -vcC \
-U <user> \
-h <host> \
-p <port> \
-d <dbname> \
< db-20211122-163508.sql
This way you don't need to use clean when you create the backup file.

clone postgres database with new name

I have a users-production database on 157.157.35.333
I would like to clone it to another host as users-sandbox
here is what I tried:
PRODUCTION_HOST=111.111.11.111
SANDBOX_HOST=222.222.22.222
echo "creating db on sanbox"
psql -h ${SANDBOX_HOST} -U postgres -c "CREATE DATABASE \"users-sandbox\";"
pg_dump -h ${PRODUCTION_HOST} -U postgres -d users-production -F c -b -v | \
pg_restore -C -c -h ${SANDBOX_HOST} -U postgres -d users-sandbox -v
but this creates the database with the old name
how do I create with a new name?
https://www.postgresql.org/docs/current/static/app-pgrestore.html
-d dbname
Connect to database dbname and restore directly into the database.
but!
-C
--create
Create the database before restoring into it. If --clean is also
specified, drop and recreate the target database before connecting to
it.
When this option is used, the database named with -d is used only to
issue the initial DROP DATABASE and CREATE DATABASE commands. All data
is restored into the database name that appears in the archive.
so remove -C from pg_restore arguments...
(formatting of quotes mine)

pg_restore ERROR: “Relation does not exist” and creating new database

I have made a backup of my specific tables that I want to restore into a new database using:
call pg_dump -Fc -h server -d database -U user -p password -v -f dump.sql -t public.table1 -t public.table2
And I have no problems.
I then want to restore the dump by creating a new database with pg_restore using:
call pg_restore --clean --create -d temp -h server -p password -U user dump.sql
This gives me a "database temp does not exist" error. This should have created the database and restored it as far as I understand.
I However then create the "temp" database manually and run:
call pg_restore --clean --create -d temp -h server -p password -U user dump.sql
This follows through, but does not create the tables and gives me an error "relation table1" and "relation table2" does not exist and only creates the corresponding id_sequences for the two tables.
What I actually want is to not have to manually create the new database and that all tables in the backup is restored into a brand new database via pg_restore using:
call pg_restore --clean --create -d temp -h server -p password -U user dump.sql
as I understand it.
In order to create the database temp, pg_restore needs to be connected to a different database first.
So it won't do to use -d temp; you must specify an existing database, typically postgres.
pg_restore will connect to that database, issue CREATE DATABASE temp, connect to temp and proceed to restore the data.

How to restore postgres database into another database name

I use the postgres today
and got a problem
I dump the database that way
pg_dump zeus_development -U test > zeus_development.dump.out
what if I wnat to restore to another database zeus_production
How could I do?
Simple, first create your database using template0 as your template database:
createdb -U test -T template0 zeus_production
Then, restore your dump on this database:
psql -U test zeus_production -f /path/to/zeus_development.dump.out
When restoring, always use template0 explicit, as it is always an empty and unmodifiable database. If you don't use an explicit template, PostgreSQL will assume template1, and if it has some objects, like a table or function that your dumped database already has, you will get some errors while restoring.
Nonetheless, even if you were restoring on a database with the same name (zeus_development) you should create (or recreate) it the same way. Unless you used -C option while dumping (or -C of pg_restore if using a binary dump), which I don't recommend, because will give you less flexibility (like restoring on a different database name).
The PostgresSQL documentation has influenced me to use the custom format. I've been using it for years and it seems to have various advantages but your mileage may vary. That said, here is what worked for me:
pg_restore --no-owner --dbname postgres --create ~/Desktop/pg_dump
psql --dbname postgres -c 'ALTER DATABASE foodog_production RENAME TO foodog_development'
There was no foodog_development nor foodog_production databases existing before the sequence.
This restores the database from the dump (~/Desktop/pg_dump) which will create it with the name it was dumped as. The rename names the DB to whatever you want.
The --no-owner may not be needed if your user name is the same on both machines. In my case, the dump was done as user1 and the restore done as user2. The new objects need to be owned by user2 and --no-owner achieves this.
Isn't it easier to simply do the following?
createdb -U test -T zeus_development zeus_production
This has an answer on dba.stackexchange, which I reproduce here:
Let's define a few variables to make the rest easier to copy/paste
old_db=my_old_database
new_db=new_database_name
db_dump_file=backups/my_old_database.dump
user=postgres
The following assumes that your backup was created with the "custom" format like this:
pg_dump -U $user -F custom $old_db > "$db_dump_file"
To restore $db_dump_file to a new database name $new_db :
dropdb -U $user --if-exists $new_db
createdb -U $user -T template0 $new_db
pg_restore -U $user -d $new_db "$db_dump_file"
Here's a hacky way of doing it, that only works if you can afford the space and time to use regular .sql format, and if you can safely sed out your database name and user.
$ pg_dump -U my_production_user -h localhost my_production > my_prod_dump.sql
$ sed -i 's/my_production_user/my_staging_user/g' my_prod_dump.sql
$ sed -i 's/my_production/my_staging/g' my_prod_dump.sql
$ mv my_prod_dump.sql my_staging_dump.sql
$ sudo su postgres -c psql
psql> drop database my_staging;
psql> create database my_staging owner my_staging_user;
psql> \c my_staging;
psql> \i my_staging_dump.sql
If your dump does not include the name, the restore will use the DB defined in DESTINATION. Both SOURCE and DESTINATION are Connection URLs.
Dump without --create
pg_dump \
--clean --if-exists \
--file ${dump_path} \
--format=directory \
--jobs 5 \
--no-acl \
--no-owner \
${SOURCE}
Restore without --create
pg_restore \
--clean --if-exists \
--dbname=${DESTINATION} \
--format=directory \
--jobs=5 \
--no-acl \
--no-owner \
$dump_path

How do I do a schema only backup and restore in PostgreSQL?

How do I take a schema level backup in PostgreSQL database and restore on the another database? Is there any single command available for this? For example, can I pg_dump and restore in single line?
pg_dump --schema=masters oldDB > masters1.sql
cat masters1.sql | psql newDB
or
in single command you can do by this
pg_dump oldDB --schema masters | psql -h localhost newDB;
Backup schema and restore it on system for postgresql as below:
Dump schema for database
pg_dump -s database_name > db.sql
Dump schema for specific table
pg_dump -s database_name -t table_name > db.sql
Restore backed up schema using below command
psql -d database_name -h localhost -U postgres < path/db.sql
-s or --schema-only to exclude data from dump
Documentation
What's wrong with the documentation?
Example from the manual:
To dump all schemas whose names start with east or west and end in
gsm, excluding any schemas whose names contain the word test:
$ pg_dump -n 'east*gsm' -n 'west*gsm' -N 'test' mydb > db.sql