How to compare a schema defined in a file with actual schema running on DB? - postgresql

I'm running a PostgreSQL DB and want to compare the schema defined in schema.sql with the schema that is created on the DB on a regular basis. I.e. to verify that no manual changes on the remote are applied or that schema.sql actually reflects what's running on the DB.
What I'm currently doing is roughly:
create a dump from schema.sql using pg_virtualenv:
$ pg_virtualenv sh -c 'psql -U $PGDATABASE -f schema.sql
$ pg_dump --schema-only --no-comments --no-owner --no-privileges --schema=public -U $PGDATABASE -f schema_local.sql'
create a dump from the remote
$ pg_dump --schema-only --no-comments --no-owner --no-privileges --schema=public -U $PGREMOTE -f schema_upstream.sql
strip out comments, SET, emtpy lines, etc. and diff them
$ diff -u <(grep -Ev '^--|^SET|^$$' schema_local.sql) <(grep -Ev '^--|^SET|^$$' schema_upstream.sql)
That works reasonably well and I can easily spot schema changes in the diff if they occur. But the solution seems a bit hacky and I wonder if there's a better way to achieve the same thing?

Related

How to restore .bak file using pg_restore command through cmd in PostgreSQL?

I have a test.bak file created with pg_dump command which has a hypertable and i have created a new database Performance_Test in postgreSQL.
The database was dumped using the command:
pg_dump -h localhost -U postgres -p 5432 -Fc -f "D:\Database Backup\temp.bak" Performace_Test
I want to restore that test.bak file in Performance_Test.
How can i do that?
You can restore doing the below(with caveats noted further on):
pg_restore -h localhost -d Performace_Test -U postgres -p 5432 "D:\Database Backup\temp.bak"
The caveats are:
If Performace_Test was created as mixed case by quoting you will need to quote the name above e.g. "Performace_Test"
If Performace_Test is not empty then you will get a bunch of errors.
Also assumes that Performace_Test is on the same cluster as you specified in the pg_dump command.
psql Performance_Test < test.bak

psql equivalent for pg_restore for .sql files

I ONLY dump my databases as *.sql files not *.dump files. As a result NONE of the pg_restore commands work. I've been reading through answers and I swear most people have a reading disability lol
I am asking for the equivalent in psql for a common pg_restore commandLine method to restore a database.
I have no intention of dumping my databases as *.dump.
my question is this:
what is the equivalent to:
pg_restore --verbose --clean --no-acl --no-owner -h localhost -U myuser -d my_db db/latest.dump
using psql
so...
something along the lines of:
psql --verbose --clean --no-acl --no-owner -h localhost -U myuser -d my_db db/latest.sql
With a SQL dump you need to decide whether you want to drop target objects, when dumping the database, not when importing it.
So, you need to use:
pg_dump --clean ....
Then the SQL dump will contain the necessary DROP statements.
Another option is to run drop owned by current_user before doing the import. This however requires that everything is owned by the user doing the import (so you can't run the import as e.g. postgres)
This can be combined with running the SQL dump:
psql -U your_user -d your_db -c 'drop owned by current_user' -f your_dump.sql

Ignoring a table in pg_dump and restore

This is what I currently do to copy a database from my local machine to a remote server.
Dump local database:
pg_dump dbname --clean -U postgres > dumpfile
Restore remote database:
psql --single-transaction dbname -U postgres < dumpfile
This does a perfect replication.
How do I modify this to ignore particular table names, both in the source and destination?
This will be useful for tables that log website visits. I want to retain my existing remote visit logs, while ignoring my local "visit" logs (which is just me visiting my own website locally).
Using the -t switch, you can be selective about what tables to include:
pg_dump <your switches> -t my_schema.awesome* -f backup.bat postgres
Which will only include those tables.
Likewise, the -T switch will do the opposite -- dump everything but the tables you specified:
pg_dump <your switches> -T my_schema.lame* -f backup.bat postgres
You can use each switch multiple times as well:
pg_dump <your switches> \
-t my_schema.awesome* \
-t my_schema.amazing* \
-t my_schema.great -f backup.bat postgres
When you restore, it won't bother trying to restore something it didn't back up, so that part should be handled somewhat natively.

Restoring postgres database except certain tables with pg_restore

I have a Postgresql dump (created with pg_dump, custom compressed format). I would like to pg_restore it onto another server except a few large tables. I have tried using -l option and remove the tables not needed from the list as shown below. Is there an effective solution as am not sure how efficient the below is.
pg_restore -l dumpfile.dmp > list.txt
egrep -v "logtable|summarytable|historytable" list.txt > listex.txt
pg_restore -Fc -v -p 5432 -d prism --use-list=listex.txt dumpfile.dmp 2>> error1.out &

How to restore postgres database into another database name

I use the postgres today
and got a problem
I dump the database that way
pg_dump zeus_development -U test > zeus_development.dump.out
what if I wnat to restore to another database zeus_production
How could I do?
Simple, first create your database using template0 as your template database:
createdb -U test -T template0 zeus_production
Then, restore your dump on this database:
psql -U test zeus_production -f /path/to/zeus_development.dump.out
When restoring, always use template0 explicit, as it is always an empty and unmodifiable database. If you don't use an explicit template, PostgreSQL will assume template1, and if it has some objects, like a table or function that your dumped database already has, you will get some errors while restoring.
Nonetheless, even if you were restoring on a database with the same name (zeus_development) you should create (or recreate) it the same way. Unless you used -C option while dumping (or -C of pg_restore if using a binary dump), which I don't recommend, because will give you less flexibility (like restoring on a different database name).
The PostgresSQL documentation has influenced me to use the custom format. I've been using it for years and it seems to have various advantages but your mileage may vary. That said, here is what worked for me:
pg_restore --no-owner --dbname postgres --create ~/Desktop/pg_dump
psql --dbname postgres -c 'ALTER DATABASE foodog_production RENAME TO foodog_development'
There was no foodog_development nor foodog_production databases existing before the sequence.
This restores the database from the dump (~/Desktop/pg_dump) which will create it with the name it was dumped as. The rename names the DB to whatever you want.
The --no-owner may not be needed if your user name is the same on both machines. In my case, the dump was done as user1 and the restore done as user2. The new objects need to be owned by user2 and --no-owner achieves this.
Isn't it easier to simply do the following?
createdb -U test -T zeus_development zeus_production
This has an answer on dba.stackexchange, which I reproduce here:
Let's define a few variables to make the rest easier to copy/paste
old_db=my_old_database
new_db=new_database_name
db_dump_file=backups/my_old_database.dump
user=postgres
The following assumes that your backup was created with the "custom" format like this:
pg_dump -U $user -F custom $old_db > "$db_dump_file"
To restore $db_dump_file to a new database name $new_db :
dropdb -U $user --if-exists $new_db
createdb -U $user -T template0 $new_db
pg_restore -U $user -d $new_db "$db_dump_file"
Here's a hacky way of doing it, that only works if you can afford the space and time to use regular .sql format, and if you can safely sed out your database name and user.
$ pg_dump -U my_production_user -h localhost my_production > my_prod_dump.sql
$ sed -i 's/my_production_user/my_staging_user/g' my_prod_dump.sql
$ sed -i 's/my_production/my_staging/g' my_prod_dump.sql
$ mv my_prod_dump.sql my_staging_dump.sql
$ sudo su postgres -c psql
psql> drop database my_staging;
psql> create database my_staging owner my_staging_user;
psql> \c my_staging;
psql> \i my_staging_dump.sql
If your dump does not include the name, the restore will use the DB defined in DESTINATION. Both SOURCE and DESTINATION are Connection URLs.
Dump without --create
pg_dump \
--clean --if-exists \
--file ${dump_path} \
--format=directory \
--jobs 5 \
--no-acl \
--no-owner \
${SOURCE}
Restore without --create
pg_restore \
--clean --if-exists \
--dbname=${DESTINATION} \
--format=directory \
--jobs=5 \
--no-acl \
--no-owner \
$dump_path