I have two databases running on the same PostgreSQL 9.3 server (running on Windows).
The two databases have the very same structure: same tables, same keys, same constraints, etc; the only difference is that data gets periodically moved from one to the other.
Think of it like a database with info on clients and orders, where every week data related to clients older than, say, 2 years is moved from one to the other.
Is there a quick way to create tables which combine data from the new and the old database? Something like:
select * from NewDatabase.MyTable
UNION
select * from OldDatabase.MyTable
to be run on about 25 of the 170 tables of the databases.
The 2 key problems I see are:
1) PostgreSQL doesn't really allow cross database queries. Foreign data wrappers or db links (as mentioned here: Possible to perform cross-database queries with postgres?) are very cumbersome to use because (if I understand correctly) I must pre-declare the entire structure of the tables I wish to access.
2) I'd need to find a way to automate this so that I can code it automatically for each of the 25 tables I need, and it runs every week.
I have no preference whatsoever on how this is done (Pentaho-Kettle, command script, sql script, etc.) as long as it can be automated efficiently.
Thanks!
I believe I may have found a way. Of course suggestions and comments are more than welcome. My idea is similar to what was suggested here: I want to restore the database with a different schema (not in the most voted answer, though).
Since I am only interested in tables in the public schema, I did the following:
Backup of the public schema of database1
"C:\Program Files\PostgreSQL\9.3\bin\pg_dump.exe" --host MYSQLHOST --port 5432 --username "JohnDoe" --format custom --section pre-data --section data --section post-data --verbose -n "public" --file "\\mypath\db1.backup" "db1"
Renamed the public schema of db2 and created a new, empty, public schema:
ALTER SCHEMA public RENAME TO public_db2 ;
CREATE SCHEMA public ;
Restore the backup of db1’s public schema into the newcly created (and empty) public schema in db2:
"C:\Program Files\PostgreSQL\9.3\bin\pg_restore.exe" --host MYSQLHOST --port 5432 --username "JohnDoe" --dbname "db2" --section pre-data --section data --section post-data --schema public --verbose "\\mypath\db1.backup"
Renamed the newly imported public schema in db2:
ALTER SCHEMA public RENAME TO public_db1 ;
My syntax above creates a binary (and automatically compressed) backup file. If you have a backup file in text format, you cannot restore just one schema. The only part I haven't managed to automate (yet) is that the shell command prompts me for a password.
I use Windows but I understand the syntax should be the same on other systems, too.
Related
Im currently using the tool pg_dump to backup my database.
I have a lot of database. I don't know their full name but I know that all DATABASES have a well defined prefix.
I would like to automate the process of backup all my DATABASES, however, i have not found a way to specify pg_dump to dump multiple DATABASE that have the same prefix.
I said database and not table nor schema, because I tried the commands in the pgsql doc which gives the options -n for schemas and -t for tables.
But that's not what I want to do, I want to save all my databases with a defined prefix.
Any help in this matter would be greatly appreciated
I have two postgres databases on the same server, a live one and another one which I use as a test database.
Periodically I need to copy my live database (both structure of the tables and their data) into the test one, but everything I've found online only copies table by table. Since I often create new tables in my live db I can't do this, otherwise I'd have to update the job every time.
Anybody knows how can I pull the whole live postgres db into the test postgres one?
Thanks!
You want to use pg_dump and pg_restore.
Example usage would be:
$ pg_dump -Fc <database_name> > dumpfile
$ pg_restore <another_database_name> < dumpfile
If you want to do this for all database you might want to consider pg_dumpall
A bit more can be found in https://www.postgresql.org/docs/current/backup-dump.html
Context: I'm new on a big project, with a huge number of PostgreSQL tables (46) and foreign keys, and no documentation. We work with Java, Spring, Hibernate and TestNG.
Goal: I'm looking for a script that's able to empty the database (to call it before every unit-test case). Unfortunately, I can't spend too much time to find foreign keys in order to empty tables in the correct order.
Question: is there a way to empty (I don't want to drop it) a table without checking constraints on foreign keys ?
As #a_horse_with_no_name commented, using truncate ... cascade for each table will work. The link also provides a function that automatically loops through all the tables in the database.
If you have large tables, with a lot of foreign keys, it will probably take a lot of time to travel through all the foreign key indexes. In that case it would probably be better to dump the schema, drop the database, recreate the database and reload the schema in a batch. There is an example in the link, but it doesn't tell you how to run them. This is a little more refined:
#!/bin/sh
# Run this as the postgres user because you need create database permissions
# I'm assuming you're using a Unix variant,
# the postgresql database is at localhost,
# and that psql trusts the Unix account.
# Otherwise add -h hostname to the command lines
# and set $PGPASSWORD variable before the first command.
pg_dump –s yourdb > /tmp/clean.sql
psql –c "drop database yourdb;"
psql –c "create database yourdb with owner you;"
psql yourdb < /tmp/clean.sql
rm –f /tmp/clean.sql
We have a dev Postgres DB that one of the developers has created an application in. Is there an existing query that will pull information from the role_table_grants table and generate all the correct statements to move into production? PGAdmin will create all the generate scripts for certain things but I haven't found a less manual way rather than just writing all the statements by hand based on the role_table_grants table. Not asking anyone to dump time into creating it, just thought I would ask if there are some existing migration scripts out there that would help.
Thanks.
Dump the schema to a file; use pg_dump or pg_dumpall with the --schema-only option.
Then use grep to get all the GRANT and REVOKE statements.
On my dev machine, I might do something like this.
$ pg_dump -h localhost -p 5435 -U postgres --schema-only sandbox > sandbox.sql
$ grep "^GRANT\|^REVOKE" sandbox.sql
REVOKE ALL ON SCHEMA public FROM PUBLIC;
REVOKE ALL ON SCHEMA public FROM postgres;
GRANT ALL ON SCHEMA public TO postgres;
[snip]
Perhaps pg_dumpall is what you need. Probably with --schema-only option in order to dump just schema, not development data.
If you need to move not all databases, you can use pg_dumpall --globals-only to dump roles (which don't belong to any particular database), and then use pg_dump to dump one certain databases.
I have a database which I have several tables (including some data) and functions on. I created these tables and functions from the command line interface.
My question is: Is there any way to have my database queries back which creates the same tables/functions again in a readable format.
PostgreSQL version: 9.1.9
pg_dump --schema-only my_database > my_database.sql
http://www.postgresql.org/docs/current/static/app-pgdump.html