I'm currently working on dumping one of our customer's database in a way that allows us to create new databases from this customer's basic structure, but without bringing along their private data.
So far, I've had success with pg_dump combined with the --exclude_table and exclude-table-data commands, which allowed me to bring only the data I'll effectively need for this task.
However, there are a few tables that mix lines which references some of the data I left behind with other lines that references data that I had to bring, and this is causing me a few issues during the restore operation. Specifically, when the dump tries to enforce FOREIGN KEY constraints for certain columns on these tables, it fails because there are some lines with keys that have no matching data on the respective foreign table - because I chose to not bring this table's data!
I know I can log into the database after the dump is complete, delete any rows that reference data that no longer exists and create the constraint myself, but I'd like to automate the process as much as possible. Is there a way to tell pg_dump or pg_restore (or any other program) to not bring rows from table A if they reference table B if and table B's data was excluded from the backup? Or to tell Postgres that I'd like to have that specific foreign key to be active before importing the table's data?
For reference, I'm working with PostgreSQL 9.2 on a HREL 7 server.
What if you disable foreign key checking when you restore your database dump? And after that remove lonely rows from the referring table.
By the way, I recommend you to fix you database schema so there is no chance wrong tuples being inserted into your database.
Related
I'm using the Oracle foreign data wrapper and would like to have local copies of some of my foreign tables locally. Is there another option than having materialized views and refreshing them manually?
Not really, unless you want to add functionality in Oracle:
If you add a trigger on the Oracle table that records all data modifications in another table, you could define a foreign table on that table. Then you can regularly run a function in PostgreSQL that takes the changes since you checked last time and applies them to a PostgreSQL table.
If you understand how “materialized view logs” work in Oracle (I don't, and I think the documentation doesn't tell), you could define a foreign table on that and use it like above. That might be cheaper.
Both of these ideas would still require you to regularly run something in PostgreSQL, but you might be cheaper. Perhaps (if you have the money) you could use Oracle Heterogenous Services to modify a PostgreSQL table whenever something changes in an Oracle table.
I'm running pg_dump -F custom for database backups, with --exclude-table-data for a very large audit table. I'm then exporting that table data in a separate dump file. It isn't referentially integral with the main dump.
As part of my restore strategy, I'd like to be able to restore the main dump, bring my app online and continue using the database immediately, then bring the audit data back in behind it. The trouble is, as soon as new audit data comes in at sequence 1, the import of the audit data fails as soon as it tries to insert over the top of the new data.
Is it possible to include the setting of the sequence in the main dump without including the table data?
I have considered removing the primary key, but there are other tables I'd also like to do this with, and they definitely do need the PK.
I'm using postgresql 13.
Instead of a sequence, which can build with a rownumber use uuids and a timestamp, so you have unique values and the order of insert doesn't matter. Uuids are a bit slower the ints.
Another possibility that you save th last audit Id in another table and set the sequence new like https://www.postgresql.org/docs/9.1/sql-altersequence.html
Apologies if this is answered in the docs or somewhere else obvious, but I have just gone through the rigmorale of having to use postgres_fdw to get data from another database (quite different to the ease of MS SQL!)
I ran an IMPORT SCHEMA statement to get the foreign table/databse data into my working database. Will the data in this schema automatially update whenever the data in the foreign database updates or is this IMPORT SCHEMA a copy of the data?
A foreign table does not contain any data. Querying it will yield the current data in the remote table. Think of it as something like a view.
If you are talking about metadata changes (ALTER TABLE), the analogy to views is also helpful: such changes will not be reflected in the foreign table, and you need to run an ALTER FOREIGN TABLE statement.
Perhaps it would make sense to put the data for these databases in a single database in two different schemas.
I need to move some old data from one database to another - both have similar schemas. After the old rows in db1.mytable are inserted into db2.mytable - these same rows should be deleted from db1.mytable.
This is reduce db size and archive data that is not really needed that much but still important.
You should to use Foreign Data Wrapper for Postgres - https://www.postgresql.org/docs/current/static/postgres-fdw.html With foreign tables, you can send a query to another database.
I have a database with around 20 normalized tables and there are a lot of foreign key constraints between these tables. The user table for instance, is being referenced in 3 other tables and the primary keys of these 3 tables are further referenced in other tables whose primary keys are further referenced in other tables.
Now, I need to implement a functionality for archiving a user which involves deleting the row from user table and deleting all its direct and indirect linkages, but at the same time saving, serializing everything in a JSON structure and dumping it in an archive table so that if the need arises to unarchive the user in future, we can do that. I was thinking of writing a function for this since it makes a lot more sense to do this entire activity in a single transaction. I was wondering if there is a better and easier alternative for doing such thing ?