I try to synchronize two databases using with bucardo. I create master to master replication and it works fine. However rows inserted before replication can not synchronize. I don't want dump one db to the other. How can I do that? Thanks for help.
Related
First of all i do not know if is possible what i want to achive, will describe below:
I have access of a remote PostgreSQL that holds the data i need (let's say Remote PostgreSQL 1)
have just credentials to read from database
What i want to achive is to create a localt PostgreSQL on my machine (let's say Local PostgreSQL 2)
Want to copy and check for missing data from Remote PostgreSQL 1 to Local PostgreSQL 2 in real time or at list to copy at the end of the day data
The scenario will work perfect with replication but the issue is that Remote PostgreSQL 1 is not owned by me and can not be used as real time DB, because of this trying to find a solution to get all the data from Remote PostgreSQL 1 to Local PostgreSQL 2.
Could be the following scenarios:
first time setup to downlaod all the database from Remote PostgreSQL 1 to Local PostgreSQL 2
after first time setup to check what data came new inside and add them in Local PostgreSQL 2
Would be great if this could be done on OS level on UBUNTU. My application is written in python 3 i could do scripts to do all this job but i speak of 100 millions of raws per table huge amount of data. Think will be problems to get everything from database and start to check everything what is missing and not.
Any ideas would be great.
If the owner of Remote Database 1 won't cooperate with you other than to give you read only access to the tables, then you don't have any efficient options. If the remote owner does or can be convinced to keep insertion/modification timestamp columns in all the tables (although then deletions would be a problem), or an in-database "audit" log for all the tables, you could use those. I think you have an organizational/political problem rather a programming problem.
I have two DB instances with postgresql 10.
First instance working with wal_level = logical, have ALL TABLE publisher. Second - have subscription to the publisher, described above.
All data from the Master DB tables are successfully sending to the replica.
There is only one issue for me - when my App add a new table on Master - I need to add the same table to the replica (and run REFRESH PUBLICATION).
My question - is there a way for replica DB to create new tables automatically?
Schema changes aren't replicated. You can see that in the documentation, at Postgresql 10 Logical Replication: Subscription
The way I handle this is to make all schema changes through a script, and to write the script so that it executes the change commands twice: once on the primary database, and once on the replica.
I am using slony-I to replicate tables from one server to another. I have to databases on the master slave that have same exact tables , and i want to replicate them to a single table in the slave. I can create the same tables in different schemas in the slave table , however i cant determine the schema in the replication set in the slave host.
I want to be able to determine the schema i am replicating to on the slave host.
How can i do this in slony?
Thank you
Unfortunately you can't choose the schema on the slave host in slony.The schema name and table name should be identical on both the master and slave. A Work around for this thing is to create another schema on both databases on slave and master and use them in your slony replication
I have two databases on Amazon RDS, both Postgres. Database 1 and 2
I need to restore an instance from a snapshot of Database 1 for my Staging environment. (Database 2 is my current Staging DB).
However, I want the data from a few of the tables in Database 2 to overwrite the tables in the newly restored snapshot. What is the best way to do this?
When restoring RDS from a Snapshot, a new database instance is created. If you only wish to copy a portion of the snapshot:
Restore the snapshot to a new (temporary) database
Connect to the new database and dump the desired tables using pg_dump
Connect to your staging server and restore the tables using pg_restore (most probably deleting any matching existing tables first)
Delete the temporary database
pg_dump actually outputs SQL commands that are then used to recreate tables and restore data. Look at the content of a dump to understand how the restore process actually works.
I hope this still works for someone else.
With my team we faced a similar issue. We also had 2 Postgres databases and we also just needed to backup some tables from db1 to db2.
What we did is to use a lambda function using Python (from AWS lambda ofc) that connected to both databases and validates if db1.table1 has the same data as db2.table1, if not, then the lambda function should write the missing data from db1.table1 into db2.table1. The approach of using lambda was because we wanted to automate the process due to the main db (let's say db1) is constantly being updated. In addition, it allowed us to only backup our desired tables (let's say 3 tables out of 10), instead of backing up the whole database.
Note: Maybe you want to do these writes using temporary tables to avoid issues with any constraints you have in your tables.
I know the way to set up a Master/Slave DB in Postgres is having 2 DB servers, but unfortunately i have only one server for now.
How can i mirror my production db into another "backup db" in "real_time"? I want to give access to another user to the mirrored db, so even if he does something there it will not affect production.
Nothing stops you setting up hot standby streaming replication, or another replication option like Londiste, between two PostgreSQL instances on the same computer.
The two copies of PostgreSQL must use different ports, but that's the only real restriction.
How to set up the second PostgreSQL instance depends on your operating system and how you installed PostgreSQL, which you have not mentioned.
You'll want to use streaming replication with hot standby if you want a read-only replica. If you want it to be read/write, then you can do a one-off copy of the database with pg_basebackup and not keep them in sync after that. Or you can use a tool like Londiste to replicate changes selectively.
You can run multiple instances of PostgreSQL on the same computer, by using different ports.