I'm using postgres 9.5.7 in RDS and want to create a slave/read replica on an EC2 box. I've figured out how to get logical replication working on RDS and am able to use pg_recvlogical to tap into the replication slot on the EC2 box.
My challenge now has been that, unfortunately, RDS doesn't support pglogical and it seems that I'm left with either test_decoding or wal2json for my output formats. Is there something out there that knows how to take either of those formats and turn them into SQL that can be executed on the slave?
Most of the guides I've found online only go as far as getting pg_recvlogical working, and don't take that extra last step of showing how to actually get those changes into the slave database.
perhaps you want to check https://wiki.postgresql.org/wiki/Logical_Decoding_Plugins#decoder_raw
this plugins output a sql statements that can be run in slave postgres
Related
I have a requirement of replicate data from AWS RDS Postgres(9.6) Database to On-Premise Postgres(9.5) Database. I have found stuff about replication from On-premise to On-premise. But How can we implement it for AWS RDS to On-premise?
I do this using Bucardo.
Check-out this: https://bucardo.org/Bucardo/
With Bucardo you can replicate RDS postgres instance to a slave postgres present somewhere, only configuring slave, so without the needs to configure RDS stuff.
Also you can do this with zero downtime.
Anyway I am not sure this will work using different versions of Postgresql. You should use same version if possible. I tested it with 9.4.x and it is working.
UPDATE
I can confirm that this is working also using different version of Postgres, for example I was able to replicate with these versions:
AWS RDS postgresql 9.4.x
On-premise postgresql 9.6.x
http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_PostgreSQL.html#PostgreSQL.Concepts.General.FeatureSupport.LogicalReplication
Beginning with PostgreSQL version 9.4, PostgreSQL supports the
streaming of WAL changes using logical replication slots. Amazon RDS
supports logical replication for a PostgreSQL DB instance version
9.4.9 and higher and 9.5.4 and higher. Using logical replication, you can set up logical replication slots on your instance and stream
database changes through these slots to a client like pg_recvlogical.
Logical slots are created at the database level and support
replication connections to a single database.
mind possible problems eg https://dba.stackexchange.com/questions/173267/aws-rds-postgres-logical-replication
I have a problem. Currently I have a 1 TB Postgres 9.6 database which is backed up with Barman with streaming.
What I need:
A replication from the production/master to the slave server:
On which I can write, I don’t care if the written data on the replica
is not sent to the master server
Which can be configured almost in real time or with little delay
On which I can use dump without locking the master database
As said above I am using Barman for backing up. However I am not able to find out how I can build a replica from Barman which is sync by the master. It was set up by a someone else and i'm not sure its the right solution for what I need.
My questions:
Is Barman the good tool for what I want ?
If no. Which tools would you suggest to me ?
If yes. Do you know how to build replica from Barman which is
sync by the master ? Could you please explain to me how to do
it?
Thanks
in master-slave mode, you can't write on slave
if you want to write on replica to you should probably use something like this
also you can make sure all of your writes on master also written on replica via synchronous-wal-streaming feature
via this feature, before wiritng on master , first master makes sure write was written successfully on replica
except for writing on slave part , barman looks a fit tool for you
writing on slave is a uncommon thing in postgresql
Currently I have 1 postgres instance which is starting to receive too much load and want create a cluster of 2 postgres nodes.
From reading the documentation for postgres and pgpool, it seems like I can only write to a master and read from a slave or run parallel queries.
What I'm looking for is a simple replication of a database but with master/slave based on which table is being updated. Is this possible? Am i missing it somewhere in the documentation?
e.g.
update users will be executed on server1 and replicated to server2
update big_table will be executed on server2 and replicated back to server1
What you are looking for is called MASTER/MASTER replication. This is supported natively (without PgPool) since 9.5. Note, that it's an "eventually consistent" architecture, so your application should be aware of possible temporary differences between the two servers.
See PG documentation for more details and setup instructions.
I want to migrate our postgres db from heroku to our own postgres on AWS.
I have tried using pg_dump and pg_restore to do the migration and it works; but it takes a really long time to do this. Our database size is around 20GB.
What's the best way to do the migration with minimal downtime?
If you mean AWS RDS PostgreSQL:
pg_dump and pg_restore
I know you don't like it, but you don't really have other options. With a lot of hoop jumping you might be able to do it with Londiste or Slony-I on a nearby EC2 instance, but it'd be ... interesting. That's not the most friendly way to do an upgrade, to say the least.
What you should be able to do is ship WAL into RDS PostgreSQL, and/or stream replication logs. However Amazon don't support this.
Hopefully Amazon will adopt some part of 9.4's logical replication and logical changeset extraction features, or better yet the BDR project - but I wouldn't hold my breath.
If you mean AWS EC2
If you're running your own EC2 instance with Pg, use replication then promote the standby into the new master.
I know the way to set up a Master/Slave DB in Postgres is having 2 DB servers, but unfortunately i have only one server for now.
How can i mirror my production db into another "backup db" in "real_time"? I want to give access to another user to the mirrored db, so even if he does something there it will not affect production.
Nothing stops you setting up hot standby streaming replication, or another replication option like Londiste, between two PostgreSQL instances on the same computer.
The two copies of PostgreSQL must use different ports, but that's the only real restriction.
How to set up the second PostgreSQL instance depends on your operating system and how you installed PostgreSQL, which you have not mentioned.
You'll want to use streaming replication with hot standby if you want a read-only replica. If you want it to be read/write, then you can do a one-off copy of the database with pg_basebackup and not keep them in sync after that. Or you can use a tool like Londiste to replicate changes selectively.
You can run multiple instances of PostgreSQL on the same computer, by using different ports.