I set up slony to replicate 3 tables from one opensuse pc (master node) to another opensuse pc (slave node). It works well at first. After running for few days, it suddenly come out the error message of -
ERROR: Slony-I: Table euprofiles is replicated and cannot be modified on a subscriber node - role=0
euprofiles is one of the tables being replicated by slony.
I know that this message may occur if you are trying to write to the table of slave node. But here I am writing to master node only.
Does anyone see similar problem?
Never had this happen. Are you certain you're connecting to the db you think you're connecting to? Slony may be a bit difficult to setup and such, but it doesn't just randomly decide a master is now a slave.
If you psql into the two databasesand do \d euprofiles on each what do they say? The source table should have something like this at the end:
Triggers:
_slony_www_logtrigger_228 AFTER INSERT OR DELETE OR UPDATE ON users FOR EACH ROW EXECUTE PROCEDURE _slony_www.logtrigger('_slony_www', '228', 'kvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv')
and the destination table should have something like this:
Triggers:
_slony_www_denyaccess_228 BEFORE INSERT OR DELETE OR UPDATE ON users FOR EACH ROW EXECUTE PROCEDURE _slony_www.denyaccess('_slony_www')
If they both look like this last trigger there's some problem. But I'm betting you're just connecting to the wrong server. Let's hope it's that simple.
Related
First of all i do not know if is possible what i want to achive, will describe below:
I have access of a remote PostgreSQL that holds the data i need (let's say Remote PostgreSQL 1)
have just credentials to read from database
What i want to achive is to create a localt PostgreSQL on my machine (let's say Local PostgreSQL 2)
Want to copy and check for missing data from Remote PostgreSQL 1 to Local PostgreSQL 2 in real time or at list to copy at the end of the day data
The scenario will work perfect with replication but the issue is that Remote PostgreSQL 1 is not owned by me and can not be used as real time DB, because of this trying to find a solution to get all the data from Remote PostgreSQL 1 to Local PostgreSQL 2.
Could be the following scenarios:
first time setup to downlaod all the database from Remote PostgreSQL 1 to Local PostgreSQL 2
after first time setup to check what data came new inside and add them in Local PostgreSQL 2
Would be great if this could be done on OS level on UBUNTU. My application is written in python 3 i could do scripts to do all this job but i speak of 100 millions of raws per table huge amount of data. Think will be problems to get everything from database and start to check everything what is missing and not.
Any ideas would be great.
If the owner of Remote Database 1 won't cooperate with you other than to give you read only access to the tables, then you don't have any efficient options. If the remote owner does or can be convinced to keep insertion/modification timestamp columns in all the tables (although then deletions would be a problem), or an in-database "audit" log for all the tables, you could use those. I think you have an organizational/political problem rather a programming problem.
I have two DB instances with postgresql 10.
First instance working with wal_level = logical, have ALL TABLE publisher. Second - have subscription to the publisher, described above.
All data from the Master DB tables are successfully sending to the replica.
There is only one issue for me - when my App add a new table on Master - I need to add the same table to the replica (and run REFRESH PUBLICATION).
My question - is there a way for replica DB to create new tables automatically?
Schema changes aren't replicated. You can see that in the documentation, at Postgresql 10 Logical Replication: Subscription
The way I handle this is to make all schema changes through a script, and to write the script so that it executes the change commands twice: once on the primary database, and once on the replica.
This may be a very basic question, but I'm struggling with it. I'm attempting to set up logical decoding between 2 separate servers.
On the Master, I've gotten it setup so that the changes that I make to a table (INSERT, UPDATE, DELETE) are sent to my logical replication slot, and I can see the changes using the pg_logical_slot_get/peek_changes functions - all on my Master server.
On the Slave, I'm attempting to run the pg_revclogical command (using command prompt), however, I can't seem to get it to receive the changes made on the Master. I've realized that no where have I told the 2 to communicate to each other. I tried to define the host as the Master to tell the replication process that I have to pull the changes from the Master to the Slave.
I've made all of the required changes to the postgresql.conf (wal_level=logical; max_replication_slots=3; max_wal_senders=3) and pg_hba.conf (whatever it told me to do to fix errors) files on both Master and Slave. The pg_recvlogical command and the resulting error message are below. Could someone please help me get my Master and Slave communicating?
pg_recvlogical --start --slot=wendy_test --plugin=test_decoding --dbname=testdb --file=C:\Logical_Decoding_Test.log --username=dbaadmin --host=127.##.##.### --port=5432
When I execute this command on the Slave, I don't receive any error messages, but it doesn't return to a command prompt either? While it's doing its thing, I can check the database and see that the wendy_test slot is active, so it appears to be doing something. However, when I INSERT a row into the table on my Master, nothing happens on my Slave (even after 5 minutes).
I can add all of the code that I have used to create the replication slots, populate the queue, etc. I was trying to limit the amount of reading and figured those to be more basic. Also, my end game is to transition this to have our AWS RDS system as our Slave, so any recommendations for that is appreciated as well. THank you for your time.
Fairly new to Postgresql, but have to get replication set up. I settled on BDR, and it works fine in the local demo, but on distributed machines it starts to get problematic, mostly because I have no real clue what the hell I am doing, and I cry myself to sleep pining for MySQL. I've gotten BDR working accross multiple servers, almost. When I run:
SELECT bdr.bdr_node_join_wait_for_ready();
on the joining nodes it hangs. This happens on both DB2 and DB3. DB1 returns a valid response. Researching this I came across the bdr_init_copy command, which apparently does everything I have been doing by hand, and then some. so tried that out. Now, when I run:
/usr/lib/postgresql/9.4/bin/bdr_init_copy -d "host=192.168.1.10 dbname=demo3" --local-dbname="host=192.168.1.23 dbname=demo3" -n db2 -D bdr_data
I get
bdr_init_copy: starting ...
Getting remote server identification ...
Detected 2 BDR database(s) on remote server
Updating BDR configuration on the remote node:
demo2: creating replication slot ...
demo2: creating node entry for local node ...
demo3: creating replication slot ...
demo3: creating node entry for local node ...
Creating base backup of the remote node...
63655/63655 kB (100%), 1/1 tablespace
Creating restore point on remote node ...
Bringing local node to the restore point ...
And it sits there. I am assuming that it is the same cause for both issues. as far as I can tell there are no log entries created on the local node (db2) but the following is present on the remote(db1)
2016-10-12 22:38:43 UTC [20808-1] postgres#demo2 LOG: logical decoding found consistent point at 0/5001F00
2016-10-12 22:38:43 UTC [20808-2] postgres#demo2 DETAIL: There are no running transactions.
2016-10-12 22:38:43 UTC [20808-3] postgres#demo2 STATEMENT: SELECT pg_create_logical_replication_slot('bdr_17163_6340711416785871202_2_17163__', 'bdr');
2016-10-12 22:38:43 UTC [20811-1] postgres#demo3 LOG: logical decoding found consistent point at 0/5002090
2016-10-12 22:38:43 UTC [20811-2] postgres#demo3 DETAIL: There are no running transactions.
2016-10-12 22:38:43 UTC [20811-3] postgres#demo3 STATEMENT: SELECT pg_create_logical_replication_slot('bdr_17939_6340711416785871202_2_17939__', 'bdr');
2016-10-12 22:38:44 UTC [20812-1] postgres#demo3 LOG: restore point "bdr_6340711416785871202" created at 0/50022A8
2016-10-12 22:38:44 UTC [20812-2] postgres#demo3 STATEMENT: SELECT pg_create_restore_point('bdr_6340711416785871202')
Any help out there?
Right, just had this issue and none of the other forums were any help. Some of them even say things like it is okay for the new node to report its status as "o" and the other nodes report the new server status as "i" because "this is just a bug and it's fine". It's NOT OKAY. The new server could receive replication updates, but no primary updates were possible on the new server. The key to solving this problem is to crank up the logging on the server you are joining to (not the new one). On the new server logs, you might see things like: 08006: could not receive data from client: Connection reset by peer, which is not very helpful, and will have you checking firewalls, etc. The real money shot will come from the source server logs when they have logs saying something like: no free replication state could be found for 11, increase max_replication_slots What's probably happened is you either have too many servers in your cluster for the default settings or, more likely, there is some junk left over from old hosts.
You need to clean things up ... ON EVERY SERVER IN THE EXISTING CLUSTER (NB!). Start by getting a list of things on the existing cluster:
select * from bdr.bdr_nodes order by node_sysid;
THEN, check the following:
select conn_sysid,conn_dboid from bdr.bdr_connections order by conn_sysid;
.. if you see old entries (that don't contain node_sysid from the first query) then delete
eg. delete from bdr.bdr_connections where conn_sysid='<from-first-query>';
select * from pg_replication_slots order by slot_name;
.. if you see old entries that don't contain an active sysid then delete
.. NB, use the function, DO NOT do a "delete from"
eg. select pg_drop_replication_slot('bdr_17213_6574566740899221664_1_17213__');
select * from pg_replication_identifier order by riname;
.. if you see old entries that don't contain an active sysid then delete
.. NB, use the function, DO NOT do a "delete from"
select pg_replication_identifier_drop('bdr_6443767151306784833_1_17210_17213_');
With any luck, after you've done this on every node, you will see your new server's BDR status go to 'r'. As you clean up each host, you should notice that the logs "08006: could not receive data from client: Connection reset by peer", matching the conn-sysid of the server you've just cleaned up, stop happening. Good luck
Currently I have 1 postgres instance which is starting to receive too much load and want create a cluster of 2 postgres nodes.
From reading the documentation for postgres and pgpool, it seems like I can only write to a master and read from a slave or run parallel queries.
What I'm looking for is a simple replication of a database but with master/slave based on which table is being updated. Is this possible? Am i missing it somewhere in the documentation?
e.g.
update users will be executed on server1 and replicated to server2
update big_table will be executed on server2 and replicated back to server1
What you are looking for is called MASTER/MASTER replication. This is supported natively (without PgPool) since 9.5. Note, that it's an "eventually consistent" architecture, so your application should be aware of possible temporary differences between the two servers.
See PG documentation for more details and setup instructions.