postgresql subscription not working for schemas other than public - postgresql

I'm trying to create a logical replication with two local postgresql servers (node1: port 5434, node2: port 5435).
I could successfully create publication and subscription on node1 and node2 for a table in public schema.
Node1:
CREATE PUBLICATION my_pub FOR TABLE t1;
GRANT SELECT ON t1 TO repuser;
Node2:
CREATE SUBSCRIPTION my_sub CONNECTION 'host=localhost port=5434 dbname=pub user=repuser password=password' PUBLICATION my_pub;
Node2 public.t1 replicates all data in node1 public.t1.
However, my problem is when I create publication and subscription with same code but in different schema, node2 fail to replicate.
Below is output of some pg_catalog query :
Node1:
pub=# select * from pg_catalog.pg_publication_tables;
pubname | schemaname | tablename
----------+------------+-----------
my_pub | public | t1
cdl_test | cdl | t1
pub_test | test | t1
Node2:
sub=# \dRs
List of subscriptions
Name | Owner | Enabled | Publication
--------------+----------+---------+-------------
cdl_sub_test | postgres | t | {cdl_test}
my_sub | postgres | t | {my_pub}
sub_test | postgres | t | {pub_test}
sub=# select * from pg_catalog.pg_replication_origin;
roident | roname
---------+----------
2 | pg_18460
1 | pg_18461
3 | pg_18466
sub=# select * from pg_catalog.pg_subscription_rel ;
srsubid | srrelid | srsubstate | srsublsn
---------+---------+------------+------------
18461 | 16386 | r | 0/3811C810
18466 | 18463 | d |
18460 | 18456 | d |
As it is shown in select * from pg_catalog.pg_subscription_rel, two subscription for test and cdl schema are in d(data is being copied) state.
Any recommendation on how to go about this problem or diagnose why the problem occurs?
As jjanes has suggested, a snippet of the log file is shown below:
2022-01-17 16:05:25.165 PST [622] WARNING: out of logical replication worker slots
2022-01-17 16:05:25.165 PST [622] HINT: You might need to increase max_logical_replication_workers.
2022-01-17 16:05:25.168 PST [970] LOG: logical replication table synchronization worker for subscription "cdl_sub_test", table "t1" has started
2022-01-17 16:05:25.245 PST [970] ERROR: could not start initial contents copy for table "cdl.t1": ERROR: permission denied for schema cdl
2022-01-17 16:05:25.247 PST [471] LOG: background worker "logical replication worker" (PID 970) exited with exit code 12022-01-17 16:05:25.797 PST [488] postgres#sub LOG: statement: /*pga4dash*/
It seems like the subscriber doesn't have permission to read cdl schema in publisher even after I gave permission for SELECT ON cdl.t1 TO repuser;.

You have to give the user repuser permission to read the table that should be replicated. That also requires USAGE permission on the schema that contains the table.

Related

slow running postgres11 logical replication from EC2 to RDS

I'm trying to move a Postgres (11.6) database on EC2 to RDS (postgres 11.6). I started replication last a couple nights ago and have now noticed the replication has slowed down considerably when I see how fast the database size in increasing on the subscriber SELECT pg_database_size('db_name')/1024/1024. Here are some stats of the environment:
Publisher Node:
Instance type: r5.24xlarge
Disk: 5Tb GP2 with 16,000 PIOPs
Database Size w/ pg_database_size/1024/1024: 2,295,955 mb
Subscriber Node:
Instance type: RDS r5.24xlarge
Disk: 3Tb GP2
Here is the current DB size for the subscriber and publisher:
Publisher:
SELECT pg_database_size('db_name')/1024/1024 db_size_publisher;
db_size_publisher
-------------------
2295971
(1 row)
Subscriber:
SELECT pg_database_size('db_name')/1024/1024 as db_size_subscriber;
db_size_subscriber
--------------------
1506348
(1 row)
The difference is still about 789GB left to replicate it seems like and I've noticed that the subsriber db is increasing at a rate of about 250kb/sec
db_name=> SELECT pg_database_size('db_name')/1024/1024, current_timestamp;
?column? | current_timestamp
----------+-------------------------------
1506394 | 2020-05-21 06:27:46.028805-07
(1 row)
db_name=> SELECT pg_database_size('db_name')/1024/1024, current_timestamp;
?column? | current_timestamp
----------+-------------------------------
1506396 | 2020-05-21 06:27:53.542946-07
(1 row)
At this rate, it would take another 30 days to finish replication, which makes me think I've set something up wrong.
Here are also some other stats from the publisher and subscriber:
Subscriber pg_stat_subscription:
db_name=> select * from pg_stat_subscription;
subid | subname | pid | relid | received_lsn | last_msg_send_time | last_msg_receipt_time | latest_end_lsn | latest_end_time
-------+----------------+-------+-------+---------------+-------------------------------+-------------------------------+----------------+-------------------------------
21562 | rds_subscriber | 2373 | 18411 | | 2020-05-20 18:41:54.132407-07 | 2020-05-20 18:41:54.132407-07 | | 2020-05-20 18:41:54.132407-07
21562 | rds_subscriber | 43275 | | 4811/530587E0 | 2020-05-21 06:15:55.160231-07 | 2020-05-21 06:15:55.16003-07 | 4811/5304BD10 | 2020-05-21 06:15:54.931053-07
(2 rows)
At this rate...it would take weeks to complete....what am I doing wrong here?

How to understand/debug bad index result on slave instance

I have the following schema and 2 postgresql instances with a slave instance replicating a master instance.
CREATE TABLE t (id serial PRIMARY KEY, c text);
CREATE INDEX ON t (upper(c));
I get this incorrect result on the slave instance.
# SELECT id, c, upper(c), upper(c) = upper('FOO') FROM t WHERE id IN (123, 456);
id | c | upper | ?column?
-----+-----+-------+----------
123 | Foo | FOO | t
456 | foo | FOO | t
(2 rows)
# SELECT id, c, upper(c), upper(c) = upper('FOO') FROM t WHERE upper(c) = upper('FOO');
id | c | upper | ?column?
----+---+-------+----------
(0 rows)
The second query should return the same rows as the first query.
However, the result is correct on the master instance.
# SELECT id, c, upper(c), upper(c) = upper('FOO') FROM t WHERE id IN (123, 456);
id | c | upper | ?column?
-----+-----+-------+----------
123 | Foo | FOO | t
456 | foo | FOO | t
(2 rows)
# SELECT id, c, upper(c), upper(c) = upper('FOO') FROM t WHERE upper(c) = upper('FOO');
id | c | upper | ?column?
-----+-----+-------+----------
123 | Foo | FOO | t
456 | foo | FOO | t
(2 rows)
Using EXPLAIN on the second query, I can see that it's using the index as expected, so I suspect the index data is somehow incorrect on the slave instance. Doing a REINDEX on the master instance does not resolve the issue and doing it on the slave instance is not possible because of the replication.
Is it possible that the index data is correct on the master instance and incorrect on the slave instance? How to further debug the issue?
UPDATE: This is the query plan of the second query on both the master and the slave instance
Index Scan using t_upper_idx on t (cost=0.43..8.46 rows=1 width=60)
Index Cond: (upper((c)::text) = 'FOO'::text)
There are ~3M rows in the t table.
UPDATE: Server version is server 11.4 (Debian 11.4-1.pgdg90+1)) on the master, and server 11.7 (Debian 11.7-0+deb10u1)) on the slave.

Postgres error on partitioned table (ERROR 42703 attribute 4 of type X has been dropped)

I'm attaching a partition to one of my partitioned tables and I'm getting an error I don't understand.
This is how I created my partition:
CREATE TABLE my_table_201906_partition (LIKE my_table_000000_partition
INCLUDING ALL)
This CREATE TABLE worked ok. my_table_000000_partition is the default partition of its parent my_table.
And this is how I'm trying to attach it to the table:
ALTER TABLE my_table ATTACH PARTITION my_table_201906_partition
FOR VALUES FROM ('2019-06-01 00:00:00+00') TO ('2019-07-01 00:00:00+00');
The full error is:
INFO: 00000: partition constraint for table "my_table_201906_partition" is implied by existing constraints
LOCATION: QueuePartitionConstraintValidation, tablecmds.c:14540
ERROR: 42703: attribute 4 of type my_table_000000_partition has been dropped
LOCATION: CheckVarSlotCompatibility, execExprInterp.c:1880
Time: 10.571 ms
Any idea what the error can mean and how to solve it?
Edit:
Result of query suggested by amil
This is the query requested by #amil:
select attrelid::regclass, attnum, attname
from pg_attribute
where attrelid in ('my_table'::regclass::oid,
'my_table_000000_partition'::regclass::oid)
and attnum > 0;
In the following results I mangled the field names, but I took care of calling some_field_1 to the same field of both tables. Field my_timestamp is the field I'm using for range partitioning.
attrelid | attnum | attname
-------------------------------+--------+------------------------------
my_table_000000_partition | 1 | id
my_table_000000_partition | 2 | cust_id
my_table_000000_partition | 3 | some_field_1
my_table_000000_partition | 4 | ........pg.dropped.4........
my_table_000000_partition | 5 | my_timestamp
my_table_000000_partition | 6 | ........pg.dropped.6........
my_table_000000_partition | 7 | some_field_2
my_table_000000_partition | 8 | some_field_3
my_table_000000_partition | 9 | some_field_4
my_table_000000_partition | 10 | some_field_5
my_table_000000_partition | 11 | some_field_6
my_table_000000_partition | 12 | some_field_7
my_table_000000_partition | 13 | some_field_8
my_table_000000_partition | 14 | some_field_9
my_table_000000_partition | 15 | some_field_10
my_table_000000_partition | 16 | some_field_11
my_table | 1 | id
my_table | 2 | cust_id
my_table | 3 | some_field_1
my_table | 4 | my_timestamp
my_table | 5 | some_field_2
my_table | 6 | some_field_3
my_table | 7 | some_field_4
my_table | 8 | some_field_5
my_table | 9 | some_field_6
my_table | 10 | some_field_7
my_table | 11 | some_field_8
my_table | 12 | some_field_9
my_table | 13 | some_field_10
my_table | 14 | some_field_11
(30 rows)
#MondoKin's answer already gives a workaround, but let me add an answer about what the error means and why it's caused in this case.
The error seen here is an internal system error thrown by the Postgres executor when it encounters an expression that's built using invalid metadata about a table's column, which in this case is a column value expression that references a dropped attribute of my_table_000000_partition.
In most cases involving such internal consistency checks (almost all of them), users are not supposed to expect errors being thrown by them. Such internal errors could only result from a corrupted internal state (on-disk or in-memory) or a bug of Postgres', which in this case, turned out to be the latter.
Thankfully, #MondoKin reported this issue to Postgres bugs mailing list. See this link to the mailing list archive, if interested: https://postgr.es/m/flat/15873-8c61945d6b3ef87c%40postgresql.org
I found a workaround to attach my new partition: you need to detach the problematic partition before trying to attach the new partition, then reattaching the problematic partition:
CREATE TABLE my_table_201906_partition (LIKE my_table_000000_partition
INCLUDING ALL);
ALTER TABLE my_table DETACH PARTITION my_table_000000_partition;
ALTER TABLE my_table
ATTACH PARTITION my_table_201906_partition
FOR VALUES FROM ('2019-06-01 00:00:00+00') TO ('2019-07-01 00:00:00+00')
ALTER TABLE my_table ATTACH PARTITION my_table_000000_partition DEFAULT;
This way, the old partition's dropped column doesn't get into the way of the ATTACH PARTITION ... FOR RANGE command. Which means I don't need to vacuum full the 273GiB table.
The only issue with this workaround is that you need to do it for every partition that you want to attach (until the bug is solved)
Edit:
I thought VACUUM FULL my_table_000000_partition would permanently solve the issue by totally removing the "deleted" columns, but no, the error persisted.

django-celery-results won't recieve results

I have celery setup and working together with django. I have some periodic tasks that run. The celery log shows that the tasks are executed and that they return something.
[2017-03-26 14:34:27,039: INFO/MainProcess] Received task: my_webapp.apps.events.tasks.clean_outdated[87994396-04f7-452b-a964-f6bdd07785e0]
[2017-03-26 14:34:28,328: INFO/PoolWorker-1] Task my_webapp.apps.events.tasks.clean_outdated[87994396-04f7-452b-a964-f6bdd07785e0] succeeded in 0.05246314400005758s: 'Removed 56 event(s)
| Removed 4 SGW(s)
'
But the result is not showing up on django-celery-results admin page.
These are my settings:
CELERY_BROKER_URL = os.environ.get('BROKER_URL')
CELERY_ACCEPT_CONTENT = ['json']
CELERY_TASK_SERIALIZER = 'json'
CELERY_RESULT_SERIALIZER = 'json'
CELERY_TIMEZONE = 'Europe/Stockholm'
CELERY_RESULT_BACKEND = 'django-cache'
CELERYBEAT_SCHEDULER = 'djcelery.schedulers.DatabaseScheduler'
CELERY_RESULT_DB_SHORT_LIVED_SESSIONS = True # Fix for low traffic sites like this one
I have also tried setting CELERY_RESULT_BACKEND = 'django-db'. I know the migrations are made (when using those settings), the table exists in the database:
my_webapp=> \dt
List of relations
Schema | Name | Type | Owner
--------+--------------------------------------+-------+----------------
...
public | django_celery_beat_crontabschedule | table | my_webapp
public | django_celery_beat_intervalschedule | table | my_webapp
public | django_celery_beat_periodictask | table | my_webapp
public | django_celery_beat_periodictasks | table | my_webapp
public | django_celery_results_taskresult | table | my_webapp
...
(26 rows)
Google won't give me much help, most answers is about old libraries like djcelery. Any idea how the get the results in the table?

Can't use PGPool with Amazon RDS Postgres

I have a Postgres 9.4 RDS instance with Multi-AZ, and there's a slave, read-only replica.
Up to this point the load balancing was made in the business layer of my app, but it's inefficient, and I was hoping to use PGPool, so the app interacts with a single Postgres connection.
It turns out that using PGPool has been a pain in the ass. If I set it to act as a load balancer, simple SELECT queries throw errors like:
SQLSTATE[HY000]: General error: 7
message contents do not agree with length in message type "N"
server sent data ("D" message)
without prior row description ("T" message)
If I set it to act in a master/slave mode with stream replication (as suggested in Postgres mail list) I get:
psql: ERROR: MD5 authentication is unsupported
in replication and master-slave modes.
HINT: check pg_hba.conf
Yeah, well, pg_hba.conf if off hands in RDS so I can't alter it.
Has anyone got PGPool to work in RDS? Are there other tools that can act as middleware to take advantage of reading replicas in RDS?
I was able to make it work here are my working config files:
You have to use md5 authentication, and sync the username/password from your database to the pool_passwd file. Also need enable_pool_hba, load_balance_mode, and master_slave_mode on.
pgpool.conf
listen_addresses = '*'
port = 9999
pcp_listen_addresses = '*'
pcp_port = 9898
pcp_socket_dir = '/tmp'
listen_backlog_multiplier = 1
backend_hostname0 = 'master-rds-database-with-multi-AZ.us-west-2.rds.amazonaws.com'
backend_port0 = 5432
backend_weight0 = 0
backend_flag0 = 'ALWAYS_MASTER'
backend_hostname1 = 'readonly-replica.us-west-2.rds.amazonaws.com'
backend_port1 = 5432
backend_weight1 = 999
backend_flag1 = 'ALWAYS_MASTER'
enable_pool_hba = on
pool_passwd = 'pool_passwd'
ssl = on
num_init_children = 1
max_pool = 2
connection_cache = off
replication_mode = off
load_balance_mode = on
master_slave_mode = on
pool_hba.conf
local all all md5
host all all 127.0.0.1/32 md5
pool_passwd
username:md5d51c9a7e9353746a6020f9602d452929
to update pool_password you can use pg_md5 or
echo username:md5`echo -n usernamepassword | md5sum`
username:md5d51c9a7e9353746a6020f9602d452929 -
Output of running example:
psql --dbname=database --host=localhost --username=username --port=9999
database=> SHOW POOL_NODES;
node_id | hostname | port | status | lb_weight | role | select_cnt | load_balance_node | replication_delay
---------+-------------------------------------------------+------+--------+-----------+---------+------------+-------------------+-------------------
0 | master-rds-database.us-west-2.rds.amazonaws.com | 8193 | up | 0.000000 | primary | 0 | false | 0
1 | readonly-replica.us-west-2.rds.amazonaws.com | 8193 | up | 1.000000 | standby | 0 | true | 0
database=> select now();
node_id | hostname | port | status | lb_weight | role | select_cnt | load_balance_node | replication_delay
---------+-------------------------------------------------+------+--------+-----------+---------+------------+-------------------+-------------------
0 | master-rds-database.us-west-2.rds.amazonaws.com | 8193 | up | 0.000000 | primary | 0 | false | 0
1 | readonly-replica.us-west-2.rds.amazonaws.com | 8193 | up | 1.000000 | standby | 1 | true | 1
database=> CREATE TABLE IF NOT EXISTS tmp_test_read_write ( data varchar(40) );
CREATE TABLE
database=> INSERT INTO tmp_test_read_write (data) VALUES (concat('',inet_server_addr()));
INSERT 0 1
database=> select data as master_ip,inet_server_addr() as replica_ip from tmp_test_read_write;
master_ip | replica_ip
--------------+---------------
172.31.37.69 | 172.31.20.121
(1 row)
You can also see from the logs id does both databases:
2018-10-16 07:56:37: pid 124528: LOG: DB node id: 0 backend pid: 21731 statement: CREATE TABLE IF NOT EXISTS tmp_test_read_write ( data varchar(40) );
2018-10-16 07:56:47: pid 124528: LOG: DB node id: 0 backend pid: 21731 statement: INSERT INTO tmp_test_read_write (data) VALUES (concat('',inet_server_addr()));
2018-10-16 07:56:52: pid 124528: LOG: DB node id: 1 backend pid: 24890 statement: select data as master_ip,inet_server_addr() as replica_ip from tmp_test_read_write;
Notice the insert used ip_address of master, and the next select used ip_address of the read only replica.
I can update after more testing, but psql client testing looks promising.
There is Citus(pgShard) that is supposed to work with standard Amazon RDS instances. It has catches though. You will have a single point of failure if you use the open source version. It's coordinator node is not duplicated.
You can get a fully HA seamless fail over version of it but you have to buy the enterprise licence, but it is CRAZY expensive. It will easily cost you $50,000 to $100,000 or more per year.
Also they are REALLY pushing their cloud version now which is even more insanely expensive.
https://www.citusdata.com/
I have also heard of people using HAProxy to balance between Postgres or MySql nodes.