I have a postgres table (300Mb size) which is logically replicated to another server. Until I've made some changes everything was perfectly good. Then master started to grow (up to 2,5 Gb at rate 15 mb at 5 minutes roughly). I tried to tune WAL settings and do a WAL cleanup, but it didn't help.
What I have done before this issue was discovered:
Rebuilt a materialized view dependent on master table a lot of times (and it is a heavy CPU consuming operation)
Added a new column on master table and slave table
Added a rule on inserts (copy a value from jsonfield to charfield)
What could have caused this issue?
I'm still unsure what caused the issue, but probably it was adding a new column. After restoring a backup and recreating publication and subscription everything worked pretty good.
Related
On my primary I ran a VACUUM then an ANALYZE on all databases, then when I check pg_stat_user_tables, the last_analyze column shows a current timestamp which is great.
When I check my replication instance, there are no values in the last_analyze column. I was assuming this timestamp would also eventually populate? Is this known behaviour?
The reason I ask is that after that VACUUM/ANALYZE on the primary, I'm running into some extremely slow queries on the replication instance. I ran an EXPLAIN plan prior to the VACUUM/ANALYZE on a query and it ran in 5 seconds... now it's taking 65 seconds. The EXPLAIN shows it's not using a lot of indexes that it should be.
PostgreSQL has two different stats systems. One records data about the distribution of values in the columns, this is transactional. It propagates to the replica via the WAL.
The other system records data about turn over on the tables and data on when the last vac/an was done. This system is used to determine when to schedule new vac/an (to prevent the first system from getting too out of date). This one is not transactional, and does not propagate to the replica.
So the replica has the latest column value distribution statistics (as soon as the WAL replays, anyway), but it doesn't know how recent they are.
We have on RDS a main Postgres server and a read replica.
We constantly write and update new data for the last couple of days.
Reading from the read-replica works fine when looking at older data but when trying to read from the last couple of days, where we keep updating the data on the main server, is painfully slow.
Queries that take 2-3 minutes on old data can timeout after 20 minutes when querying data from the last day or two.
Looking at the monitors like CPU I don't see any extra load on the read replica.
Is there a solution for this?
You are accessing over 65 buffers for ever 1 visible row found in the index scan (and over 500 buffers for each row which is returned by the index scan, since 90% are filtered out by the mmsi criterion).
One issue is that your index is not as well selective as it could be. If you had the index on (day, mmsi) rather than just (day) it should be about 10 times faster.
But it also looks like you have a massive amount of bloat.
You are probably not vacuuming the table often enough. With your described UPDATE pattern, all the vacuum needs are accumulating in the newest data, but the activity counters are evaluated based on the full table size, so autovacuum is not done often enough to suit the needs of the new data. You could lower the scale factor for this table:
alter table simplified_blips set (autovacuum_vacuum_scale_factor = 0.01)
Or if you partition the data based on "day", then the partitions for newer days will naturally get vacuumed more often because the occurrence of updates will be judged against the size of each partition, it won't get diluted out by the size of all the older inactive partitions. Also, each vacuum run will take less work, as it won't have to scan all of the indexes of the entire table, just the indexes of the active partitions.
As suggested, the problem was bloat.
When you update a record in an ACID database the database creates a new version of the record with the new updated record.
After the update you end with a "dead record" (AKA dead tuple)
Once in a while the database will do autovacuum and clean the table from the dead tuples.
Usually the autovacuum should be fine but if your table is really large and updated often you should consider changing the autovacuum analysis and size to be more aggressive.
I have a large and fast-growing PostgreSQL table (166gb Index and 72 GB database). And I want to set up a logical replication of this table. Version 11.4 on both sides.
I'm trying to do it for 2 weeks, but the only thing I have is infinite syncing and growing table size on the replica (already 293 Gb index and 88Gb table, more than original, and there are no errors in the log).
I also have tried to take a dump, restore it and start syncing - but got errors with existing primary keys.
Backend_xmin value of replication stats is changing once in a week, but the sync state is still "startup". The network connection between those servers is not used at all (they are in the same datacenter), actual speed like 300-400Kb (looks like it's mostly streaming part of replication process).
So the question is How to set up a Logical replication of large and fast-growing table properly, is it possible somehow? Thank you.
I'm trying to do it for 2 weeks, but the only thing I have is infinite syncing and growing table size on the replica (already 293 Gb index and 88Gb table, more than original, and there are no errors in the log).
Drop the non-identity indexes on the replica until after the sync is done.
The problem is exactly the same
Check the logs I found the following error:
ERROR: could not receive data from WAL stream: ERROR: canceling statement due to statement timeout
Due to large tables, replication fell off by timeout
By increasing the timeouts, the problem went away
PS Ideally, it would be cooler to set up separate timeouts for replication and for the main base.
I am trying to run a PostgreSQL (10.6) logical replication between two servers on one table only. The table has Id(int2) as a primary key. This is intentional and the table acts as a rolling window for some IoT time series data. It is heavy on writing on the Master node. The whole table has roughly 10 minutes worth of sensor data. And that is the design we like to keep.
Logical replication between Master and Replica nodes works great until there is a network outage lasting more than 1 hour. In the meantime, PostgreSQL on Master node is collecting WAL files with step by step insert/update on the table. So effectively WAL files might contain even hours of data which we are not interested in and they take forever to replay, step by step, from Master -> Replica when the connection restores. It is basically replaying records which long time don't exist in the database table!
How can I set it up so that only relevant data got replayed? If that is hard to do is there a way to throw away WAL files older than, say 10 minutes, so that they simply won't be sent?
I have tried to play with postgresql.conf settings. I am not sure if there is a flag I can limit the WAL files storing in a case of replication slot disconnect.
This is how the table looks like:
CREATE TABLE iot_ts (id int2 not null, time timestamp(0) not null, value real, primary key(id));
I would like to have a logical replication of such table set up so that when a long internet outage occurs the restoration is fast and contains only the most recent data.
We have slony replication set up, and the replication on the slave has fallen behind by 10 days.
On investigating I noticed that the sl_log_1 table has 25K records, but the sl_log_2 table has over 100 million rows, and they keep going up.
How do I go about troubleshooting this?
I am a newbie to slony, and would appreciate all the help that I can get
Check if all Slony processes are running, or better - restart them. After restarting check the logs, that can be caused sometimes by some DDL query, that was performed by hand on master. If there are some serious errors on slaves, or the lag is not falling down too fast, you could always reinit the replication (but beware, that will delete all data from all tables on the slave databases).