PostgresSQL 14 materialized views not updating

PostgresSQL 14 materialized views not updating - postgresql

I have 20+ materialized views that I use as some sort of pre-calculated cache for a number of operations. Each night at 3 AM I do automated sync with an external source, followed by a series of materialized views refreshing.
This is where it gets weird. Each morning, at the beginning of the workday, I execute
SELECT * FROM dbo.materialized_view
and then execute the actual materialized view's query (specified during the CREATE MATERIALIZED VIEW).
I get different results.
The database source does not change after the sync process completes at around 3:10 AM. Originally, REFRESH SQL queries were getting executed from within the same Java sync process after the transaction commits, but since that didn't work, I created a cron job at 4 AM to do the same. That still did not help.
As soon as I manually refresh views in the morning (around 6-7 AM), the results update
Is there something incredibly funky I am missing about how materialized views refresh?
Update:
Here is one of the materialized views I refresh:
create materialized view dbo.ind_newborn_daily as
SELECT dt.date_time::DATE AS date,
s.source_id AS staff_id,
b.status,
COUNT(DISTINCT b.id) AS count
FROM dbo.newborn nb
JOIN dbo.date_time dt ON nb.created_datetime_id = dt.id
JOIN dbo.staff s ON nb.staff_id = s.id
JOIN dbo.baby b ON nb.baby_id = b.id
WHERE nb.current = TRUE
GROUP BY CUBE ((dt.date_time::DATE), s.source_id, b.status);
And I update it via:
REFRESH MATERIALIZED VIEW dbo.ind_newborn_daily;
Update 2
Upon reviewing the Postgres statement logs, I've found that refresh statements were executed. though, I failed to mention those are processed by a parallel worker queue (4 threads).
As a result, each refresh statement is wrapped in a separate transaction AND those transitions are shown somewhat intertwined in logs:
2022-09-24 03:06:36.679 CEST,"XZY","XZYdb",17629,"127.0.0.1:44326",632e581c.44dd,3,"BEGIN",2022-09-24 03:06:36 CEST,5/78,0,LOG,00000,"execute <unnamed>: BEGIN",,,,,,,,,"PostgreSQL JDBC Driver","client backend",,0
2022-09-24 03:06:36.679 CEST,"XZY","XZYdb",17629,"127.0.0.1:44326",632e581c.44dd,4,"REFRESH MATERIALIZED VIEW",2022-09-24 03:06:36 CEST,5/78,0,LOG,00000,"execute <unnamed>: REFRESH MATERIALIZED VIEW dbo.view1",,,,,,,,,"PostgreSQL JDBC Driver","client backend",,0
2022-09-24 03:06:36.695 CEST,,,17579,,632e581b.44ab,1,,2022-09-24 03:06:35 CEST,13/3,6157,LOG,00000,"temporary file: path ""base/pgsql_tmp/pgsql_tmp17579.0"", size 3219456",,,,,,"REFRESH MATERIALIZED VIEW dbo.view2",,,"PostgreSQL JDBC Driver","parallel worker",17570,0
2022-09-24 03:06:36.696 CEST,,,17578,,632e581b.44aa,1,,2022-09-24 03:06:35 CEST,12/3,6157,LOG,00000,"temporary file: path ""base/pgsql_tmp/pgsql_tmp17578.0"", size 4087808",,,,,,"REFRESH MATERIALIZED VIEW dbo.view2",,,"PostgreSQL JDBC Driver","parallel worker",17570,0
2022-09-24 03:06:36.713 CEST,"XZY","XZYdb",17631,"127.0.0.1:44330",632e581c.44df,1,"SET",2022-09-24 03:06:36 CEST,6/29,0,LOG,00000,"execute <unnamed>: SET extra_float_digits = 3",,,,,,,,,"","client backend",,0
2022-09-24 03:06:36.713 CEST,"XZY","XZYdb",17631,"127.0.0.1:44330",632e581c.44df,2,"SET",2022-09-24 03:06:36 CEST,6/30,0,LOG,00000,"execute <unnamed>: SET application_name = 'PostgreSQL JDBC Driver'",,,,,,,,,"","client backend",,0
2022-09-24 03:06:36.713 CEST,"XZY","XZYdb",17631,"127.0.0.1:44330",632e581c.44df,3,"BEGIN",2022-09-24 03:06:36 CEST,6/31,0,LOG,00000,"execute <unnamed>: BEGIN",,,,,,,,,"PostgreSQL JDBC Driver","client backend",,0
2022-09-24 03:06:36.713 CEST,"XZY","XZYdb",17631,"127.0.0.1:44330",632e581c.44df,4,"REFRESH MATERIALIZED VIEW",2022-09-24 03:06:36 CEST,6/31,0,LOG,00000,"execute <unnamed>: REFRESH MATERIALIZED VIEW dbo.view3",,,,,,,,,"PostgreSQL JDBC Driver","client backend",,0
2022-09-24 03:06:36.737 CEST,"XZY","XZYdb",17570,"127.0.0.1:44232",632e581b.44a2,5,"REFRESH MATERIALIZED VIEW",2022-09-24 03:06:35 CEST,7/7,6157,LOG,00000,"temporary file: path ""base/pgsql_tmp/pgsql_tmp17570.2"", size 4038656",,,,,,"REFRESH MATERIALIZED VIEW dbo.view2",,,"PostgreSQL JDBC Driver","client backend",,0
2022-09-24 03:06:36.838 CEST,"XZY","XZYdb",17615,"127.0.0.1:44304",632e581c.44cf,5,"COMMIT",2022-09-24 03:06:36 CEST,4/189,6177,LOG,00000,"execute S_1: COMMIT",,,,,,,,,"PostgreSQL JDBC Driver","client backend",,0
2022-09-24 03:06:36.968 CEST,"XZY","XZYdb",17570,"127.0.0.1:44232",632e581b.44a2,6,"REFRESH MATERIALIZED VIEW",2022-09-24 03:06:35 CEST,7/7,6157,LOG,00000,"temporary file: path ""base/pgsql_tmp/pgsql_tmp17570.4"", size 2703360",,,,,,"REFRESH MATERIALIZED VIEW dbo.view2",,,"PostgreSQL JDBC Driver","client backend",,0
2022-09-24 03:06:36.992 CEST,"XZY","XZYdb",17631,"127.0.0.1:44330",632e581c.44df,5,"COMMIT",2022-09-24 03:06:36 CEST,6/31,6183,LOG,00000,"execute S_1: COMMIT",,,,,,,,,"PostgreSQL JDBC Driver","client backend",,0
Later on, at 4 AM, there goes the cron job, which is synchronous (one view at a time), though I did not wrap these in a transaction:
2022-09-24 04:00:08.866 CEST,"postgres","Xdb",20646,"[local]",632e64a1.50a6,422,"idle",2022-09-24 04:00:01 CEST,3/2638,0,LOG,00000,"statement: REFRESH MATERIALIZED VIEW dbo.view1;",,,,,,,,,"psql","client backend",,0
2022-09-24 04:00:09.842 CEST,,,20662,,632e64a8.50b6,1,,2022-09-24 04:00:08 CEST,5/93,6269,LOG,00000,"temporary file: path ""base/pgsql_tmp/pgsql_tmp20662.0"", size 3620864",,,,,,"REFRESH MATERIALIZED VIEW dbo.view1;",,,"psql","parallel worker",20646,0
2022-09-24 04:00:09.842 CEST,,,20661,,632e64a8.50b5,1,,2022-09-24 04:00:08 CEST,4/208,6269,LOG,00000,"temporary file: path ""base/pgsql_tmp/pgsql_tmp20661.0"", size 3612672",,,,,,"REFRESH MATERIALIZED VIEW dbo.view1;",,,"psql","parallel worker",20646,0
2022-09-24 04:00:09.875 CEST,"postgres","Xdb",20646,"[local]",632e64a1.50a6,423,"REFRESH MATERIALIZED VIEW",2022-09-24 04:00:01 CEST,3/2638,6269,LOG,00000,"temporary file: path ""base/pgsql_tmp/pgsql_tmp20646.42"", size 4554752",,,,,,"REFRESH MATERIALIZED VIEW dbo.view1;",,,"psql","client backend",,0
2022-09-24 04:00:10.354 CEST,"postgres","Xdb",20646,"[local]",632e64a1.50a6,424,"REFRESH MATERIALIZED VIEW",2022-09-24 04:00:01 CEST,3/2638,6269,LOG,00000,"temporary file: path ""base/pgsql_tmp/pgsql_tmp20646.41"", size 10829824",,,,,,"REFRESH MATERIALIZED VIEW dbo.view1;",,,"psql","client backend",,0
2022-09-24 04:00:10.687 CEST,"postgres","Xdb",20646,"[local]",632e64a1.50a6,425,"REFRESH MATERIALIZED VIEW",2022-09-24 04:00:01 CEST,3/2638,6269,LOG,00000,"temporary file: path ""base/pgsql_tmp/pgsql_tmp20646.8.sharedfileset/i2of16.p0.0"", size 655360",,,,,,"REFRESH MATERIALIZED VIEW dbo.view1;",,,"psql","client backend",,0
...
...
2022-09-24 04:00:10.701 CEST,"postgres","Xdb",20646,"[local]",632e64a1.50a6,518,"REFRESH MATERIALIZED VIEW",2022-09-24 04:00:01 CEST,3/2638,6269,LOG,00000,"temporary file: path ""base/pgsql_tmp/pgsql_tmp20646.43"", size 10829824",,,,,,"REFRESH MATERIALIZED VIEW dbo.view1;",,,"psql","client backend",,0
2022-09-24 04:00:10.702 CEST,"postgres","Xdb",20646,"[local]",632e64a1.50a6,519,"REFRESH MATERIALIZED VIEW",2022-09-24 04:00:01 CEST,3/2638,6269,LOG,00000,"temporary file: path ""base/pgsql_tmp/pgsql_tmp20646.40"", size 3588096",,,,,,"REFRESH MATERIALIZED VIEW dbo.view1;",,,"psql","client backend",,0
As you can see, logs are pretty much littered with the usage of temporary files, but I expected those as views are big.
Now comes the manual refresh at 8:11 AM from DataGrip:
2022-09-24 08:11:04.997 CEST,"X","Xdb",1088,"127.0.0.1:44432",632e9f78.440,1,"SET",2022-09-24 08:11:04 CEST,3/3703,0,LOG,00000,"execute <unnamed>: SET extra_float_digits = 3",,,,,,,,,"","client backend",,0
2022-09-24 08:11:05.034 CEST,"X","Xdb",1088,"127.0.0.1:44432",632e9f78.440,2,"SET",2022-09-24 08:11:04 CEST,3/3704,0,LOG,00000,"execute <unnamed>: SET application_name = ''",,,,,,,,,"","client backend",,0
2022-09-24 08:11:05.063 CEST,"X","Xdb",1088,"127.0.0.1:44432",632e9f78.440,3,"SELECT",2022-09-24 08:11:04 CEST,3/3705,0,LOG,00000,"execute <unnamed>: select version()",,,,,,,,,"","client backend",,0
2022-09-24 08:11:05.096 CEST,"X","Xdb",1088,"127.0.0.1:44432",632e9f78.440,4,"SET",2022-09-24 08:11:04 CEST,3/3706,0,LOG,00000,"execute <unnamed>: SET application_name = 'DataGrip 2022.2.4'",,,,,,,,,"","client backend",,0
2022-09-24 08:11:05.172 CEST,"X","Xdb",1088,"127.0.0.1:44432",632e9f78.440,5,"SHOW",2022-09-24 08:11:04 CEST,3/3708,0,LOG,00000,"execute <unnamed>: SHOW TRANSACTION ISOLATION LEVEL",,,,,,,,,"DataGrip 2022.2.4","client backend",,0
2022-09-24 08:11:05.232 CEST,"X","Xdb",1088,"127.0.0.1:44432",632e9f78.440,6,"REFRESH MATERIALIZED VIEW",2022-09-24 08:11:04 CEST,3/3709,0,LOG,00000,"execute <unnamed>: refresh materialized view dbo.view1",,,,,,,,,"DataGrip2022.2.4","client backend",,0
2022-09-24 08:11:06.236 CEST,,,1089,,632e9f79.441,1,,2022-09-24 08:11:05 CEST,4/274,6372,LOG,00000,"temporary file: path ""base/pgsql_tmp/pgsql_tmp1089.0"", size 3604480",,,,,,"refresh materialized view dbo.view1",,,"DataGrip 2022.2.4","parallel worker",1088,0
2022-09-24 08:11:06.237 CEST,,,1090,,632e9f79.442,1,,2022-09-24 08:11:05 CEST,5/147,6372,LOG,00000,"temporary file: path ""base/pgsql_tmp/pgsql_tmp1090.0"", size 3645440",,,,,,"refresh materialized view dbo.view1",,,"DataGrip 2022.2.4","parallel worker",1088,0
2022-09-24 08:11:06.263 CEST,"X","Xdb",1088,"127.0.0.1:44432",632e9f78.440,7,"REFRESH MATERIALIZED VIEW",2022-09-24 08:11:04 CEST,3/3709,6372,LOG,00000,"temporary file: path ""base/pgsql_tmp/pgsql_tmp1088.2"", size 4554752",,,,,,"refresh materialized view dbo.view1",,,"DataGrip 2022.2.4","client backend",,0
2022-09-24 08:11:06.742 CEST,"X","Xdb",1088,"127.0.0.1:44432",632e9f78.440,8,"REFRESH MATERIALIZED VIEW",2022-09-24 08:11:04 CEST,3/3709,6372,LOG,00000,"temporary file: path ""base/pgsql_tmp/pgsql_tmp1088.1"", size 10829824",,,,,,"refresh materialized view dbo.view1",,,"DataGrip 2022.2.4","client backend",,0
...
...
2022-09-24 08:11:07.072 CEST,"X","Xdb",1088,"127.0.0.1:44432",632e9f78.440,102,"REFRESH MATERIALIZED VIEW",2022-09-24 08:11:04 CEST,3/3709,6372,LOG,00000,"temporary file: path ""base/pgsql_tmp/pgsql_tmp1088.3"", size 10829824",,,,,,"refresh materialized view dbo.view1",,,"DataGrip 2022.2.4","client backend",,0
2022-09-24 08:11:07.072 CEST,"X","Xdb",1088,"127.0.0.1:44432",632e9f78.440,103,"REFRESH MATERIALIZED VIEW",2022-09-24 08:11:04 CEST,3/3709,6372,LOG,00000,"temporary file: path ""base/pgsql_tmp/pgsql_tmp1088.0"", size 3579904",,,,,,"refresh materialized view dbo.view1",,,"DataGrip 2022.2.4","client backend",,0
The only difference I spotted so far was that both Java-based and manual refresh had:
execute <unnamed>: REFRESH MATERIALIZED VIEW
whereas the one from cron-job only had:
statement: REFRESH MATERIALIZED VIEW
Update #3
2022-09-24 03:00:00.814 CEST,"X","Xdb",17112,"127.0.0.1:44060",632e5690.42d8,1,"SET",2022-09-24 03:00:00 CEST,3/2370,0,LOG,00000,"execute <unnamed>: SET extra_float_digits = 3",,,,,,,,,"","client backend",,0
2022-09-24 03:00:00.815 CEST,"X","Xdb",17112,"127.0.0.1:44060",632e5690.42d8,2,"SET",2022-09-24 03:00:00 CEST,3/2371,0,LOG,00000,"execute <unnamed>: SET application_name = 'PostgreSQL JDBC Driver'",,,,,,,,,"","client backend",,0
2022-09-24 03:00:00.815 CEST,"X","Xdb",17112,"127.0.0.1:44060",632e5690.42d8,3,"SHOW",2022-09-24 03:00:00 CEST,3/2372,0,LOG,00000,"execute <unnamed>: SHOW TRANSACTION ISOLATION LEVEL",,,,,,,,,"PostgreSQL JDBC Driver","client backend",,0
2022-09-24 03:00:00.815 CEST,"X","Xdb",17112,"127.0.0.1:44060",632e5690.42d8,4,"SET",2022-09-24 03:00:00 CEST,3/2373,0,LOG,00000,"execute <unnamed>: SET SESSION CHARACTERISTICS AS TRANSACTION ISOLATION LEVEL READ UNCOMMITTED",,,,,,,,,"PostgreSQL JDBC Driver","client backend",,0
...
After a while sync job completes
...
2022-09-24 03:06:35.311 CEST,"X","Xdb",17112,"127.0.0.1:44060",632e5690.42d8,77697,"COMMIT",2022-09-24 03:00:00 CEST,3/2374,6153,LOG,00000,"execute S_35: COMMIT",,,,,,,,,"PostgreSQL JDBC Driver","client backend",,0
2022-09-24 03:06:35.322 CEST,"X","Xdb",17112,"127.0.0.1:44060",632e5690.42d8,77698,"SET",2022-09-24 03:00:00 CEST,3/2375,0,LOG,00000,"execute <unnamed>: SET SESSION CHARACTERISTICS AS TRANSACTION ISOLATION LEVEL READ COMMITTED",,,,,,,,,"PostgreSQL JDBC Driver","client backend",,0
2022-09-24 03:06:35.355 CEST,"X","Xdb",17568,"127.0.0.1:44228",632e581b.44a0,1,"SET",2022-09-24 03:06:35 CEST,4/171,0,LOG,00000,"execute <unnamed>: SET extra_float_digits = 3",,,,,,,,,"","client backend",,0
2022-09-24 03:06:35.355 CEST,"X","Xdb",17568,"127.0.0.1:44228",632e581b.44a0,2,"SET",2022-09-24 03:06:35 CEST,4/172,0,LOG,00000,"execute <unnamed>: SET application_name = 'PostgreSQL JDBC Driver'",,,,,,,,,"","client backend",,0
2022-09-24 03:06:35.355 CEST,"X","Xdb",17571,"127.0.0.1:44234",632e581b.44a3,1,"SET",2022-09-24 03:06:35 CEST,6/17,0,LOG,00000,"execute <unnamed>: SET extra_float_digits = 3",,,,,,,,,"","client backend",,0
2022-09-24 03:06:35.355 CEST,"X","Xdb",17571,"127.0.0.1:44234",632e581b.44a3,2,"SET",2022-09-24 03:06:35 CEST,6/18,0,LOG,00000,"execute <unnamed>: SET application_name = 'PostgreSQL JDBC Driver'",,,,,,,,,"","client backend",,0
2022-09-24 03:06:35.355 CEST,"X","Xdb",17569,"127.0.0.1:44230",632e581b.44a1,1,"SET",2022-09-24 03:06:35 CEST,5/48,0,LOG,00000,"execute <unnamed>: SET extra_float_digits = 3",,,,,,,,,"","client backend",,0
2022-09-24 03:06:35.356 CEST,"X","Xdb",17569,"127.0.0.1:44230",632e581b.44a1,2,"SET",2022-09-24 03:06:35 CEST,5/49,0,LOG,00000,"execute <unnamed>: SET application_name = 'PostgreSQL JDBC Driver'",,,,,,,,,"","client backend",,0
2022-09-24 03:06:35.356 CEST,"X","Xdb",17569,"127.0.0.1:44230",632e581b.44a1,3,"BEGIN",2022-09-24 03:06:35 CEST,5/50,0,LOG,00000,"execute <unnamed>: BEGIN",,,,,,,,,"PostgreSQL JDBC Driver","client backend",,0
... The refresh statements go on and on
It turns out that the sync job runs in a transaction isolation level READ UNCOMMITTED, but view refresh tasks run at READ COMMITTED level
On the other hand, DataGrip does not even set the isolation level, but only queries it...

Answer you are looking for is that materialized views are not updated immediately. They are updated in the background. You can try to run the following query to see if the materialized view is still updating:
SELECT pg_is_in_recovery(); -- returns true if the server is a standby server
SELECT pg_last_xact_replay_timestamp(); -- returns the timestamp of the last transaction that was replayed on the standby server
SELECT pg_last_xact_replay_timestamp() > pg_last_xact_replay_timestamp(); -- returns true if the last transaction that was replayed on the standby server is newer than the last transaction that was replayed on the primary server

Edit:
I realize that my "solution" is kinda hacky and not really correct. Frustration with this problem caused me to jump to conclusion, but it is in fact invalid.
As stated here: https://wiki.postgresql.org/wiki/Don't_Do_This#Don't_use_timestamp_(without_time_zone)
I later realized that my Linux box is set (and Postgres) in CET TZ, however, all my timestamps are in UTC and my clients use UTC.
Original answer:
After several months of the original question, I can finally put an end to this. Up until now, every single morning, I had to refresh views manually in order to work. Finally, I dedicated some time to investigating and solving this.
Root cause:
The database server is running in the CET timezone
The application server is running in the CET timezone
All clients (JetBrains Data Grip / Tableau) are running in UTC timezone
Application server executed code against materialized views
Tableau executed arbitrary SQL
I stored all datetimes WITH TIMEZONE
All events happing after 11 PM, were pushed to the next day...
I heard this one before: Always store the datetimes in UTC (or without timezone). But mistakes happen.
I guess this is a meme-worthy mistake:
Thank you all for the elaborate discussion and for bearing with at the time non-sense "facts" :)

Related

extract/separate duration value from postgresql log in elk

I have elk(elasticsearch, logstash , kibana) for monitor logs. postgresql logs with filebeat send to logstash and show logs in kibana.
I enabled log_duration in postgresql.conf and logged correctly.
for example, the postgresql log is below :
2023-01-11 06:17:09.754 EST [19751] user#books LOG: duration: 0.014 ms execute <unnamed>: SET SESSION CHARACTERISTICS AS TRANSACTION READ ONLY
2023-01-11 06:17:09.755 EST [19751] user#books LOG: duration: 0.016 ms bind S_1: BEGIN
2023-01-11 06:17:09.755 EST [19751] user#books LOG: duration: 0.006 ms execute S_1: BEGIN
2023-01-11 06:17:09.756 EST [19751] user#books LOG: duration: 0.488 ms parse <unnamed>: select * from books
but in kibana, The duration value of the field is not separate and it is displayed as one with the message field, and there is no possibility of aggregation and other opetaions.
how to extract and split duration value from message ???

How to read transaction from PostgreSQL statement log

I am enabling postgresSQL statement log by SET log_statement = 'all'; and received output like below for an Java application that is using Hibernate. Default log format is used and the Postgres version is 13.1
I can guess the number in square bracket is threadID? I see that there is multiple BEGIN and multiple COMMIT scattered without explicit ID or order
Is there a way I can see the scope of a transaction from this log? (Actual statement and values are removed)
2021-11-08 05:45:52.827 UTC [107] LOG: execute S_4: BEGIN
2021-11-08 05:45:52.841 UTC [107] LOG: execute <unnamed>:
RETURNING *
2021-11-08 05:45:52.841 UTC [107] DETAIL: parameters:
2021-11-08 05:45:52.927 UTC [107] LOG: execute <unnamed>:
RETURNING *
2021-11-08 05:45:52.927 UTC [107] DETAIL: parameters:
2021-11-08 05:45:52.975 UTC [107] LOG: execute <unnamed>:
RETURNING *
2021-11-08 05:45:52.975 UTC [107] DETAIL: parameters:
2021-11-08 05:45:54.209 UTC [107] LOG: execute <unnamed>:
2021-11-08 05:45:54.209 UTC [107] DETAIL: parameters:
2021-11-08 05:45:54.251 UTC [107] LOG: execute <unnamed>:
2021-11-08 05:45:54.251 UTC [107] DETAIL: parameters:
2021-11-08 05:45:54.297 UTC [107] LOG: execute <unnamed>:
2021-11-08 05:45:54.297 UTC [107] DETAIL: parameters:
2021-11-08 05:45:54.565 UTC [107] LOG: execute <unnamed>:
2021-11-08 05:45:54.565 UTC [107] DETAIL: parameters:
2021-11-08 05:45:55.164 UTC [194] LOG: execute <unnamed>: SET extra_float_digits = 3
2021-11-08 05:45:55.209 UTC [194] LOG: execute <unnamed>: SET application_name = 'PostgreSQL JDBC Driver'
2021-11-08 05:45:55.253 UTC [194] LOG: execute <unnamed>: BEGIN
2021-11-08 05:45:55.257 UTC [194] LOG: execute <unnamed>:
2021-11-08 05:45:55.257 UTC [194] DETAIL:
2021-11-08 05:45:55.308 UTC [194] LOG: execute <unnamed>:
2021-11-08 05:45:55.308 UTC [194] DETAIL: parameters:
2021-11-08 05:45:55.354 UTC [194] LOG: execute <unnamed>:
RETURNING *
2021-11-08 05:45:55.354 UTC [194] DETAIL: parameters:
2021-11-08 05:45:55.412 UTC [195] LOG: execute <unnamed>: SET extra_float_digits = 3
2021-11-08 05:45:55.451 UTC [195] LOG: execute <unnamed>: SET application_name = 'PostgreSQL JDBC Driver'
2021-11-08 05:45:55.577 UTC [194] LOG: execute <unnamed>:
2021-11-08 05:45:55.577 UTC [194] DETAIL:
2021-11-08 05:45:55.617 UTC [194] LOG: execute S_1: COMMIT
2021-11-08 05:45:56.393 UTC [196] LOG: execute <unnamed>: SET extra_float_digits = 3
2021-11-08 05:45:56.435 UTC [196] LOG: execute <unnamed>: SET application_name = 'PostgreSQL JDBC Driver'
2021-11-08 05:45:56.486 UTC [196] LOG: execute <unnamed>:
2021-11-08 05:45:56.486 UTC [196] DETAIL: parameters:
2021-11-08 05:45:56.529 UTC [196] LOG: execute <unnamed>:
2021-11-08 05:45:56.529 UTC [196] DETAIL: parameters:
2021-11-08 05:45:56.571 UTC [196] LOG: execute <unnamed>: BEGIN
2021-11-08 05:45:56.572 UTC [196] LOG: execute <unnamed>:
RETURNING *
2021-11-08 05:45:56.572 UTC [196] DETAIL: parameters:
2021-11-08 05:45:56.622 UTC [196] LOG: execute S_1: COMMIT
2021-11-08 05:45:56.650 UTC [197] LOG: execute <unnamed>: SET extra_float_digits = 3
2021-11-08 05:45:56.677 UTC [107] LOG: execute <unnamed>:
2021-11-08 05:45:56.677 UTC [107] DETAIL: parameters:
2021-11-08 05:45:56.690 UTC [197] LOG: execute <unnamed>: SET application_name = 'PostgreSQL JDBC Driver'
2021-11-08 05:45:56.724 UTC [107] LOG: execute <unnamed>:
2021-11-08 05:45:56.724 UTC [107] DETAIL: parameters:
2021-11-08 05:45:58.603 UTC [196] LOG: execute <unnamed>: BEGIN READ ONLY
2021-11-08 05:45:58.604 UTC [196] LOG: execute <unnamed>:
2021-11-08 05:45:58.604 UTC [196] DETAIL: parameters:
2021-11-08 05:45:58.653 UTC [196] LOG: execute <unnamed>:
2021-11-08 05:45:58.653 UTC [196] DETAIL: parameters:
2021-11-08 05:45:58.710 UTC [196] LOG: execute <unnamed>:
2021-11-08 05:45:58.710 UTC [196] DETAIL: parameters:
2021-11-08 05:45:58.751 UTC [196] LOG: execute S_1: COMMIT
2021-11-08 05:45:58.792 UTC [196] LOG: execute <unnamed>: BEGIN READ ONLY
2021-11-08 05:45:58.792 UTC [196] LOG: execute <unnamed>:
2021-11-08 05:45:58.792 UTC [196] DETAIL: parameters:
2021-11-08 05:45:58.836 UTC [196] LOG: execute S_1: COMMIT
2021-11-08 05:45:58.880 UTC [196] LOG: execute <unnamed>: BEGIN READ ONLY
2021-11-08 05:45:58.881 UTC [196] LOG: execute <unnamed>:
2021-11-08 05:45:58.881 UTC [196] DETAIL: parameters:
2021-11-08 05:45:58.924 UTC [196] LOG: execute S_1: COMMIT
2021-11-08 05:45:58.965 UTC [196] LOG: execute <unnamed>:
2021-11-08 05:45:58.965 UTC [196] DETAIL: parameters:
2021-11-08 05:45:59.008 UTC [196] LOG: execute <unnamed>:
2021-11-08 05:45:59.008 UTC [196] DETAIL: parameters:
2021-11-08 05:45:59.192 UTC [107] LOG: execute S_1: COMMIT

According to Postgres document in log_line_prefix, You can set how Postgres print log for example log data, process id, username, etc
You can use %x for log transaction-id

PostgreSQL failed with vacuum and autovacuum

Postgres v11.9
There are many errors on Postgres log like this:
2020-09-05 17:35:37 GMT [22464]: #: [6-1] ERROR: uncommitted xmin 636700836 from before xid cutoff 809126794 needs to be frozen
2020-09-05 17:35:37 GMT [22464]: #: [7-1] CONTEXT: automatic vacuum of table "table_nane"
Manual vacuum fails with this error too.
What can I do to fix this error?

Export the database with pg_dump.
Create a new database cluster and restore the dump into it.
Remove the original database cluster.

Streaming replication is failing with "WAL segment has already been moved"

I am trying to implement Master/Slave streaming replication on Postgres 11.5. I ran the following steps -
On Master
select pg_start_backup('replication-setup',true);
On Slave
Stopped the postgres 11 database and ran
rsync -aHAXxv --numeric-ids --progress -e "ssh -T -o Compression=no -x" --exclude pg_wal --exclude postgresql.pid --exclude pg_log MASTER:/var/lib/postgresql/11/main/* /var/lib/postgresql/11/main
On Master
select pg_stop_backup();
On Slave
rsync -aHAXxv --numeric-ids --progress -e "ssh -T -o Compression=no -x" MASTER:/var/lib/postgresql/11/main/pg_wal/* /var/lib/postgresql/11/main/pg_wal
I created the recovery.conf file on slave ~/11/main folder
standby_mode = 'on'
primary_conninfo = 'user=postgres host=MASTER port=5432 sslmode=prefer sslcompression=1 krbsrvname=postgres'
primary_slot_name='my_repl_slot'
When I start Postgres on Slave, I get the error on both MASTER and SLAVE logs -
019-11-08 09:03:51.205 CST [27633] LOG: 00000: database system was interrupted; last known up at 2019-11-08 02:53:04 CST
2019-11-08 09:03:51.205 CST [27633] LOCATION: StartupXLOG, xlog.c:6388
2019-11-08 09:03:51.252 CST [27633] LOG: 00000: entering standby mode
2019-11-08 09:03:51.252 CST [27633] LOCATION: StartupXLOG, xlog.c:6443
2019-11-08 09:03:51.384 CST [27634] LOG: 00000: started streaming WAL from primary at 12DB/C000000 on timeline 1
2019-11-08 09:03:51.384 CST [27634] LOCATION: WalReceiverMain, walreceiver.c:383
2019-11-08 09:03:51.384 CST [27634] FATAL: XX000: could not receive data from WAL stream: ERROR: requested WAL segment 00000001000012DB0000000C has already been removed
2019-11-08 09:03:51.384 CST [27634] LOCATION: libpqrcv_receive, libpqwalreceiver.c:772
2019-11-08 09:03:51.408 CST [27635] LOG: 00000: started streaming WAL from primary at 12DB/C000000 on timeline 1
2019-11-08 09:03:51.408 CST [27635] LOCATION: WalReceiverMain, walreceiver.c:383
The problem is the START WAL - 00000001000012DB0000000C is available right until I run the pg_stop_backup() and is getting archived and no longer available, once the pg_stop_backup() is executed. So this is not an issue of the WAL being archived out due to low WAL_KEEP_SEGMENTS.
postgres#SLAVE:~/11/main/pg_wal$ cat 00000001000012DB0000000C.00000718.backup
START WAL LOCATION: 12DB/C000718 (file 00000001000012DB0000000C)
STOP WAL LOCATION: 12DB/F4C30720 (file 00000001000012DB000000F4)
CHECKPOINT LOCATION: 12DB/C000750
BACKUP METHOD: pg_start_backup
BACKUP FROM: master
START TIME: 2019-11-07 15:47:26 CST
LABEL: replication-setup-mdurbha
START TIMELINE: 1
STOP TIME: 2019-11-08 08:48:35 CST
STOP TIMELINE: 1
My MASTER has archive_command set, and I have the missing WALs available. I copied them into a restore directory on the SLAVE and tried the recovery.conf below, but it still fails with the MASTER reporting the same WAL segment has already been moved error.
Any idea how I can address this issue? I have used rsync to setup replication without any issues in the past on Postgres 9.6, but have been experiencing this issue on Postgres 11.
standby_mode = 'on'
primary_conninfo = 'user=postgres host=MASTER port=5432 sslmode=prefer sslcompression=1 krbsrvname=postgres'
restore_command='cp /var/lib/postgresql/restore/%f %p'

Put a restore_command into recovery.conf that can restore archived WAL files and you are fine.

Postgresql 10 logical replication not working

I install postgresql 10 using scripts:
$ wget -q https://www.postgresql.org/media/keys/ACCC4CF8.asc -O - | sudo apt-key add -
$ sudo sh -c 'echo "deb http://apt.postgresql.org/pub/repos/apt/ `lsb_release -cs`-pgdg main" >> /etc/apt/sources.list.d/pgdg.list'
$ sudo apt-get update
$ sudo apt-get install postgresql postgresql-contrib
On Master server : xx.xxx.xxx.xx-:
And after that in postgresql.conf:
set wal_level = logical
On Slave server -:
in postgresql.conf:
set wal_level = logical
And after all i use below query on master server:
create table t1 (id integer primary key, val text);
create user replicant with replication;
grant select on t1 to replicant;
insert into t1 (id, val) values (10, 'ten'), (20, 'twenty'), (30, 'thirty');
create publication pub1 for table t1;
And at Slave server:
create table t1 (id integer primary key, val text, val2 text);
create subscription sub1 connection 'dbname=dbsrc user=replicant' publication pub1;
But the problem which is i am facing is table are not syncing and as per logical replication when i insert new row on master server, slave server not getting that row.
I am new to postgresql please help me.
Thanks for your precious time.
Now here is my postgresql log for Master server:
2017-10-17 11:06:16.644 UTC [10713] replicant#postgres LOG: starting logical decoding for slot "sub_1"
2017-10-17 11:06:16.644 UTC [10713] replicant#postgres DETAIL: streaming transactions committing after 1/F45EB0C8, reading WAL from 1/F45EB0C8
2017-10-17 11:06:16.645 UTC [10713] replicant#postgres LOG: logical decoding found consistent point at 1/F45EB0C8
2017-10-17 11:06:16.645 UTC [10713] replicant#postgres DETAIL: There are no running transactions.
Here is my slave server postgresql log:
2017-10-17 19:14:45.622 CST [7820] WARNING: out of logical replication worker slots
2017-10-17 19:14:45.622 CST [7820] HINT: You might need to increase max_logical_replication_workers.
2017-10-17 19:14:45.670 CST [7821] WARNING: out of logical replication worker slots
2017-10-17 19:14:45.670 CST [7821] HINT: You might need to increase max_logical_replication_workers.
2017-10-17 19:14:45.680 CST [7822] WARNING: out of logical replication worker slots
2017-10-17 19:14:45.680 CST [7822] HINT: You might need to increase max_logical_replication_workers.
2017-10-17 19:14:50.865 CST [7820] WARNING: out of logical replication worker slots
2017-10-17 19:14:50.865 CST [7820] HINT: You might need to increase max_logical_replication_workers.
2017-10-17 19:14:50.917 CST [7821] WARNING: out of logical replication worker slots
2017-10-17 19:14:50.917 CST [7821] HINT: You might need to increase max_logical_replication_workers.
2017-10-17 19:14:50.928 CST [7822] WARNING: out of logical replication worker slots
2017-10-17 19:14:50.928 CST [7822] HINT: You might need to increase max_logical_replication_workers.
2017-10-17 19:14:55.871 CST [7820] WARNING: out of logical replication worker slots
2017-10-17 19:14:55.871 CST [7820] HINT: You might need to increase max_logical_replication_workers.
And after increasing the max_logical_replication_workers i am getting this:
2017-10-17 19:44:45.898 CST [7987] LOG: logical replication table synchronization worker for subscription "sub2", table "t1" has started
2017-10-17 19:44:45.982 CST [7988] LOG: logical replication table synchronization worker for subscription "myadav_test", table "test_replication" h$
2017-10-17 19:44:45.994 CST [7989] LOG: logical replication table synchronization worker for subscription "sub3", table "t1" has started
2017-10-17 19:44:48.621 CST [7987] ERROR: could not start initial contents copy for table "staging.t1": ERROR: permission denied for schema staging
2017-10-17 19:44:48.623 CST [7962] LOG: worker process: logical replication worker for subscription 20037 sync 20027 (PID 7987) exited with exit co$
2017-10-17 19:44:48.705 CST [7988] ERROR: could not start initial contents copy for table "staging.test_replication": ERROR: permission denied for$
2017-10-17 19:44:48.707 CST [7962] LOG: worker process: logical replication worker for subscription 20025 sync 20016 (PID 7988) exited with exit co$
2017-10-17 19:44:48.717 CST [7989] ERROR: duplicate key value violates unique constraint "t1_pkey"
2017-10-17 19:44:48.717 CST [7989] DETAIL: Key (id)=(10) already exists.
2017-10-17 19:44:48.717 CST [7989] CONTEXT: COPY t1, line 1
2017-10-17 19:44:48.718 CST [7962] LOG: worker process: logical replication worker for subscription 20038 sync 20027 (PID 7989) exited with exit co$
2017-10-17 19:44:51.629 CST [8008] LOG: logical replication table synchronization worker for subscription "sub2", table "t1" has started
2017-10-17 19:44:51.712 CST [8009] LOG: logical replication table synchronization worker for subscription "myadav_test", table "test_replication" h$
2017-10-17 19:44:51.722 CST [8010] LOG: logical replication table synchronization worker for subscription "sub3", table "t1" has started
Now i finally realize that logical replication is working for postgres database but not for my other database on same server. I am getting permission issue on schema that is is log.

The row changes are applied using the rights of the user who owns the subscription. By default that's the user who created the subscription.
So make sure the subscription is owned by a user with sufficient rights. Grant needed rights to tables, or if you can't be bothered, make the subscription owned by a superuser who has full rights to everything.
See:
CREATE SUBSCRIPTION
logical replication - security
logical replication

We Keep Coding

iphone swift flutter scala powershell matlab mongodb postgresql perl eclipse

PostgresSQL 14 materialized views not updating - postgresql

Related

extract/separate duration value from postgresql log in elk

How to read transaction from PostgreSQL statement log

PostgreSQL failed with vacuum and autovacuum

Streaming replication is failing with "WAL segment has already been moved"

Postgresql 10 logical replication not working

Categories

Resources