PostgreSQL PANIC: WAL contains references to invalid pages - postgresql

I have problem with PostgreSQL database running as replica of the master database server. Database on master runs without any problems. But replica database runs only for few hours (it is random time) and after that crashing down by this reason:
WARNING: page 3318889 of relation base/16389/19632 is uninitialized
...
PANIC: WAL contains references to invalid pages
Have you any idea what is wrong please? I'm not able to solve this problem for many days! Thanks.

There was more Postgres bugs with these symptoms. Lot of was fixed already. Please, check if your Postgres is latest minor release. And if it is, then report this issue to mailing list https://www.postgresql.org/list/pgsql-hackers/.

Related

Postgres: some process is cleaning all my table records

I have a Postgres installed in a Centos and another application is using Postgres the save data.
For sometime, and I can't find the reason, all the database tables become empty on the weekends.
I have been searching a lot to try to find some clues of the reason of that behaviour, but logs are not giving me that info.
I am pretty sure the application is not executing anything to clean the records, my thoughts are pointing to some process for some reason in the Postgres side.
The pg_log only shows this warning the day it happens:
HINT: Consider increasing the configuration parameter "checkpoint_segments".
LOG: checkpoints are occurring too frequently (11 seconds apart)
Apart from that I have no other clues.
Performing a VACUUM ANALYZE VERBOSE it says there is no dead data so it has nothing to delete.
Can you tell me what should I look to get the reason? Should it be any Postgres process to do it?
LOG: checkpoints are occurring too frequently (11 seconds apart)
This log message should also include all the information log_line_prefix tells it to include. So you should set log_line_prefix to include more information, like application name (voluntarily supplied by the client), database username, and host name/IP from which the connection came.
But perhaps more directly at issue, if things are connecting to your database and doing things you don't understand or approve of, it is time to change your passwords.

Firebird error: cannot start transaction for password database

I am looking for some help - today my firebird database has stopped opening the database files with the following error:
engine code: 335544654
engine message: cannot start transaction for password database
I've tried googling but can't find any resources that have any relevant information.
Thanks
I eventually got this resolved by doing a number of restores of the security2.fdb database.
In my case, there was not enough data space in /var.
I cleared the contents of "/var/spool/clientmqueue/".

postgres index contains "unexpected zero page at block" exceptions

I have following errors in pg_log file several thousand times. How to resolve them.
index "meeting_pkey" contains unexpected zero page at block 410.Please REINDEX it.
index "faultevent_props_pkey" contains unexpected zero page at block 37290.
index "faultevent_pkey" contains unexpected zero page at block 1704
The cause of the issue is due to bad index pages and its unable to read it.
Reindex the problematic index to overcome the issue.
Reindex index <schema_name>.<index_name>;
Here you have some hits.
Your database is corrupt.
Try to run pg_dumpall to get a logical dump of the database.
If that fails, buy support from somebody who can salvage data from corrupted databases.
If it succeeds:
Check the hardware, particularly storage and RAM.
Once you are certain the hardware is ok, install the latest patch update for your PostgreSQL version.
Create a new database cluster with initdb.
Restore the dump into the new cluster.
Did you have crashes recently?
Did you test if your storage handles fsync requests properly?
Do you have any dangerous settings like fsync = off?
I ran into this issue and after a lot of reading I decided to do a complete DB reindex:
reindex DATABASE <DATABASE NAME>
and it solved the issue for me. Hope this helps you.

Postgresql lost tables

I have a PostgreSQL database on my centos server.
Unfortunately since yesterday, in all existing schemas, all the tables are lost.
I saw log file and there was an unexpected reboot in recent days, probably a crash of the o.s.
Now the Postgres server it start correctly and I can view triggers, sequences and there aren't other problems.
Can I do something to recovery these tables?
Thanks.

missing chunk number 0 for toast value 37946637 in pg_toast_2619

Main Issue:
Getting "ERROR: missing chunk number 0 for toast value 37946637 in pg_toast_2619" while selecting from tables.
Steps that led to the issue:
- Used pg_basebackup from a Primary db and tried to restore it onto a Dev host.
- Did a pg_resetxlog -f /${datadir} and started up the Dev db.
- After starting up the Dev db, when I query a varchar column, I keep getting:
psql> select text_col_name from big_table;
ERROR: missing chunk number 0 for toast value 37946637 in pg_toast_2619
This seems to be happening for most varchar columns in the restored db.
Has anyone else seen it?
Does anyone have ideas of why it happens and how to fix it?
pg_resetxlog is a bit of a last resort utility which you should prefer not to use. Easiest way to make a fully working backup is to use pg_basebackup with the -X s option. That is an uppercase X. What this does is that basebackup opens two connections. One to copy all the data files and one to receive all of the wal that is written during the duration of the backup. This way you cannot run into the problem that parts of the wal you need are already deleted.
I tried a few things since by original question. I can confirm that the source of my error "ERROR: missing chunk number 0 for toast value 37946637 in pg_toast_2619" was doing a pg_resetxlog during the restore process.
I re-did the restore today but this time, applied the pg_xlog files from Primary using recovery.conf. The restored db started up fine now and all queries are running as expected.