We are experiencing a problem on a Postgres instance where one long running transaction on one database prevent the vacuum process to remove dead tuples on a tables of another database (of the same instance).
It seems crazy to me that xmin is shared cross database.
So here are my questions :
Is that a normal behavior or did we misconfigured something ?
Why is it so ?
Is there a workaround ?
Thanks folks
postgres version 12.4
Related
We are using Golang and GORM for ORM and database schema migrations. Using the defautl AUtoMigrate is nice to add tables/columns/indexes that are net new. One big issue with Postgres, specifically AWS AUrora Postgres is it seems even a simple alter table add column with no default and no not null constraints still creates a full on AccessExclusiveLock. This caused us downtime in production for what is otherwise a deployable infrastructure anytime with zero downtime.
Postgres claims this wont cause lock, but think Aurora has some innodb engine magic that does cause one, probably for read replica consistency ?
Regardless, in our deploy shell script trying to see if we can ask GORM to give us the DDL it would run, like a preview, without running it. Anyway to do that?
Currently I am trying to upgrade my postgres 9.1 to 10 using logical replication. As 9.1 does not support native logical_replication, I tried slony and made a replica successfully.
P.S. The above replica I created is using a sample dump from an year ago which is only of 800mb.
Now I have few questions.
How can I validate whether the replica has all the data replicated successfully. Few suggested to put the master on maintenance mode(few downtime) and do a last N items comparison with both the database on all the tables.
I tried with 800mb. Will there be any problem when I go and try with 100+ GB?
Please share your personal experience here. I have been trying to document what are the things that could go wrong so I can always try to anticipate the next course of action.
You may use the Data Validator that is shipped with trial version of EDB Postgres Replication server for validating the data between old and new PostgreSQL databases.
You may read the details of Data Validator at Data Validator Document
To download the EDB Replication Server please follow this link: EDB Replication Server
Disclosure: I work for EnterpriseDB (EDB)
Heroku warns of using Multiple Schemas with Postgres. But does not specify numerous operational problems caused.
As posted on Heroku docs:
The most common use case for using multiple schemas in a database is
building a software-as-a-service application wherein each customer has
their own schema. While this technique seems compelling, we strongly
recommend against it as it has caused numerous cases of operational
problems. For instance, even a moderate number of schemas (> 50) can
severely impact the performance of Heroku’s database snapshots tool,
PG Backups.
I think, the problem of Backups can be solved by adding a follower db.
I have 60 tables per schema, so with 1000 schemas I will have 60,000 tables. How will this impact database performance? What kind of problems I can expect while scaling?
The first issues with running a large volume of schemas and/or tables don't typically interfere with a running database. The major problem that operators encounter is that they will be unable to create logical backups of the database. Running heroku pg:backups is likely to fail as is running pg_dump manually. Typically, this is the error you'll see in the attempted backup logs:
ERROR: out of shared memory HINT: You might need to increase max_locks_per_transaction
The large volume of locks needed end up causing OOM conditions for the database. This isn't always a problem on Heroku. If you're using a production database you can rely on their point-in-time recovery option as your DR solution. That being said, exporting the data off Heroku will be difficult if you can't run a logical backup since they don't currently support external replication. It's less than ideal but hypothetically you could try to dump the database schema by schema in order avoid the OOM conditions.
I have a PostgreSQL database on my centos server.
Unfortunately since yesterday, in all existing schemas, all the tables are lost.
I saw log file and there was an unexpected reboot in recent days, probably a crash of the o.s.
Now the Postgres server it start correctly and I can view triggers, sequences and there aren't other problems.
Can I do something to recovery these tables?
Thanks.
we are using Heroku Postgres with Ruby on Rails 3.2.
A few days before, we deleted important data by mistake using 'heroku run db:load' with misconfigured data.yml, that is, drop tables and the recreate tables with almost no data.
Backup data is only available 2 weeeks before, so we lost data of 2 weeks.
So We need to recover not by PG Backup/pg_dump but by postgresql's system data files.
I think, the only way to recover data is to restore data from xlog or archive file, but of course we don't have permission to be Super User/Replication Role to copy postgres database on heroku (or Amazon EC2) to local server.
Is there anyone who confronted such a case and resolved the problem?
Your only option is the backups provided by the PgBackups service (if you had that running). If not, Heroku support might have more options available.
At a minimum, you will have some data loss, but you can guarantee you won't do it again ;)