Postgresql - Why is DROP VIEW command hanging? - postgresql

I want to perform a simple DROP VIEW ... but it hangs.
I have run this query SELECT * FROM pg_locks WHERE NOT granted taken from this page on Lock Monitoring.
However the following query they suggest returns no results:
SELECT bl.pid AS blocked_pid,
a.usename AS blocked_user,
kl.pid AS blocking_pid,
ka.usename AS blocking_user,
a.query AS blocked_statement
FROM pg_catalog.pg_locks bl
JOIN pg_catalog.pg_stat_activity a ON a.pid = bl.pid
JOIN pg_catalog.pg_locks kl ON kl.transactionid = bl.transactionid AND kl.pid != bl.pid
JOIN pg_catalog.pg_stat_activity ka ON ka.pid = kl.pid
WHERE NOT bl.granted;
Where should I look now ?

Finally I figure out what was wrong. Here are the steps to find the root cause:
Solution
Step 1 : List requested locks not granted
select * from pg_locks where not granted;
In my case, an attempt to lock, with the mode AccessExclusiveLock, the view I want to drop was not granted. This is why my DROP VIEW... hangs.
Step 2 : Find which other process(es) held a conflicting lock
select * from pg_locks where relation = <oid_of_view>
Here I list all processes locking or trying to lock on my view. I found out two processes, the one that want to drop the view and... another one.
Step 3 : Find out what other process(es) is/are doing now
select xact_start,query_start,backend_start,state_change,state from pg_stat_activity where pid in (<list_of_other_process(es)_pid>);
I had only one process holding a lock in my case. Surprisingly, its state was : idle in transaction
I was not able to drop the view because another process was idle in transaction. I simply kill it to solve my issue. For example, if the procpid was 8484 and let's suppose my postgresql server runs on a Linux box, then in the shell, I execute the following command:
$ kill -9 8484
Discussion
If you face similar issue, you can quickly find out what's going on by reproducing steps 1,2,3. You may need to customize Step 2 in order to find other conflicting process(es).
References
Lock Monitoring
Lock Dependency Information
View Postgresql Locks

I had a similar problem but the accepted answer didn't work for me as I do not have admin access to kill any process. Instead, this is how I managed to solve the problem:
Issue SELECT * FROM pg_stat_activity; to get the stats about the PostgreSQL activities.
In query column, look for the queries that read from that view. You may choose to narrow down your search by only looking into the rows related to your username (using username column) or query_start if you know when the issue emerged. There could be more than one row associated with your unwanted view.
Identify all pid from the rows in the above step and plug them into SELECT pg_terminate_backend(<pid>); (instead of <pid>) one by one and run them.
Now you should be able to drop your view.
Please note that as you terminate the backend processes using pg_terminate_backend(), you may face some errors. The reason is that terminating some process may automatically end other processes. Therefore, some of the identified PIDs might be invalid by the time.

As a summary, this solution from comments worked for me:
Step 1: Find the pid:
select * from pg_stat_activity
where pid in
(select pid from pg_locks
where relation =
(select relation from pg_locks where not granted));
Step 2: kill pid:
kill -9 pid

Related

postgres delete query is hanging forever, although its associated transaction is in progress and all locks are granted

I have a query doing some rolling delete from old entries from my database:
delete from my_table where last_updated_at < now() - '1 day'::interval
This query gets triggered every 15 Minutes. As the query stopped working some time ago, we started investigating our database and noticed that there is one transaction having all required locks to do its work, but it hangs and blocks all subsequent ones.
The results of our analysis follow:
Using this script we got the following results:
It clearly shows, that one of the delete queries, whose pid is 2674 somehow blocked all other delete queries causing a "traffic jam". In order to see, whether the process 2674 is waiting for any locks, we issued following query:
select relation::regclass, * from pg_locks l
join pg_stat_activity a on l.pid = a.pid
where not granted and a.pid = 2674 order by query_start ;
but the result was empty. A quick look in the lock table revealed to us, that the transaction id associated with this pid is 1838047967:
select l.transactionid from pg_locks l where l.pid = 2674 and l.transactionid is not null;
1838047967
Quick check of the transaction status shows, that the transaction is running:
SELECT pg_xact_status (xid8 '1838047967');
in progress
The problem is, that it has not been doing anything for more than 2 days.
Funny enough, after canceling the backend 2674 via command
SELECT pg_cancel_backend(2674);
the next backend showed in the tree above, with pid 4679, overtakes the role of the canceled backend and all queries listed in this post yield similar results for this new backend (i.e. in progress transaction with all locks granted)
Following is the result of the query:
SELECT a.pid, pg_blocking_pids(a.pid), a.query, a.query_start,
l.transactionid,
l.mode,
l.GRANTED
FROM pg_stat_activity a
join pg_locks l on l.pid = a.pid
WHERE backend_type = 'client backend'
order by a.query_start ;
after canceling backend 2674, with backend 4679 being the offending one (marked in blue). It is clearly visible that backend 4679 got all locks and is blocking subsequently the backend with pid 4555 and therefore the latter cannot acquire ShareLock.
The entry in pg_stat_activity for this backed looks like following:
To summarize:
We have an in progress transaction holding all required locks to do its delete work, but this transaction has been hanging for more than 2 days, and it blocked all subsequent transactions. The question is, what can we check to find out what is causing this transaction to hang.
Postgres Version: 14.3

AccessExclusive lock origin

I am running the following query in first transaction:
BEGIN ISOLATION LEVEL repeatable read;
SELECT balance from "Users" WHERE id = 1 FOR UPDATE;
UPDATE "Users" SET "balance"="balance"+1 WHERE id = 1;
Then the second, from different connection which is exactly same. On SELECT ... FOR UPDATE it's waiting.
Then I run the following to see the locks
select t.relname,l.locktype,l.tuple,page,virtualtransaction,pid,mode,granted from pg_locks l, pg_stat_all_tables t where l.relation=t.relid order by relation asc;
It shows row locks (ROW SHARE, ROW EXCLUSIVE), which is fine. But I also see
Users tuple 9 0 11/17085 199957 AccessExclusiveLock TRUE
According to documentation AccessExclusiveLock comes from:
Acquired by the ALTER TABLE, DROP TABLE, TRUNCATE, REINDEX, CLUSTER,
and VACUUM FULL commands. This is also the default lock mode for LOCK
TABLE statements that do not specify a mode explicitly.
I am not explicitly doing any of them and I can't find in docs how implicitly this lock is acquired. Moreover, what means tuple lock type and 9?
update #1:
I used the following query to get more info:
SELECT a.datname,
l.relation::regclass,
l.transactionid,
l.mode,
l.GRANTED,
a.usename,
a.query,
a.query_start,
age(now(), a.query_start) AS "age",
a.pid
FROM pg_stat_activity a
JOIN pg_locks l ON l.pid = a.pid
ORDER BY a.query_start;
So indeed database is same, query is the one that is blocked.
mydb "Users" AccessExclusiveLock TRUE appuserdev SELECT balance from "Users" WHERE id = 1 for update; 2021-08-30 08:06:38.864007+00 00:08:06.978082 205464
The docs you are reading apply to AccessExclusiveLock on tables. But what you are seeing is not a table lock. I don't think the lock modes for non-tables are explicitly documented anywhere, but you can generally figure it out through analogy.
On the other hand, I also don't see what you see. I see an ExclusiveLock, not an AccessExclusiveLock, being held on the tuple.

pgpool-II does not cache some queries even though it should

I have a PostgreSQL server running on 10.6 with an openSUSE distribution of Linux, and I just set up pgpool-II, to allow me to cache queries. It works mostly fine, but for unknown reasons, sometimes I get this warning message :
WARNING: memcache: adding table oid maps, failed to create directory:"/var/log/pgpool/oiddir". error:"No such file or directory"
I already created the directory, changed the owner to the user that runs the pgpool server, and allowed read, write, and execution on this directory to the same user.
This message appears when a query is not yet cached, and it doesn't seem to have any impact, i.e. the query is cached as it should and if I do it again, the result is directly pulled from the cache.
But I also have another problem, and I don't know if it's related to the first one :
When I write big queries (using JOINs, subqueries, lots of conditions, etc), pgpool-II will not cache their results (or cache it but not use it, I have no idea), even though the results are not big (less than 500 rows). Also, in this case, I don't get the "oid" warning message. I tried raising the different limits of shared memory pgpool-II is allowed to use (see documentation) but it changed nothing as I expected, because when pgpool-II fails to cache a query due to the lack of available shared memory, it's supposed to return a message like this one :
LOG: pid 13756: pool_add_temp_query_cache: data size exceeds memqcache_maxcache. current:4095 requested:111 memq_maxcache:4096
But in my case, I don't get any message. Examples of both cases below.
1st problem
Simple query, result is cached, "oid" error :
SELECT *
FROM some_table
WARNING: memcache: adding table oid maps, failed to create directory:"/var/log/pgpool/oiddir". error:"No such file or directory"
-- Doing it a second time will just give me the cached result without any warning, as expected
2nd problem
Complex query, result is not cached (or cached and not used), no warning/error message :
SELECT A.geom, A.id, to_char(C.timestamp, 'DD/MM/YY') as date, C.timestamp::time as time, ROUND(C.value) as value
FROM segments A, lines B, ( SELECT DISTINCT ON (B.id) B.id, A.timestamp, ROUND(A.nb1+A.nb2+A.nb3) as value
FROM records A
CROSS JOIN LATERAL (SELECT *
FROM points B
WHERE A.id = B.id
AND A.direction = B.direction
ORDER BY A.position <-> B.geom
LIMIT 1) AS B
ORDER BY B.id, A.timestamp DESC) AS C
WHERE A.id = B.id
AND B.id = C.id
AND A.direction = B.direction
AND B.direction = C.direction
-- Doing it a second time will just directly request the PostgreSQL server again, instead of pulling the result from the cache, as it should if the result was cached.
Your first issue looks like a permissions issue on the oiddir. Basically pgpool can't create the necessary directories for caching. Depending on how you start pgpool (through pgpoolAdmin or command line), you need to
Create the oiddir folder, mkdir /var/log/pgpool/oiddir
Give permissions of the folder to pgpool
Assuming pgpoolAdmin starts pgpool:
chown -R _www /var/log/pgpool/oiddir (I use pgpool on mac os, so the apache user is "_www", change accordingly)
Assuming you start pgpool from command line:
chown -R postgres /var/log/pgpool/oiddir (or you can use your current user instead of postgres)
Your second issue looks based on the first issue. Basically you can't cache because the oiddir isn't created. Fixing the first should allow you to cache.

POSTGRES: Find out how long a lock has been acquired

i'm currently using Postgres as a database engine for an application.
I currently have a situation where I have a lot of READ locks (AccessSharedLocks) that are present. I run the following query to check for the locks:
SELECT t.schemaname,
t.relname,
l.locktype,
l.page,
l.virtualtransaction,
l.pid,
l.mode,
l.granted
FROM pg_locks l
JOIN pg_stat_all_tables t ON l.relation = t.relid
WHERE t.schemaname <> 'pg_toast'::name AND t.schemaname <> 'pg_catalog'::name
What I would like to know is how long the lock was acquired for by a table. Is there any way that I can retrieve this information?
Thank you in advance.
The time when a lock has been taken is not available in PostgreSQL.
The best you can do is to take the transaction start time xact_start from pg_stat_activity, that is a lower boundary for the age of the lock.
Transactions should always be short, because long transactions they hold locks and keep autovacuum from doing its job.
If you have any long running transactions, that might be the problem that you have to fix. Then the locks won't be such a problem.

Postgresql Check if query is still running

At my work, I needed to build a new join table in a postgresql database that involved doing a lot of computations on two existing tables. The process was supposed to take a long time so I set it up to run over the weekend before I left on Friday. Now, I want to check to see if the query finished or not.
How can I check if an INSERT command has finished yet while not being at the computer I ran it on? (No, I don't know how many rows it was suppose to add.)
Select * from pg_stat_activity where state not ilike 'idle%' and query ilike 'insert%'
This will return all non-idle sessions where the query begins with insert, if your query does not show in this list then it is no longer running.
pg_stat_activity doc
You can have a look at the table pg_stat_activity which contains all database connections including active query, owner etc.
At https://gist.github.com/rgreenjr/3637525 there is a copy-able example how such a query could look like.