I have a Spring Data application connected to PostgreSQL and I enabled the slow log (statements of 100 ms). I noticed a number of these kinds of queries, a few per second:
2022-02-12 15:43:40.247 GMT [29629] LOG: duration: 130.888 ms
bind <unnamed>: SELECT * FROM
(SELECT n.nspname,c.relname,a.attname,a.atttypid,a.attnotnull OR (t.typtype = 'd' AND t.typnotnull)
AS attnotnull,a.atttypmod,a.attlen,t.typtypmod,row_number()
OVER (PARTITION BY a.attrelid ORDER BY a.attnum)
AS attnum, nullif(a.attidentity, '') as attidentity,pg_catalog.pg_get_expr(def.adbin, def.adrelid) AS adsrc,dsc.description,t.typbasetype,t.typtype
FROM pg_catalog.pg_namespace n
JOIN pg_catalog.pg_class c ON (c.relnamespace = n.oid)
JOIN pg_catalog.pg_attribute a ON (a.attrelid=c.oid)
JOIN pg_catalog.pg_type t ON (a.atttypid = t.oid)
LEFT JOIN pg_catalog.pg_attrdef def ON (a.attrelid=def.adrelid AND a.attnum = def.adnum)
LEFT JOIN pg_catalog.pg_description dsc ON (c.oid=dsc.objoid AND a.attnum = dsc.objsubid)
LEFT JOIN pg_catalog.pg_class dc ON (dc.oid=dsc.classoid AND dc.relname='pg_class')
LEFT JOIN pg_catalog.pg_namespace dn ON (dc.relnamespace=dn.oid AND dn.nspname='pg_catalog')
WHERE c.relkind in ('r','p','v','f','m') and a.attnum > 0 AND NOT a.attisdropped
AND n.nspname LIKE 'public' AND c.relname LIKE 'assertion') c WHERE true
ORDER BY nspname,c.relname,attnum
This is not a query my code issuesl my guess is that this query is being issued as part of a JDBC result set mapping. But it looks very inefficient running the same query repeatedly.
The property_assertion table is one of mine, but this query is issued for a number of tables in addition to this one. So this isn't confined to just one query meaning there's no specific code I snippet I can narrow this down to.
Is it a problem seeing these repeatedly, at 100ms a go? Might it be the result of misconfiguration?
Related
"select * from pg_stat_activity" for listing the existing connections, there are 100+ open connections with the state "idle in transaction" with the below query
Query :
SELECT c.oid, a.attnum, a.attname, c.relname, n.nspname,
a.attnotnull OR (t.typtype = 'd' AND t.typnotnull),
a.attidentity != '' OR pg_catalog.pg_get_expr(d.adbin, d.adrelid) LIKE '%nextval(%'
FROM pg_catalog.pg_class c
JOIN pg_catalog.pg_namespace n ON (c.relnamespace = n.oid)
JOIN pg_catalog.pg_attribute a ON (c.oid = a.attrelid)
JOIN pg_catalog.pg_type t ON (a.atttypid = t.oid)
LEFT JOIN pg_catalog.pg_attrdef d ON (d.adrelid = a.attrelid AND d.adnum = a.attnum)
JOIN (SELECT 21826 AS oid , 9 AS attnum UNION ALL SELECT 21826, 3) vals ON (c.oid = vals.oid AND a.attnum = vals.attnum)
Google says, its the query for getting the metadata and its generated by framework.
I have checked my code base and there is no place where I called this query. I wonder, why these connections are still in *"idle in transaction" state and is still open; which causing the busy connection.
How can I find all Views and Tables with rules which depend on a given Table?
I need this in order to find which views and tables I have to check if I want to alter/drop that given Table.
Get Views or Tables which refer to a given table TABLENAME:
SELECT cl_r.relname AS ref_table
FROM pg_rewrite AS r
JOIN pg_class AS cl_r ON r.ev_class=cl_r.oid
JOIN pg_depend AS d ON r.oid=d.objid
JOIN pg_class AS cl_d ON d.refobjid=cl_d.oid
WHERE cl_d.relkind IN ('r','v') AND cl_d.relname='TABLENAME'
GROUP BY cl_r.relname
ORDER BY cl_r.relname;
I did it by reversing the following answer: https://stackoverflow.com/a/4337615
If I understand correctly, you want the rules not the relations. If you want the custom defined ones you can:
SELECT * from pg_rules WHERE tablename = 'TABLENAME'
if you want to see the system defines ones (for examples the one from the views) you can:
SELECT
N.nspname AS schemaname,
C.relname AS tablename,
R.rulename AS rulename,
pg_get_ruledef(R.oid) AS definition
FROM (pg_rewrite R JOIN pg_class C ON (C.oid = R.ev_class))
LEFT JOIN pg_namespace N ON (N.oid = C.relnamespace)
WHERE C.relname = 'TABLENAME';
I just took the query from the definition of pg_rules that is actually a system view (with \d+ pg_rules), and changed the WHERE clause, that is normally:
WHERE r.rulename <> '_RETURN'::name;
Assume I have a query like this:
SELECT *
FROM clients c
INNER JOIN clients_balances cb ON cb.id_clients = c.id
LEFT JOIN clients com ON com.id = c.id_companies
LEFT JOIN clients com_real ON com_real.id = c.id_companies_real
LEFT JOIN rate_tables rt_orig ON rt_orig.id = c.orig_rate_table
LEFT JOIN rate_tables rt_term ON rt_term.id = c.term_rate_table
LEFT JOIN payment_terms pt ON pt.id = c.id_payment_terms
LEFT JOIN paygw_clients_profiles cpgw ON (cpgw.id_clients = c.id AND cpgw.id_companies = c.id_companies_real)
WHERE
EXISTS (SELECT * FROM accounts WHERE (name LIKE 'x' OR accname LIKE 'x' OR ani LIKE 'x') AND id_clients = c.id)
AND c."type" = '0'
AND c."id" > 0
ORDER BY c."name";
This query takes around 35 seconds to run when used in the production environment ("clients" has about 1 million records). However, if I take out ANY join - the query will take only about 300 ms to execute.
I've played around with the query planner settings, but to no avail.
Here are a few explain analyze outputs:
http://explain.depesz.com/s/hzy (slow - 48049.574 ms)
http://explain.depesz.com/s/FWCd (fast - 286.234 ms, rate_tables JOIN removed)
http://explain.depesz.com/s/MyRf (fast - 539.733 ms, paygw_clients_profiles JOIN removed)
It looks like in the fast case the planner starts from the EXISTS statement and has to perform join for only two rows in total. However, in the slow case it will first join all the tables and then filter by EXISTS.
What I need to do is to make this query run in a reasonable time with all seven join in place.
Postgres version is 9.3.10 on CentOS 6.3.
Thanks.
UPDATE
Rewriting the query like this:
SELECT *
FROM clients c
INNER JOIN clients_balances cb ON cb.id_clients = c.id
INNER JOIN accounts a ON a.id_clients = c.id AND (a.name = 'x' OR a.accname = 'x' OR a.ani = 'x')
LEFT JOIN clients com ON com.id = c.id_companies
LEFT JOIN clients com_real ON com_real.id = c.id_companies_real
LEFT JOIN rate_tables rt_orig ON rt_orig.id = c.orig_rate_table
LEFT JOIN rate_tables rt_term ON rt_term.id = c.term_rate_table
LEFT JOIN payment_terms pt ON pt.id = c.id_payment_terms
LEFT JOIN paygw_clients_profiles cpgw ON (cpgw.id_clients = c.id AND cpgw.id_companies = c.id_companies_real)
WHERE
c."type" = '0' AND c.id > 0
ORDER BY c."name";
makes it run fast, however, this is not acceptable, as account filtration parameters are optional, and I still need the result if there are no matches in that table. Using "LEFT JOIN accounts" instead of "INNER JOIN accounts" kills the performance again.
As suggested by Tome Lane, I've changed the following two parameters: join_collapse_limit and from_collapse_limit to 10 instead of the default 8, and this solved the issue.
I have this query and taking so much time to run , is there any way i can improve it , with creating any new views ,,, i can`t create temp db tables as they are avoid by DBA because of load on server
SELECT DISTINCT o.email AS eml_addr,
UPPER(max(FORMAT(cl.clicktime, 'dd-MMM-yyyy'))) AS lst_clk_mnth,
UPPER(max(FORMAT(o.opentime, 'dd-MMM-yyyy'))) AS lst_opn_mnth
FROM [open] o WITH (NOLOCK)
INNER JOIN view_mailing m WITH (NOLOCK)
ON (o.mailingid = m.mailingid)
INNER JOIN view_campaign c WITH (NOLOCK)
ON (c.campaignid = m.campaignid)
LEFT JOIN click cl WITH (NOLOCK)
ON (
o.mailingid = cl.mailingid
AND o.email = cl.email
)
WHERE c.clientid IN (
SELECT clientid
FROM client
WHERE sameclient IN (
SELECT sameclient
FROM client
WHERE clientid IN (219)
)
)
AND CONVERT(VARCHAR(8), cast(o.opentime AS DATETIME), 112) & gt;= 20130826
GROUP BY o.email;
Thanks
I don't see the purpose of both a distinct and group by
A derived table does not use indexes
Date in a varchar is not optimal
SELECT o.email AS eml_addr,
UPPER(max(FORMAT(cl.clicktime, 'dd-MMM-yyyy'))) AS lst_clk_mnth,
UPPER(max(FORMAT(o.opentime, 'dd-MMM-yyyy'))) AS lst_opn_mnth
FROM [open] o WITH (NOLOCK)
LEFT JOIN click cl WITH (NOLOCK)
ON o.mailingid = cl.mailingid
AND o.email = cl.email
JOIN view_mailing m WITH (NOLOCK)
ON o.mailingid = m.mailingid
JOIN view_campaign c WITH (NOLOCK)
ON c.campaignid = m.campaignid
join client with (NOLOCK)
on c.clientid = client.sameclient
and client.clientid IN (219)
WHERE CONVERT(VARCHAR(8), cast(o.opentime AS DATETIME), 112) & gt;= 20130826
GROUP BY o.email;
Totally confused and I have been working at this for 2 hours
I thought restriction on the left side of the join are honored
On this query I am getting [docSVsys].[visibility] 1 and <> 1
I thought this would restrict [docSVsys].[visibility] to 1
select top 1000
[docSVsys].[sID], [docSVsys].[visibility]
,[Table].[sID],[Table].[enumID],[Table].[valueID]
from [docSVsys] with (nolock)
left Join [DocMVenum1] as [Table] with (nolock)
on [docSVsys].[visibility] in (1)
and [Table].[sID] = [docSVsys].[sID]
and [Table].[enumID] = '140'
and [Table].[valueID] in (1,7)
This works
select top 1000
[docSVsys].[sID], [docSVsys].[visibility]
,[Table].[sID],[Table].[enumID],[Table].[valueID]
from [docSVsys] with (nolock)
left Join [DocMVenum1] as [Table] with (nolock)
on [Table].[sID] = [docSVsys].[sID]
and [Table].[enumID] = '140'
and [Table].[valueID] in (1,7)
where [docSVsys].[visibility] in (1)
I am just having a really off day as I had it in my mind the left side honored the join
SELECT *
FROM A
LEFT JOIN B ON Condition
is equivalent to
SELECT *
FROM A
CROSS JOIN B
WHERE Condition
UNION ALL
SELECT A.*, NULL AS B
FROM A
WHERE NOT EXISTS (SELECT * FROM B WHERE Condition)
Some rough pseudo-code...
Note, that all rows from A get through. It's just that the columns from B can be NULL if the join fails for some particular row of A.
Put the filter on docSVsys into the WHERE clause.
LEFT JOINs preserve all rows from the left (first) table, no matter what. The condition in the ON clause is only for matching which rows from the right/second table should be paired with rows from the left/first table.
If you want to exclude some rows from the firs table, use the WHERE clause:
select top 1000
[docSVsys].[sID], [docSVsys].[visibility]
,[Table].[sID],[Table].[enumID],[Table].[valueID]
from [docSVsys] with (nolock)
left Join [DocMVenum1] as [Table] with (nolock)
on [Table].[sID] = [docSVsys].[sID]
and [Table].[enumID] = '140'
and [Table].[valueID] in (1,7)
where [docSVsys].[visibility] in (1)