I am trying to put row level lock on a table in one postgres function.
do $$
declare tabname text :='locktest' ;
begin
execute 'create temp table temp1 as
select v.* from (
select row_number() over (partition by a.id) as row_num,a.*
from '||tabname||' a,locktest2 b where a.id=b.id and b.val=111
union all
select row_number() over (partition by a.id) as row_num,a.*
from '||tabname||' a,locktest2 b where a.id=b.id and b.val=222
)v where v.row_num=1 for update';
raise notice 'Completed';
end $$;
But while compiling it , getting below error.
ERROR: FOR UPDATE is not allowed with UNION/INTERSECT/EXCEPT
Please suggest.
The way the statement is written, the database does not know in which table to lock the rows.
Rewrite the query along these lines:
SELECT ... FROM
(SELECT ...
FROM tab1
WHERE ...
FOR NO KEY UPDATE) AS t1
UNION ALL
(SELECT ...
FROM tab2
WHERE ...
FOR NO KEY UPDATE) AS t2;
FOR NO KEY UPDATE is the correct lock if you plan to update. FOR UPDATE is the lock if you intend to delete.
Related
When conflict value goes to this query, update is not executed (concerning to updated_at field). But insert works perfect with such subquery as SELECT with joins
INSERT INTO history (tx_hash, user_address, address_from, address_to, value, type, status, submitted_at,
coin_id, network_id)
SELECT :tx_hash, :user_address, :address_from, :address_to, :value, :type, :status, :submitted_at,
c.id, n.id
FROM (VALUES (:coin_symbol, :network_symbol) ) x
INNER JOIN coins c on c.symbol = :coin_symbol
INNER JOIN networks n on n.symbol = :network_symbol
ON CONFLICT (tx_hash)
DO UPDATE SET (confirmations, status, updated_at) =
(excluded.confirmations, excluded.status, now());
Table structure gist
The table has many columns, but for the problematic part let's assume only two, [ID] and [dependency].
[Dependency] means which [ID]s should be listed before this [ID].
Each row has its unique [ID] but it might have none, one or more dependencies in the [Dependency] column
ID
Dependency
1
4
2
null
3
1,2
4
2
5
1,2,4
Expected Result
ID
Dependency
2
null
4
2
1
4
3
1,2
5
1,2,4
I have no prior experience in Postgres, I found this very useful:
SELECT aa.dep_id::integer FROM unnest(string_to_array(ss.dependency, ',')) aa(dep_id)
But still, I can't make it right.
EDIT: Added one row with 3 dependencies'
http://sqlfiddle.com/#!15/894c3/4
Use a recursive CTE:
WITH RECURSIVE o AS (
SELECT ss.id, ss.dependency,
1 AS level
FROM ss
WHERE dependency IS NULL
UNION ALL
SELECT ss.id, ss.dependency,
o.level + 1
FROM ss
JOIN o
ON o.id IN (SELECT x
FROM unnest(ss.dependency) AS x(x)
)
)
SELECT o.id, o.dependency
FROM o
GROUP BY o.id, o.dependency
ORDER BY max(o.level);
My working solution, I hope it can be improved
DO $do$
DECLARE v_RowCountInt Int;
--copy all values to temp table
begin
--This is the origin table, all rows to sort
drop table if exists _tempAll;
create temp table _tempAll as select id, dependency from XXXXX;
-- create temp table for results
drop table if exists _tempSort;
create temp table _tempSort (
id integer
,dependency varchar
,pos serial primary key);
-- move all IDs with no depencencies
WITH tmp AS (DELETE FROM _tempAll where dependency is null RETURNING id, dependency)
INSERT INTO _tempSort ( id, dependency)
SELECT id, dependency FROM tmp;
GET DIAGNOSTICS v_RowCountInt = ROW_COUNT;
while v_RowCountInt>0 loop
-- move rows with all dependencies met
WITH tmp AS (DELETE FROM _tempAll a
where a.id in(SELECT a.id FROM _tempSort s inner join _tempAll a on
s.id in (select split.dep_sid::integer from unnest(string_to_array(a.dependency, ',')) split(dep_sid))
group by a.id, a.dependency
-- verify all dependencies are already sorted
having count(*)=(CHAR_LENGTH(a.dependency) - CHAR_LENGTH(REPLACE(a.dependency, ',', ''))+1))
RETURNING a.id, a.dependency)
INSERT INTO _tempSort (id, dependency)
SELECT id, dependency FROM tmp;
GET DIAGNOSTICS v_RowCountInt = ROW_COUNT;
end loop;
end;
$do$
We have a static database we constantly update with loader scripts. These loader scripts get current information from third party sources, clean it and upload it to database.
I have already made some SQL scripts to ensure schemas and tables required exists. Now I'd like to check that each table has the expected row count.
I did something like this:
select case when count(*) = <someNumber>
then 'someSchema.someTable OK'
else 'someSchema.someTable BAD row count' end
from someSchema.someTable;
But doing these kind of queries for ~300 tables is cumbersome.
Now I was thinking maybe there's a way to have a table like:
create table expected_row_count (
schema_name varchar,
table_name varchar,
row_count bigint
);
And somehow test all listed tables and only output the ones that fail the count check. But I'm kind of missing now... Should I try to write a function? Can a table like this be used to build queries and execute them?
Whole credit goes to #a-horse_with*_no_name , I'm posting a reply for completeness:
Check row count
First let's create some data to test the query:
create schema if not exists data;
create table if not exists data.test1 (nothing int);
create table if not exists data.test2 (nothing int);
insert into data.test1 (nothing)
(select random() from generate_series(1, 28));
insert into data.test2 (nothing)
(select random() from generate_series(1, 55));
create table if not exists public.expected_row_count (
table_schema varchar not null default '',
table_name varchar not null default '',
row_count bigint not null default 0
);
insert into public.expected_row_count (table_schema, table_name, row_count) values
('data', 'test1', (select count(*) from data.test1)),
('data', 'test2', (select count(*) from data.test2))
;
Now the query to check the data:
select * from (
select
table_schema,
table_name,
(xpath('/row/cnt/text()', xml_count))[1]::text::int as row_count
from (
select
table_schema,
table_name,
query_to_xml(format('select count(*) as cnt from %I.%I', table_schema, table_name), false, true, '') as xml_count
from information_schema.tables
where table_schema = 'data' --<< change here for the schema you want
) infs ) as r
inner join expected_row_count erc
on r.table_schema = erc.table_schema
and r.table_name = erc.table_name
and r.row_count != erc.row_count
;
Previous query should give an empty results if all counts are ok, and the
tables with missing data if not. To check it, update the count for some
table on expected_row_count and re-run the query. For example:
update expected_row_count set row_count = 666 where table_name = 'test1';
I have a query for create table like below
DROP TABLE IF EXISTS BAJUL;
CREATE TABLE BAJUL AS (
SELECT dt_trx, row_number() OVER (ORDER BY dt_trx DESC) AS row_number
FROM stock_trx_idx
WHERE dt_trx BETWEEN '2017-01-01' AND '2017-02-28'
GROUP BY 1
ORDER BY 1 DESC);
How able to create above table with stored function in Postgresql?
I tried with below script
CREATE OR REPLACE FUNCTION my_function (dt1 DATE, dt2 DATE)
RETURNS VOID AS
$func$
BEGIN
EXECUTE format('
DROP TABLE IF EXISTS tblq;
CREATE TABLE IF NOT EXISTS tblq AS(
SELECT dt_trx, row_number() OVER (ORDER BY dt_trx DESC) AS row_number
FROM stock_trx
WHERE dt_trx BETWEEN dt1 AND dt2
GROUP BY 1
ORDER BY 1 DESC
)' );
END
$func$ LANGUAGE plpgsql;
but when I try to execute SF like below
SELECT my_function ('2017-01-01', '2017-02-28');
I got error --> ERROR: column "dt1" does not exist
Would like to seek your help.
Thanks & rgds,
Bayu
Use
format('CREATE ... WHERE dt_trx BETWEEN %L AND %L ...', dt1, dt2)
You error is obvious. In the SELECT statement,
SELECT dt_trx, row_number() OVER (ORDER BY dt_trx DESC) AS row_number
FROM stock_trx
WHERE dt_trx BETWEEN dt1 AND dt2
GROUP BY 1
ORDER BY 1 DESC
The dt1 column doesn't exist. You didn't tell him you wanted to use your variable. Try concatenate your string with your variables.
By the way, you can drop your ORDER BY if your creating a table with that statement.
how to retrieve duplicate records and delete them in table A, also insert these retrieved duplicate records in another table B (in postgres db)
SQL query's are required for my project.
To delete duplicates without having a unique column you can use the ctid virtual column which is essentially the same thing as the rowid in Oracle:
delete from table_A t1
where ctid <> (select min(t2.ctid)
from table_A t2
where t1.unique_column = t2.unique_column);
You can use the returning clause to get the deleted rows and insert them into the other table:
with deleted as (
delete from table_A x1
where ctid <> (select min(t2.ctid)
from table_A t2
where t1.unique_column = t2.unique_column);
returning *
)
insert into table_B (col_1, col_2)
select unique_column, some_other_column
from deleted;
If you further want to see those deleted rows, you can throw in another CTE:
with deleted as (
delete from table_A x1
where ctid <> (select min(t2.ctid)
from table_A t2
where t1.unique_column = t2.unique_column);
returning *
), moved as (
insert into table_B (col_1, col_2)
select unique_column, some_other_column
from deleted
returning *
)
select *
from moved;