sqlite3 trigger to convert hex text into binary blob equivalent - triggers

I'd like an after insert trigger to convert a hex text string into its binary blob equivalent.
I've tried something like this:
CREATE TABLE data
(
t_hex TEXT,
b_hex BLOB
);
CREATE TRIGGER data_insert_trigger AFTER INSERT ON data
BEGIN
UPDATE data SET b_hex = "x''"||t_hex||"''" WHERE rowid = new.rowid;
END;
INSERT into data(t_hex) VALUES ('A5A5');
this results in:
sqlite> select * from data;
t_hex = A5A5
b_hex = x''A5A5''
also tried
CREATE TRIGGER data_insert_trigger AFTER INSERT ON data
BEGIN
UPDATE data SET b_hex = x''||t_hex||'' WHERE rowid = new.rowid;
END;
this results in:
sqlite> select * from data;
t_hex = A5A5
b_hex = A5A5
anyone know of a way to utilize the x'value' syntax yet reference an existing column value or perhaps some other SQL-based mechanism?
** EDIT **
Taking into consideration the custom function, thanks CL and LS_dev. For the SQL only solution that LS_dev provided here's where I'm at. I tweaked data_t_update_trigger to
CREATE TRIGGER data_t_update_trigger
AFTER UPDATE ON data
WHEN NEW.t_hex IS NOT OLD.t_hex
BEGIN
UPDATE data SET b_hex = x'' WHERE ROWID = NEW.ROWID;
END;
and my test set generated:
sqlite> insert into data(t_hex) values('A5A5');
sqlite> select t_hex, hex(b_hex) from data;
A5A5|A5A5
sqlite> update data set t_hex = 'FF';
sqlite> select t_hex, hex(b_hex) from data;
FF|FF
sqlite> update data set t_hex = 'FFFE';
sqlite> select t_hex, hex(b_hex) from data;
FFFE|FFFE3F
sqlite> update data set t_hex = '00';
Error: too many levels of trigger recursion
working through that I qualified a few lines in the data_h_update_trigger as:
CREATE TRIGGER data_b_update_trigger
AFTER UPDATE ON data
WHEN LENGTH(NEW.t_hex)>LENGTH(NEW.b_hex)*2
BEGIN
UPDATE data SET b_hex = NEW.b_hex||COALESCE((
SELECT b FROM _hb WHERE h=SUBSTR(NEW.t_hex, (LENGTH(NEW.b_hex)*2)+1, 2)
), CAST('?' AS BLOB)) WHERE ROWID = NEW.ROWID;
END;
and now my test set yields:
sqlite> select t_hex, hex(b_hex) from data;
A5A5|A5A5
sqlite> update data set t_hex = 'FF';
sqlite> select t_hex, hex(b_hex) from data;
FF|FF
sqlite> update data set t_hex = 'FFFE';
sqlite> select t_hex, hex(b_hex) from data;
FFFE|FFFE
sqlite> update data set t_hex = '00';
Error: too many levels of trigger recursion
so still dealing with some unexplained recursion. FWIW, this also happens with a statement like this:
sqlite> update data set t_hex = 'DEADBEEF';
Error: too many levels of trigger recursion
running:
SQLite version 3.7.9 2011-11-01 00:52:41

Using a user defined function would be much easier and inexpensive, but I got a solution.
First, you need a look-up table as this:
CREATE TABLE _hb(h TEXT COLLATE NOCASE, b BLOB);
BEGIN;
INSERT INTO _hb VALUES('00', x'00');
INSERT INTO _hb VALUES('01', x'01');
(...)
INSERT INTO _hb VALUES('A4', x'A4');
INSERT INTO _hb VALUES('A5', x'A5');
INSERT INTO _hb VALUES('A6', x'A6');
(...)
INSERT INTO _hb VALUES('FE', x'FE');
INSERT INTO _hb VALUES('FF', x'FF');
COMMIT;
Then, enabling recursive triggers (SQLite>=3.6.18):
PRAGMA RECURSIVE_TRIGGERS=1;
you may create a trigger which will progressively append bytes to b_hex:
CREATE TRIGGER data_h_update_trigger
AFTER UPDATE ON data
WHEN LENGTH(NEW.t_hex)>LENGTH(NEW.b_hex)*2
BEGIN
UPDATE data SET b_hex = b_hex||COALESCE((
SELECT b FROM _hb WHERE h=SUBSTR(NEW.t_hex, LENGTH(b_hex)*2+1, 2)
), CAST('?' AS BLOB)) WHERE ROWID = NEW.ROWID;
END;
Two more triggers to trigger data_h_update_trigger upon data insertion or t_hex update:
CREATE TRIGGER data_insert_trigger
AFTER INSERT ON data
BEGIN
UPDATE data SET b_hex = x'' WHERE ROWID = NEW.ROWID;
END;
CREATE TRIGGER data_t_update_trigger
AFTER UPDATE OF t_hex ON data
BEGIN
UPDATE data SET b_hex = x'' WHERE ROWID = NEW.ROWID;
END;
Limitations: single step blob calculation limited to SQLITE_MAX_TRIGGER_DEPTH (1000 by default) bytes.

Blob literals can be used only directly in the SQL statement; they cannot be constructed dynamically from SQL code.
To convert a hex string into a blob, you would have to install your own user-defined function.

Related

how to iterate in all schemas and find count from all tables present in all schemas with same table name for every 5mins?

imagine there are 5 schemas in my database and in every schema there is a common name table (ex:- table1) after every 5mins records get inserted in table1, how I can iterate in all schemas n calculate the count of table1[i have to automate the process so i am going to write the code in function and call that function after every 5mins using crontab].
Basically 2 options: Hard code schema.table and union the results. So something like:
create or replace function count_rows_in_each_table1()
returns table (schema_name text, number_or_rows integer)
language sql
as $$
select 'schema1', count(*) from schema1.table1 union all
select 'schema2', count(*) from schema2.table1 union all
select 'schema3', count(*) from schema3.table1 union all
...
select 'scheman', count(*) from scheman.table1;
$$;
The alternative being building the query dynamically from information_scheme.
create or replace function count_rows_in_each_table1()
returns table (schema_name text, number_of_rows bigint)
language plpgsql
as $$
declare
c_rows_count cursor is
select table_schema::text
from information_schema.tables
where table_name = 'table1';
l_tbl record;
l_sql_statement text = '';
l_connector text = '';
l_base_select text = 'select ''%s'', count(*) from %I.table1';
begin
for l_tbl in c_rows_count
loop
l_sql_statement = l_sql_statement ||
l_connector ||
format (l_base_select, l_tbl.table_schema, l_tbl.table_schema);
l_connector = ' union all ';
end loop;
raise notice E'Running Query: \n%', l_sql_statement;
return query execute l_sql_statement;
end;
$$;
Which is better. With few schema and few schema add/drop, opt for the first. It is direct and easily shows what you are doing. If you add/drop schema often then opt for the second. If you have many schema, but seldom add/drop them then modify the second to generate the first, save and schedule execution of the generated query.
NOTE: Not tested

Is it worth Parallel/Concurrent INSERT INTO... (SELECT...) to the same Table in Postgres?

I was attempting an INSERT INTO.... ( SELECT... ) (inserting a batch of rows from SELECT... subquery), onto the same table in my database. For the most part it was working, however, I did see a "Deadlock" exception logged every now and then. Does it make sense to do this or is there a way to avoid a deadlock scenario? On a high-level, my queries both resemble this structure:
CREATE OR REPLACE PROCEDURE myConcurrentProc() LANGUAGE plpgsql
AS $procedure$
DECLARE
BEGIN
LOOP
EXIT WHEN row_count = 0
WITH cte AS (SELECT *
FROM TableA tbla
WHERE EXISTS (SELECT 1 FROM TableB tblb WHERE tblb.id = tbla.id)
INSERT INTO concurrent_table (SELECT id FROM cte);
COMMIT;
UPDATE log_tbl
SET status = 'FINISHED',
WHERE job_name = 'tblA_and_B_job';
END LOOP;
END
$procedure$;
And the other script that runs in parallel and INSERTS... also to the same table is also basically:
CREATE OR REPLACE PROCEDURE myConcurrentProc() LANGUAGE plpgsql
AS $procedure$
DECLARE
BEGIN
LOOP
EXIT WHEN row_count = 0
WITH cte AS (SELECT *
FROM TableC c
WHERE EXISTS (SELECT 1 FROM TableD d WHERE d.id = tblc.id)
INSERT INTO concurrent_table (SELECT id FROM cte);
COMMIT;
UPDATE log_tbl
SET status = 'FINISHED',
WHERE job_name = 'tbl_C_and_D_job';
END LOOP;
END
$procedure$;
So you can see I'm querying two different tables in each script, however inserting into the same some_table. I also have the UPDATE... statement that writes to a log table so I suppose that could also cause issues. Is there any way to use BEGIN... END here and COMMIT to avoid any deadlock/concurrency issues or should I just create a 2nd table to hold the "tbl_C_and_D_job" data?

Postgres create universal function on all table

I have a few db tables.
I want write universtal postgres function on copy rows to history tables
I have tables:
table1
table1_h
table2
table2_h
I wrote function (with help stackoverflow)
CREATE OR REPLACE FUNCTION copy_history_f() RETURNS TRIGGER AS
$BODY$
DECLARE
tablename_h text:= TG_TABLE_NAME || '_h';
BEGIN
EXECUTE 'INSERT INTO ' || quote_ident(TG_TABLE_SCHEMA) || '.' || quote_ident(tablename_h) || ' VALUES (' || OLD.* ||')';
RETURN NULL;
END;
$BODY$
LANGUAGE plpgsql VOLATILE;
And functions was create, but after update is error.
ERROR: syntax error at or near ","
ROW 1: ...RT INTO table1_h VALUES ((12,,,0,,"Anto...
I know where is error in this insert but I don't know how I repair that.
Structure tables table1 and table1_h are identical but table1_h has one more column (id_h)
Can you help me, how I have create psql function?
Thnak you.
drop table if exists t;
drop table if exists t_h;
drop function if exists ftg();
create table t(i serial, x numeric);
insert into t(x) values(1.1),(2.2);
create table t_h(i int, x numeric);
create function ftg() returns trigger language plpgsql as $ftg$
declare
tablename_h text:= TG_TABLE_NAME || '_h';
begin
execute format($q$ insert into %I.%I select $1.*; $q$, TG_TABLE_SCHEMA, tablename_h) using old;
return null;
end $ftg$;
create trigger tg_t after delete on t for each row execute procedure ftg();
delete from t where i = 1;
select * from t_h;
dbfiddle
Update It solves your problem, but I think that you want to have a bit more info in your history tables. It will be more complex a bit:
drop table if exists t;
drop table if exists t_h;
drop function if exists ftg();
create table t(i serial, x numeric);
insert into t(x) values(1.1),(2.2);
create table t_h(
hi serial, -- just ID
hd timestamp, -- timestamp
hu text, -- user who made changes
ha text, -- action
i int, x numeric
);
create function ftg() returns trigger language plpgsql as $ftg$
declare
tablename_h text:= TG_TABLE_NAME || '_h';
begin
execute format(
$q$
insert into %I.%I
select
nextval(%L || '_hi_seq'),
clock_timestamp(),
current_user,
%L,
$1.*
$q$, TG_TABLE_SCHEMA, tablename_h, tablename_h, TG_OP) using old;
return null;
end $ftg$;
create trigger tg_t after delete or update on t for each row execute procedure ftg();
update t set x = x * 2;
update t set x = x * 2 where i = 2;
delete from t where i = 1;
select * from t_h;
dbfiddle
I assume you are inserting the 'old' values from table1 into table1_h.
The additional column is your problem. When you using an insert without naming columns you must use a matching number and type for the insert.
You must use column referencing.
eg.
Insert into table1_h(column1, column2, column3)
values (a,b,c)
Consider a default value for the additional column in table table1_h.

PostgreSQL trigger not firing on update

I am using PostgreSQL 9.3.4
There are two linked tables updated remotely: table1 and table2
table2 uses foreign key dependance from table1.
Tables are updated in the following manner on remote server:
Insert into table1 returning id;
Insert into table2 using previous id
Query is sent ( 1 transaction with 2 insert statements)
I need to duplicate new rows to remote db using dblink so I created two 'before update' triggers for table1 and table2;
The problem is that only table2 trigger is firing; the first isn't ( from remote update;
doing test query from pgadmin under the same user, I get both triggers fired OK )
I assumed it is because the update is being processed in 1 transaction/query on remote server. So I tried to process both tables in second trigger, but still no luck - only table2 is processed.
What could be the reason ?
Thanks
P.S.
Trigger codes
Version 1
PROCEDURE fn_replicate_data:
DECLARE
BEGIN
PERFORM DBLINK_EXEC('myconn','INSERT INTO table1(dataid,sessionid,uid) VALUES('||new.dataid||','||new.sessionid||',
'||new.uid||') ');
RETURN new;
END;
PROCEDURE fn_replicate_data2:
DECLARE
BEGIN
PERFORM DBLINK_EXEC('myconn','INSERT INTO table2(dataid,data) VALUES('||new.dataid||','''||new.data||''') ');
RETURN new;
END;
CREATE TRIGGER tr_remote_insert_data
BEFORE INSERT OR UPDATE ON table1
FOR EACH ROW EXECUTE PROCEDURE fn_replicate_data();
CREATE TRIGGER tr_remote_insert_data2
BEFORE INSERT OR UPDATE ON table2
FOR EACH ROW EXECUTE PROCEDURE fn_replicate_data2();
VERSION2
PROCEDURE fn_replicate_data:
DECLARE
var table1%ROWTYPE;
BEGIN
select * from table1 into var where dataid = new.dataid;
PERFORM DBLINK_EXEC('myconn','INSERT INTO table1(dataid,sessionid,uid) VALUES('||var.dataid||','||var.sessionid||','||var.uid||') ');
PERFORM
DBLINK_EXEC('myconn','INSERT INTO table2(dataid,data) VALUES('||new.dataid||','''||new.data||''') ');
RETURN new;
END;
CREATE TRIGGER tr_remote_insert_data
BEFORE INSERT OR UPDATE ON table2
FOR EACH ROW EXECUTE PROCEDURE fn_replicate_data();
The reason was in NULL value if uid field. It had bigint type and had no default value in db, which causes the trigger to not work properly.
The fix is either
IF (NEW.uid IS NULL )
THEN
uid := 'DEFAULT';
else
uid := NEW.uid;
END IF;
before insert query; or (simpler) adding default value to db;

Postgresql 9.1 select from all schemas

I have a Postgresql 9.1 database with couple hundred schemas. All have same structure, just different data. I need to perform a select on a table and get data from each schema. Unfortunately I haven't found a decent way to do it.
I tried setting the search path to schema_1,schema_2, etc and then perform a select on the table but it only selects data from the first schema.
The only way I managed to do it so far is by generating a big query like:
select * from schema_1.table
union
select * from schema_2.table
union
(...another 100 lines....)
Is there any other way to do this in a more reasonable fashion? If this is not possible, can I at least find out which of the schemas has records in that table without performing this select?
Different schemas mean different tables, so if you have to stick to this structure, it'll mean unions, one way or the other. That can be pretty expensive. If you're after partitioning through the convenience of search paths, it might make sense to reverse your schema:
Store a big table in the public schema, and then provision views in each of the individual schemas.
Check out this sqlfiddle that demonstrates my concept:
http://sqlfiddle.com/#!12/a326d/1
Also pasted inline for posterity, in case sqlfiddle is inaccessible:
Schema:
CREATE SCHEMA customer_1;
CREATE SCHEMA customer_2;
CREATE TABLE accounts(id serial, name text, value numeric, customer_id int);
CREATE INDEX ON accounts (customer_id);
CREATE VIEW customer_1.accounts AS SELECT id, name, value FROM public.accounts WHERE customer_id = 1;
CREATE VIEW customer_2.accounts AS SELECT id, name, value FROM public.accounts WHERE customer_id = 2;
INSERT INTO accounts(name, value, customer_id) VALUES('foo', 100, 1);
INSERT INTO accounts(name, value, customer_id) VALUES('bar', 100, 1);
INSERT INTO accounts(name, value, customer_id) VALUES('biz', 150, 2);
INSERT INTO accounts(name, value, customer_id) VALUES('baz', 75, 2);
Queries:
SELECT SUM(value) FROM public.accounts;
SET search_path TO 'customer_1';
SELECT * FROM accounts;
SET search_path TO 'customer_2';
SELECT * FROM accounts;
Results:
425
1 foo 100
2 bar 100
3 biz 150
4 baz 75
If you have to know some about data in tables, you have to do SELECT. There is no any other way. Schema is just logical addressing - for your case is important, so you use lot of tables, and you have to do massive UNION.
search_path works as expected. It has no meaning - return data from mentioned schemes, but it specify a order for searching not fully qualified table. Searching ends on first table, that has requested name.
Attention: massive unions can require lot of memory.
you can use a dynamic SQL and stored procedures with temp table:
postgres=# DO $$
declare r record;
begin
drop table if exists result;
create temp table result as select * from x.a limit 0; -- first table;
for r in select table_schema, table_name
from information_schema.tables
where table_name = 'a'
loop
raise notice '%', r;
execute format('insert into result select * from %I.%I',
r.table_schema,
r.table_name);
end loop;
end; $$;
result:
NOTICE: (y,a)
NOTICE: (x,a)
DO
postgres=# select * from result;
a
----
1
2
3
4
5
..
Here's one approach. You will need to pre-feed it all the schema names you are targeting. You could change this to just loop through all the schemas as Pavel shows if you know you want every schema. In my example I have three schemas that I care about each containing a table called bar. The logic will run a select on each schema's bar table and insert the value into a result table. At the end you have a table with all the data from all the tables. You could change this to update, delete, or do DDL. I chose to keep it simple and just collect the data from each table in each schema.
--START SETUP AKA Run This Section Once
create table schema3.bar(bar_id SERIAL PRIMARY KEY,
bar_name VARCHAR(50) NOT NULL);
insert into schema1.bar(bar_name) select 'One';
insert into schema2.bar(bar_name) select 'Two';
insert into schema3.bar(bar_name) select 'Three';
--END SETUP
DO $$
declare r record;
DECLARE l_id INTEGER = 1;
DECLARE l_schema_name TEXT;
begin
drop table if exists public.result;
create table public.result (bar_id INTEGER, bar_name TEXT);
drop table if exists public.schemas;
create table public.schemas (id serial PRIMARY KEY, schema_name text NOT NULL);
INSERT INTO public.schemas(schema_name)
VALUES ('schema1'),('schema2'),('schema3');
for r in select *
from public.schemas
loop
raise notice '%', r;
SELECT schema_name into l_schema_name
FROM public.schemas
WHERE id = l_id;
raise notice '%', l_schema_name;
EXECUTE 'set search_path TO ' || l_schema_name;
EXECUTE 'INSERT into public.result(bar_id, bar_name) select bar_id, bar_name from ' || l_schema_name || '.bar';
l_id = l_id + 1;
end loop;
end; $$;
--DEBUG
select * from schema1.bar;
select * from schema2.bar;
select * from schema3.bar;
select * from public.result;
select * from public.schemas;
--CLEANUP
--DROP TABLE public.result;
--DROP TABLE public.schemas;