I need a stored function which returns table and being returned table gathers 1 rows from all 9000 tables.
Prerequisites:
There are same structure 9000 tables.
Table name are subsquent, like X1000 X1001 .... X9999.
Issue query to all table and get 1 rows from all table.
Gather all rows and return as 1 table.
Envirnment are PostgreSQL 9.3 and PL/Pgsql
I hope you will give me some easy example.
I think I can not understand how to deal variable value.
I made myself, however it didn't work.
create or replace function ListOfInc
(ddate int) returns table(col1 int, col2 dec) as $$
declare
numofrow integer;
begin
for i in 1000..9999 loop
numofrow := execute 'select count(*) from x' || i || ';';
if numofrow > 0 then
return query execute 'select ' || i || ' Trunc(100 * (CAST(Lag(adjust) over (order by'
|| %1 || ') as dec) / CAST(adjust as dec) - 1) ,2) from x' || i || ' order by ' || %1 || ' desc limit 1;';
end if;
end loop;
end;
$$ language plpgsql;
This is an ideal job for PostgreSQL's inheritance features.
If you want to avoid inheritance, you'll need to build a union all of all the tables using dynamic SQL, or loop over each table using RETURN QUERY EXECUTE ....
Personally I recommend consolodating all the data into one table. Create views to emulate the old structure, so your apps don't notice the change.
Related
When initializing my process, it runs the PL/pgSQL statement below creating two functions. However, every time I create multiple processes simultaneously as part of an end-to-end test, parallel execution of this statement leads to a tuple concurrently updated error that I can't seem to get around. Any help would be much appreciated.
CREATE OR REPLACE FUNCTION
count_rows(schema text, tablename text) returns integer
AS
$body$
DECLARE
result integer;
query varchar;
BEGIN
query := 'SELECT count(1) FROM "' || schema || '"."' || tablename || '"';
execute query into result;
return result;
END;
$body$
LANGUAGE plpgsql;
CREATE OR REPLACE FUNCTION
delete_if_empty(schema text, tablename text) RETURNS INTEGER
AS
$$
DECLARE
result integer;
query varchar;
BEGIN
query := 'SELECT COUNT(*) FROM "' || schema || '"."' || tablename || '"';
execute query into result;
IF result = 0 THEN
EXECUTE 'DROP TABLE "' || schema || '"."' || tablename || '" CASCADE;';
EXECUTE 'NOTIFY "' || schema || '", ''DESTROY_TABLE:' || tablename || ''';';
RETURN 1;
END IF;
RETURN 0;
END;
$$
LANGUAGE plpgsql;
SELECT version()
As described here, postgres doesn't currently allow you to use CREATE FUNCTION concurrently:
It'd be necessary to add
some kind of locking scheme if you want to avoid "tuple concurrently
updated" errors. This is not really any different from the situation
where two transactions both want to update the same row in a user table:
unless the application takes extra steps to serialize the updates, you're
going to get "tuple concurrently updated" errors.
We do have such locking for DDL on tables/indexes, but the theory in the
past has been that it's not worth the trouble for objects represented by
single catalog rows, such as functions or roles.
A solution to this is to ensure that no two transaction try to do the CREATE FUNCTION at the same time.
You can use posgres advisory locks for that.
A good introduction to advisory locks can be found here: https://vladmihalcea.com/how-do-postgresql-advisory-locks-work/
For example, you can use:
BEGIN; -- start of transaction
SELECT pg_advisory_xact_lock(2142616474639426746); -- random 64-bit signed ('bigint') lock number
CREATE OR REPLACE FUNCTION myfunction ...
COMMIT;
This takes a transaction-level exclusive advisory lock, so that no two concurrent transaction can run create the function at the same time. At the end of the transaction, the lock is automatically released.
I am trying to create crosstab queries in PostgreSQL such that it automatically generates the crosstab columns instead of hardcoding it. I have written a function that dynamically generates the column list that I need for my crosstab query. The idea is to substitute the result of this function in the crosstab query using dynamic sql.
I know how to do this easily in SQL Server, but my limited knowledge of PostgreSQL is hindering my progress here. I was thinking of storing the result of function that generates the dynamic list of columns into a variable and use that to dynamically build the sql query. It would be great if someone could guide me regarding the same.
-- Table which has be pivoted
CREATE TABLE test_db
(
kernel_id int,
key int,
value int
);
INSERT INTO test_db VALUES
(1,1,99),
(1,2,78),
(2,1,66),
(3,1,44),
(3,2,55),
(3,3,89);
-- This function dynamically returns the list of columns for crosstab
CREATE FUNCTION test() RETURNS TEXT AS '
DECLARE
key_id int;
text_op TEXT = '' kernel_id int, '';
BEGIN
FOR key_id IN SELECT DISTINCT key FROM test_db ORDER BY key LOOP
text_op := text_op || key_id || '' int , '' ;
END LOOP;
text_op := text_op || '' DUMMY text'';
RETURN text_op;
END;
' LANGUAGE 'plpgsql';
-- This query works. I just need to convert the static list
-- of crosstab columns to be generated dynamically.
SELECT * FROM
crosstab
(
'SELECT kernel_id, key, value FROM test_db ORDER BY 1,2',
'SELECT DISTINCT key FROM test_db ORDER BY 1'
)
AS x (kernel_id int, key1 int, key2 int, key3 int); -- How can I replace ..
-- .. this static list with a dynamically generated list of columns ?
You can use the provided C function crosstab_hash for this.
The manual is not very clear in this respect. It's mentioned at the end of the chapter on crosstab() with two parameters:
You can create predefined functions to avoid having to write out the
result column names and types in each query. See the examples in the
previous section. The underlying C function for this form of crosstab
is named crosstab_hash.
For your example:
CREATE OR REPLACE FUNCTION f_cross_test_db(text, text)
RETURNS TABLE (kernel_id int, key1 int, key2 int, key3 int)
AS '$libdir/tablefunc','crosstab_hash' LANGUAGE C STABLE STRICT;
Call:
SELECT * FROM f_cross_test_db(
'SELECT kernel_id, key, value FROM test_db ORDER BY 1,2'
,'SELECT DISTINCT key FROM test_db ORDER BY 1');
Note that you need to create a distinct crosstab_hash function for every crosstab function with a different return type.
Related:
PostgreSQL row to columns
Your function to generate the column list is rather convoluted, the result is incorrect (int missing after kernel_id), it can be replaced with this SQL query:
SELECT 'kernel_id int, '
|| string_agg(DISTINCT key::text, ' int, ' ORDER BY key::text)
|| ' int, DUMMY text'
FROM test_db;
And it cannot be used dynamically anyway.
#erwin-brandstetter: The return type of the function isn't an issue if you're always returning a JSON type with the converted results.
Here is the function I came up with:
CREATE OR REPLACE FUNCTION report.test(
i_start_date TIMESTAMPTZ,
i_end_date TIMESTAMPTZ,
i_interval INT
) RETURNS TABLE (
tab JSON
) AS $ab$
DECLARE
_key_id TEXT;
_text_op TEXT = '';
_ret JSON;
BEGIN
-- SELECT DISTINCT for query results
FOR _key_id IN
SELECT DISTINCT at_name
FROM report.company_data_date cd
JOIN report.company_data_amount cda ON cd.id = cda.company_data_date_id
JOIN report.amount_types at ON cda.amount_type_id = at.id
WHERE date_start BETWEEN i_start_date AND i_end_date
AND interval_type_id = i_interval
LOOP
-- build function_call with datatype of column
IF char_length(_text_op) > 1 THEN
_text_op := _text_op || ', ' || _key_id || ' NUMERIC(20,2)';
ELSE
_text_op := _text_op || _key_id || ' NUMERIC(20,2)';
END IF;
END LOOP;
-- build query with parameter filters
RETURN QUERY
EXECUTE '
SELECT array_to_json(array_agg(row_to_json(t)))
FROM (
SELECT * FROM crosstab(''SELECT date_start, at.at_name, cda.amount ct
FROM report.company_data_date cd
JOIN report.company_data_amount cda ON cd.id = cda.company_data_date_id
JOIN report.amount_types at ON cda.amount_type_id = at.id
WHERE date_start between $$' || i_start_date::TEXT || '$$ AND $$' || i_end_date::TEXT || '$$
AND interval_type_id = ' || i_interval::TEXT || ' ORDER BY date_start'')
AS ct (date_start timestamptz, ' || _text_op || ')
) t;';
END;
$ab$ LANGUAGE 'plpgsql';
So, when you run it, you get the dynamic results in JSON, and you don't need to know how many values were pivoted:
select * from report.test(now()- '1 week'::interval, now(), 1);
tab
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
[{"date_start":"2015-07-27T08:40:01.277556-04:00","burn_rate":0.00,"monthly_revenue":5800.00,"cash_balance":0.00},{"date_start":"2015-07-27T08:50:02.458868-04:00","burn_rate":34000.00,"monthly_revenue":15800.00,"cash_balance":24000.00}]
(1 row)
Edit: If you have mixed datatypes in your crosstab, you can add logic to look it up for each column with something like this:
SELECT a.attname as column_name, format_type(a.atttypid, a.atttypmod) AS data_type
FROM pg_attribute a
JOIN pg_class b ON (a.attrelid = b.relfilenode)
JOIN pg_catalog.pg_namespace n ON n.oid = b.relnamespace
WHERE n.nspname = $$schema_name$$ AND b.relname = $$table_name$$ and a.attstattarget = -1;"
I realise this is an older post but struggled for a little while on the same issue.
My Problem Statement:
I had a table with muliple values in a field and wanted to create a crosstab query with 40+ column headings per row.
My Solution was to create a function which looped through the table column to grab values that I wanted to use as column headings within the crosstab query.
Within this function I could then Create the crosstab query. In my use case I added this crosstab result into a separate table.
E.g.
CREATE OR REPLACE FUNCTION field_values_ct ()
RETURNS VOID AS $$
DECLARE rec RECORD;
DECLARE str text;
BEGIN
str := '"Issue ID" text,';
-- looping to get column heading string
FOR rec IN SELECT DISTINCT field_name
FROM issue_fields
ORDER BY field_name
LOOP
str := str || '"' || rec.field_name || '" text' ||',';
END LOOP;
str:= substring(str, 0, length(str));
EXECUTE 'CREATE EXTENSION IF NOT EXISTS tablefunc;
DROP TABLE IF EXISTS temp_issue_fields;
CREATE TABLE temp_issue_fields AS
SELECT *
FROM crosstab(''select issue_id, field_name, field_value from issue_fields order by 1'',
''SELECT DISTINCT field_name FROM issue_fields ORDER BY 1'')
AS final_result ('|| str ||')';
END;
$$ LANGUAGE plpgsql;
The approach described here worked well for me.
Instead of retrieving the pivot table directly. The easier approach is to let the function generate a SQL query string. Dynamically execute the resulting SQL query string on demand.
I have multiple tables with each two rows of interest: connection_node_start_id and connection_node_end_id. My goal is to get a collection of all those IDs, either as a flat ARRAY or as a new TABLE consisting of one row.
Example output ARRAY:
result = {1,4,7,9,2,5}
Example output TABLE:
IDS
-------
1
4
7
9
2
5
My fist attempt is somewhat clumsy and does not work properly as the SELECT statement just returns one row. It seems there must be a simple way to do this, can someone point me into the right direction?
CREATE OR REPLACE FUNCTION get_connection_nodes(anyarray)
RETURNS anyarray AS
$$
DECLARE
table_name varchar;
result integer[];
sel integer[];
BEGIN
FOREACH table_name IN ARRAY $1
LOOP
RAISE NOTICE 'table_name(%)',table_name;
EXECUTE 'SELECT ARRAY[connection_node_end_id,
connection_node_start_id] FROM ' || table_name INTO sel;
RAISE NOTICE 'sel(%)',sel;
result := array_cat(result, sel);
END LOOP;
RETURN result;
END
$$
LANGUAGE 'plpgsql';
Test table:
connection_node_start_id | connection_node_end_id
--------------------------------------------------
1 | 4
7 | 9
Call:
SELECT get_connection_nodes(ARRAY['test_table']);
Result:
{1,4} -- only 1st row, rest is missing
For Postgres 9.3+
CREATE OR REPLACE FUNCTION get_connection_nodes(text[])
RETURNS TABLE (ids int) AS
$func$
DECLARE
_tbl text;
BEGIN
FOREACH _tbl IN ARRAY $1
LOOP
RETURN QUERY EXECUTE format('
SELECT t.id
FROM %I, LATERAL (VALUES (connection_node_start_id)
, (connection_node_end_id)) t(id)'
, _tbl);
END LOOP;
END
$func$ LANGUAGE plpgsql;
Related answer on dba.SE:
SELECT DISTINCT on multiple columns
Or drop the loop and concatenate a single query. Probably fastest:
CREATE OR REPLACE FUNCTION get_connection_nodes2(text[])
RETURNS TABLE (ids int) AS
$func$
BEGIN
RETURN QUERY EXECUTE (
SELECT string_agg(format(
'SELECT t.id FROM %I, LATERAL (VALUES (connection_node_start_id)
, (connection_node_end_id)) t(id)'
, tbl), ' UNION ALL ')
FROM unnest($1) tbl
);
END
$func$ LANGUAGE plpgsql;
Related:
Loop through like tables in a schema
LATERAL was introduced with Postgres 9.3.
For older Postgres
You can use the set-returning function unnest() in the SELECT list, too:
CREATE OR REPLACE FUNCTION get_connection_nodes2(text[])
RETURNS TABLE (ids int) AS
$func$
BEGIN
RETURN QUERY EXECUTE (
SELECT string_agg(
'SELECT unnest(ARRAY[connection_node_start_id
, connection_node_end_id]) FROM ' || tbl
, ' UNION ALL '
)
FROM (SELECT quote_ident(tbl) AS tbl FROM unnest($1) tbl) t
);
END
$func$ LANGUAGE plpgsql;
Should work with pg 8.4+ (or maybe even older). Works with current Postgres (9.4) as well, but LATERAL is much cleaner.
Or make it very simple:
CREATE OR REPLACE FUNCTION get_connection_nodes3(text[])
RETURNS TABLE (ids int) AS
$func$
BEGIN
RETURN QUERY EXECUTE (
SELECT string_agg(format(
'SELECT connection_node_start_id FROM %1$I
UNION ALL
SELECT connection_node_end_id FROM %1$I'
, tbl), ' UNION ALL ')
FROM unnest($1) tbl
);
END
$func$ LANGUAGE plpgsql;
format() was introduced with pg 9.1.
Might be a bit slower with big tables because each table is scanned once for every column (so 2 times here). Sort order in the result is different, too - but that does not seem to matter for you.
Be sure to sanitize escape identifiers to defend against SQL injection and other illegal syntax. Details:
Table name as a PostgreSQL function parameter
The EXECUTE ... INTO statement can only return data from a single row:
If multiple rows are returned, only the first will be assigned to the INTO variable.
In order to concatenate values from all rows you have to aggregate them first by column and then append the arrays:
EXECUTE 'SELECT array_agg(connection_node_end_id) ||
array_agg(connection_node_start_id) FROM ' || table_name INTO sel;
You're probably looking for something like this:
CREATE OR REPLACE FUNCTION d (tblname TEXT [])
RETURNS TABLE (c INTEGER) AS $$
DECLARE sql TEXT;
BEGIN
WITH x
AS (SELECT unnest(tblname) AS tbl),
y AS (
SELECT FORMAT('
SELECT connection_node_end_id
FROM %s
UNION ALL
SELECT connection_node_start_id
FROM %s
', tbl, tbl) AS s
FROM x)
SELECT string_agg(s, ' UNION ALL ')
INTO sql
FROM y;
RETURN QUERY EXECUTE sql;
END;$$
LANGUAGE plpgsql;
CREATE TABLE a (connection_node_end_id INTEGER, connection_node_start_id INTEGER);
INSERT INTO A VALUES (1,2);
CREATE TABLE b (connection_node_end_id INTEGER, connection_node_start_id INTEGER);
INSERT INTO B VALUES (100, 101);
SELECT * from d(array['a','b']);
c
-----
1
2
100
101
(4 rows)
I have a database with multiple identical schemas. There is a number of tables all named 'tran_...' in each schema. I want to loop through all 'tran_' tables in all schemas and pull out records that fall within a specific date range. This is the code I have so far:
CREATE OR REPLACE FUNCTION public."configChanges"(starttime timestamp, endtime timestamp)
RETURNS SETOF character varying AS
$BODY$DECLARE
tbl_row RECORD;
tbl_name VARCHAR(50);
tran_row RECORD;
out_record VARCHAR(200);
BEGIN
FOR tbl_row IN
SELECT * FROM pg_tables WHERE schemaname LIKE 'ivr%' AND tablename LIKE 'tran_%'
LOOP
tbl_name := tbl_row.schemaname || '.' || tbl_row.tablename;
FOR tran_row IN
SELECT * FROM tbl_name
WHERE ch_edit_date >= starttime AND ch_edit_date <= endtime
LOOP
out_record := tbl_name || ' ' || tran_row.ch_field_name;
RETURN NEXT out_record;
END LOOP;
END LOOP;
RETURN;
END;
$BODY$
LANGUAGE plpgsql;
When I attempt to run this, I get:
ERROR: relation "tbl_name" does not exist
LINE 1: SELECT * FROM tbl_name WHERE ch_edit_date >= starttime AND c...
#Pavel already provided a fix for your basic error.
However, since your tbl_name is actually schema-qualified (two separate identifiers in : schema.table), it cannot be escaped as a whole with %I in format(). You have to escape each identifier individually.
Aside from that, I suggest a different approach. The outer loop is necessary, but the inner loop can be replaced with a simpler and more efficient set-based approach:
CREATE OR REPLACE FUNCTION public.config_changes(_start timestamp, _end timestamp)
RETURNS SETOF text AS
$func$
DECLARE
_tbl text;
BEGIN
FOR _tbl IN
SELECT quote_ident(schemaname) || '.' || quote_ident(tablename)
FROM pg_tables
WHERE schemaname LIKE 'ivr%'
AND tablename LIKE 'tran_%'
LOOP
RETURN QUERY EXECUTE format (
$$
SELECT %1$L || ' ' || ch_field_name
FROM %1$s
WHERE ch_edit_date BETWEEN $1 AND $2
$$, _tbl
)
USING _start, _end;
END LOOP;
RETURN;
END
$func$ LANGUAGE plpgsql;
You have to use dynamic SQL to parametrize identifiers (or code), like #Pavel already told you. With RETURN QUERY EXECUTE you can return the result of a dynamic query directly. Examples:
Return SETOF rows from PostgreSQL function
Refactor a PL/pgSQL function to return the output of various SELECT queries
Remember that identifiers have to be treated as unsafe user input in dynamic SQL and must always be sanitized to avoid syntax errors and SQL injection:
Table name as a PostgreSQL function parameter
Note how I escape table and schema separately:
quote_ident(schemaname) || '.' || quote_ident(tablename)
Consequently I just use %s to insert the already escaped table name in the later query. And %L to escape it a string literal for output.
I like to prepend parameter and variable names with _ to avoid naming conflicts with column names. No other special meaning.
There is a slight difference compared to your original function. This one returns an escaped identifier (double-quoted only where necessary) as table name, e.g.:
"WeIRD name"
instead of
WeIRD name
Much simpler yet
If possible, use inheritance to obviate the need for above function altogether. Complete example:
Select (retrieve) all records from multiple schemas using Postgres
You cannot use a plpgsql variable as SQL table name or SQL column name. In this case you have to use dynamic SQL:
FOR tran_row IN
EXECUTE format('SELECT * FROM %I
WHERE ch_edit_date >= starttime AND ch_edit_date <= endtime', tbl_name)
LOOP
out_record := tbl_name || ' ' || tran_row.ch_field_name;
RETURN NEXT out_record;
END LOOP;
I'm trying to write a function in PL/PgSQL that have to work with a table it receives as a parameter.
I use EXECUTE..INTO..USING statements within the function definition to build dynamic queries (it's the only way I know to do this) but ... I encountered a problem with RECORD data types.
Let's consider the follow (extremely simplified) example.
-- A table with some values.
DROP TABLE IF EXISTS table1;
CREATE TABLE table1 (
code INT,
descr TEXT
);
INSERT INTO table1 VALUES ('1','a');
INSERT INTO table1 VALUES ('2','b');
-- The function code.
DROP FUNCTION IF EXISTS foo (TEXT);
CREATE FUNCTION foo (tbl_name TEXT) RETURNS VOID AS $$
DECLARE
r RECORD;
d TEXT;
BEGIN
FOR r IN
EXECUTE 'SELECT * FROM ' || tbl_name
LOOP
--SELECT r.descr INTO d; --IT WORK
EXECUTE 'SELECT ($1)' || '.descr' INTO d USING r; --IT DOES NOT WORK
RAISE NOTICE '%', d;
END LOOP;
END;
$$ LANGUAGE plpgsql STRICT;
-- Call foo function on table1
SELECT foo('table1');
It output the following error:
ERROR: could not identify column "descr" in record data type
although the syntax I used seems valid to me. I can't use the static select (commented in the example) because I want to dinamically refer the columns names.
So..someone know what's wrong with the above code?
It's true. You cannot to use type record outside PL/pgSQL space.
RECORD value is valid only in plpgsql.
you can do
EXECUTE 'SELECT $1.descr' INTO d USING r::text::xx;
$1 should be inside the || ,like || $1 || and give spaces properly then it will work.
BEGIN
EXECUTE ' delete from ' || quote_ident($1) || ' where condition ';
END;