The function is created fine, but when I try to execute it, I get this error:
ERROR: relation "column1" does not exist
SQL state: 42P01
Context: SQL statement "ALTER TABLE COLUMN1 ADD COLUMN locationZM geography (POINTZM, 4326)"
PL/pgSQL function addlocationzm() line 6 at SQL statement
Code:
CREATE OR REPLACE FUNCTION addlocationZM()
RETURNS void AS
$$
DECLARE
COLUMN1 RECORD;
BEGIN
FOR COLUMN1 IN SELECT f_table_name FROM *schema*.geography_columns WHERE type LIKE 'Point%' LOOP
ALTER TABLE COLUMN1 ADD COLUMN locationZM geography (POINTZM, 4326);
END LOOP;
END;
$$
LANGUAGE 'plpgsql';
SELECT addlocationZM()
I'm probably just being dumb, but I've been at this for a while now and I just can't get it. The SELECT f_table_name ... statement executed on its own returns 58 rows of a single column, each of which is the name of a table in my schema. The idea of this is to create a new column, type PointZM, in each table pulled by the SELECT.
The function would work like this:
CREATE OR REPLACE FUNCTION addlocationZM()
RETURNS void AS
$func$
DECLARE
_tbl text;
BEGIN
FOR _tbl IN
SELECT f_table_name FROM myschema.geography_columns WHERE type LIKE 'Point%'
LOOP
EXECUTE
format('ALTER TABLE %I ADD COLUMN location_zm geography(POINTZM, 4326)', _tbl);
END LOOP;
END
$func$ LANGUAGE plpgsql;
Note how I use a simple text variable to simplify matters. You don't need the record to begin with.
If it's a one-time operation, use a DO command instead of creating a function:
DO
$do$
BEGIN
EXECUTE (
SELECT string_agg(
format(
'ALTER TABLE %I ADD COLUMN location_zm geography(POINTZM, 4326);'
, f_table_name)
, E'\n')
FROM myschema.geography_columns
WHERE type LIKE 'Point%'
);
END
$do$;
This is concatenating a single string comprised of all commands (separated with ;) for a single EXECUTE.
Or, especially while you are not familiar with plpgsql and dynamic SQL, just generate the commands, copy/paste the result and execute as 2nd step:
SELECT 'ALTER TABLE '
|| quote_ident(f_table_name)
|| ' ADD COLUMN locationZM geography(POINTZM, 4326);'
FROM myschema.geography_columns
WHERE type LIKE 'Point%';
(Demonstrating quote_ident() this time.)
Related:
Table name as a PostgreSQL function parameter
Aside: Unquoted CaMeL-case identifiers like locationZM or your function name addlocationZM may not be such a good idea:
Are PostgreSQL column names case-sensitive?
Related
I have a fields table to store column information for other tables:
CREATE TABLE public.fields (
schema_name varchar(100),
table_name varchar(100),
column_text varchar(100),
column_name varchar(100),
column_type varchar(100) default 'varchar(100)',
column_visible boolean
);
And I'd like to create a function to fetch data for a specific table.
Just tried sth like this:
create or replace function public.get_table(schema_name text,
table_name text,
active boolean default true)
returns setof record as $$
declare
entity_name text default schema_name || '.' || table_name;
r record;
begin
for r in EXECUTE 'select * from ' || entity_name loop
return next r;
end loop;
return;
end
$$
language plpgsql;
With this function I have to specify columns when I call it!
select * from public.get_table('public', 'users') as dept(id int, uname text);
I want to pass schema_name and table_name as parameters to function and get record list, according to column_visible field in public.fields table.
Solution for the simple case
As explained in the referenced answers below, you can use registered (row) types, and thus implicitly declare the return type of a polymorphic function:
CREATE OR REPLACE FUNCTION public.get_table(_tbl_type anyelement)
RETURNS SETOF anyelement AS
$func$
BEGIN
RETURN QUERY EXECUTE format('TABLE %s', pg_typeof(_tbl_type));
END
$func$ LANGUAGE plpgsql;
Call:
SELECT * FROM public.get_table(NULL::public.users); -- note the syntax!
Returns the complete table (with all user columns).
Wait! How?
Detailed explanation in this related answer, chapter
"Various complete table types":
Refactor a PL/pgSQL function to return the output of various SELECT queries
TABLE foo is just short for SELECT * FROM foo:
Is there a shortcut for SELECT * FROM?
2 steps for completely dynamic return type
But what you are trying to do is strictly impossible in a single SQL command.
I want to pass schema_name and table_name as parameters to function and get record list, according to column_visible field in
public.fields table.
There is no direct way to return an arbitrary selection of columns (return type not known at call time) from a function - or any SQL command. SQL demands to know number, names and types of resulting columns at call time. More in the 2nd chapter of this related answer:
How do I generate a pivoted CROSS JOIN where the resulting table definition is unknown?
There are various workarounds. You could wrap the result in one of the standard document types (json, jsonb, hstore, xml).
Or you generate the query with one function call and execute the result with the next:
CREATE OR REPLACE FUNCTION public.generate_get_table(_schema_name text, _table_name text)
RETURNS text AS
$func$
SELECT format('SELECT %s FROM %I.%I'
, string_agg(quote_ident(column_name), ', ')
, schema_name
, table_name)
FROM fields
WHERE column_visible
AND schema_name = _schema_name
AND table_name = _table_name
GROUP BY schema_name, table_name
ORDER BY schema_name, table_name;
$func$ LANGUAGE sql;
Call:
SELECT public.generate_get_table('public', 'users');
This create a query of the form:
SELECT usr_id, usr FROM public.users;
Execute it in the 2nd step. (You might want to add column numbers and order columns.)
Or append \gexec in psql to execute the return value immediately. See:
How to force evaluation of subquery before joining / pushing down to foreign server
Be sure to defend against SQL injection:
INSERT with dynamic table name in trigger function
Define table and column names as arguments in a plpgsql function?
Asides
varchar(100) does not make much sense for identifiers, which are limited to 63 characters in standard Postgres:
Maximum characters in labels (table names, columns etc)
If you understand how the object identifier type regclass works, you might replace schema and table name with a singe regclass column.
I think you just need another query to get the list of columns you want.
Maybe something like (this is untested):
create or replace function public.get_table(_schema_name text, _table_name text, active boolean default true) returns setof record as $$
declare
entity_name text default schema_name || '.' || table_name;
r record;
columns varchar;
begin
-- Get the list of columns
SELECT string_agg(column_name, ', ')
INTO columns
FROM public.fields
WHERE fields.schema_name = _schema_name
AND fields.table_name = _table_name
AND fields.column_visible = TRUE;
-- Return rows from the specified table
RETURN QUERY EXECUTE 'select ' || columns || ' from ' || entity_name;
RETURN;
end
$$
language plpgsql;
Keep in mind that column/table references may need to be surrounded by double quotes if they have certain characters in them.
I am trying to log changes, if any were made to a table, but I am stuck when trying to loop through column names. I am receiving a "array value must start with "{" ... line 6 at FOR over SELECT rows" error. I do not understand why this is happening.. The function compiles ok but running an update gives that error.
CREATE TABLE test(x varchar(50))
CREATE OR REPLACE FUNCTION testF()
RETURNS trigger
AS $$
DECLARE
col varchar[255]; //don't know if this is the right variable type to use
BEGIN
IF OLD.* IS DISTINCT FROM NEW.* THEN
FOR col in SELECT column_name FROM information_schema.columns WHERE table_schema = TG_TABLE_SCHEMA AND table_name = TG_TABLE_NAME LOOP
INSERT INTO test(x) VALUES(col||'oldValue:'||OLD.col||'newValue:'||NEW.col); //I want to put the name and the old and new values in a varchar field
END LOOP;
END IF;
RETURN NULL;
END $$ LANGUAGE plpgsql;
CREATE TRIGGER testT AFTER UPDATE
ON "triggerTable" FOR EACH ROW EXECUTE PROCEDURE testF();
to get at the columns of OLD or NEW by name you'll have to use exec an a bunch of typecasts.
something like this:
execute '('||quote_literal(NEW::text)||'::'||quote_ident(pg_typeof(NEW))||
').'||quote_ident(col)||'::text';
this may get you imprecise values for some floats
I have created the following stored procedure, which basically receives a name of table, and a prefix. The function then finds all columns that share this prefix and returns as an output a 'select' query command ('myoneliner').
as follows:
CREATE OR REPLACE FUNCTION mytext (mytable text, myprefix text)
RETURNS text AS $myoneliner$
declare
myoneliner text;
BEGIN
SELECT 'SELECT ' || substr(cols,2,length(cols)-2) ||' FROM '||mytable
INTO myoneliner
FROM (
SELECT array(
SELECT DISTINCT quote_ident(column_name::text)
FROM information_schema.columns
WHERE table_name = mytable
AND column_name LIKE myprefix||'%'
order by quote_ident
)::text cols
) sub;
RETURN myoneliner;
END;
$myoneliner$ LANGUAGE plpgsql;
Call:
select mytext('dkj_p_k27ac','enri');
As a result of running this stored procedure and the 'select' that is following it, I get the following output at the Data Output window (all within one cell, named "mytext text"):
'SELECT enrich_d_dkj_p_k27ac,enrich_lr_dkj_p_k27ac,enrich_r_dkj_p_k27ac
FROM dkj_p_k27ac'
I would like to basically be able to take the output command line that I received as an output and execute it. In other words, I would like to be able and execute the output of my stored procedure.
How can I do so?
I tried the following:
CREATE OR REPLACE FUNCTION mytext (mytable text, myprefix text)
RETURNS SETOF RECORD AS $$
declare
smalltext text;
myoneliner text;
BEGIN
SELECT 'SELECT ' || substr(cols,2,length(cols)-2) ||' FROM '||mytable
INTO myoneliner
FROM (
SELECT array(
SELECT DISTINCT quote_ident(column_name::text)
FROM information_schema.columns
WHERE table_name = mytable
AND column_name LIKE myprefix||'%'
order by quote_ident
)::text cols
) sub;
smalltext=lower(myoneliner);
raise notice '%','my additional text '||smalltext;
RETURN QUERY EXECUTE smalltext;
END;
$$ LANGUAGE plpgsql;
Call function:
SELECT * from mytext('dkj_p_k27ac','enri');
But I'm getting the following error message, could you please advise what should I change in order for it to execute?:
ERROR: a column definition list is required for functions returning "record"
LINE 26: SELECT * from mytext('dkj_p_k27ac','enri');
********** Error **********
ERROR: a column definition list is required for functions returning "record"
SQL state: 42601
Character: 728
Your first problem was solved by using dynamic SQL with EXECUTE like Craig advised.
But the rabbit hole goes deeper:
CREATE OR REPLACE FUNCTION myresult(mytable text, myprefix text)
RETURNS SETOF RECORD AS
$func$
DECLARE
smalltext text;
myoneliner text;
BEGIN
SELECT INTO myoneliner
'SELECT '
|| string_agg(quote_ident(column_name::text), ',' ORDER BY column_name)
|| ' FROM ' || quote_ident(mytable)
FROM information_schema.columns
WHERE table_name = mytable
AND column_name LIKE myprefix||'%'
AND table_schema = 'public'; -- schema name; might be another param
smalltext := lower(myoneliner); -- nonsense
RAISE NOTICE 'My additional text: %', myoneliner;
RETURN QUERY EXECUTE myoneliner;
END
$func$ LANGUAGE plpgsql;
Major points
Don't cast the whole statement to lower case. Column names might be double-quoted with upper case letters, which are case-sensitive in this case (no pun intended).
You don't need DISTINCT in the query on information_schema.columns. Column names are unique per table.
You do need to specify the schema, though (or use another way to single out one schema), or you might be mixing column names from multiple tables of the same name in multiple schemas, resulting in nonsense.
You must sanitize all identifiers in dynamic code - including table names: quote_ident(mytable). Be aware that your text parameter to the function is case sensitive! The query on information_schema.columns requires that, too.
I untangled your whole construct to build the list of column names with string_agg() instead of the array constructor. Related answer:
Update multiple columns that start with a specific string
The assignment operator in plpgsql is :=.
Simplified syntax of RAISE NOTICE.
Core problem impossible to solve
All of this still doesn't solve your main problem: SQL demands a definition of the columns to be returned. You can circumvent this by returning anonymous records like you tried. But that's just postponing the inevitable. Now you have to provide a column definition list at call time, just like your error message tells you. But you just don't know which columns are going to be returned. Catch 22.
Your call would work like this:
SELECT *
FROM myresult('dkj_p_k27ac','enri') AS f (
enrich_d_dkj_p_k27ac text -- replace with actual column types
, enrich_lr_dkj_p_k27ac text
, enrich_r_dkj_p_k27ac text);
But you don't know number, names (optional) and data types of returned columns, not at creation time of the function and not even at call time. It's impossible to do exactly that in a single call. You need two separate queries to the database.
You could return all columns of any given table dynamically with a function using polymorphic types, because there is a well defined type for the whole table. Last chapter of this related answer:
Refactor a PL/pgSQL function to return the output of various SELECT queries
I need to do the same deletion or purge operation (based on several conditions) on a set of tables. For that I am trying to pass the table names in an array to a function. I am not sure if I am doing it right. Or is there a better way?
I am pasting just a sample example this is not the real function I have written but the basic is same as below:
CREATE OR REPLACE FUNCTION test (tablename text[]) RETURNS int AS
$func$
BEGIN
execute 'delete * from '||tablename;
RETURN 1;
END
$func$ LANGUAGE plpgsql;
But when I call the function I get an error:
select test( {'rajeev1'} );
ERROR: syntax error at or near "{"
LINE 10: select test( {'rajeev1'} );
^
********** Error **********
ERROR: syntax error at or near "{"
SQL state: 42601
Character: 179
Array syntax
'{rajeev1, rajeev2}' or ARRAY['rajeev1', 'rajeev2']. Read the manual.
TRUNCATE
Since you are deleting all rows from the tables, consider TRUNCATE instead. Per documentation:
Tip: TRUNCATE is a PostgreSQL extension that provides a faster
mechanism to remove all rows from a table.
Be sure to study the details. If TRUNCATE works for you, the whole operation becomes very simple, since the command accepts multiple tables:
TRUNCATE rajeev1, rajeev2, rajeev3, ..
Dynamic DELETE
Else you need dynamic SQL like you already tried. The scary missing detail: you are completely open to SQL injection and catastrophic syntax errors. Use format() with %I (not %s to sanitize identifiers like table names. Or, better yet in this particular case, use an array of regclass as parameter instead:
CREATE OR REPLACE FUNCTION f_del_all(_tbls regclass)
RETURNS void AS
$func$
DECLARE
_tbl regclass;
BEGIN
FOREACH _tbl IN ARRAY _tbls LOOP
EXECUTE format('DELETE * FROM %s', _tbl);
END LOOP;
END
$func$ LANGUAGE plpgsql;
Call:
SELECT f_del_all('{rajeev1,rajeev2,rajeev3}');
Explanation here:
Table name as a PostgreSQL function parameter
You used wrong syntax for text array constant in the function call. But even if it was right, your function is not correct.
If your function has text array as argument you should loop over the array to execute query for each element.
CREATE OR REPLACE FUNCTION test (tablenames text[]) RETURNS int AS
$func$
DECLARE
tablename text;
BEGIN
FOREACH tablename IN ARRAY tablenames LOOP
EXECUTE FORMAT('delete * from %s', tablename);
END LOOP;
RETURN 1;
END
$func$ LANGUAGE plpgsql;
You can then call the function for several tables at once, not only for one.
SELECT test( '{rajeev1, rajeev2}' );
If you do not need this feature, simply change the argument type to text.
CREATE OR REPLACE FUNCTION test (tablename text) RETURNS int AS
$func$
BEGIN
EXECUTE format('delete * from %s', tablename);
RETURN 1;
END
$func$ LANGUAGE plpgsql;
SELECT test('rajeev1');
I recommend using the format function.
If you want to execute a function (say purge_this_one_table(tablename)) on a group of tables identified by similar names you can use this construction:
create or replace function purge_all_these_tables(mask text)
returns void language plpgsql
as $$
declare
tabname text;
begin
for tabname in
select relname
from pg_class
where relkind = 'r' and relname like mask
loop
execute format(
'purge_this_one_table(%s)',
tabname);
end loop;
end $$;
select purge_all_these_tables('agg_weekly_%');
It should be:
select test('{rajeev1}');
I have the following script to dynamically create views into a PostgreSQL database.
CREATE OR REPLACE FUNCTION cs_refresh_mviews() RETURNS integer AS $$
DECLARE
mviews RECORD;
query text;
park_name text;
ppstatements int;
BEGIN
RAISE NOTICE 'Creating views...';
FOR mviews IN SELECT name FROM "Canadian_Parks" LOOP
park_name := mviews.name;
RAISE NOTICE 'Creating or replace view %s...', mviews.name;
query := 'CREATE OR REPLACE VIEW %_view AS
SELECT * from "Canadian_Parks" where name=''%'';
ALTER TABLE %_view OWNER TO postgres', park_name, park_name, park_name;
-- RAISE NOTICE query;
EXECUTE query;
END LOOP;
RAISE NOTICE 'Done refreshing materialized views.';
RETURN 1;
END;
$$ LANGUAGE plpgsql;
I have confirmed integrity of the string, such as
CREATE OR REPLACE VIEW Saguenay_St__Lawrence_view AS
SELECT * from "Canadian_Parks" where name='Saguenay_St__Lawrence';
ALTER TABLE Saguenay_St__Lawrence_view OWNER TO postgres
assigned to the query variable by manually submitting this to the database and getting a successful response.
However, if I attempt to execute the function using
SELECT cs_refresh_mviews();
the followig error is displayed:
ERROR: query "SELECT 'CREATE OR REPLACE VIEW %_view AS SELECT * from "Canadian_Parks" where name=''%''; ALTER TABLE %_view OWNER TO postgres', park_name, park_name, park_name" returned 4 columns
CONTEXT: PL/pgSQL function "cs_refresh_mviews" line 32 at assignment
********** Error **********
ERROR: query "SELECT 'CREATE OR REPLACE VIEW %_view AS SELECT * from "Canadian_Parks" where name=''%''; ALTER TABLE %_view OWNER TO postgres', park_name, park_name, park_name" returned 4 columns
SQL state: 42601
Context: PL/pgSQL function "cs_refresh_mviews" line 32 at assignment
Why has this been converted to a SELECT statement, instead of a pure CREATE?
You setup is pretty twisted. Why would you save part of the name of a view in a composite type of a table instead of saving it in a plain text column?
Anyhow, it could work like this:
Setup matching question:
CREATE SCHEMA x; -- demo in test schema
SET search_path = x;
CREATE TYPE mviews AS (id int, name text); -- composite type used in table
CREATE TABLE "Canadian_Parks" (name mviews);
INSERT INTO "Canadian_Parks"(name) VALUES
('(1,"canadian")')
,('(2,"islandic")'); -- composite types, seriously?
SELECT name, (name).* from "Canadian_Parks";
CREATE OR REPLACE FUNCTION cs_refresh_mviews()
RETURNS int LANGUAGE plpgsql SET search_path = x AS -- search_path for test
$func$
DECLARE
_parkname text;
BEGIN
FOR _parkname IN SELECT (name).name FROM "Canadian_Parks" LOOP
EXECUTE format('
CREATE OR REPLACE VIEW %1$I AS
SELECT * FROM "Canadian_Parks" WHERE (name).name = %2$L;
ALTER TABLE %1$I OWNER TO postgres'
, _parkname || '_view', _parkname);
END LOOP;
RETURN 1;
END
$func$;
SELECT cs_refresh_mviews();
DROP SCHEMA x CASCADE; -- clean up
Major points
As you are executing text with execute, you need to safeguard against SQL injection. I use the format() function for identifiers and the literal
I use the syntax SELECT (name).name to cope with your weird setup and extract the name we need right away.
Similarly, the VIEW needs to read WHERE (name).name = .. to work in this setup.
I removed a lot of noise that is irrelevant to the question.
It's also probably pointless to have the function RETURN 1. Just define the function with RETURNS void. I kept it, though, to match the question.
Untangled setup
How it probably should be:
CREATE SCHEMA x;
SET search_path = x;
CREATE TABLE canadian_parks (id serial primary key, name text);
INSERT INTO canadian_parks(name) VALUES ('canadian'), ('islandic');
SELECT * from canadian_parks;
CREATE OR REPLACE FUNCTION cs_refresh_mviews()
RETURNS void LANGUAGE plpgsql SET search_path = x AS
$func$
DECLARE
parkname text;
BEGIN
FOR parkname IN SELECT name FROM canadian_parks LOOP
EXECUTE format('
CREATE OR REPLACE VIEW %1$I AS
SELECT * FROM canadian_parks WHERE name = %2$L;
ALTER TABLE %1$I OWNER TO postgres'
, parkname || '_view', parkname);
END LOOP;
END
$func$;
SELECT cs_refresh_mviews();
DROP SCHEMA x CASCADE;
You've misunderstood usage of commas in assignment expression.
It turns query to array (RECORD) instead of scalar.
Use concatenation:
park_name := quote_ident(mviews.name||'_view');
query := 'CREATE OR REPLACE VIEW '||park_name||' AS SELECT * from "Canadian_Parks" where name='||quote_literal(mviews.name)||'; ALTER TABLE '||park_name||' OWNER TO postgres';