I'm trying to invoke a SQL script within another SQL script using pgsql.
I already saw that I can use
\i 'path/to/file.sql'
where path/to/files.sql is between single quotes.
I was trying to replace 'path/to/file.sql' with a variable instead, like
DO $$
DECLARE
ls_id INT := 271195;
tokens VARCHAR(20);
BEGIN
tokens := CONCAT(ls_id, '_tokens.sql');
\i tokens
END $$;
Is this possible some way or another?
It's not something you can do directly in sql because \i is a psql command and not actually SQL at all.
But the following should demonstrate how you can go about it.
SELECT ('something' || '_and_' || 'something_else.sql') as filename \gset
\i :filename
The gset will create a psql variable for you based on the result of the query. Again the \gset is actually just for psql rather than being sent to the backend.
Then you can reference the psql variable prefixed by a colon.
Note that this is just a macro-style text substitution. You will need to deal with quoting spaces and backslashes and so on yourself.
I have few functions containing select queries and dbms_output.put_line statements. Further, I am trying to execute them via a script (master.sql) and need to send their corresponding dbms_output results to their respective text files.
Sample function:
CREATE OR REPLACE FUNCTION myschema.my_func1() RETURNS VOID AS $func$
DECLARE
cur_fetch_my_data CURSOR IS
select col1,col2 from myschema.table1;
BEGIN
DBMS_OUTPUT.put_line('column 1'||CHR(9)||'column 2');
for data_cur in cur_fetch_my_data LOOP
DBMS_OUTPUT.put_line(data_cur.col1||CHR(9)||data_cur.col2);
END LOOP;
END;
$func$ LANGUAGE sql;
Script for executing functions:
--master.sql
begin
dbms_output.serveroutput(true);
end;
\o DUMP_OF_func1.txt
SELECT myschema.my_func1();
\o DUMP_OF_func2.txt
SELECT myschema.my_func2();
After above script execution, all dbms_output results are printed on console but not in text file. Only thing which is dumped into text file is
my_func1
------------
(1 row)
Further, i tried changing the master script. I changed
dbms_output.serveroutput(true)
to
dbms_output.serveroutput(false)
After doing this, console output did not print. But dump into text file remained same.
Can anyone please tell me how I can divert dbms_output results to their text files?
I have a Postgres script that I have that I intend to run against multiple databases. The sample SQL has functions and tables like
CREATE FUNCTION point() RETURNS trigger
LANGUAGE plpgsql
AS $$
BEGIN
NEW.the_geom:=ST_SetSRID(geom, :CRS) ;
RETURN NEW;
END
$$;
CREATE TABLE admin (
gid integer NOT NULL,
geom geometry(Polygon,:CRS)
);
I have put a variable inside so that I can substitute it on runtime as
psql -d db -p 5432 -U username -h localhost -f test.sql --variable=CRS=3857
Why does the variable only get properly substituted in table definitions and not function definition
https://www.postgresql.org/docs/current/static/app-psql.html#APP-PSQL-INTERPOLATION
Therefore, a construction such as ':foo' doesn't work to produce a
quoted literal from a variable's value (and it would be unsafe if it
did work, since it wouldn't correctly handle quotes embedded in the
value).
a function definition is put between quotes - in your case $$ instead of ', so it fails. You can change ST_SetSRID(geom, :CRS) to ST_SetSRID(geom, $$:CRS) to see it start being handled again (of course ruining the function body)
As a workaround you can try using bash variable instead:
vao#vao-VirtualBox:~$ export CRS=33
vao#vao-VirtualBox:~$ psql t << EOF
> CREATE OR REPLACE FUNCTION point() RETURNS trigger
> LANGUAGE plpgsql
> AS \$\$
> BEGIN
> NEW.the_geom:=ST_SetSRID(geom, $CRS) ;
> RETURN NEW;
> END
> \$\$;
> \sf point()
> EOF
CREATE FUNCTION
CREATE OR REPLACE FUNCTION public.point()
RETURNS trigger
LANGUAGE plpgsql
AS $function$
BEGIN
NEW.the_geom:=ST_SetSRID(geom, 33) ;
RETURN NEW;
END
$function$
Of course it can be scripted to a file that you can call as bash with argument
Shell substitution, as Vao Tsun suggests, might work fine for small psql scripts, but it's still not a general solution. For instance, shells are not aware of psql and pgSQL syntaxes and may substitute sub-strings that are not meant to be substituted in the psql script:
-- If CA and AU are undefined environment variable (likely),
-- the following SQL, after shell substitution, will result in:
-- ... VALUES ('Australian Dollar', ''), (Canadian Dollar, '');
INSERT (name, symbol) INTO currencies VALUES
('Australian Dollar', '$AU'),
('Canadian Dollar', '$CA');
This will become unmanageable with large scripts without additional measures. A better approach is to use string concatenation. Something like this would be ideal:
-- This won't work; syntax error
CREATE FUNCTION point() RETURNS trigger
LANGUAGE plpgsql
AS $$
BEGIN
NEW.the_geom := ST_SetSRID(geom, $$ || :'CRS' || $$);
RETURN NEW;
END
$$;
Unfortunately, pgSQL syntax expects literal strings as function bodies, not arbitrary string expressions. Fortunately, however, PostgreSQL implements as part of its SQL syntax implicit concatenation between single-quoted strings separated by only whitespaces and at least one line break:
-- This gets parsed as "SELECT 'mystring';"
SELECT 'my'
'string';
The catch though is that it only works with single-quoted strings, not dollar-quoted strings. The following will work fine, but it's an eye-sore:
-- This will work, but it's ugly
CREATE FUNCTION point() RETURNS trigger
LANGUAGE plpgsql
AS
-- Mind the trailing space below
'BEGIN '
'NEW.the_geom := ST_SetSRID(geom, $$'
:'CRS'
'$$);'
'RETURN NEW;'
'END';
This is perhaps a more eye-pleasing variant:
-- This will work
-- (assign appropriate ownership and permissions if CRS contains sentitive data)
CREATE FUNCTION _get_crs() RETURNS text LANGUAGE sql AS 'SELECT $$'
:'crs'
'$$';
CREATE FUNCTION point() RETURNS trigger
LANGUAGE plpgsql
AS $$
BEGIN
NEW.the_geom := ST_SetSRID(geom, _get_crs());
RETURN NEW;
END
$$;
And here's another variant:
-- This will also work
-- (may expose sensitive data in the catalog)
SET "app.crs" = :'crs';
CREATE FUNCTION point() RETURNS trigger
LANGUAGE plpgsql
AS $$
BEGIN
NEW.the_geom := ST_SetSRID(geom, current_setting('app.crs'));
RETURN NEW;
END
$$;
The function body is a sting literal. psql has hardly any string modification functionality. However, PostgreSQL has plenty of it. With psql variables and \gset (since PostgreSQL 9.3) you can utilize this to first build your command and then execute it:
--for demo purpose set CRS directly. The OP did pass it from outside to psql
\set CRS 1234
--here we wrap the whole CREATE FUNCTION call in a literal separated by $_$
--this improves the readabilty and covers the inner $$. The result will be
--stored in psql variable func. This is workaround for a multi-line \set
SELECT $_$
CREATE FUNCTION point() RETURNS trigger
LANGUAGE plpgsql
AS $$
BEGIN
NEW.the_geom:=ST_SetSRID(geom, :CRS) ;
RETURN NEW;
END
$$;
$_$ AS func \gset
--replace the substring ':CRS' with content of variable CRS
--and store the result in func again
SELECT replace(:'func', ':CRS', :'CRS') AS func \gset
--we can now have a look to the resulting DDL
\echo :func
--and we can execute it:
:func
--clean up psql variables
\unset func
\unset CRS
with the help of \gexec (since PostgreSQL 9.6) we can also combine the last two steps:
--replace the substring ':CRS' with content of variable CRS
--and execute the result
SELECT replace(:'func', ':CRS', :'CRS') \gexec
or even do all in one command:
SELECT replace($_$
CREATE FUNCTION point() RETURNS trigger
LANGUAGE plpgsql
AS $$
BEGIN
NEW.the_geom:=ST_SetSRID(geom, :CRS) ;
RETURN NEW;
END
$$;
$_$, ':CRS', :'CRS') \gexec
I want to do copy csv to database using stored procedure. My function is follows;
CREATE FUNCTION gis_portal.copycsv(IN path text) RETURNS void AS
'COPY gis_portal.temp_excel FROM path WITH DELIMITER'
LANGUAGE sql VOLATILE LEAKPROOF;
query is
COPY gis_portal.temp_excel FROM path WITH DELIMITER ',' CSV HEADER
with parameter path.
It is giving error as syntax error near path while creating function.
Please help me.
'COPY gis_portal.temp_excel FROM path WITH DELIMITER'
^^^^^^^^^^
What delimiter? You must specify one if you use that keyword.
Try:
CREATE FUNCTION gis_portal.copycsv(IN path text) RETURNS void AS $$
COPY gis_portal.temp_excel FROM path WITH DELIMITER ','
$$ LANGUAGE sql;
or whatever delimiter you want.
Additionally, in SQL functions you can't use the identifier you must use a positional parameter, like $1. But because COPY isn't a plannable statement you cannot use parameters in it. You will have to use PL/PgSQL and EXECUTE to run it as dynamic SQL:
CREATE FUNCTION gis_portal.copycsv(IN path text) RETURNS void AS $$
BEGIN
EXECUTE format('COPY gis_portal.temp_excel FROM %L WITH DELIMITER '',''', path);
END;
$$ LANGUAGE plpgsql;
Note the doubled quotes around the delimiter because it's now an SQL string.
I may be contorting postgres here, but fundamentally, what I would like to do is take a string variable and pass it to an sql command (in this case COPY) using only psql.
So this is what I came up with. The commands are separated into 2 files because I want to be able to use the mydb_functions in other situations:
file one: mydb_functions--1.0.sql (in share/extension and with mydb_functions.control also setup as described in the manual. Given a filename, returns a full filepath. this is done solely to make the COPY statements in add_data.sql below, neater.
CREATE OR REPLACE FUNCTION fpn(filename text) RETURNS TEXT as '
DECLARE
mypath text := ''/path/to/my/directory/'';
BEGIN
RETURN mypath || filename;
END
' LANGUAGE plpgsql;
file two: add_data.sql . This exists solely to copy data into existing postgres tables using psql at the command line. Note: running psql with superuser privileges is required because of the CREATE EXTENSION command.
CREATE EXTENSION IF NOT EXISTS mydb_functions;
-- haven't figured out how to create a function without arguments yet.
CREATE OR REPLACE FUNCTION test (dummyvar text) RETURNS text as '
DECLARE
filepath RECORD;
BEGIN
SELECT * INTO filepath from fpn(''mydatafile.data'');
COPY tablename (columnname) FROM filepath;
END
' LANGUAGE plpgsql;
The part I am stuck on is how to extract a text from the filepath record to use in the COPY command. Any tips on an easier way to achieve this are also welcome. I think creating a table to store the variable is far easier than this. But I would like to finish the last step.
If your issue is running COPY with a dynamic target path, use EXECUTE to run a formatted SQL query.
CREATE OR REPLACE FUNCTION test () RETURNS text as $$
DECLARE
filepath RECORD;
BEGIN
SELECT * INTO filepath from fpn('mydatafile.data');
EXECUTE format('COPY tablename (columnname) FROM %L', filepath);
END
$$ LANGUAGE plpgsql;
See, this fails:
DO
$$
DECLARE
somevar text;
BEGIN
somevar := '/tmp/somepath.csv';
COPY tablename(columnname) FROM somevar;
END;
$$;
but this works:
DO
$$
DECLARE
somevar text;
BEGIN
somevar := '/tmp/somepath.csv';
EXECUTE format('COPY tablename(columnname) FROM %L', somevar);
END;
$$;
See:
EXECUTE
format()
format()'s %L specifier auto-quotes literals. %I does the same for identifiers.
If as I originally thought, you're talking about getting data into psql from outside, you may find psql variables and variable interpolation useful:
$ psql -v filepath=/path/to/my/directory/mydatafile.data regress
regress=> SELECT :'filepath';
?column?
---------------------------------------
/path/to/my/directory/mydatafile.data
(1 row)
Note that the colon is unquoted, then the variable name is quoted. Odd syntax, I know. This only works in psql; it won't work in (say) PgAdmin-III.
Alternately, you can use a here-document (unix-like shell specific, won't work in Windows' cmd.exe) to do quoted text interpolation:
$ FILEPATH=/path/to/my/directory/mydatafile.data
$ psql regress <<__END__
SELECT '$FILEPATH';
__END__
?column?
---------------------------------------
/path/to/my/directory/mydatafile.data
(1 row)
Both of these show how to insert a variable from the shell level into psql. From there, it's a matter of using it in your function. Like this, say, with -v filename=myfilename.csv:
CREATE OR REPLACE FUNCTION test() RETURNS text as $$
DECLARE
filepath RECORD;
BEGIN
SELECT * INTO filepath from fpn(:'filename');
COPY tablename (columnname) FROM filepath;
END
$$ LANGUAGE plpgsql;
or just this with -v filepath=/full/path/to/myfilename.csv:
$ psql -v filepath=/path/to/my/directory/mydatafile.data regress
regress=> COPY tablename (columnname) FROM :'filepath';