I'm writing my own regression test, and I'm using \o command to output my query result to a temporary file. For example
create or replace function aaaa()
returns integer as
$$
BEGIN
RAISE NOTICE 'ssss';
RETURN 1;
END;
$$ language plpgsql;
\o a.tmp
select aaaa();
With those commands, I'm able to output the return value (1 in this case) to file a.tmp. However, the message NOTICE: ssss is print directly to my psql client.
I'm wondering if there is a way to print the NOTICE: ssss also to a.tmp.
Related
I have few functions containing select queries and dbms_output.put_line statements. Further, I am trying to execute them via a script (master.sql) and need to send their corresponding dbms_output results to their respective text files.
Sample function:
CREATE OR REPLACE FUNCTION myschema.my_func1() RETURNS VOID AS $func$
DECLARE
cur_fetch_my_data CURSOR IS
select col1,col2 from myschema.table1;
BEGIN
DBMS_OUTPUT.put_line('column 1'||CHR(9)||'column 2');
for data_cur in cur_fetch_my_data LOOP
DBMS_OUTPUT.put_line(data_cur.col1||CHR(9)||data_cur.col2);
END LOOP;
END;
$func$ LANGUAGE sql;
Script for executing functions:
--master.sql
begin
dbms_output.serveroutput(true);
end;
\o DUMP_OF_func1.txt
SELECT myschema.my_func1();
\o DUMP_OF_func2.txt
SELECT myschema.my_func2();
After above script execution, all dbms_output results are printed on console but not in text file. Only thing which is dumped into text file is
my_func1
------------
(1 row)
Further, i tried changing the master script. I changed
dbms_output.serveroutput(true)
to
dbms_output.serveroutput(false)
After doing this, console output did not print. But dump into text file remained same.
Can anyone please tell me how I can divert dbms_output results to their text files?
I have a Postgres script that I have that I intend to run against multiple databases. The sample SQL has functions and tables like
CREATE FUNCTION point() RETURNS trigger
LANGUAGE plpgsql
AS $$
BEGIN
NEW.the_geom:=ST_SetSRID(geom, :CRS) ;
RETURN NEW;
END
$$;
CREATE TABLE admin (
gid integer NOT NULL,
geom geometry(Polygon,:CRS)
);
I have put a variable inside so that I can substitute it on runtime as
psql -d db -p 5432 -U username -h localhost -f test.sql --variable=CRS=3857
Why does the variable only get properly substituted in table definitions and not function definition
https://www.postgresql.org/docs/current/static/app-psql.html#APP-PSQL-INTERPOLATION
Therefore, a construction such as ':foo' doesn't work to produce a
quoted literal from a variable's value (and it would be unsafe if it
did work, since it wouldn't correctly handle quotes embedded in the
value).
a function definition is put between quotes - in your case $$ instead of ', so it fails. You can change ST_SetSRID(geom, :CRS) to ST_SetSRID(geom, $$:CRS) to see it start being handled again (of course ruining the function body)
As a workaround you can try using bash variable instead:
vao#vao-VirtualBox:~$ export CRS=33
vao#vao-VirtualBox:~$ psql t << EOF
> CREATE OR REPLACE FUNCTION point() RETURNS trigger
> LANGUAGE plpgsql
> AS \$\$
> BEGIN
> NEW.the_geom:=ST_SetSRID(geom, $CRS) ;
> RETURN NEW;
> END
> \$\$;
> \sf point()
> EOF
CREATE FUNCTION
CREATE OR REPLACE FUNCTION public.point()
RETURNS trigger
LANGUAGE plpgsql
AS $function$
BEGIN
NEW.the_geom:=ST_SetSRID(geom, 33) ;
RETURN NEW;
END
$function$
Of course it can be scripted to a file that you can call as bash with argument
Shell substitution, as Vao Tsun suggests, might work fine for small psql scripts, but it's still not a general solution. For instance, shells are not aware of psql and pgSQL syntaxes and may substitute sub-strings that are not meant to be substituted in the psql script:
-- If CA and AU are undefined environment variable (likely),
-- the following SQL, after shell substitution, will result in:
-- ... VALUES ('Australian Dollar', ''), (Canadian Dollar, '');
INSERT (name, symbol) INTO currencies VALUES
('Australian Dollar', '$AU'),
('Canadian Dollar', '$CA');
This will become unmanageable with large scripts without additional measures. A better approach is to use string concatenation. Something like this would be ideal:
-- This won't work; syntax error
CREATE FUNCTION point() RETURNS trigger
LANGUAGE plpgsql
AS $$
BEGIN
NEW.the_geom := ST_SetSRID(geom, $$ || :'CRS' || $$);
RETURN NEW;
END
$$;
Unfortunately, pgSQL syntax expects literal strings as function bodies, not arbitrary string expressions. Fortunately, however, PostgreSQL implements as part of its SQL syntax implicit concatenation between single-quoted strings separated by only whitespaces and at least one line break:
-- This gets parsed as "SELECT 'mystring';"
SELECT 'my'
'string';
The catch though is that it only works with single-quoted strings, not dollar-quoted strings. The following will work fine, but it's an eye-sore:
-- This will work, but it's ugly
CREATE FUNCTION point() RETURNS trigger
LANGUAGE plpgsql
AS
-- Mind the trailing space below
'BEGIN '
'NEW.the_geom := ST_SetSRID(geom, $$'
:'CRS'
'$$);'
'RETURN NEW;'
'END';
This is perhaps a more eye-pleasing variant:
-- This will work
-- (assign appropriate ownership and permissions if CRS contains sentitive data)
CREATE FUNCTION _get_crs() RETURNS text LANGUAGE sql AS 'SELECT $$'
:'crs'
'$$';
CREATE FUNCTION point() RETURNS trigger
LANGUAGE plpgsql
AS $$
BEGIN
NEW.the_geom := ST_SetSRID(geom, _get_crs());
RETURN NEW;
END
$$;
And here's another variant:
-- This will also work
-- (may expose sensitive data in the catalog)
SET "app.crs" = :'crs';
CREATE FUNCTION point() RETURNS trigger
LANGUAGE plpgsql
AS $$
BEGIN
NEW.the_geom := ST_SetSRID(geom, current_setting('app.crs'));
RETURN NEW;
END
$$;
The function body is a sting literal. psql has hardly any string modification functionality. However, PostgreSQL has plenty of it. With psql variables and \gset (since PostgreSQL 9.3) you can utilize this to first build your command and then execute it:
--for demo purpose set CRS directly. The OP did pass it from outside to psql
\set CRS 1234
--here we wrap the whole CREATE FUNCTION call in a literal separated by $_$
--this improves the readabilty and covers the inner $$. The result will be
--stored in psql variable func. This is workaround for a multi-line \set
SELECT $_$
CREATE FUNCTION point() RETURNS trigger
LANGUAGE plpgsql
AS $$
BEGIN
NEW.the_geom:=ST_SetSRID(geom, :CRS) ;
RETURN NEW;
END
$$;
$_$ AS func \gset
--replace the substring ':CRS' with content of variable CRS
--and store the result in func again
SELECT replace(:'func', ':CRS', :'CRS') AS func \gset
--we can now have a look to the resulting DDL
\echo :func
--and we can execute it:
:func
--clean up psql variables
\unset func
\unset CRS
with the help of \gexec (since PostgreSQL 9.6) we can also combine the last two steps:
--replace the substring ':CRS' with content of variable CRS
--and execute the result
SELECT replace(:'func', ':CRS', :'CRS') \gexec
or even do all in one command:
SELECT replace($_$
CREATE FUNCTION point() RETURNS trigger
LANGUAGE plpgsql
AS $$
BEGIN
NEW.the_geom:=ST_SetSRID(geom, :CRS) ;
RETURN NEW;
END
$$;
$_$, ':CRS', :'CRS') \gexec
I may be contorting postgres here, but fundamentally, what I would like to do is take a string variable and pass it to an sql command (in this case COPY) using only psql.
So this is what I came up with. The commands are separated into 2 files because I want to be able to use the mydb_functions in other situations:
file one: mydb_functions--1.0.sql (in share/extension and with mydb_functions.control also setup as described in the manual. Given a filename, returns a full filepath. this is done solely to make the COPY statements in add_data.sql below, neater.
CREATE OR REPLACE FUNCTION fpn(filename text) RETURNS TEXT as '
DECLARE
mypath text := ''/path/to/my/directory/'';
BEGIN
RETURN mypath || filename;
END
' LANGUAGE plpgsql;
file two: add_data.sql . This exists solely to copy data into existing postgres tables using psql at the command line. Note: running psql with superuser privileges is required because of the CREATE EXTENSION command.
CREATE EXTENSION IF NOT EXISTS mydb_functions;
-- haven't figured out how to create a function without arguments yet.
CREATE OR REPLACE FUNCTION test (dummyvar text) RETURNS text as '
DECLARE
filepath RECORD;
BEGIN
SELECT * INTO filepath from fpn(''mydatafile.data'');
COPY tablename (columnname) FROM filepath;
END
' LANGUAGE plpgsql;
The part I am stuck on is how to extract a text from the filepath record to use in the COPY command. Any tips on an easier way to achieve this are also welcome. I think creating a table to store the variable is far easier than this. But I would like to finish the last step.
If your issue is running COPY with a dynamic target path, use EXECUTE to run a formatted SQL query.
CREATE OR REPLACE FUNCTION test () RETURNS text as $$
DECLARE
filepath RECORD;
BEGIN
SELECT * INTO filepath from fpn('mydatafile.data');
EXECUTE format('COPY tablename (columnname) FROM %L', filepath);
END
$$ LANGUAGE plpgsql;
See, this fails:
DO
$$
DECLARE
somevar text;
BEGIN
somevar := '/tmp/somepath.csv';
COPY tablename(columnname) FROM somevar;
END;
$$;
but this works:
DO
$$
DECLARE
somevar text;
BEGIN
somevar := '/tmp/somepath.csv';
EXECUTE format('COPY tablename(columnname) FROM %L', somevar);
END;
$$;
See:
EXECUTE
format()
format()'s %L specifier auto-quotes literals. %I does the same for identifiers.
If as I originally thought, you're talking about getting data into psql from outside, you may find psql variables and variable interpolation useful:
$ psql -v filepath=/path/to/my/directory/mydatafile.data regress
regress=> SELECT :'filepath';
?column?
---------------------------------------
/path/to/my/directory/mydatafile.data
(1 row)
Note that the colon is unquoted, then the variable name is quoted. Odd syntax, I know. This only works in psql; it won't work in (say) PgAdmin-III.
Alternately, you can use a here-document (unix-like shell specific, won't work in Windows' cmd.exe) to do quoted text interpolation:
$ FILEPATH=/path/to/my/directory/mydatafile.data
$ psql regress <<__END__
SELECT '$FILEPATH';
__END__
?column?
---------------------------------------
/path/to/my/directory/mydatafile.data
(1 row)
Both of these show how to insert a variable from the shell level into psql. From there, it's a matter of using it in your function. Like this, say, with -v filename=myfilename.csv:
CREATE OR REPLACE FUNCTION test() RETURNS text as $$
DECLARE
filepath RECORD;
BEGIN
SELECT * INTO filepath from fpn(:'filename');
COPY tablename (columnname) FROM filepath;
END
$$ LANGUAGE plpgsql;
or just this with -v filepath=/full/path/to/myfilename.csv:
$ psql -v filepath=/path/to/my/directory/mydatafile.data regress
regress=> COPY tablename (columnname) FROM :'filepath';
I'm trying to get a PSQL script running using variables in an example like the one below without declaring functions and having to call them.
DECLARE
result TEXT;
BEGIN
SELECT INTO result name
FROM test;
RAISE NOTICE result;
END;
Where table test only has 1 row and column. Is this possible without having to wrap this script inside a function. This will allow me to call the script via say command line easier.
Thanks guys.
You can use DO to create and execute an anonymous function:
DO executes an anonymous code block, or in other words a transient anonymous function in a procedural language.
Something like this:
do $$
declare result text;
begin
select name into result from test;
raise notice '%', result;
end;
$$;
I also fixed your raise notice.
If you just want to dump the single value from the table to the standard output in a minimal format (i.e. easy to parse), then perhaps --tuples-only will help:
-t
--tuples-only
Turn off printing of column names and result row count footers, etc. This is equivalent to the \t command.
So you could say things like this from the shell:
result=$(echo 'select name from test;' | psql -t ...)
This PostgreSQL COPY command works:
copy tablename from E'c:\\abc\\a.txt';
but I want the tablename to be dynamically generated. How can I do this?
You need to build a string, concatenating in the dynamic table name, and then use execute. Note that you escape the ' by ''. This also includes a dynamic name to save the file too. You need to replace savedir with the actual directory you are using.
CREATE OR REPLACE FUNCTION dynamicCopy(tablename text, outname text) RETURNS VOID AS $$
DECLARE STATEMENT TEXT;
BEGIN
STATEMENT := 'COPY (select * from ' || quote_ident(tablename) || ') to ''savedir' || outname ||'.txt''';
EXECUTE STATEMENT;
END;
$$ LANGUAGE 'plpgsql';
EDIT:
Since I first wrote this, I have discovered the format function, which I think is generally easier to read than SQL generated with the concatenation operator || and more flexible.
CREATE OR REPLACE FUNCTION dynamicCopy(tablename text, outname text) RETURNS VOID AS
$BODY$
BEGIN
EXECUTE FORMAT('COPY (SELECT * FROM %s) TO ''savedir%s.csv''',
tablename,
outname);
END
$BODY$
LANGUAGE plpgsql;
See the official docs for a full discussion: https://www.postgresql.org/docs/current/static/plpgsql-statements.html#PLPGSQL-STATEMENTS-EXECUTING-DYN
You can copy from multi csv files to an table via shell script.
Make an script file: vim csvtotable
write csvtotable script. Here is my example:
#!/bin/sh DBNAME=postgres files=$1 for file in ${files}; do psql -d ${DBNAME} -c "\copy parent_tree(parent_id, some_text) FROM '${file}' delimiters ',' csv header" done
Execute the script.
./csv2table "$(ls *.out.csv)"
Obviously, Local CSV file should be match with table. Then It will import from csv to database tables if csv file name ending with .out.csv.
I am not sure csv name match is global or just at present directory match.
Reference link:
-c command reference https://www.postgresql.org/docs/current/app-psql.html#APP-PSQL-PATTERNS
chomd command: https://linuxize.com/post/chmod-command-in-linux/
bash shanebang: https://linuxize.com/post/bash-shebang/