I want to do copy csv to database using stored procedure. My function is follows;
CREATE FUNCTION gis_portal.copycsv(IN path text) RETURNS void AS
'COPY gis_portal.temp_excel FROM path WITH DELIMITER'
LANGUAGE sql VOLATILE LEAKPROOF;
query is
COPY gis_portal.temp_excel FROM path WITH DELIMITER ',' CSV HEADER
with parameter path.
It is giving error as syntax error near path while creating function.
Please help me.
'COPY gis_portal.temp_excel FROM path WITH DELIMITER'
^^^^^^^^^^
What delimiter? You must specify one if you use that keyword.
Try:
CREATE FUNCTION gis_portal.copycsv(IN path text) RETURNS void AS $$
COPY gis_portal.temp_excel FROM path WITH DELIMITER ','
$$ LANGUAGE sql;
or whatever delimiter you want.
Additionally, in SQL functions you can't use the identifier you must use a positional parameter, like $1. But because COPY isn't a plannable statement you cannot use parameters in it. You will have to use PL/PgSQL and EXECUTE to run it as dynamic SQL:
CREATE FUNCTION gis_portal.copycsv(IN path text) RETURNS void AS $$
BEGIN
EXECUTE format('COPY gis_portal.temp_excel FROM %L WITH DELIMITER '',''', path);
END;
$$ LANGUAGE plpgsql;
Note the doubled quotes around the delimiter because it's now an SQL string.
Related
I'm trying to get FULL definition text of a function, but the query that everyone suggests returns the functions script without the header and footer parts.
Here's what I'm using:
select prosrc from pg_proc where proname='my_func';
Here's what is returned:
DECLARE
some_var1 TEXT;
some_var2 TEXT;
Status character varying;
BEGIN
--SCRIPT CODE HERE
RETURN retval;
END;
But what I want is this:
CREATE OR REPLACE FUNCTION my_func(logdate_utc date)
RETURNS character varying
LANGUAGE 'plpgsql'
COST 100
VOLATILE
AS $BODY$
DECLARE
some_var1 TEXT;
some_var2 TEXT;
Status character varying;
BEGIN
--SCRIPT CODE HERE
RETURN retval;
END;
$BODY$;
ALTER FUNCTION my_func(date)
OWNER TO some_owner;
Searched around and haven't found much. How can I get that result?
Using SQL:
select pg_catalog.pg_get_functiondef('my_func(date)'::regprocedure::oid);
Doc
Note that you should to use regprocedure type if you specify the parameters list and regproc type if you specify only function name (like 'my_func'::regproc::oid)
And using command line psql with \sf meta-command:
This command fetches and shows the definition of the named function, in the form of a CREATE OR REPLACE FUNCTION command. The definition is printed to the current query output channel, as set by \o.
The target function can be specified by name alone, or by name and arguments, for example foo(integer, text). The argument types must be given if there is more than one function of the same name.
from the documentation.
For example:
psql -c '\sf my_func(date)'
And to retrieve function's owner name you can use next query:
select proowner::regrole
from pg_proc
where oid = 'my_func(date)'::regprocedure;
Use pg_get_functiondef(func_oid):
SELECT pg_get_functiondef('my_func'::regproc);
SELECT pg_get_functiondef(( SELECT oid FROM pg_proc where proname= 'my_func'));
I'm trying to get FULL definition text of a function, but the query that everyone suggests returns the functions script without the header and footer parts.
Here's what I'm using:
select prosrc from pg_proc where proname='my_func';
Here's what is returned:
DECLARE
some_var1 TEXT;
some_var2 TEXT;
Status character varying;
BEGIN
--SCRIPT CODE HERE
RETURN retval;
END;
But what I want is this:
CREATE OR REPLACE FUNCTION my_func(logdate_utc date)
RETURNS character varying
LANGUAGE 'plpgsql'
COST 100
VOLATILE
AS $BODY$
DECLARE
some_var1 TEXT;
some_var2 TEXT;
Status character varying;
BEGIN
--SCRIPT CODE HERE
RETURN retval;
END;
$BODY$;
ALTER FUNCTION my_func(date)
OWNER TO some_owner;
Searched around and haven't found much. How can I get that result?
Using SQL:
select pg_catalog.pg_get_functiondef('my_func(date)'::regprocedure::oid);
Doc
Note that you should to use regprocedure type if you specify the parameters list and regproc type if you specify only function name (like 'my_func'::regproc::oid)
And using command line psql with \sf meta-command:
This command fetches and shows the definition of the named function, in the form of a CREATE OR REPLACE FUNCTION command. The definition is printed to the current query output channel, as set by \o.
The target function can be specified by name alone, or by name and arguments, for example foo(integer, text). The argument types must be given if there is more than one function of the same name.
from the documentation.
For example:
psql -c '\sf my_func(date)'
And to retrieve function's owner name you can use next query:
select proowner::regrole
from pg_proc
where oid = 'my_func(date)'::regprocedure;
Use pg_get_functiondef(func_oid):
SELECT pg_get_functiondef('my_func'::regproc);
SELECT pg_get_functiondef(( SELECT oid FROM pg_proc where proname= 'my_func'));
Being completely new to PL/pgSQL , what is the meaning of double dollar signs in this function:
CREATE OR REPLACE FUNCTION check_phone_number(text)
RETURNS boolean AS $$
BEGIN
IF NOT $1 ~ e'^\\+\\d{3}\\ \\d{3} \\d{3} \\d{3}$' THEN
RAISE EXCEPTION 'Wrong formated string "%". Expected format is +999 999';
END IF;
RETURN true;
END;
$$ LANGUAGE plpgsql STRICT IMMUTABLE;
I'm guessing that, in RETURNS boolean AS $$, $$ is a placeholder.
The last line is a bit of a mystery: $$ LANGUAGE plpgsql STRICT IMMUTABLE;
By the way, what does the last line mean?
These dollar signs ($$) are used for dollar quoting, which is in no way specific to function definitions. It can be used to replace single quotes enclosing string literals (constants) anywhere in SQL scripts.
The body of a function happens to be such a string literal. Dollar-quoting is a PostgreSQL-specific substitute for single quotes to avoid escaping of nested single quotes (recursively). You could enclose the function body in single-quotes just as well. But then you'd have to escape all single-quotes in the body:
CREATE OR REPLACE FUNCTION check_phone_number(text)
RETURNS boolean
LANGUAGE plpgsql STRICT IMMUTABLE AS
'
BEGIN
IF NOT $1 ~ e''^\\+\\d{3}\\ \\d{3} \\d{3} \\d{3}$'' THEN
RAISE EXCEPTION ''Malformed string "%". Expected format is +999 999'';
END IF;
RETURN true;
END
';
This isn't such a good idea. Use dollar-quoting instead. More specifically, also put a token between the $$ to make each pair unique - you might want to use nested dollar-quotes inside the function body. I do that a lot, actually.
CREATE OR REPLACE FUNCTION check_phone_number(text)
RETURNS boolean
LANGUAGE plpgsql STRICT IMMUTABLE AS
$func$
BEGIN
...
END
$func$;
See:
Insert text with single quotes in PostgreSQL
As to your second question:
Read the most excellent manual on CREATE FUNCTION to understand the last line of your example.
The $$ is a delimiter you use to indicate where the function definition starts and ends. Consider the following,
CREATE TABLE <name> <definition goes here> <options go here, eg: WITH OIDS>
The create function syntax is similar, but because you are going to use all sorts of SQL in your function (especially the end of statement ; character), the parser would trip if you didn't delimit it. So you should read your statement as:
CREATE OR REPLACE FUNCTION check_phone_number(text)
RETURNS boolean AS <code delimited by $$> LANGUAGE plpgsql STRICT IMMUTABLE;
The stuff after the actual definition are options to give the database more information about your function, so it can optimize its usage.
In fact, if you look under "4.1.2.4. Dollar-Quoted String Constants" in the manual, you will see that you can even use characters in between the dollar symbols and it will all count as one delimiter.
I may be contorting postgres here, but fundamentally, what I would like to do is take a string variable and pass it to an sql command (in this case COPY) using only psql.
So this is what I came up with. The commands are separated into 2 files because I want to be able to use the mydb_functions in other situations:
file one: mydb_functions--1.0.sql (in share/extension and with mydb_functions.control also setup as described in the manual. Given a filename, returns a full filepath. this is done solely to make the COPY statements in add_data.sql below, neater.
CREATE OR REPLACE FUNCTION fpn(filename text) RETURNS TEXT as '
DECLARE
mypath text := ''/path/to/my/directory/'';
BEGIN
RETURN mypath || filename;
END
' LANGUAGE plpgsql;
file two: add_data.sql . This exists solely to copy data into existing postgres tables using psql at the command line. Note: running psql with superuser privileges is required because of the CREATE EXTENSION command.
CREATE EXTENSION IF NOT EXISTS mydb_functions;
-- haven't figured out how to create a function without arguments yet.
CREATE OR REPLACE FUNCTION test (dummyvar text) RETURNS text as '
DECLARE
filepath RECORD;
BEGIN
SELECT * INTO filepath from fpn(''mydatafile.data'');
COPY tablename (columnname) FROM filepath;
END
' LANGUAGE plpgsql;
The part I am stuck on is how to extract a text from the filepath record to use in the COPY command. Any tips on an easier way to achieve this are also welcome. I think creating a table to store the variable is far easier than this. But I would like to finish the last step.
If your issue is running COPY with a dynamic target path, use EXECUTE to run a formatted SQL query.
CREATE OR REPLACE FUNCTION test () RETURNS text as $$
DECLARE
filepath RECORD;
BEGIN
SELECT * INTO filepath from fpn('mydatafile.data');
EXECUTE format('COPY tablename (columnname) FROM %L', filepath);
END
$$ LANGUAGE plpgsql;
See, this fails:
DO
$$
DECLARE
somevar text;
BEGIN
somevar := '/tmp/somepath.csv';
COPY tablename(columnname) FROM somevar;
END;
$$;
but this works:
DO
$$
DECLARE
somevar text;
BEGIN
somevar := '/tmp/somepath.csv';
EXECUTE format('COPY tablename(columnname) FROM %L', somevar);
END;
$$;
See:
EXECUTE
format()
format()'s %L specifier auto-quotes literals. %I does the same for identifiers.
If as I originally thought, you're talking about getting data into psql from outside, you may find psql variables and variable interpolation useful:
$ psql -v filepath=/path/to/my/directory/mydatafile.data regress
regress=> SELECT :'filepath';
?column?
---------------------------------------
/path/to/my/directory/mydatafile.data
(1 row)
Note that the colon is unquoted, then the variable name is quoted. Odd syntax, I know. This only works in psql; it won't work in (say) PgAdmin-III.
Alternately, you can use a here-document (unix-like shell specific, won't work in Windows' cmd.exe) to do quoted text interpolation:
$ FILEPATH=/path/to/my/directory/mydatafile.data
$ psql regress <<__END__
SELECT '$FILEPATH';
__END__
?column?
---------------------------------------
/path/to/my/directory/mydatafile.data
(1 row)
Both of these show how to insert a variable from the shell level into psql. From there, it's a matter of using it in your function. Like this, say, with -v filename=myfilename.csv:
CREATE OR REPLACE FUNCTION test() RETURNS text as $$
DECLARE
filepath RECORD;
BEGIN
SELECT * INTO filepath from fpn(:'filename');
COPY tablename (columnname) FROM filepath;
END
$$ LANGUAGE plpgsql;
or just this with -v filepath=/full/path/to/myfilename.csv:
$ psql -v filepath=/path/to/my/directory/mydatafile.data regress
regress=> COPY tablename (columnname) FROM :'filepath';
This PostgreSQL COPY command works:
copy tablename from E'c:\\abc\\a.txt';
but I want the tablename to be dynamically generated. How can I do this?
You need to build a string, concatenating in the dynamic table name, and then use execute. Note that you escape the ' by ''. This also includes a dynamic name to save the file too. You need to replace savedir with the actual directory you are using.
CREATE OR REPLACE FUNCTION dynamicCopy(tablename text, outname text) RETURNS VOID AS $$
DECLARE STATEMENT TEXT;
BEGIN
STATEMENT := 'COPY (select * from ' || quote_ident(tablename) || ') to ''savedir' || outname ||'.txt''';
EXECUTE STATEMENT;
END;
$$ LANGUAGE 'plpgsql';
EDIT:
Since I first wrote this, I have discovered the format function, which I think is generally easier to read than SQL generated with the concatenation operator || and more flexible.
CREATE OR REPLACE FUNCTION dynamicCopy(tablename text, outname text) RETURNS VOID AS
$BODY$
BEGIN
EXECUTE FORMAT('COPY (SELECT * FROM %s) TO ''savedir%s.csv''',
tablename,
outname);
END
$BODY$
LANGUAGE plpgsql;
See the official docs for a full discussion: https://www.postgresql.org/docs/current/static/plpgsql-statements.html#PLPGSQL-STATEMENTS-EXECUTING-DYN
You can copy from multi csv files to an table via shell script.
Make an script file: vim csvtotable
write csvtotable script. Here is my example:
#!/bin/sh DBNAME=postgres files=$1 for file in ${files}; do psql -d ${DBNAME} -c "\copy parent_tree(parent_id, some_text) FROM '${file}' delimiters ',' csv header" done
Execute the script.
./csv2table "$(ls *.out.csv)"
Obviously, Local CSV file should be match with table. Then It will import from csv to database tables if csv file name ending with .out.csv.
I am not sure csv name match is global or just at present directory match.
Reference link:
-c command reference https://www.postgresql.org/docs/current/app-psql.html#APP-PSQL-PATTERNS
chomd command: https://linuxize.com/post/chmod-command-in-linux/
bash shanebang: https://linuxize.com/post/bash-shebang/