I am generating a few CSV files using COPY statements from postgres in a stored function.
CREATE OR REPLACE FUNCTION "public"."create_csv" () RETURNS INT AS
$$
BEGIN
EXECUTE format('COPY (SELECT * FROM test) TO %L DELIMITER '','' CSV HEADER', '/tmp/test.csv');
EXECUTE format('COPY (SELECT * FROM test2) TO %L DELIMITER '','' CSV HEADER', '/tmp/test2.csv');
RETURN 0;
END;
$$
LANGUAGE 'plpgsql' SECURITY DEFINER
This function is called from a parent one, after the execution of that I want to zip the generated CSV files into a zip file, how can I accomplish this using stored functions in the data base? Is there a way to do so using plpgsql? Or should i use plpythonu language?
Postgres version 9.6.
Thanks!
#a_horse_with_no_name gave you an alert about what you should do use copy ... PROGRAM, just add the code at the end of your function, in this case in tar format:
CREATE OR REPLACE FUNCTION "public"."create_csv" () RETURNS INT AS
$$
BEGIN
EXECUTE format('COPY (SELECT * FROM test) TO %L DELIMITER '','' CSV HEADER', '/tmp/test.csv');
EXECUTE format('COPY (SELECT * FROM test2) TO %L DELIMITER '','' CSV HEADER', '/tmp/test2.csv');
--add copy commando with PROGRAM
COPY (SELECT 1) TO PROGRAM 'tar cvf /tmp/mytarfile.tar --remove-files /tmp/*.csv';
RETURN 0;
END;
$$
LANGUAGE 'plpgsql' SECURITY DEFINER
Related
I use below command to export table to csv
COPY (
SELECT
*
FROM
table_name
TO '/data/test/test.csv' WITH CSV DELIMITER ',' HEADER;
I used it in Postgres function but i cannot pass string to file path
CREATE OR REPLACE FUNCTION backup (filePath CHARACTER VARYING)
RETURNS void
LANGUAGE SQL
AS $$
-- Export data to csv
COPY (
SELECT
*
FROM
table_name)
TO filePath WITH CSV DELIMITER ',' HEADER;
$$
err: Query 1 ERROR: ERROR: syntax error at or near "filePath" LINE 16: TO filePath WITH CSV DELIMITER ',' HEADER;
See here SQL Functions:
SQL function arguments can only be used as data values, not as identifiers. Thus for example this is reasonable: ...
I'm going to say the filePath is being seen as an identifier. You will need to use plpgsql like so:
CREATE OR REPLACE FUNCTION public.backup(filepath character varying)
RETURNS void
LANGUAGE plpgsql
AS $function$
BEGIN
-- Export data to csv
EXECUTE 'COPY ( SELECT * FROM table_name) TO ' || quote_literal(filePath) || ' WITH CSV DELIMITER '','' HEADER';
RETURN;
END;
$function$
I have copied function(from one of the webportal and modified accordingly) to copy data from csv file to table.
create or replace function public.load_csv_file
(
target_table text,
csv_path text,
col_count integer
)
returns void as $$
declare
iter integer; -- dummy integer to iterate columns with
col text; -- variable to keep the column name at each iteration
col_first text; -- first column name, e.g., top left corner on a csv file or spreadsheet
begin
set schema 'public';
create table insert_from_csv ();
-- add just enough number of columns
for iter in 1..col_count
loop
execute format('alter table insert_from_csv add column col_%s text;', iter);
end loop;
-- copy the data from csv file
execute format('copy insert_from_csv from %L with delimiter '','' quote ''"'' csv ', csv_path);
iter := 1;
col_first := (select col_1 from insert_from_csv limit 1);
-- update the column names based on the first row which has the column names
for col in execute format('select unnest(string_to_array(trim(temp_table::text, ''()''), '','')) from temp_table where col_1 = %L', col_first)
loop
execute format('alter table insert_from_csv rename column col_%s to %s', iter, col);
iter := iter + 1;
end loop;
-- delete the columns row
execute format('delete from insert_from_csv where %s = %L', col_first, col_first);
-- change the temp table name to the name given as parameter, if not blank
if length(target_table) > 0 then
execute format('alter table insert_from_csv rename to %I', target_table);
end if;
end;
$$ language plpgsql;
And
passing parameters as
select load_csv_file('Customer','C:\Insert_postgres.csv' ,4)
but getting error message
ERROR: must be superuser to COPY to or from a file
Hint: Anyone can COPY to stdout or from stdin. psql's \copy command also works for anyone.
The idea is, i will create a automated test, and if i want to test on different instance then test should automatically create function and copy data from csv file.
Is there any work around to copy data without superuser?
Looks Insert_postgres.csv is in C drive which usually does not have Read/Write permission. Move the file to your directory where Read/Write given atleast to some groups or everyone.
Hope it will resolve the issue
Try running the following command in terminal. This allows us to use COPY command, since you can't use COPY command without being a super user.
ALTER USER <username> WITH SUPERUSER;
I have copied function(from one of the webportal and modified accordingly) to copy data from csv file to table.
create or replace function public.load_csv_file
(
target_table text,
csv_path text,
col_count integer
)
returns void as $$
declare
iter integer; -- dummy integer to iterate columns with
col text; -- variable to keep the column name at each iteration
col_first text; -- first column name, e.g., top left corner on a csv file or spreadsheet
begin
set schema 'public';
create table insert_from_csv ();
-- add just enough number of columns
for iter in 1..col_count
loop
execute format('alter table insert_from_csv add column col_%s text;', iter);
end loop;
-- copy the data from csv file
execute format('copy insert_from_csv from %L with delimiter '','' quote ''"'' csv ', csv_path);
iter := 1;
col_first := (select col_1 from insert_from_csv limit 1);
-- update the column names based on the first row which has the column names
for col in execute format('select unnest(string_to_array(trim(temp_table::text, ''()''), '','')) from temp_table where col_1 = %L', col_first)
loop
execute format('alter table insert_from_csv rename column col_%s to %s', iter, col);
iter := iter + 1;
end loop;
-- delete the columns row
execute format('delete from insert_from_csv where %s = %L', col_first, col_first);
-- change the temp table name to the name given as parameter, if not blank
if length(target_table) > 0 then
execute format('alter table insert_from_csv rename to %I', target_table);
end if;
end;
$$ language plpgsql;
And
passing parameters as
select load_csv_file('Customer','C:\Insert_postgres.csv' ,4)
but getting error message
ERROR: could not open file "C:\Insert_postgres.csv" for reading: No such file or directory
CONTEXT: SQL statement "copy insert_from_csv from E'C:\Insert_postgres.csv' with delimiter ',' quote '"' csv "
PL/pgSQL function load_csv_file(text,text,integer) line 21 at EXECUTE
Any help appreciated.
I tried to moving file to some other location, 'C:\testData\Insert_postgres.csv'
Initially i was trying below
select load_csv_file('Customer','C:\Insert_postgres.csv' ,4)
and received error
After several tries i thought i will create a folder on c drive and put the csv file in that folder and it worked.
select load_csv_file('Customer','C:\testData\Insert_postgres.csv' ,4).
It seems postgres couldnt access file directly from c drive, but if file is put in a folder on c: drive then worked successfully.
I'm trying to compile a postgresql function
CREATE OR REPLACE FUNCTION LoadData(tablename varchar(25), filepath varchar(35))
RETURNS void AS $$
declare
BEGIN
RAISE NOTICE 'Data is being loaded from an external file, please wait...';
COPY tablename FROM filepath DELIMITER ',' CSV HEADER;
RAISE NOTICE 'Data loaded successfully!!';
END;
$$ LANGUAGE plpgsql;
but it's giving an error like this
ERROR: syntax error at or near "filepath"
LINE 9: COPY tablename FROM filepath DELIMITER ',' CSV HEADER;
How to use both the in parameters in the COPY command?
To execute dynamically built SQL queries use execute. format makes it easy and safe
create or replace function loaddata(
tablename varchar(25), filepath varchar(35)
) returns void as $$
begin
raise notice 'data is being loaded from an external file, please wait...';
execute format($copy$
copy %I from %L delimiter ',' csv header
$copy$,
tablename, filepath
);
raise notice 'data loaded successfully!!';
end;
$$ language plpgsql;
http://www.postgresql.org/docs/current/static/plpgsql-statements.html#PLPGSQL-STATEMENTS-EXECUTING-DYN
I would like to repeat the following query 8760 times, replacing ‘2’ with 1 to 8760 for every hour in the year. The idea is to create a separate CSV file for each hour for further processing.
COPY
(SELECT *
FROM
public.completedsolarirad2012
WHERE
completedsolarirad2012."UniquetmstmpID" = 2)
TO 'C:\temp\2012hour2.csv' WITH DELIMITER ','
CSV HEADER
I have put together the following function (testing with only a few hours):
CREATE OR REPLACE FUNCTION everyhour()
RETURNS void AS
$BODY$BEGIN
FOR i IN 0..5 LOOP
EXECUTE $x$
COPY (
SELECT *
FROM
public.completedsolarirad2012
WHERE
completedsolarirad2012."UniquetmstmpID" = i
)
TO $concat$ 'C:\temp.' || i::text
|| '.out.csv' WITH DELIMITER ',' CSV HEADER $concat$
$x$;
END LOOP;
END;
$BODY$
LANGUAGE plpgsql VOLATILE
COST 100;
ALTER FUNCTION everyhour()
OWNER TO postgres;
I seem to be having two separate problems:
Firstly, I’m getting:
Error: column "i" does not exist.
Secondly, when I test the concatenation statement only by replacing “i” with e.g. “2”, I get:
Error: relative path not allowed for COPY to file
I am the postgres superuser, so I do not understand why I am having this problem.
Note: Removing the $concat$ double-quoting around the concatenation statement gives the following error:
ERROR: syntax error at or near "||"
LINE 9: TO 'C:\temp.' || i::text
I would be very grateful for any help.
Assuming your server OS is Windows.
CREATE OR REPLACE FUNCTION everyhour()
RETURNS void AS
$func$
BEGIN
FOR i IN 0..5 LOOP
EXECUTE '
COPY (
SELECT *
FROM public.completedsolarirad2012 c
WHERE c."UniquetmstmpID" = ' || i || $x$)
TO 'C:/temp.'$x$ || i || $x$'.out.csv' WITH DELIMITER ',' CSV HEADER$x$;
END LOOP;
END
$func$ LANGUAGE plpgsql;
You had one layer of dollar-quoting too many.
You also accidentally concatenated the letter "i" instead of its value.
Use forward-slashes, even with windows. Or you may have to double up the backslashes, depending on your settings. More in this related answer:
PostgreSQL: export resulting data from SQL query to Excel/CSV
Simpler with format()
Since Postgres 9.1 you can use format() to simplify complex concatenations:
CREATE OR REPLACE FUNCTION everyhour()
RETURNS void AS
$func$
BEGIN
FOR i IN 0..5 LOOP
EXECUTE format($x$COPY (
SELECT *
FROM public.completedsolarirad2012 c
WHERE c."UniquetmstmpID" = %1$s)
TO 'C:/temp.%1$s.out.csv' WITH DELIMITER ',' CSV HEADER$x$, i);
END LOOP;
END
$func$ LANGUAGE plpgsql;