I am trying to write a procedure that loads a csv into a table. I create a dynamic sql and execute it.
v_sql = '\copy test_table from' || ' ''' || p_temp_dir || ''' ' || ' with csv header;';
--insert into test
--select v_sql;
EXECUTE v_sql;
When I check the generated string it is fine and I can run it properly, but if I run it from a procedure, I get an error:
syntax error at or near "\"
If I remove the \ character it works (throws an error that there is no such file; this job is intended to work on the local machine). How can I execute this SQL?
\copy is a psql command, not an SQL statement.
You can only execute SQL statements with PL/pgSQL's EXECUTE statement.
You cannot import a file from the client machine in a PostgreSQL function that runs on the server. You will have to write client code that does COPY ... FROM STDIN.
Related
I'm attempting to dynamically create a script that gets saved as a bat file that will be scheduled to execute daily via Windows Task Scheduler. The script performs full database backups for each Postgres database using pg_dump.
The current script is as follows:
COPY (SELECT 'pg_dump '|| datname || ' > e:\postgresbackups\FULL\' || datname || '_%date:~4,2%-%date:~7,2%-%date:~10,4%_%time:~0,2%_%time:~3,2%_%time:~6,2%.dump' FROM pg_database) TO 'E:\PostgresBackups\Script\FULL_Postgres_Backup_Job_TEST.bat' (format csv, delimiter ';');
An example of the output is as follows:
pg_dump postgres > e:\postgresbackups\FULL\postgres_%date:~4,2%-%date:~7,2%-%date:~10,4%%time:~0,2%%time:~3,2%_%time:~6,2%.dump
I need help with updating my code so that the output will include double quotes around the name of the dump file; however, when I add this to my COPY script it adds more than what is necessary to the output. I would like the output to look like the following which includes the double-quotes:
pg_dump postgres > "e:\postgresbackups\FULL\postgres_%date:~4,2%-%date:~7,2%-%date:~10,4%%time:~0,2%%time:~3,2%_%time:~6,2%.dump"
Any help would be greatly appreciated!
Thanks to #Mike Organek's comment, my issue has been resolved by switching the format from CSV to TEXT. Now when I enclose the dump filename in double quotes, the output is more of what is expected and works as intended. The only odd thing now is that in the output it creates a second backslash in the filename. My code has been updated as follows:
COPY (SELECT 'pg_dump '|| datname || ' > "e:\postgresbackups\FULL\' || datname || '_%date:~4,2%-%date:~7,2%-%date:~10,4%_%time:~0,2%_%time:~3,2%_%time:~6,2%.dump"' FROM pg_database) TO 'E:\PostgresBackups\Script\FULL_Postgres_Backup_Job.bat' (format text, delimiter ';');
An example of the output that gets created within the bat file is as follows:
pg_dump postgres > "e:\\postgresbackups\\FULL\\postgres_%date:~4,2%-%date:~7,2%-%date:~10,4%_%time:~0,2%_%time:~3,2%_%time:~6,2%.dump"
As you can see, it adds a double backslash; however, the pg_dump executes successfully!
below is my procedure executed to upload file into table and do joins etc.
CREATE OR REPLACE FUNCTION sp_product_price()
RETURNS void
LANGUAGE 'plpgsql'
COST 100
AS $BODY$
BEGIN
truncate table prd_product_data;
truncate table price_import;
COPY price_import FROM 'C:\Users\Ram\Documents\prices.csv' CSV HEADER;
truncate table active_product_price;
insert into active_product_price(key,name,price)
SELECT prd.key,prd.name,prd.price FROM prd_product_data prd JOIN price_import import ON prd.name = import.name;
raise notice 'success';
END
$BODY$;
Above procedure giving error could not open file "C:\Users\Ram\Documents\prices.csv" for reading: No such file or directory HINT: COPY FROM instructs the PostgreSQL server process to read a file. You may want a client-side facility such as psql's \copy.
I have given access to file for everyone in file properties.
I tried \copy in procedure but this gives error syntax error at or near "\" .
\copy is working when I ran cmd in psql but not in above procedure.
Is there a way to import file into table in above procedure/functions ?
The procedure and the COPY statement are running on the database server, so the file C:\Users\Ram\Documents\prices.csv must be on the database server as well (and your database user must either be a superuser or a member of pg_read_server_files).
The only way you can use COPY to load data from the client is COPY ... FROM STDIN, and you won't be able to use that in a procedure.
\copy is a psql command, not an SQL statement, so it cannot be used in a procedure either.
My suggestion is to use \copy, but to forget about the procedure.
I want to run some sql scripts if a specific schema exists. This is my attempt:
do $$
begin
if exists (select 1 from information_schema.schemata where schema_name = 'public')
then
\ir ./script.sql
end if;
end
$$;
But I get this error:
$ psql my_database -f test.sql
psql:test.sql:8: ERROR: syntax error at or near "\"
LINE 5: \ir ./script.sql
The same script without the if statement works and is able to trigger ./script.sql.
Adding the code from script.sql directly in the if statement also works, there doesn't seem to be anything wrong with the if syntax.
How can you conditionally trigger SQL scripts?
You cannot do that in PL/pgSQL.
You could write a function in PL/PerlU or PL/Python that does that, or you can install the PL/sh language.
Basically, what I want to achieve is something like this:
mypath:= '~/Desktop/' || my_variable || '.csv'
COPY (SELECT * FROM my_table) TO mypath CSV DELIMITER ',' HEADER;
Where the value of my_variable will change dynamically.
Can someone help me on this problem?
Here and here you can find two different options (either via a bash script or a sql script with variables) to solve your problem.
Since you are using windows the only viable solution is the one with the variable in the sql file. With Windows you should also be able to use the psql command line utility to execute your sqlfile and pass the path as a parameter like this:
-- Example sql file content:
COPY (SELECT * FROM my_table) TO :path CSV DELIMITER ',' HEADER;
Command line example
psql -f /tmp/file.sql -v path='~/Desktop/' || my_variable || '.csv'
Another option i just found would be to output the content of the csv to stdout and forwarding that to file on the windows cmd level.
Example code (based on this answer Linux Example code):
psql -c "COPY (SELECT * FROM my_table) TO STDOUT CSV DELIMITER ',' HEADER;" > '~/Desktop/' || my_variable || '.csv'
EDIT1 based on the new requirement that the variable comes out of the postgresql database:
I quickly built a For loop which can loop over the result of a separate query and then execute a sql query for each of the values in the result:
Example code:
DO $$
declare
result record;
BEGIN
FOR result IN Select * FROM (VALUES ('one'), ('two'), ('three')) AS t (path) LOOP
RAISE NOTICE 'path: ~/Desktop/%.csv', test.path;
END LOOP;
END; $$;
This produces the following output:
NOTICE: path: ~/Desktop/one.csv
NOTICE: path: ~/Desktop/two.csv
NOTICE: path: ~/Desktop/three.csv
You can substitue the Select * FROM (VALUES ('one'), ('two'), ('three')) AS t (path) with any query which produces the table result which contains one path per row.
Second you can substitue the RAISE NOTICE 'path: ~/Desktop/%.csv', test.path; with your copy query
You can use EXECUTE command. A working example is:
sql_copy_command= '
COPY abc_schema_name.xyz_table_name FROM ''' || masterFilePath ||''' DELIMITER '';'' CSV HEADER ' ;
RAISE NOTICE 'sql_copy_command: % ', sql_copy_command;
EXECUTE sql_copy_command;
is possible in PSQL console export file with current date on the end of the file name?
The name of the exported file should be like this table_20140710.csv is it possible to do this dynamically? - the format of the date can be different than the above it isn't so much important.
This is example what i mean:
\set curdate current_date
\copy (SELECT * FROM table) To 'C:/users/user/desktop/table_ ' || :curdate || '.csv' WITH DELIMITER AS ';' CSV HEADER
The exception of the \copy meta command not expanding variables is (meanwhile) documented
Unlike most other meta-commands, the entire remainder of the line is always taken to be the arguments of \copy, and neither variable interpolation nor backquote expansion are performed in the arguments.
To workaround you can build, store and execute the command in multiple steps (similar to the solution Clodoaldo Neto has given):
\set filename 'my fancy dynamic name'
\set command '\\copy (SELECT * FROM generate_series(1, 5)) to ' :'filename'
:command
With this, you need to double (escape) the \ in the embedded meta command. Keep in mind that \set concatenates all further arguments to the second one, so quote spaces between the arguments. You can show the command before execution (:command) with \echo :command.
As an alternative to the local \set command, you could also build the command server side with SQL (the best way depends on where the dynamic content is originating):
SELECT '\copy (SELECT * FROM generate_series(1, 5)) to ''' || :'filename' || '''' AS command \gset
Dynamically build the \copy command and store it in a file. Then execute it with \i
First set tuples only output
\t
Set the output to a file
\o 'C:/users/user/desktop/copy_command.txt'
Build the \copy command
select format(
$$\copy (select * from the_table) To 'C:/users/user/desktop/table_%s.csv' WITH DELIMITER AS ';' CSV HEADER$$
, current_date
);
Restore the output to stdout
\o
Execute the generated command from the file
\i 'C:/users/user/desktop/copy_command.txt'