I have a simple query that i run inside a Postgres SQL Query that runs a bunch of queries then extracts it to CSV, which [without the bunch of queries above it] is
COPY (SELECT * FROM public.view_report_teamlist_usertable_wash)
TO 'd:/sf/Reports/view_report_teamlist_usertable_wash.csv'
DELIMITER ','
CSV HEADER encoding 'UTF8' FORCE QUOTE *;
Can I alter the above at all to append the date/time [now()] to the filename? ie. 'd:/sf/Reports/view_report_teamlist_usertable_wash_2017-08-23 14:30:28.288912+10.csv'
I have googled it many times but only come up with solutions that runs it from a command line
If you want to use it once or not regularly, you could use postgres DO. But, if you use the script regularly, you should write a PL.
Either way, it should be like this:
DO $$
DECLARE variable text;
BEGIN
variable := to_char(NOW(), 'YYYY-MM-DD_HH24:MI:SS');
EXECUTE format ('COPY (SELECT * FROM public.view_report_teamlist_usertable_wash)
TO ''d:/sf/Reports/view_report_teamlist_usertable_wash_%s.csv''
DELIMITER '',''
CSV HEADER encoding ''UTF8'' FORCE QUOTE *',-- // %s will be replaced by string variable
variable -- File name
);
END $$;
-- // NOTE the '' for escaping '
EDIT: DO runs inline with other queries.
Related
I have a dynamic SQL statement I'm building up that is something like
my_interval_var := interval - '60 days';
sql_query := format($f$ AND NOT EXISTS (SELECT 1 FROM some_table st WHERE st.%s = %s.id AND st.expired_date >= %L )$f$, var1, var2, now() - my_interval_var)
Regarding my first question, it seemed to insert the Timestamp correctly (it seems) after the now() - my_interval_var computation. However, I just want to make sure I don't need to cast anything or something, because the only way I could get it work was if I used %L, which is the string literal Identifer. Or does postgres allow direct comparisons with Strings that represent Time without a cast?, like
some_column <= '2021-12-31 00:00:00'; // is ::timestamp cast needed?
Second of all, regarding the sql_query variable that I concatenated an SQL String into above, I actually wanted to skip the Format I did, and directly inject this sql_query variable into an EXECUTE...FORMAT...USING statement.
I couldn't get it to work, but something like this:
EXECUTE format($f$ SELECT *
FROM %I tbl_alias
WHERE tbl_alias.%s = %L
%s ) USING var1, var2, var3, sql_query;
Is it possible to leave the Dynamic SQL Identifiers %I %L and %s inside the variable and FORMAT it at the EXECUTE... level? Something tells me this isn't possible, but it would be really cool.
Also last question I didn't want to add, but I feel someone might have a quick answer.
I was using the ]
FOR temprecord IN
SELECT myCol1, myCol2, myCol3
FROM %I tbl',var1)
LOOP
EXECUTE temprecord.someColumnOnMyTbl;
END LOOP;
...but I could not for the life of get the EXECUTE temprecord.someColumnOnMyTbl statement to work when I made the query dynamic. I tried everything identifier, using FORMAT, USING...
I thought columns were strings like %s because I do that for columns all the time when they are aliased like alias.%s = 'some string literal'
ANyway, I couldn't get it to work, I wanted to make the column name dynamic but tried all these things
EXECUTE format($f$ %I.%s $f$, var1, var2);
EXECUTE format($f$ %$1.%$2 $f$) USING var1, var2;
EXECUTE format($f$ %I.someColumnOn%s $f$, var1, var2);
EXECUTE format($f$ $1.someColumnOn$2 $f$) USING var1, var2;
Anyway, I tried more stuff than that, but I actually got some data from the DB when I made the temprecord variable an %I but I am Selecting 3 columns and it looked like sommething got jacked up with the second identifier because I got a syntax error and it looked like it was trying to concatenate all 3 columns of the query results...
I did try hardcoding it and that worked fine... any help appreciated!
String literal is unknown type value. Postgres always does cast to some target binary format. The type is deduced from context. When you use function format, and %L placeholder, then any binary value is converted to string, and escaped to Postgres's string literal (protection against syntax errors, and SQL injection). When you use USING clause, then the binary value is passed directly to executor. It is little bit faster, and there is not possibility to lost some information under cast to string. Without these points, the real effect of %L and USING clause is almost same.
Your type of variable is timestamp. Probably type of expired_date column is date type. So some conversion timestamp->date is necessary.
Function format is just string function. It just make string. For better readability it supports placeholders, that ensure correct escaping and correct result SQL string. %L is same like calling function quote_literal and %I is same like quote_ident (for column, table names). %s inserts string without escaping and quoting. The result of format function (when you use it in EXECUTE command) should be valid SQL statement. You can use it in RAISE NOTICE command, and you can print result to debug output. Usually it is good idea
DECLARE
query text;
x date DEFAULT current_date
y int;
BEGIN
query := format('.... WHERE inserted = $1', ...);
RAISE NOTICE 'dynamic query will be: %', query);
EXECUTE query USING x INTO y;
...
Clause USING allows using parameters in dynamic SQL (EXECUTE clause). Usually, the format's placeholdres should be used for table or column names, and USING for any other.
For types date and timestamp (scalars basic types) the following execution will be on 99.99% same:
EXECUTE format('select count(*) from foo where inserted = %L', current_date) INTO ..
EXECUTE 'select count(*) from foo where inserted = $1' USING current_date INTO ..
You cannot to use query parameters on column name or table name positions. This is limit of USING clause. But for any other cases, this clause should be used primary.
I have a requirement where i need to copy the records from Table A to CSV file with copy command via procedures.
Directly using this command returns success:
COPY (SELECT * FROM <Table_Name>)
TO '<CSV file location>'
DELIMITER ',' CSV HEADER;
But when I use below command it gives syntax error:
syntax error at or near "SELECT"
BEGIN
COPY (SELECT * FROM <Table_Name>) TO '<CSV File location>' DELIMITER ',' CSV HEADER;
END;
Also Below procedures didn't help .
create or replace procedure copy_table as
declare
sql varchar;
begin
sql = 'COPY (SELECT * FROM personsaddrs limit 10) TO ''/tmp/Shivangi_File2.csv'' DELIMITER ',' CSV HEADER';
perform sql;
end;
BEGIN
copy_table;
END;
This also returns an error
You are mixing up PL/pgSQL's BEGIN, which starts a block, and SQL's BEGIN, which is short for START TRANSACTION and starts a transaction.
You forgot the semicolon after BEGIN. But a transaction that contains only a single statement need not be explicit; simply omit it.
In your PL/pgSQL code you forgot to double the quotes in
DELIMITER ','
You created a PROCEDURE, you need to use it with a CALL; CALL copy_table;. If it where a FUNCTION then you would do: SELECT copy_table;.
I've got a PostGIS database of points in Postgres, and I would like to extract the points in several geographically distinct areas to CSV files, one file per area.
I have set up an area table with area polygons, and area titles and I would like to effectively loop through that table, using something like Postgis' st_intersects() to select the data to go in each CSV file, and get the filename for the CSV file from the title in the area table.
I'm comfortable with the details of doing the intersection code, and setting up the CSV output - what I don't know is how to do it for each area. Is it possible to do something like this with some sort of join? Or do I need to do it with a stored procedure, and use a loop construct in plpgsql?
You can loop over rows in your area table in plpgsql. But be careful to get quoting of identifiers and values right:
Assuming this setup:
CREATE TABLE area (
title text PRIMARY KEY
, area_polygon geometry
);
CREATE TABLE points(
point_id serial PRIMARY KEY
, the_geom geometry);
You can use this plpgsql block:
DO
$do$
DECLARE
_title text;
BEGIN
FOR _title IN
SELECT title FROM area
LOOP
EXECUTE format('COPY (SELECT p.*
FROM area a
JOIN points p ON ST_INTERSECTS(p.the_geom, a.area_polygon)
WHERE a.title = %L) TO %L (FORMAT csv)'
, _title
, '/path/to/' || _title || '.csv');
END LOOP;
END
$do$;
Use format with %L (for string literal) to get properly quoted strings to avoid syntax errors and possible SQL injection. You still need to use strings in area.title that work for file names.)
Also careful to quote the filename as a whole, not just the title part of it.
You must concatenate the whole command as string. The "utility command" COPY does not allow variable substitution. That's only possible with the core DML commands SELECT, INSERT, UPDATE, and DELETE. See:
Error when setting n_distinct using a plpgsql variable
So don't read out area.area_polygon in the loop. It would have to be cast to text to concatenate it into the query string, where the text representation would be cast back to geometry (or whatever your actual undisclosed data type is). That's prone to errors.
Instead I only read area.title to uniquely identify the row and handle the rest in the query internally.
You can use a plpgsql function or an inline do (if you only need to do it once and you don't want to store a function.)
do $body$
DECLARE i int;
BEGIN FOR i IN SELECT DISTINCT city FROM table
LOOP RAISE
NOTICE 'foo';
EXECUTE format($$COPY (SELECT * FROM foo WHERE x='%s') TO /tmp/%s$$, i, i);
END LOOP;
RETURN;
END;
$body$ LANGUAGE plpgsql;
According to docs of PostgreSQL it is possible to copy data to csv file right from a query without using an intermediate table. I am curious how to do that.
CREATE OR REPLACE FUNCTION m_tbl(my_var integer)
RETURNS void AS
$BODY$
DECLARE
BEGIN
COPY (
select my_var
)
TO 'c:/temp/out.csv';
END;
$$ LANGUAGE plpgsql;
I get an error: no such column 'my_var'.
Yes, it is possible to COPY from any query, whether or not it refers to a table.
However, COPY is a non-plannable statement, a utility statement. It doesn't support query parameters - and query parameters are how PL/PgSQL implements the insertion of variables into statements.
So you can't use PL/PgSQL variables with COPY.
You must instead use dynamic SQL with EXECUTE. See the Pl/PgSQL documentation for examples. There are lots of examples here on Stack Overflow and on https://dba.stackexchange.com/ too.
Something like:
EXECUTE format('
COPY (
select %L
)
TO ''c:/temp/out.csv'';
', my_var);
The same applies if you want the file path to be dynamic - you'd use:
EXECUTE format('
COPY (
select %L
)
TO %L;
', my_var, 'file_name.csv');
It also works for dynamic column names but you would use %I (for identifier, like "my_name") instead of %L for literal like 'my_value'. For details on %I and %L, see the documentation for format.
In other statistical programs, it's possible to create a log file that shows the output issued as a result of a command. Is it possible to do something similar in SQL?
In particular, I'd like to have a single .sql file with many queries and to then output each result to a text file.
I'm using PostgreSQL and Navicat.
plpgsql function and COPY
One way would be to put the SQL script into a plpgsql function, where you can write the individual return values to files with COPY and compile a report from intermediary results just like you need it.
This has additional effect that may or may not be desirable. Like, you can grant or revoke permission to the whole function to arbitrary roles. Read about SECURITY DEFINER in the manual. And the syntax will be verified when you save the function - however, only superficially (there are plans to change that in the future). More details in this answer on dba.SE.
Basic example:
CREATE OR REPLACE FUNCTION func()
RETURNS void AS
$BODY$
BEGIN
COPY (SELECT * FROM tbl WHERE foo) TO '/path/to/my/file/tbl.csv';
COPY (SELECT * FROM tbl2 WHERE NOT bar) TO '/path/to/my/file/tbl2.csv';
END;
$BODY$
LANGUAGE plpgsql;
Of course, you need to have the necessary privileges in the database and in the file system.
Call it from the shell:
psql mydb -c 'SELECT func();'
psql switching between meta commands and SQL
#!/bin/sh
BASEDIR='/var/lib/postgresql/outfiles/'
echo "
\\o $OUTDIR/file1.txt \\\\\\\\ SELECT * FROM tbl1;
\\o $OUTDIR/file2.txt \\\\\\\\ SELECT * FROM tbl2;
\\o $OUTDIR/file3.txt \\\\\\\\ SELECT * FROM tbl3;" | psql event -p 5432 -t -A
That's right, 8 backslashes. Results from a double backslash that gets interpreted two times, so you have to double them two times.
I quote the manual about the meta-commands \o:
Saves future query results to the file filename or ...
and \\:
command must be either a command string that is completely parsable by
the server (i.e., it contains no psql-specific features), or a single
backslash command. Thus you cannot mix SQL and psql meta-commands with
this option. To achieve that, you could pipe the string into psql,
like this: echo '\x \\ SELECT * FROM foo;' | psql. (\\ is the
separator meta-command.)
Don't know about navicat, but you can do it with psql. Check the various --echo-X command-line options and the \o command if you just want temporary output to a file.