is possible in PSQL console export file with current date on the end of the file name?
The name of the exported file should be like this table_20140710.csv is it possible to do this dynamically? - the format of the date can be different than the above it isn't so much important.
This is example what i mean:
\set curdate current_date
\copy (SELECT * FROM table) To 'C:/users/user/desktop/table_ ' || :curdate || '.csv' WITH DELIMITER AS ';' CSV HEADER
The exception of the \copy meta command not expanding variables is (meanwhile) documented
Unlike most other meta-commands, the entire remainder of the line is always taken to be the arguments of \copy, and neither variable interpolation nor backquote expansion are performed in the arguments.
To workaround you can build, store and execute the command in multiple steps (similar to the solution Clodoaldo Neto has given):
\set filename 'my fancy dynamic name'
\set command '\\copy (SELECT * FROM generate_series(1, 5)) to ' :'filename'
:command
With this, you need to double (escape) the \ in the embedded meta command. Keep in mind that \set concatenates all further arguments to the second one, so quote spaces between the arguments. You can show the command before execution (:command) with \echo :command.
As an alternative to the local \set command, you could also build the command server side with SQL (the best way depends on where the dynamic content is originating):
SELECT '\copy (SELECT * FROM generate_series(1, 5)) to ''' || :'filename' || '''' AS command \gset
Dynamically build the \copy command and store it in a file. Then execute it with \i
First set tuples only output
\t
Set the output to a file
\o 'C:/users/user/desktop/copy_command.txt'
Build the \copy command
select format(
$$\copy (select * from the_table) To 'C:/users/user/desktop/table_%s.csv' WITH DELIMITER AS ';' CSV HEADER$$
, current_date
);
Restore the output to stdout
\o
Execute the generated command from the file
\i 'C:/users/user/desktop/copy_command.txt'
Related
I'm attempting to dynamically create a script that gets saved as a bat file that will be scheduled to execute daily via Windows Task Scheduler. The script performs full database backups for each Postgres database using pg_dump.
The current script is as follows:
COPY (SELECT 'pg_dump '|| datname || ' > e:\postgresbackups\FULL\' || datname || '_%date:~4,2%-%date:~7,2%-%date:~10,4%_%time:~0,2%_%time:~3,2%_%time:~6,2%.dump' FROM pg_database) TO 'E:\PostgresBackups\Script\FULL_Postgres_Backup_Job_TEST.bat' (format csv, delimiter ';');
An example of the output is as follows:
pg_dump postgres > e:\postgresbackups\FULL\postgres_%date:~4,2%-%date:~7,2%-%date:~10,4%%time:~0,2%%time:~3,2%_%time:~6,2%.dump
I need help with updating my code so that the output will include double quotes around the name of the dump file; however, when I add this to my COPY script it adds more than what is necessary to the output. I would like the output to look like the following which includes the double-quotes:
pg_dump postgres > "e:\postgresbackups\FULL\postgres_%date:~4,2%-%date:~7,2%-%date:~10,4%%time:~0,2%%time:~3,2%_%time:~6,2%.dump"
Any help would be greatly appreciated!
Thanks to #Mike Organek's comment, my issue has been resolved by switching the format from CSV to TEXT. Now when I enclose the dump filename in double quotes, the output is more of what is expected and works as intended. The only odd thing now is that in the output it creates a second backslash in the filename. My code has been updated as follows:
COPY (SELECT 'pg_dump '|| datname || ' > "e:\postgresbackups\FULL\' || datname || '_%date:~4,2%-%date:~7,2%-%date:~10,4%_%time:~0,2%_%time:~3,2%_%time:~6,2%.dump"' FROM pg_database) TO 'E:\PostgresBackups\Script\FULL_Postgres_Backup_Job.bat' (format text, delimiter ';');
An example of the output that gets created within the bat file is as follows:
pg_dump postgres > "e:\\postgresbackups\\FULL\\postgres_%date:~4,2%-%date:~7,2%-%date:~10,4%_%time:~0,2%_%time:~3,2%_%time:~6,2%.dump"
As you can see, it adds a double backslash; however, the pg_dump executes successfully!
I am using the \copy command for migrating my data . But the table size is 30GB and it is taking hours to migrate. Can I use a where clause so that I can migrate only data that was available a month back?
\copy hotel_room_types TO | (select hotel_room_types.* from hotel_room_types limit 1) $liocation CSV DELIMITER ',';
ERROR: syntax error at or near "."
LINE 1: ...otel_room_types TO STDOUT (select hotel_room_types.* from h...
You can specify a query with psql's \copy just like you can with the SQL command COPY:
\copy (SELECT ... WHERE ...) TO 'filename'
After all, \copy just calls COPY ... TO STDOUT under the hood.
I am trying to write a procedure that loads a csv into a table. I create a dynamic sql and execute it.
v_sql = '\copy test_table from' || ' ''' || p_temp_dir || ''' ' || ' with csv header;';
--insert into test
--select v_sql;
EXECUTE v_sql;
When I check the generated string it is fine and I can run it properly, but if I run it from a procedure, I get an error:
syntax error at or near "\"
If I remove the \ character it works (throws an error that there is no such file; this job is intended to work on the local machine). How can I execute this SQL?
\copy is a psql command, not an SQL statement.
You can only execute SQL statements with PL/pgSQL's EXECUTE statement.
You cannot import a file from the client machine in a PostgreSQL function that runs on the server. You will have to write client code that does COPY ... FROM STDIN.
Basically, what I want to achieve is something like this:
mypath:= '~/Desktop/' || my_variable || '.csv'
COPY (SELECT * FROM my_table) TO mypath CSV DELIMITER ',' HEADER;
Where the value of my_variable will change dynamically.
Can someone help me on this problem?
Here and here you can find two different options (either via a bash script or a sql script with variables) to solve your problem.
Since you are using windows the only viable solution is the one with the variable in the sql file. With Windows you should also be able to use the psql command line utility to execute your sqlfile and pass the path as a parameter like this:
-- Example sql file content:
COPY (SELECT * FROM my_table) TO :path CSV DELIMITER ',' HEADER;
Command line example
psql -f /tmp/file.sql -v path='~/Desktop/' || my_variable || '.csv'
Another option i just found would be to output the content of the csv to stdout and forwarding that to file on the windows cmd level.
Example code (based on this answer Linux Example code):
psql -c "COPY (SELECT * FROM my_table) TO STDOUT CSV DELIMITER ',' HEADER;" > '~/Desktop/' || my_variable || '.csv'
EDIT1 based on the new requirement that the variable comes out of the postgresql database:
I quickly built a For loop which can loop over the result of a separate query and then execute a sql query for each of the values in the result:
Example code:
DO $$
declare
result record;
BEGIN
FOR result IN Select * FROM (VALUES ('one'), ('two'), ('three')) AS t (path) LOOP
RAISE NOTICE 'path: ~/Desktop/%.csv', test.path;
END LOOP;
END; $$;
This produces the following output:
NOTICE: path: ~/Desktop/one.csv
NOTICE: path: ~/Desktop/two.csv
NOTICE: path: ~/Desktop/three.csv
You can substitue the Select * FROM (VALUES ('one'), ('two'), ('three')) AS t (path) with any query which produces the table result which contains one path per row.
Second you can substitue the RAISE NOTICE 'path: ~/Desktop/%.csv', test.path; with your copy query
You can use EXECUTE command. A working example is:
sql_copy_command= '
COPY abc_schema_name.xyz_table_name FROM ''' || masterFilePath ||''' DELIMITER '';'' CSV HEADER ' ;
RAISE NOTICE 'sql_copy_command: % ', sql_copy_command;
EXECUTE sql_copy_command;
I'm trying to write a script that copies data from a crosstab query to a .csv file in Postgres 8.4. I am able to run the command in the psql command line but when I put the command in a file and run it using the -f option, I get a syntax error.
Here's an example of what I'm looking at (from this great answer):
CREATE TEMP TABLE t (
section text
,status text
,ct integer
);
INSERT INTO t VALUES
('A', 'Active', 1), ('A', 'Inactive', 2)
,('B', 'Active', 4), ('B', 'Inactive', 5)
, ('C', 'Inactive', 7);
\copy (
SELECT * FROM crosstab(
'SELECT section, status, ct
FROM t
ORDER BY 1,2'
,$$VALUES ('Active'::text), ('Inactive')$$)
AS ct ("Section" text, "Active" int, "Inactive" int)
) TO 'test.csv' HEADER CSV
I then run this and get the following syntax error:
$ psql [system specific] -f copy_test.sql
CREATE TABLE
INSERT 0 5
psql:copy_test.sql:12: \copy: parse error at end of line
psql:copy_test.sql:19: ERROR: syntax error at or near ")"
LINE 7: ) TO 'test.csv' HEADER CSV
^
A similar exercise doing just a simple query without crosstab works without incident.
What is causing the syntax error and how can I copy this table to a csv file using script file?
psql thinks your first command is just \copy ( and the lines below that are from another unrelated statement. Meta-commands aren't spread on multiple lines, because newline is is a terminator for them.
Relevant excerpts from psql manpage with some emphasis added:
Meta-Commands
Anything you enter in psql that begins with an unquoted backslash is a
psql meta-command that is processed by psql itself. These commands
make psql more useful for administration or scripting. Meta-commands
are often called slash or backslash commands.
....
....(skipped)
Parsing for arguments stops at the end of the line, or when another
unquoted backslash is found. An unquoted backslash is taken as the
beginning of a new meta-command. The special sequence \\ (two
backslashes) marks the end of arguments and continues parsing SQL
commands, if any. That way SQL and psql commands can be freely mixed
on a line. But in any case, the arguments of a meta-command cannot
continue beyond the end of the line.
So the first error is that \copy ( failing, then the lines below are interpreted as an independent SELECT which looks fine until line 7 when there is a spurious closing parenthesis.
As told in the comments, the fix would be to cram the whole meta-command into a single line.
As with this answer, create a multi-line VIEW with a single-line \copy command, e.g.:
CREATE TEMP TABLE t (
section text
,status text
,ct integer
);
INSERT INTO t VALUES
('A', 'Active', 1), ('A', 'Inactive', 2)
,('B', 'Active', 4), ('B', 'Inactive', 5)
, ('C', 'Inactive', 7);
CREATE TEMP VIEW v1 AS
SELECT * FROM crosstab(
'SELECT section, status, ct
FROM t
ORDER BY 1,2'
,$$VALUES ('Active'::text), ('Inactive')$$)
AS ct ("Section" text, "Active" int, "Inactive" int);
\copy (SELECT * FROM v1) TO 'test.csv' HEADER CSV
-- optional
DROP VIEW v1;
The answers listed here explain the reasoning quite clearly. Here is a small hack that allows you to have your sql contain multiple lines and work with psql.
# Using a file
psql -f <(tr -d '\n' < ~/s/test.sql )
# or
psql < <(tr -d '\n' < ~/s/test.sql )
# Putting the SQL using a HEREDOC
cat <<SQL | tr -d '\n' | \psql mydatabase
\COPY (
SELECT
provider_id,
provider_name,
...
) TO './out.tsv' WITH( DELIMITER E'\t', NULL '', )
SQL
According to the psql documentation:
-f filename
--file filename
Use the file filename as the source of commands instead of reading commands interactively. After the file is processed, psql terminates. This is in many ways equivalent to the internal command \i.
If filename is - (hyphen), then standard input is read.
Using this option is subtly different from writing psql < filename. In general, both will do what you expect, but using -f enables some nice features such as error messages with line numbers. There is also a slight chance that using this option will reduce the start-up overhead. On the other hand, the variant using the shell's input redirection is (in theory) guaranteed to yield exactly the same output that you would have gotten had you entered everything by hand.
This would be one of those cases where the -f option treats your input differently from the command line. Removing your newlines worked, redirecting the original file to psql's stdin would likely have worked as well.