I want to copy a .csv file into a postgresql table, where the file name is a variable. It fails with a "no such file or directory" error if \COPY and a user other than postgres is used. However, the copy succeeds if COPY and the postgres user is used.
The failing script:
martin#opensuse1:~> ./test1.sh
Null display is "¤".
'/home/martin/20180423.csv'
psql:load.sql:2: :load_csv: No such file or directory
martin#opensuse1:~> cat test1.sh
load_csv=/home/martin/20180423.csv
psql -d test1 -e -f load.sql --variable=load_csv="'$load_csv'"
martin#opensuse1:~> cat load.sql
\echo :load_csv
\copy test_table (col1, col2, col3) FROM :load_csv delimiter ';' encoding 'LATIN1' NULL '';
martin#opensuse1:~>
The working script:
martin#opensuse1:~> ./test1.sh
Null display is "¤".
'/home/martin/20180423.csv'
copy test_table (col1, col2, col3) FROM '/home/martin/20180423.csv' delimiter ';' encoding 'LATIN1' NULL '';
COPY 3
martin#opensuse1:~> cat test1.sh
load_csv=/home/martin/20180423.csv
psql -w postgres -d test1 -e -f load.sql --variable=load_csv="'$load_csv'"
martin#opensuse1:~> cat load.sql
\echo :load_csv
copy test_table (col1, col2, col3) FROM :load_csv delimiter ';' encoding 'LATIN1' NULL '';
martin#opensuse1:~>
What can I do to make this script run without having to use the postgres user?
Martin
It seems that the psql variables are not substituted in the \copy command.
A solution is to write the \copy command to a file and execute that file.
The part from my script (load the table par from the tsv-file with the
name stored in :input_file) is:
-- Tuples only:
\t on
-- Output file:
\o load_cmd.sql
select concat('\copy par from ''', :'input_file', '.tsv'';');
-- Standard output again:
\o
-- Normal decoration of tables:
\t off
-- Now execute the file with the \copy command:
\i load_cmd.sql
Related
I have a batch script that I want to be able to run on both my laptop and desktop. Depending on which machine I'm running it on, I need to pass 1 of 2 file paths to a postgresql script.
if "%computername%"=="LAPTOP" (
set init_path=C:\path1
) else (
set init_path=C:\path2
)
psql -U postgres -d dbname -v full_path=%init_path%\csvfile.csv -qf sql/sqlfile.sql
My postgresql script looks like this:
begin;
drop table if exists schm.table_name;
create table schm.table_name (
var1 date,
var2 float,
var3 text,
var4 float
);
truncate table schm.table_name;
\set quoted_path '\'' :full_path '\''
\copy schm.table_name from :quoted_path with delimiter as ',' csv quote as '"';
commit;
When I run it, I get the error:
:quoted_path: No such file or directory
Is this the correct way to quote a variable? I was basing my syntax on this thread.
I have a SQL file my_query.sql:
select * from my_table
Using psql, I can read in this sql file:
\i my_query.sql
Or pass it in as an arg:
psql -f my_query.sql
And I can output the results of a query string to a csv:
\copy (select * from my_table) to 'output.csv' with csv header
Is there a way to combine these so I can output the results of a query from a SQL file to a CSV?
Unfortunately there's no baked-in functionality for this, so you need a little bash-fu to get this to work properly.
CONN="psql -U my_user -d my_db"
QUERY="$(sed 's/;//g;/^--/ d;s/--.*//g;' my_query.sql | tr '\n' ' ')"
echo "\\copy ($QUERY) to 'out.csv' with CSV HEADER" | $CONN
The sed fun removes all semicolons, comment lines, and end of line comments, and tr converts newlines to spaces (as mentioned in a comment by #abelisto):
-- my_query.sql
select *
from my_table
where timestamp < current_date -- only want today's records
limit 10;
becomes:
select * from my_table where timestamp < current_date limit 10
which then gets passed in to the valid psql command:
\copy (select * from my_table where timestamp < current_date) to 'out.csv' with csv header
Here's a script:
sql_to_csv.sh
#!/bin/bash
# sql_to_csv.sh
CONN="psql -U my_user -d my_db"
QUERY="$(sed 's/;//g;/^--/ d;s/--.*//g;' $1 | tr '\n' ' ')"
echo "$QUERY"
echo "\\copy ($QUERY) to '$2' with csv header" | $CONN > /dev/null
./sql_to_csv.sh my_query.sql out.csv
I think the simplest way is to take advantage of the shell's variable expansion capabilities:
psql -U my_user -d my_db -c "COPY ($(cat my_query.sql)) TO STDOUT WITH CSV HEADER" > my_query_results.csv
You could do it using a bash script.
dump_query_to_csv.sh:
#!/bin/bash
# Takes an sql query file as an argument and dumps its results
# to a CSV file using psql \copy command.
#
# Usage:
#
# dump_query_to_csv.sh <sql_query_file> [<csv_output_filesname>]
SQL_FILE=$1
[ -z $SQL_FILE ] && echo "Must supply query file" && exit
shift
OUT_FILE=$1
[ -z $OUT_FILE ] && OUT_FILE="output.csv" # default to "output.csv" if no argument is passed
TMP_TABLE=ttt_temp_table_xx # some table name that will not collide with existing tables
## Build a psql script to do the work
PSQL_SCRIPT=temp.psql
# create a temporary database table using the SQL from the query file
echo "DROP TABLE IF EXISTS $TMP_TABLE;CREATE TABLE $TMP_TABLE AS" > $PSQL_SCRIPT
cat $SQL_FILE >> $PSQL_SCRIPT
echo ";" >> $PSQL_SCRIPT
# copy the temporary table to the output CSV file
echo "\copy (select * from $TMP_TABLE) to '$OUT_FILE' with csv header" >> $PSQL_SCRIPT
# drop the temporary table
echo "DROP TABLE IF EXISTS $TMP_TABLE;" >> temp.sql
## Run psql script using psql
psql my_database < $PSQL_SCRIPT # replace my_database and add user login credentials as necessary
## Remove the psql script
rm $PSQL_SCRIPT
You'll need to edit the psql line in the script to connect to your database. The script could also be enhanced to take the database and account credentials as arguments.
The accepted solution is correct, but I had Windows and had to make it run via a batch (command) file. Posting it here if someone needs that
#echo off
echo 'Reading file %1'
set CONN="C:\Program Files\PostgreSQL\11\bin\psql.exe" -U dbusername -d mydbname
"C:\Program Files\Git\usr\bin\sed.exe" 's/;//g;/^--/ d;s/--.*//g;' %1 | "C:\Program Files\Git\usr\bin\tr.exe" '\n' ' ' > c:\temp\query.txt
set /p QUERY=<c:\temp\query.txt
echo %QUERY%
echo \copy (%QUERY%) to '%2' WITH (FORMAT CSV, HEADER) | %CONN%
I am trying to copy (not unload) a table from redshift to a local file.
I run in psql:
\copy my_schema.my_table to 'my_file.csv' with csv;
I get the error
ERROR: COPY TO file from Xen-tables not supported
Running
\copy (select * from my_schema.my_table) to 'my_file.csv' with csv;
raises syntax error:
ERROR: syntax error at or near "("
How should I perform the copy?
Thanks,
Dafna
You can redirect the psql output to a local file:
psql [your connection options go here] -F, -A \
-c 'select * from my_schema.my_table' >my_file.csv
-F, sets the field separator to a comma
-A gives you unaligned/unformatted output
To specify a different delimiter like pipe, use '|' instead of the -F.
Note: The above command won't tolerate newlines in text fields, they are not encoded and terminate the line prematurely.
Is it possible to export a table to csv, but to append multiple selections to the same file?
I would like to export (for instance):
SELECT * FROM TABLE WHERE a > 5
Then, later:
SELECT * FROM TABLE WHERE b > 2
This must go to the same file.
Thanks in advance!
The only way that I know of to do this is from the command-line, redirecting output.
psql -d dbname -t -A -F"," -c "SELECT * FROM TABLE WHERE a > 5" >> output.csv
then later
psql -d dbname -t -A -F"," -c "SELECT * FROM TABLE WHERE b > 2" >> output.csv
You can look up the command line options here.
http://www.postgresql.org/docs/9.0/static/app-psql.html
Use \o <filename> to output to a file. All your SELECT statements after using \o will be appended to <file> until you set \o to something else.
Using \o in combination with \copy to STDOUT seems to work. For example:
db=> \o /tmp/test.csv
db=> \copy (select 'foo','bar') to STDOUT with CSV;
db=> \copy (select 'foo','bar') to STDOUT with CSV;
db=> \q
$ cat /tmp/test.csv
foo,bar
foo,bar
I need to export a big table to csv file and compress it.
I can export it using COPY command from postgres like -
COPY foo_table to '/tmp/foo_table.csv' delimiters',' CSV HEADER;
And then can compress it using gzip like -
gzip -c foo_table.csv > foo.gz
The problem with this approach is, I need to create this intermediate csv file, which itself is huge, before I get my final compressed file.
Is there a way of export table in csv and compressing the file in one step?
Regards,
Sujit
The trick is to make COPY send its output to stdout, then pipe the output through gzip:
psql -c "COPY foo_table TO stdout DELIMITER ',' CSV HEADER" \
| gzip > foo_table.csv.gz
You can use directly, as per docs, https://www.postgresql.org/docs/9.4/sql-copy.html
COPY foo_table to PROGRAM 'gzip > /tmp/foo_table.csv' delimiter ',' CSV HEADER;
Expanding a bit on #Joey's answer, below adds support for a couple more features available in the manual.
psql -c "COPY \"Foo_table\" (column1, column2) TO stdout DELIMITER ',' CSV HEADER" \
| gzip > foo_table.csv.gz
If you have capital letters in your table name (woe be onto you), you need the \" before and after the table name.
The second thing I've added is column listing.
Also note from the docs:
This operation is not as efficient as the SQL COPY command because all data must pass through the client/server connection. For large amounts of data the SQL command might be preferable.
PostgreSQL 13.4
psql command \copy also works combined with SELECT column_1, column_2, ... and timestamp date +"%Y-%m-%d_%H%M%S" for filename dump.
\copy (SELECT id, column_1, column_2, ... FROM foo_table) \
TO PROGRAM 'gzip > ~/Downloads/foo_table_dump_`date +"%Y-%m-%d_%H%M%S"`.csv.gz' \
DELIMITER ',' CSV HEADER ;