I was connecting to another team's Postgres DB and was trying to export data for some debugging purposes.
I noticed when I ran the below command, I got 50K rows
psql -t "sslmode=require host=MY_HOST port=MY_PORT dbname=MY_DB user=MY_USER password=MY_PASSWORD" -c "SELECT * FROM MYTABLE;" -o EXPORT.txt
$ cat EXPORT.txt | wc -l
50000
But when I log in and check the row count, it was 94K
psql -t "sslmode=require host=MY_HOST port=MY_PORT dbname=MY_DB user=MY_USER password=MY_PASSWORD"
MY_DB=> SELECT count(*) FROM MYTABLE;
94508
What is the row count for exporting the table using psql shell?
Is this a hard limit?
I wrote a bash script to loop using limit and offset as a work around.
I need to execute a query from a file and write the output of it to a CSV file. I want to keep the delimiter as ;. I have tried below queries.
psql -h localhost -p 5432 -U postgres -d postgres -f \path\to\sqlQuery.sql -o \path\to\result\result.csv
This query puts the result into a file but it is in a postgres result tabular format.
psql -h localhost -p 5432 -U postgres -d postgres -f copy(\path\to\sqlQuery.sql) to \path\to\result.csv csv header;
Above query is giving me a syntax error.
I'm looking for a way to use COPY command and \f or -f together so I'll be able to exceute the query from a file and also write the output to another CSV file with specified delimiter.
Change your SQL script to use COPY:
COPY (/* your query */) TO STDOUT
(FORMAT 'csv', DELIMITER ';');
I am trying to store a Greenplum query result into a csv but it is storing only one row. I want to store multiple rows. Can anyone help with this.
I tried the below already but it is storing only one row.
r= `psql -A -t -q -h $hostname -U $username -d $dbname <<-THE_END
COPY(select * From ${gschema}.${gtable} order by ${key} limit 3) TO STDOUT with NULL as ' '
THE_END` > /home/gp.csv
I also tried below which is storing the result but I am unable to pass table name as parameter in the below query. For standard table names the output is as desired.
psql -h $hostname -U $username -d $dbname -c "COPY (select * from table_name order by key limit 3) TO STDOUT "> /home/gp.csv
Can anyone help me with this please.
please note that I am trying to embed the above in a shell script
Try this:
psql -A -t -q -h $hostname -U $username -d $dbname -c "select * from $gschema.$gtable order by $key limit 3" > /home/gp.csv
You could also put the SQL in a file:
example.sql
select * from :gschema.:gtable order by :key limit 3
And then to use the sql file:
psql -A -t -q -h $hostname -U $username -d $dbname -f example.sql -v gschema=public -v gtable=customer -v key=id > /home/gp.csv
I am having an issue trying to run \copy command where my column name is desc
I've tried the following:
psql -U user -p 1234 -h hostname -c "\copy schema.table (desc) from /my/file.txt WITH DELIMITER '|' HEADER CSV " db
psql -U user -p 1234 -h hostname -c "\copy schema.table ('desc') from /my/file.txt WITH DELIMITER '|' HEADER CSV " db
psql -U user -p 1234 -h hostname -c "\copy schema.table ("desc") from /my/file.txt WITH DELIMITER '|' HEADER CSV db
I get the same error everytime:
ERROR: syntax error at or near "desc"
I can't seem to get around it. Is there something that I am missing?
just escape double quotes, like here:
vao#vao-X102BA:~$ psql -c "copy s1(\"desc\") from stdin;"
Enter data to be copied followed by a newline.
End with a backslash and a period on a line by itself.
>> b
>> \.
COPY 1
vao#vao-X102BA:~$ psql
psql (9.6.3)
Type "help" for help.
vao=# select * from s1;
i | desc
---+------
| a
| b
(2 rows)
I'd like to execute a query from the shell (not in the interactive psql client) and have it print the CSV or TSV representation of the output to STDOUT. How do you do that with psql or one of the PostgreSQL command-line tools?
If you are using PostgreSQL 8.2 or newer, use this for CSV:
psql -c "COPY (<select query>) TO STDOUT WITH CSV"
and this of TSV, with proper NULLs:
psql -c "COPY (<select query>) TO STDOUT WITH NULL AS ''"
The CSV form will properly quote any fields that contain the double-quote character. See the PostgreSQL documentation of your specific version for more details and options for COPY.
Starting from Bohemian's answer, I found these flags useful:
psql my_database -U myuser -A -F , -X -t -f /path/to/query.sql -o /path/to/output.csv
Unaligned output mode: -A
Use comma as field delimiter: -F ,
Do not read psqlrc: -X
Tuples only (no header/footer): -t
File containing SQL query: -f
Output file: -o
EDITED: Using -F
Use commas via -F and use "unaligned table output mode" -A:
psql my_database -U myuser -A -F , -c "select * from mytable"
To specify a tsv use the delimiter '\t'
psql my_database -U myuser -F'\t' --no-align -f mysqlfile.sql -o outputfile.tsv
To specify a csv use the delimiter ','
psql my_database -U myuser -F',' --no-align -f mysqlfile.sql -o outputfile.csv
The simplest way (using psql) seems to be by using --csv flag:
psql --csv -c "SELECT * FROM sometable"
Also possible is the copy command which allows you to specify header, delimiters and quoting options
psql my_database -U my_user -c "copy (select a.id,b.id from my_table_a as a inner join my_table_b as b on b.id = a.id) to STDOUT"
You can specify the field separator with the -F command line parameter to psql
Export AS TSV WITH HEADER
You can include the HEADER as follows:
\COPY (SELECT * FROM tca) TO '/.../metab/tca.tsv' WITH DELIMITER E'\t' CSV HEADER;
\COPY (SELECT * FROM tca) TO '/...a/metab/tca.tsv' WITH NULL AS '' DELIMITER E'\t' CSV HEADER;
E.g. (PSQL):
[metabolism]# \COPY (SELECT * FROM tca) TO '/mnt/Vancouver/programming/data/metabolism/tca.tsv' WITH NULL AS '' DELIMITER E'\t' CSV HEADER;
COPY 22
BASH:
[victoria#victoria tsv]$ pwd
/mnt/Vancouver/programming/data/metabolism/tsv
[victoria#victoria tsv]$ head -n3 tca.tsv
uuid src tgt rel rel_type
878b87de-0ca8-49a8-9f77-a24353e251d2 oxalosuccinic acid oxoglutaric acid 1.1.1.42 2
7fd9cf88-495b-491b-956e-294f19097923 isocitric acid oxoglutaric acid 1.1.1.41 2
[victoria#victoria csv]$