Export to CSV and Compress with GZIP in postgres - postgresql

I need to export a big table to csv file and compress it.
I can export it using COPY command from postgres like -
COPY foo_table to '/tmp/foo_table.csv' delimiters',' CSV HEADER;
And then can compress it using gzip like -
gzip -c foo_table.csv > foo.gz
The problem with this approach is, I need to create this intermediate csv file, which itself is huge, before I get my final compressed file.
Is there a way of export table in csv and compressing the file in one step?
Regards,
Sujit


The trick is to make COPY send its output to stdout, then pipe the output through gzip:
psql -c "COPY foo_table TO stdout DELIMITER ',' CSV HEADER" \
| gzip > foo_table.csv.gz


You can use directly, as per docs, https://www.postgresql.org/docs/9.4/sql-copy.html
COPY foo_table to PROGRAM 'gzip > /tmp/foo_table.csv' delimiter ',' CSV HEADER;


Expanding a bit on #Joey's answer, below adds support for a couple more features available in the manual.
psql -c "COPY \"Foo_table\" (column1, column2) TO stdout DELIMITER ',' CSV HEADER" \
| gzip > foo_table.csv.gz
If you have capital letters in your table name (woe be onto you), you need the \" before and after the table name.
The second thing I've added is column listing.
Also note from the docs:
This operation is not as efficient as the SQL COPY command because all data must pass through the client/server connection. For large amounts of data the SQL command might be preferable.


PostgreSQL 13.4
psql command \copy also works combined with SELECT column_1, column_2, ... and timestamp date +"%Y-%m-%d_%H%M%S" for filename dump.
\copy (SELECT id, column_1, column_2, ... FROM foo_table) \
TO PROGRAM 'gzip > ~/Downloads/foo_table_dump_`date +"%Y-%m-%d_%H%M%S"`.csv.gz' \
DELIMITER ',' CSV HEADER ;

Related

Skipping first column data in CSV while using copy command in PostgreSQL

I have a pipe delimited data file with no headers. I need to import data into a PostgreSQL table starting with the data from second column in the file i.e skip the data before the first '|' for each line. How do I achieve this using the COPY command?

Use the cut command to remove the first column and then import.
cut -d "|" -f 2- file1.csv > file2.csv
psql -d test -h localhost -c "\copy table(f1,f2,f3) from 'file2.csv' delimiter '|' csv header;"
Not an answer as such related to postgresql but more about command line tools.

Loading zipped CSV(filename.csv.gz) file into PostgreSQL table

How to load larger amount of csv.gz file into Postgresql without unzipping to csv file ,since I tried pipeline command (mkfifo pipelinename) but it doesn't work for me. Is there any other solution to solve this issue ?
I have try to load it from local to postgresql using following command
command : zcat file.csv.gz | psql -U username -d database;
Result : out of memory
Need : I want to load a big size csv.gz (around 15+ GB) file from centos to postgresql database.

Note that this should also work from inside psql:
\copy TABLE_NAME FROM PROGRAM 'gzip -dc FILENAME.csv.gz' DELIMITER ',' CSV HEADER NULL ''

Just to share my simple examples using zcat instead of gzip. Simply less typing. I am using zcat to expand the gzipped file.
\copy tmptable from program 'zcat O1variant.tab.gz' with (format csv, delimiter E'\t', header TRUE)

Error in using postgres COPY in windows batch file

This is working perfectly. All the columns in table1 is successfully exported as a .csv file.
(
echo \copy table1 TO '%HOME%\net\CSV_OUTPUT.csv' DELIMITER ',' CSV HEADER;
) | "C:\Program Files\PostgreSQL\9.3\bin\psql.exe" -h %DB_HOST% -p 5432 -U %DB_USER% -d %DB_NAME%
But I wanted to export only selected columns to the .csv file but what I have tried so far does not work.
I have tried using SELECT but it returns an error that says "TO was unexpected at this time"
echo \copy (SELECT red_foo FROM table1) TO '%HOME%\net\CSV_OUTPUT.csv' DELIMITER ',' CSV HEADER;
I have also tried to remove the parentheses but it returns an error that says
"\copy: parse error at "red_foo""
What should be the correct way for this implementation? Your suggestions and ideas are highly appreciated.

Using the suggestion of #a_horse_with_no_name in the comments, I have successfully exported a .csv file with only selected columns.
A separate SQL script that contains the \copy statement is used on the psql command on my windows batch script and it works like a charm.

PostgreSQL - copy first 5 rows of all tables and output to a file

I need to grab the first 5 rows of every table in PostgreSQL and output them to my computer in .csv and (preferably) .sql. There are 275 total tables.
Is this possible to do via CLI in a single scripted command?
So far I'm able to copy a single table at a time, but it's taking forever.
\COPY (SELECT * from table-name limit 5) TO '/vagrant/testexport.csv' DELIMITER ',' CSV HEADER;

bash file:
tables=$(psql -d a -tXa -c "COPY(select concat(schemaname,'.',tablename) as tables from pg_tables) to '/tmp/tlist'")
for i in $(cat /tmp/tlist); do
psql -d a -tXa -c "\COPY (SELECT * from $i limit 5) TO '/tmp/$i.csv' DELIMITER ',' CSV HEADER;";
done

How to export table as CSV with headings on Postgresql?

I'm trying to export a PostgreSQL table with headings to a CSV file via command line, however I get it to export to CSV file, but without headings.
My code looks as follows:
COPY products_273 to '/tmp/products_199.csv' delimiters',';

COPY products_273 TO '/tmp/products_199.csv' WITH (FORMAT CSV, HEADER);
as described in the manual.

From psql command line:
\COPY my_table TO 'filename' CSV HEADER
no semi-colon at the end.

instead of just table name, you can also write a query for getting only selected column data.
COPY (select id,name from tablename) TO 'filepath/aa.csv' DELIMITER ',' CSV HEADER;
with admin privilege
\COPY (select id,name from tablename) TO 'filepath/aa.csv' DELIMITER ',' CSV HEADER;

When I don't have permission to write a file out from Postgres I find that I can run the query from the command line.
psql -U user -d db_name -c "Copy (Select * From foo_table LIMIT 10) To STDOUT With CSV HEADER DELIMITER ',';" > foo_data.csv

This works
psql dbname -F , --no-align -c "SELECT * FROM TABLE"

The simplest way (using psql) seems to be by using --csv flag:
psql --csv -c "SELECT * FROM products_273" > '/tmp/products_199.csv'

For version 9.5 I use, it would be like this:
COPY products_273 TO '/tmp/products_199.csv' WITH (FORMAT CSV, HEADER);

This solution worked for me using \copy.
psql -h <host> -U <user> -d <dbname> -c "\copy <table_name> FROM '<path to csvfile/file.csv>' with (format csv,header true, delimiter ',');"

Heres how I got it working power shell using pgsl connnect to a Heroku PG database:
I had to first change the client encoding to utf8 like this: \encoding UTF8
Then dumped the data to a CSV file this:
\copy (SELECT * FROM my_table) TO C://wamp64/www/spider/chebi2/dump.csv CSV DELIMITER '~'
I used ~ as the delimiter because I don't like CSV files, I usually use TSV files, but it won't let me add '\t' as the delimiter, so I used ~ because its a rarely used characeter.

The COPY command isn't what is restricted. What is restricted is directing the output from the TO to anywhere except to STDOUT. However, there is no restriction on specifying the output file via the \o command.
\o '/tmp/products_199.csv';
COPY products_273 TO STDOUT WITH (FORMAT CSV, HEADER);

copy (anysql query datawanttoexport) to 'fileablsoutepathwihname' delimiter ',' csv header;
Using this u can export data also.

I am posting this answer because none of the other answers given here actually worked for me. I could not use COPY from within Postgres, because I did not have the correct permissions. So I chose "Export grid rows" and saved the output as UTF-8.
The psql version given by #Brian also did not work for me, for a different reason. The reason it did not work is that apparently the Windows command prompt (I was using Windows) was meddling around with the encoding on its own. I kept getting this error:
ERROR: character with byte sequence 0x81 in encoding "WIN1252" has no equivalent in encoding "UTF8"
The solution I ended up using was to write a short JDBC script (Java) which read the CSV file and issued insert statements directly into my Postgres table. This worked, but the command prompt also would have worked had it not been altering the encoding.

Try this:
"COPY products_273 FROM '\tmp\products_199.csv' DELIMITER ',' CSV HEADER"

In pgAdmin, highlight your query statement just like when you use F5 to execute and press F9 - this will open the file browser so you can pick where you save your CSV.
If you are using Azure Data Studio, the instruction are here: Azure Data Studio: Save As CSV.
I know this isn't a universal solution, but most of the time you just want to grab the file by hand.