Postgres COPY command with literal delimiter - postgresql

I was trying to import a CSV file into a PostgreSQL table using the COPY command. The delimiter of the CSV file is comma (,). However, there's also a text field with a comma in the value. For example:
COPY schema.table from '/folder/foo.csv' delimiter ',' CSV header
Here's the content of the foo.csv file:
Name,Description,Age
John,Male\,Tall,30
How to distinguish between the literal comma and the delimiter?
Thanks for your help.


To have the \ to be recognized as a escape character it is necessary to use the text format
COPY schema.table from '/folder/foo.csv' delimiter ',' TEXT
But then it is also necessary to delete the first line as the HEADER option is only valid for the CSV format.

Related

PostgreSQL exclude extra comma from being used as a delimiter while parsing csv file

I am trying to copy data from a csv file into my database table. But the problem is that a column named title has some values that contain a comma within.
How can I exclude it from being used as a delimiter while parsing?
Sample data:
"Hold On, I'm Coming", The Canettes Blues Band, On Tap & In the Can, 34, 100, 282
I don't want the comma after "Hold on" to be used as a delimiter.

If you are using copy or psql's \copy then use the options
format csv quote '"'
this will make the import ignore commas inside quoted values (which is what your sample data uses)
e.g. in psql
\copy target_table from the_input_file.txt with (format csv quote '"')

Escape \n when exporting CSV using COPY command

I need to export various tables in CSV for AWS Glue Catalog and I just noticed a major showstopper:
COPY command does not escape new line inputs in columns, only quotes them.
What confuses me even more is that I can switch to TEXT and get the format right - escape the special characters - but I cannot have HEADER in that format!
COPY (%s) TO STDOUT DELIMITER ',' NULL ''
Is there a way to get both HEADER and to escape the new line through COPY command?
I'm hoping that it's my overlook as the code is obviously there.

The text format does not produce CSV, that is why you cannot get headers or change the delimiter. It is the “internal” tab-separated format of PostgreSQL.
There are no provisions to replace newlines with \n in a CSV file, and indeed that would produce invalid CSV (according to what most people think; there is no standard).
You'll have to post-process the file.

postgresql copy from with csv array literal whose delimiter is ','

I'd like to copy from with csv file with Postgres.
That csv file has array literal whose delimiter is ;.
example: a,b,c,{1;2;3}
I did that with replacing delimiter of csv file , to | and set option delimiter | and replacing delimiter of array literal ; to ,.
example: a|b|c|{1,2,3}
I think there may be option to set delimiter of array literal.
If so, I don't have to replace delimiter of csv file.
Are there any smarter way?

There is no option to configure the separator between the elements of an array in its text representation.
But if you have any control over how the CSV file is generated, you can escape the array literal:
a,b,c,"{1,2,3}"
That would work fine with COPY.

COPY command and newline error

I have my simple CSV file, delimited by commas and no strange character, and when trying to do a \COPY table FROM '/srv/www/.../CSV.csv' WITH DELIMITER ',' I get the error
ERROR: literal newline found in data
HINT: Use "\n" to represent newline.
Its not the first time I execute a COPY and it never happened to me. I exported the CSV from excel to Notepad and saved here as CSV (I know some programs add extra info, but AFAIK notepad does not)

If you want to import CSV, you must specify FORMAT CSV. A literal newline is not legal in PostgreSQL's default delimited input.
\COPY table FROM '/srv/www/.../CSV.csv' WITH (FORMAT CSV, DELIMITER ',');

Trying to import a CSV file into postgres with comma as delimeter

I am trying to import a CSV file into Posgres that has a comma as delimiter. I do:
\COPY products(title, department) from 'toyd.csv' with (DELIMITER ',');
All super cool.
However, title and department are both strings. I have some commas that are in these columns that I don't want to be interpreted as delimiters. So I pass the strings in quotes. But this doesn't work. Postgres still thinks they are delimiters. What are my missing?
Here is a snippet from the CSV that causes the problem:
"Light","Reading, Writing & Spelling"
Any ideas?

You aren't using CSV format there, just a comma-delimited one.
Tell it you want FORMAT CSV and it should default to quoted text - you could also change the quoting character if necessary.