I'm trying to insert some data into my table using the copy command :
copy otype_cstore from '/tmp/otype_fdw.csv' delimiter ';' quote '"' csv;
And I have this answer :
ERROR: unterminated CSV quoted field
There is the line in my CSV file where I have the problem :
533696;PoG;-251658240;from id GSW C";
This is the only line a double quote and I can't remove it, so do you have some advice for me ?
Thank you in advance
If you have lines like this in your csv:
533696;PoG;-251658240;from id GSW C";
this actually means/shows the fields are not quoted, which is still perfectly valid csv as long as there are no separators inside the fields.
In this case the parser should be told the fields are not quoted.
So, instead of using quote '"' (which is actually telling the parser the fields are quoted and why you get the error), you should use something like quote 'none', or leave the quote parameter out (I don't know Postgres, so I can't give you the exact option to do this).
Ok, I did a quick lookup of the parameters. It looks like there is not really an option to turn quoting off. The only option left would be to provide a quote character that is never used in the data.
quote E'\b' (backspace) seems to work ok.
Bit late to reply, but I was facing same issue and found that for double quote we need to add one more double quote to escape with along with prefix and suffix double quote to that special characater. for your case it would be
input data is : from id GSW C"
change data to : from id GSW C""""
note there are 4 consecutive double quotes.
first and last double quote is prefix and postfix as per documentation.
middle 2 double quotes is data with one escape double quote.
Hope this helps for readers with similar issue going forward.
So for every double quote in data it needs to be escaped with one escape character (default double quote). This is as per documentation.
Related
Question (because I can't work it out), should ""hello world"" be a valid field value in a CSV file according to the specification?
i.e should:
1,""hello world"",9.5
be a valid CSV record?
(If so, then the Perl CSV-XS parser I'm using is mildly broken, but if not, then $line =~ s/\342\200\234/""/g; is a really bad idea ;) )
The weird thing is is that this code has been running without issue for years, but we've only just hit a record that started with both a left double quote and contained no comma (the above is from a CSV pre-parser).
The canonical format definition of CSV is https://www.rfc-editor.org/rfc/rfc4180.txt. It says:
Each field may or may not be enclosed in double quotes (however
some programs, such as Microsoft Excel, do not use double quotes
at all). If fields are not enclosed with double quotes, then
double quotes may not appear inside the fields. For example:
"aaa","bbb","ccc" CRLF
zzz,yyy,xxx
Fields containing line breaks (CRLF), double quotes, and commas
should be enclosed in double-quotes. For example:
"aaa","b CRLF
bb","ccc" CRLF
zzz,yyy,xxx
If double-quotes are used to enclose fields, then a double-quote
appearing inside a field must be escaped by preceding it with
another double quote. For example:
"aaa","b""bb","ccc"
Last rule means your line should have been:
1,"""hello world""",9.5
But not all parsers/generators follow this standard perfectly, so you might need for interoperability reasons to relax some rules. It all depends on how much you control the CSV format writing and CSV format parsing parts.
That depends on the escape character you use. If your escape character is '"' (double quote) then your line should look like
1,"""hello world""",9.5
If your escape character is '\' (backslash) then your line should look like
1,"\"hello world\"",9.5
Check your parser/environment defaults or explicitly configure your parser with the escape character you need e.g. to use backslash do:
my $csv = Text::CSV_XS->new ({ quote_char => '"', escape_char => "\\" });
I'm using Powershell with ODBC to transfer table data between Sage 50 and MariaDb.
Some data rows in a text column in Sage can contain both single and double quotes that need to be retained when the data is imported into MariaDb.
I'm struggling to get PowerShell to replace for the Values portion of the Insert statement:
I need to replace a single quote ' with backslash single quote \', and also a double quote " with backslash double quote \"
For anyone else searching for this the answer is :
For single quotes
[regex]::replace($text, "'", "\'")
For double quotes
[regex]::replace($DETAILS, "`"", "\'")
I found this URL helpful https://vwiki.co.uk/MySQL_and_PowerShell and as stated you may wish to put this in a function.
This is a very simple problem, I am using the psql terminal command COPY as shown bellow
COPY tbname FROM '/tmp/file.csv'
delimiter '|' csv;
However this file.csv contains data such as
random|stuff|32"
as well as
random|other "stuff"|15
I tried to use the double quote to escape the quotes as the Postgres site suggested
random|stuff|32""
random|other ""stuff""|15
This seems to remove the quotes completely which I don't want.
Is there a way to get the import to just treat these quotes as regular characters so that they appear in the database as they do in the csv file?
According to the documentation, the default quote symbol is ", so you need to provide a QUOTE argument with a different symbol. The quote symbol has to be a single one-byte character.
COPY tbname FROM '/tmp/file.csv'
delimiter '|' QUOTE '}' csv; -- use a symbol you know does not appear in your file.
All,
I am trying to clean up data in a SQLite database, that contains tons of escaped single and double quotes.
Example: \"Text\" has \'junk\' like this all over the place.
When I do a SELECT to see if the REPLACE function has the correct pattern, the REPLACE is not working:
-- replace escaped single quotes
SELECT id,subject,body,replace(body,"\\'","'") as `clean-body`
FROM article
WHERE id=1118
Thanks for looking
There is no need to escape the backslash. Try this in place of your original call to replace:
replace(body,"\'","'")
I'm trying to import a tab-delimited file into my PostgreSQL database. One of the fields in my file is a "title" field, which occasionally contains actual quotation marks. For example, my tsv might look like:
id title
5 Hello/Bleah" Foo
(Yeah, there's just that one quotation mark in the title.)
When I try importing the file into my database:
copy articles from 'articles.tsv' with delimiter E'\t' csv header;
I get this error, referencing that line:
ERROR: unterminated CSV quoted field
How do I fix this? Quotation marks are never used to surround entire fields in the file. I tried copy articles from 'articles.tsv' with delimiter E'\t' escape E'\\' csv header; but I get the same error on the same line.
Assuming the file never actually tries to quote its fields:
The option you want is "with quote", see http://www.postgresql.org/docs/8.2/static/sql-copy.html
Unfortunately, I'm not sure how to turn off quote processing altogether, one kludge would be to specify a character that does not appear in your file at all.
Tab separated is the default format for copy statements. Treating them as CSV is just silly. (do you take this path just to skip the header ?)
copy articles from 'articles.tsv';
does exactly what you want.
I struggled with the same error and a few more. Finally gathering knowledge from few SO questions I came up with the following setup for making COPY TO/FROM successful even for quite sophisticated JSON columns:
COPY "your_schema_name.yor_table_name" (your, column_names, here)
FROM STDIN WITH CSV DELIMITER E'\t' QUOTE '\b' ESCAPE '\';
--here rows data
\.
the most important parts:
QUOTE '\b' - quote with backspace (thanks a lot #grautur!)
DELIMITER E'\t' - delimiter with tabs
ESCAPE '\' - and escape with a backslash