Related
I am trying to import a TSV file into Postgresql. I have created a table:
CREATE TABLE description (
id TEXT
, effective_time DATE
, active INT
, module_id TEXT
, concept_id TEXT
, language_code TEXT
, type_id TEXT
, term TEXT
, case_significance_id TEXT
);
I have a TSV file like so:
id effectiveTime active moduleId conceptId languageCode typeId term caseSignificanceId
12118017 20170731 1 900000000000207008 6708002 en 900000000000013009 Intrauterine cordocentesis 900000000000448009
12119013 20020131 1 900000000000207008 6709005 en 900000000000013009 Gentamicin 2''-nucleotidyltransferase 900000000000020002
12119013 20170731 1 900000000000207008 6709005 en 900000000000013009 Gentamicin 2''-nucleotidyltransferase 900000000000448009
12120019 20020131 1 900000000000207008 6710000 en 900000000000013009 Nitric oxide 900000000000020002
Note that the middle two entries have two consecutive single quotes acting as the symbol for double-prime (Gentamicin 2''-nucleotidyltransferase).
If I run
psql=# \copy description FROM /path/to/foo.txt WITH DELIMITER AS E'\t';
I get ERROR: missing data for column "effective_time". I think that's because the '' is screwing up the parsing of the column boundaries.
I have tried finding and replacing the '' instances with either \'\' or '''' and using CSV QUOTE E'\'' or CSV QUOTE '''', respectively, but I get the same error.
How do I edit the file or alter the \copy command to import the file correctly?
Haleemur Ali correctly points out that the original file—whose README purports it to comprise "UTF-8 encoded tab-delimited flat files which can be imported into any database"—is in fact not tab-separated, which may be my editor's fault. It works once I fix that.
I am wondering what the delimiter from this .csv file is. I am trying to import the .csv via the COPY FROM Statement, but somehow it throws always an error. When I change the delimiter to E'\t' it throws an error. When I change the delimiter to '|' it throws a different error. I have been trying to import a silly .csv file for 3 days and I cannot achieve a success. I really need your help. Here is my .csv file: Download here, please
My code on postgresql looks like this:
CREATE TABLE movie
(
imdib varchar NOT NULL,
name varchar NOT NULL,
year integer,
rating float ,
votes integer,
runtime varchar ,
directors varchar ,
actors varchar ,
genres varchar
);
MY COPY Statement:
COPY movie FROM '/home/max/Schreibtisch/imdb_top100t_2015-06-18.csv' (DELIMITER E'\t', FORMAT CSV, NULL '', ENCODING 'UTF8');
When I use SHOW SERVER_ENCODING it says "UTF8". But why the hell can't postgre read the datas from the columns? I really do not get it. I use Ubuntu 64 bit, the .csv file has all the permissions it needs, postgresql has also. Please help me.
These are my errors:
ERROR: missing data for column "name"
CONTEXT: COPY movie, line 1: "tt0468569,The Dark Knight,2008,9,1440667,152 mins.,Christopher Nolan,Christian Bale|Heath Ledger|Aar..."
********** Error **********
ERROR: missing data for column "name"
SQL state: 22P04
Context: COPY movie, line 1: "tt0468569,The Dark Knight,2008,9,1440667,152 mins.,Christopher Nolan,Christian Bale|Heath Ledger|Aar..."
Use this code instead it is working fine on Linux as well on windows
\COPY movie(imdib,name,year,rating,votes,runtime,directors,actors,genres) FROM 'D:\test.csv' WITH DELIMITER '|' CSV HEADER;
and one more thing insert header in your csv file like shown below:
imdib|name|year|rating|votes|runtime|directors|actors|genres
tt0111161|The Shawshank Redemption|1994|9.3|1468273|142 mins.|Frank Darabont|Tim Robbins|Morgan Freeman
and use single byte delimiter like ',','|' etc.
Hope this will work for you ..!
The following works for me:
COPY movie (imdib,name,year,rating,votes,runtime,directors,actors,genres)
FROM 'imdb_top100t_2015-06-18.csv'
WITH (format csv, header false, delimiter E'\t', NULL '');
Unfortunately the file is invalid because on line 12011 the column year contains the value 2015 Video and thus the import fails because this can't be converted to an integer. And then further down (line 64155) there is an invalid value NA for the rating which can't be converted to a float and then one more for the votes.
But if you create the table with all varchar columns the above command worked for me.
Ok, I've managed to solve the problem. I had comas on the problematic lines which was causing the script to not run. I had to go through each one to fix it, but it worked in the end. Thanks
I created a table using
CREATE TABLE customers_transfer
(
customer text,
phone text,
alt_phone text,
fax text,
street1 text,
street2 text,
city text,
state text,
zip text,
email text,
)
WITH (
OIDS=FALSE
);
and I have a .csv file that matches it. But when I try running COPY customers_transfer FROM 'C:/customer_transfer.csv' DELIMITER ',' I get this error. I've tried adding a single ',' to every last line but it still happens.
Access America Transport,423.821.8044,,423.678.7782,"2515 East 43rd St, Suite B",,Chattanooga,TN,37407,invoices#accessamericatransport.com
it should be
COPY customers_transfer FROM 'C:/customer_transfer.csv' CSV
the DELIMITER ',' makes it split the text "2515 East 43rd St, Suite B" into two columns
Running COPY results in ERROR: invalid input syntax for integer: "" error message for me. What am I missing?
My /tmp/people.csv file:
"age","first_name","last_name"
"23","Ivan","Poupkine"
"","Eugene","Pirogov"
My /tmp/csv_test.sql file:
CREATE TABLE people (
age integer,
first_name varchar(20),
last_name varchar(20)
);
COPY people
FROM '/tmp/people.csv'
WITH (
FORMAT CSV,
HEADER true,
NULL ''
);
DROP TABLE people;
Output:
$ psql postgres -f /tmp/sql_test.sql
CREATE TABLE
psql:sql_test.sql:13: ERROR: invalid input syntax for integer: ""
CONTEXT: COPY people, line 3, column age: ""
DROP TABLE
Trivia:
PostgreSQL 9.2.4
ERROR: invalid input syntax for integer: ""
"" isn't a valid integer. PostgreSQL accepts unquoted blank fields as null by default in CSV, but "" would be like writing:
SELECT ''::integer;
and fail for the same reason.
If you want to deal with CSV that has things like quoted empty strings for null integers, you'll need to feed it to PostgreSQL via a pre-processor that can neaten it up a bit. PostgreSQL's CSV input doesn't understand all the weird and wonderful possible abuses of CSV.
Options include:
Loading it in a spreadsheet and exporting sane CSV;
Using the Python csv module, Perl Text::CSV, etc to pre-process it;
Using Perl/Python/whatever to load the CSV and insert it directly into the DB
Using an ETL tool like CloverETL, Talend Studio, or Pentaho Kettle
I think it's better to change your csv file like:
"age","first_name","last_name"
23,Ivan,Poupkine
,Eugene,Pirogov
It's also possible to define your table like
CREATE TABLE people (
age varchar(20),
first_name varchar(20),
last_name varchar(20)
);
and after copy, you can convert empty strings:
select nullif(age, '')::int as age, first_name, last_name
from people
Just came across this while looking for a solution and wanted to add I was able to solve the issue by adding the "null" parameter to the copy_from call:
cur.copy_from(f, tablename, sep=',', null='')
I got this error when loading '|' separated CSV file although there were no '"' characters in my input file. It turned out that I forgot to specify FORMAT:
COPY ... FROM ... WITH (FORMAT CSV, DELIMITER '|').
Use the below command to copy data from CSV in a single line without casting and changing your datatype.
Please replace "NULL" by your string which creating error in copy data
copy table_name from 'path to csv file' (format csv, null "NULL", DELIMITER ',', HEADER);
I had this same error on a postgres .sql file with a COPY statement, but my file was tab-separated instead of comma-separated and quoted.
My mistake was that I eagerly copy/pasted the file contents from github, but in that process all the tabs were converted to spaces, hence the error. I had to download and save the raw file to get a good copy.
CREATE TABLE people (
first_name varchar(20),
age integer,
last_name varchar(20)
);
"first_name","age","last_name"
Ivan,23,Poupkine
Eugene,,Pirogov
copy people from 'file.csv' with (delimiter ';', null '');
select * from people;
Just in first column.....
Ended up doing this using csvfix:
csvfix map -fv '' -tv '0' /tmp/people.csv > /tmp/people_fixed.csv
In case you know for sure which columns were meant to be integer or float, you can specify just them:
csvfix map -f 1 -fv '' -tv '0' /tmp/people.csv > /tmp/people_fixed.csv
Without specifying the exact columns, one may experience an obvious side-effect, where a blank string will be turned into a string with a 0 character.
this ought to work without you modifying the source csv file:
alter table people alter column age type text;
copy people from '/tmp/people.csv' with csv;
There is a way to solve "", the quoted null string as null in integer column,
use FORCE_NULL option :
\copy table_name FROM 'file.csv' with (FORMAT CSV, FORCE_NULL(column_name));
see postgresql document, https://www.postgresql.org/docs/current/static/sql-copy.html
All in python (using psycopg2), create the empty table first then use copy_expert to load the csv into it. It should handle for empty values.
import psycopg2
conn = psycopg2.connect(host="hosturl", database="db_name", user="username", password="password")
cur = conn.cursor()
cur.execute("CREATE TABLE schema.destination_table ("
"age integer, "
"first_name varchar(20), "
"last_name varchar(20)"
");")
with open(r'C:/tmp/people.csv', 'r') as f:
next(f) # Skip the header row. Or remove this line if csv has no header.
conn.cursor.copy_expert("""COPY schema.destination_table FROM STDIN WITH (FORMAT CSV)""", f)
Incredibly, my solution to the same error was to just re-arrange the columns. For anyone else doing the above solutions and still not getting past the error.
I apparently had to arrange the columns in my CSV file to match the same sequence in the table listing in PGADmin.
I'm building a database for chemical names. My main table contains two attributes, something like this:
CREATE TABLE chemicals
(
chemical id int PRIMARY KEY,
names varchar(200)[]
);
The table is then filled with COPY
COPY chemicals FROM chemicals_file delimiters E'\t';
I have formated the chemicals_file like this:
chemical_id { name1 name2 name3 }
......................................
As you know chemical names (IUPAC names) contain the character ',' inside, so I cannot use it as a delimiter for the array of names. I can't find on the online documentation any way to specify another delimiter for the array. Could you please help me?
Another option I would have here, since the original chemicals file has one entry per name, like:
chemical_id_1 name1
chemical_id_1 name2
.......................
chemical_id_6 name3
.......................
would be to create a trigger BEFORE COPY (can this be done?) so that if the chemical_id exits in the table the name is added into the existing array of names, instead of creating a whole new row, but I don't know how these could be specified.
I have created already a trigger BEFORE INSERT OR UPDATE but it doesn't seem to work for COPY
Thanks a lot for the help ;)
Try text format like this:
chemical_id_1 { "one,two,three", "four", "five,six" }
...
Mainly you want a multivalued column, but that is evil. I'd use this:
chemical(id, name)
chemical_name(id, chemical_id, name)
And in chemical you have a handy preferred name to use in JOINs