I have a database whereas the first column is labeled as serial not null primary key. The table creation and automatic sequence table creation is successful. However, whenever I do:
copy <table_name> from '/path/to/file' delimiter ',' CSV HEADER;
PostgreSQL tries to read my first column into the serial column, which fails because my first column in my CSV file contains characters (not an integer).
How can I tell the COPY command to populate using a serial column as the first column?
I determined that if I specified the header names and named my columns exactly like the header names in my CSV file, that the import worked:
copy <table_name>(column1, column2, etc) from '/path/to/file' delimiter ',' CSV HEADER;
Related
When I tried to copy a very large txt file into my postgres database, I got a following error below.
Note that I created a table with a single column and am not using any delimiter when importing the txt file.
db1=# create table travis_2018_data (v text);
db1=# \COPY travis_2018_data FROM 'C:\Users\testu\Downloads\travis_2018\2018-Certification\PROP.txt';
The error:
ERROR: extra data after last expected column
CONTEXT: COPY travis_2018_data, line 295032: "000000561125P 02018000000000000
I'm wondering why I still get the error about the extra data (or column) on line 295032 ?
Your text probably contains a tab character which is the default column delimiter for the TEXT format when using \copy (or copy) without specifying a format.
So \copy thinks the line contains two column but only expects one, hence the error message
You need to specify a delimiter that will not occur in the file. The character with the ASCII value 1 is a highly unlikely to occur in such a file, so you can try:
\COPY travis_2018_data FROM '.....' DELIMITER E'\x01'
I use basketball data tables to get some understanding of Postgres 9.2 & phppgadmin. Therefore I would like to import csv tables into that database. However, I get:
ERROR: missing data for column "year"
CONTEXT: COPY coaches, line 1: ""coachid";"year";"yr_order";"firstname";"lastname";"season_win";"season_loss";"playoff_win";"playoff..."
with command:
\copy coaches FROM '/Users/Desktop/Database/NBAPostGres/DataOriginal/coaches_data.csv' DELIMITER ',' CSV;
The current table has no missings. So my questions are:
What did I wrong and if using a table with missing values?
How to import such table or handle such structure generally(also in respect to missing values)?
Data structure:
coachid year yr_order firstname lastname season_win
HAMBLFR01 204 2 Frank Hamblen 10
RUSSEJO01 1946 1 John Russell 22
I used:
varchar integer integer character character integer
You can have columns missing for the whole table. Tell COPY (or the psql wrapper \copy) to only fill selected columns by appending a column list to the table:
\copy coaches (coachid, yr_order, firstname)
FROM '/Users/.../coaches_data.csv' (FORMAT csv, HEADER, DELIMITER ',');
Missing values are filled in with column defaults. The manual:
If there are any columns in the table that are not in the column list,
COPY FROM will insert the default values for those columns.
But you cannot have values missing for just some rows. That's not possible. The text representation of NULL can be used (overruling respective column defaults).
It's all in the manual, really:
SQL-COPY
psql \copy
ERROR: missing data for column "year" CONTEXT: COPY coaches, line 1:
""coachid";"year";"yr_order";"firstname";"lastname";"season_win";"season_loss";"playoff_win";"playoff..."
This type of error is also the result of a Table-mismatch. The table you are importing the text file into either has more columns or less columns than the text file has.
My table has a NOT NULL column called 'created' with DEFAULT CLOCK_TIMESTAMP(). My CSV file from which I am copying intentionally does not have a column called 'created' because I want the database to use the default value for it. My copy command is:
\copy table_name from '/local/path/to/file.csv' delimiter ',' CSV HEADER
The error I receive is:
missing data for column "created_"
The PostgreSQL documentation says: "If there are any columns in the table that are not in the column list, COPY FROM will insert the default values for those columns."
Thanks for any help
The clue is in the "that are not in the column list".
You need to specify the "column list" for this to work and leave out the one with the default value:
\copy table_name (column1, column2, ...) from '/local/path/to/file.csv' delimiter ',' CSV HEADER
I use basketball data tables to get some understanding of Postgres 9.2 & phppgadmin. Therefore I would like to import csv tables into that database. However, I get:
ERROR: missing data for column "year"
CONTEXT: COPY coaches, line 1: ""coachid";"year";"yr_order";"firstname";"lastname";"season_win";"season_loss";"playoff_win";"playoff..."
with command:
\copy coaches FROM '/Users/Desktop/Database/NBAPostGres/DataOriginal/coaches_data.csv' DELIMITER ',' CSV;
The current table has no missings. So my questions are:
What did I wrong and if using a table with missing values?
How to import such table or handle such structure generally(also in respect to missing values)?
Data structure:
coachid year yr_order firstname lastname season_win
HAMBLFR01 204 2 Frank Hamblen 10
RUSSEJO01 1946 1 John Russell 22
I used:
varchar integer integer character character integer
You can have columns missing for the whole table. Tell COPY (or the psql wrapper \copy) to only fill selected columns by appending a column list to the table:
\copy coaches (coachid, yr_order, firstname)
FROM '/Users/.../coaches_data.csv' (FORMAT csv, HEADER, DELIMITER ',');
Missing values are filled in with column defaults. The manual:
If there are any columns in the table that are not in the column list,
COPY FROM will insert the default values for those columns.
But you cannot have values missing for just some rows. That's not possible. The text representation of NULL can be used (overruling respective column defaults).
It's all in the manual, really:
SQL-COPY
psql \copy
ERROR: missing data for column "year" CONTEXT: COPY coaches, line 1:
""coachid";"year";"yr_order";"firstname";"lastname";"season_win";"season_loss";"playoff_win";"playoff..."
This type of error is also the result of a Table-mismatch. The table you are importing the text file into either has more columns or less columns than the text file has.
I defined a new datatype, say
CREATE TABLE IF NOT EXISTS face_axis ( face INTEGER, normal real[]);
where normal should be a vector.
I have already write them on harddisk, like
5, 1,0,0,
....
Then I want to use
COPY face_axis FROM 'face_axis.csv' with csv;
but it reports error
ERROR: extra data after last expected column
what's wrong with it? thanks,
There are two problems:
The array contents must be enclosed in brackets in all cases
Ambiguous use of comma as both the CSV delimiter and delimiter of values inside the array.
You may use a different CSV separator or quote the contents according to CSV rules:
To be loaded with COPY table FROM file with csv delimiter ';'
1;{1.0,2.0}
To be loaded with COPY table FROM file with csv
1,"{1.0,2.0}"