Importing CSV to postgres with new columns - postgresql

Question

I have an issue importing csv data to a postgres database with geo data / postgis enabled via the following command on my database 'landmarks':
CREATE EXTENSION postgis;
So.... the story goes:
I am following this tutorial.
I am trying to import a csv with these columns
name conf capital venture latitude longitude
the first line, as an example of the data, is:
example, 1, 1, 1, 51.51923, -0.12205
I have set the table up following the tutorial except adding conf, capital and venture instead of the columns in his data (address, date_built, architect, landmark). i.e.:
CREATE TABLE landmarks
(
gid serial NOT NULL,
name character varying(50),
conf character varying(10),
capital character varying(10),
venture character varying(10),
the_geom geometry,
CONSTRAINT landmarks_pkey PRIMARY KEY (gid),
CONSTRAINT enforce_dims_the_geom CHECK (st_ndims(the_geom) = 2),
CONSTRAINT enforce_geotype_geom CHECK (geometrytype(the_geom) = 'POINT'::text OR the_geom IS NULL),
CONSTRAINT enforce_srid_the_geom CHECK (st_srid(the_geom) = 4326)
);
then
CREATE INDEX landmarks_the_geom_gist
ON landmarks
USING gist
(the_geom );
The data is essentially the same otherwise to his example.
I've set the table up properly and enabled the postgis extension to deal with geom data fine.
However, the problem comes when I try to import my csv:
landmarks=# \copy landmarks(name,conf,capital,venture,latitude,longitude) FROM '../../../../../var/tmp/map2.csv' DELIMITERS ',' CSV HEADER;
ERROR: column "latitude" of relation "landmarks" does not exist
Now, I noticed that when he creates the table, he doesn't add latitude or longitude columns... so I wondered if that was the issue and tried to create another table with those columns as well as integers, however that just gives me this error:
ptmap3=# \copy landmarks(name,conf,capital,venture,latitude,longitude) FROM '../../../../../var/tmp/map2.csv' DELIMITERS ',' CSV HEADER;
ERROR: invalid input syntax for integer: "51.51923"
CONTEXT: COPY landmarks, line 2, column latitude: "51.51923"
So... it seems that if I add the latitude column then it works, but fails with the data? After checking the csv for errors using this
od -c map2.csv
...there is nothing wrong with my csv (no hidden characters or errors)... so what's the deal?
If anyone can help me import my csv to this db I'd be very grateful!

Answer 1

You need to process in two steps to import your (latitude,longitude) data into the geometry column of your database.
1. Import the data into two float columns latitude and longitude:
From your original table landmarks:
CREATE TABLE landmarks
(
gid serial NOT NULL,
name character varying(50),
conf character varying(10),
capital character varying(10),
venture character varying(10),
the_geom geometry,
CONSTRAINT landmarks_pkey PRIMARY KEY (gid),
CONSTRAINT enforce_dims_the_geom CHECK (st_ndims(the_geom) = 2),
CONSTRAINT enforce_geotype_geom CHECK (geometrytype(the_geom) = 'POINT'::text OR the_geom IS NULL),
CONSTRAINT enforce_srid_the_geom CHECK (st_srid(the_geom) = 4326)
);
CREATE INDEX landmarks_the_geom_gist
ON landmarks
USING gist
(the_geom );
Add the temporary two columns latitude and longitude:
ALTER TABLE landmarks ADD COLUMN latitude double precision;
ALTER TABLE landmarks ADD COLUMN longitude double precision;
Then import your data:
\copy landmarks(name,conf,capital,venture,latitude,longitude) FROM '../../../../../var/tmp/map2.csv' DELIMITERS ',' CSV HEADER;
2. Populate the geometry column by creating POINT geometries from latitude and longitude:
UPDATE landmarks SET geom = ST_SetSRID(ST_MakePoint(longitude,latitude),4326);
Finally, delete the temporary columns:
ALTER TABLE landmarks DROP COLUMN latitude;
ALTER TABLE landmarks DROP COLUMN longitude;

Answer 2

There is an alternate method.
Google pgfutter and download appropriate version of the executable file. link for windows but other releases available as well.
Make sure pgfutter file and yourdata.csv file are in the same directory.
Set directory in terminal and run the following code.
This will create a new table as well so just enter a fresh table name.
pgfutter.exe --host "localhost" --port "5432" --db "YourDatabaseName" --schema "public" --table "TableName" --user "postgres" --pw "YourPassword" csv YourData.csv