I am trying import the following csv file which is imported from PgAdmin.
CSV File:
"id";"email";"password";"permissions";"activated";"activation_code";"activated_at";"last_login";"persist_code";"reset_password_code";"first_name";"last_name";"created_at";"updated_at"
"11";"test.teset#gmail.com";"$2y$10$gNfDFVCgqhQzCgKoiLuKfeskyCUPzK1akJvk6PdXt1DmMJooCuYpi";"";"t";"XY56HNcFHeAcAwzUhleYhvVbxxmOaMr57ifYEhxiUd";"";"2014-05-27 08:47:33";"$2y$10$g0LnAStmA/kEWuhNDfWleOjQeyo9maGvvIlfiJms/KpRiPAdfBWHm";"";"";"";"2014-05-27 07:51:07";"2014-05-27 08:47:33"
"5";"test#gmail.com";"$2y$10$dXODoI520pddcmiSXcS/OuiH.4K/87LEXeQRzvUl2k/Uth2HumpNy";"";"t";"4k8s1TbgrPfAMcNozVEP19MOQkCApQ0LA8bhGkF55A";"";"2014-05-21 21:18:06";"$2y$10$CefSnbQIzAJBo5PfbMdzKuhzpW17fHqh/frWabmljzJvv0A5Vkt1O";"";"";"";"2014-05-21 21:17:45";"2014-05-22 19:12:01"
And this the command I am using to import CSV file,
DROP TABLE users;
CREATE TABLE users
(
id serial NOT NULL,
email character varying(255) NOT NULL,
password character varying(255) NOT NULL,
permissions text,
activated boolean NOT NULL DEFAULT true,
activation_code character varying(255),
activated_at timestamp without time zone,
last_login timestamp without time zone,
persist_code character varying(255),
reset_password_code character varying(255),
first_name character varying(255),
last_name character varying(255),
created_at timestamp without time zone,
updated_at timestamp without time zone
);
COPY users (email,password,permissions,activated,activation_code,activated_at,last_login,persist_code,reset_password_code,first_name,last_name,created_at,updated_at)
FROM 'D:/test/db_backup/data_csv.csv' WITH CSV HEADER delimiter ';'
;
But I am getting following error, not sure why.
**ERROR: extra data after last expected column SQL state: 22P04**
Please let me know what is the issue here?
Because you have 14 columns of data but only 13 inside your copy statement. Specifically, you are missing the ID column in your copy statment. If the data are in the same order as they are declared in the table, there is no need to put the column names in the copy table from statement.
COPY users FROM 'D:/test/db_backup/data_csv.csv' CSV HEADER delimiter ';' null '\N';
will work just fine in your case -- note you don't need the WITH either.
EDIT. You need to explicitly set the null value, as in, null '\N'. If you use "", you can get an error such as: "CSV quote character must not appear in the NULL specification". Also, you can only specify HEADER in CSV mode, so if you want to use HEADER, and CSV, then you cannot use "" as a delimiter, and must use an explicit null '\N'.
I have updated my original answer to include this and imported your sample data, after replacing all the "" with \N.
Related
A table in my database (PostgreSQL 9.6) has a mixture of NULL and not null values, which I need to COALESCE() as a part of the creation of another attribute during insert into a resulting dimension table. However, Postgres seems unable to recognize the NULL values as NULL.
SELECT DISTINCT name, description
FROM my_table
WHERE name IN('STUDIO', 'ONE BEDROOM')
AND description IS NOT NULL;
returns
name
description
STUDIO
NULL
ONE BEDROOM
NULL
Whereas
SELECT DISTINCT name, description
FROM my_table
WHERE name IN('STUDIO', 'ONE BEDROOM')
AND description IS NULL;
returns
name
description
as such, something like
SELECT DISTINCT name, COALESCE(description, 'N/A')
FROM my_table
WHERE name IN('STUDIO', 'ONE BEDROOM');
will return
name
coalesce
STUDIO
NULL
ONE BEDROOM
NULL
instead of the expected
name
coalesce
STUDIO
N/A
ONE BEDROOM
N/A
The DDL for these attributes is fairly straightforward:
...
name text COLLATE pg_catalog."default",
description text COLLATE pg_catalog."default",
...
I've already checked whether the attribute was filled with 'NULL' rather than being an actual NULL value, and that's not the case. I've also tried quoting the attribute in question as "description" and that hasn't made a difference. Casting to VARCHAR hasn't helped (I thought it might be the fact that it's a TEXT attribute). If I nullify some values in the other text column (name) I'm able to coalesce with a test value, so that one is seemingly behaving as expected leading me to think it's not a data type issue. This table exists in multiple databases on multiple servers and exhibits the same behavior in all of them.
I've tried inserting into a new table that has different attribute definitions:
...
floorplan_name "character varying(128)" COLLATE pg_catalog."default" NOT NULL DEFAULT 'Unknown'::character varying,
floorplan_desc "character varying(256)" COLLATE pg_catalog."default" NOT NULL DEFAULT 'Not Provided'::character varying,
...
resulting in
name
coalesce
STUDIO
NULL
ONE BEDROOM
NULL
so, not only is the default value unable to populate, leaving the values NULL in an attribute that is defined as NOT NULL, but the example SELECT statements above all behave in exactly the same way when run against the new table.
Does anyone have any idea what might be causing this?
It turns out that the source database is writing empty strings instead of proper NULLs. Adding NULLIF(description, '') before trying to COALESCE() solves the problem.
Thanks to everyone!
I have a json file as:
[xyz#innolx20122 ~]$ cat test_cgs.json
{"technology":"AAA","vendor":"XXX","name":"RBNI","temporal_unit":"hour","regional_unit":"cell","dataset_metadata":"{\"name\": \"RBNI\", \"intervals_epoch_seconds\": [[1609941600, 1609945200]], \"identifier_column_names\": [\"CELLID\", \"CELLNAME\", \"NETWORK\"], \"vendor\": \"XXX\", \"timestamp_column_name\": \"COLLECTTIME\", \"regional_unit\": \"cell\"}","rk":1}
which I am trying to upload to below table in Postgres
CREATE TABLE temp_test_table
(
technology character varying(255),
vendor character varying(255),
name character varying(255),
temporal_unit character varying(255),
regional_unit character varying(255),
dataset_metadata json,
rk character varying(255)
);
and here is my copy command
db-state=> \copy temp_test_table(technology,vendor,name,temporal_unit,regional_unit,dataset_metadata,rk) FROM '/home/eksinvi/test_cgs.json' WITH CSV DELIMITER ',' quote E'\b' ESCAPE '\';
ERROR: extra data after last expected column
CONTEXT: COPY temp_test_table, line 1: "{"technology":"AAA","vendor":"XXX","name":"RBNI","temporal_unit":"hour","regional_unit":"cell","data..."
I even tried loading this file to big query table but no luck
bq load --autodetect --source_format=NEWLINE_DELIMITED_JSON --allow_quoted_newlines --allow_jagged_rows --ignore_unknown_values test-project:vikrant_test_dataset.cg_test_table "gs://test-bucket-01/test/test_cgs.json"
any of the solution would work for me. I want to load this json either to Postgres table or bigquery table.
I had similar problems. In my case, it was related to NULL columns and encoding of the file. I also had to specify a custom delimiter because my columns sometimes included the default limiter and it would make the copy fail.
\\copy mytable FROM 'filePath.dat' (DELIMITER E'\\t', FORMAT CSV, NULL '', ENCODING 'UTF8' );
In my case, I was exporting data to a CSV file from SQL Server and importing it to postgres. In SQL Server, we had unicode characters that would show up as "blanks" but that would screw up the copy command. I had to search the SQL table for those characters with regex queries and eliminate invalid characters. It's an edge case but that was part of the problem in my case.
I've successfully created the following table:
CREATE TABLE tampadocs (
fname varchar(255),
lname varchar(255),
pracName varchar(255),
address varchar(255),
city varchar(255),
state varchar(255),
zip varchar(255),
spec varchar(255),
phone varchar(255),
totalMD integer,
avgPt integer,
mdName varchar(255),
notes varchar(255));
Then run the following to import data from a CSV to the table:
COPY tampadocs
FROM 'C:\Users\bam\Desktop\tampadocs.csv' DELIMITERS ',' CSV;
I receive the following error:
ERROR: invalid input syntax for integer: "Total MDs"
CONTEXT: COPY tampadocs, line 1, column totalmd: "Total MDs"
I've looked at each value in the Total MDs column but they are just numbers, so I'm not sure what I'm missing. Any help would be greatly appreciated!
Sounds like your file 'tampadocs.csv' has a header line. Can you check on that? The copy syntax you're using is assuming there's only data in the file.
If there's a header line in your file, you can try the following:
COPY tampadocs
FROM 'C:\Users\bam\Desktop\tampadocs.csv'
WITH (FORMAT CSV, DELIMITER ',', HEADER);
That lets the copy statement know to expect a header line in the file. The full syntax for COPY is available here.
When trying to use the COPY command via SQL in Postgres 9.5.1 in a simple example database…
I am getting this error:
ERROR: invalid input syntax for integer: "Sally"
CONTEXT: COPY customer_, line 2, column id_: "Sally"
********** Error **********
ERROR: invalid input syntax for integer: "Sally"
SQL state: 22P02
Context: COPY customer_, line 2, column id_: "Sally"
…when importing this data in CSV (comma-separated value):
"first_name_","last_name_","phone_","email_"
"Sally","Jones","425.555.1324","s.jones#acme.com"
"Jarrod","Barkley","206.555.3454","j.barkley#example.com"
"Wendy","Melvin","415.555.2343","wendy#wendyandlisa.com"
"Lisa","Coleman","425.555.7282","lisa#wendyandlisa.com"
"Jesse","Johnson","507.555.7865","j.j#guitar.com"
"Jean-Luc","Martin","212.555.2244","jean-luc.martin#example.com"
…being imported via the following SQL executed in pgAdmin:
COPY customer_
FROM '/home/parallels/Downloads/customer_.csv'
CSV
HEADER
;
…into this table:
-- Table: public.customer_
-- DROP TABLE public.customer_;
CREATE TABLE public.customer_
(
id_ integer NOT NULL DEFAULT nextval('customer__id__seq'::regclass),
first_name_ text NOT NULL,
last_name_ text NOT NULL,
phone_ text NOT NULL DEFAULT ''::text,
email_ text NOT NULL DEFAULT ''::text,
CONSTRAINT pkey_customer_ PRIMARY KEY (id_)
)
WITH (
OIDS=FALSE
);
ALTER TABLE public.customer_
OWNER TO postgres;
COMMENT ON TABLE public.customer_
IS 'Represents a person whose pets visit our clinic.';
So it seems the first row containing the names of the columns is being processed successfully. The failure point is with the first data value in the first data line of the CSV. None of my imported data is of integer type, so the I am befuddled by the error message. The only integer is the id_ primary key, auto-incrementing SERIAL.
I did read the Question page on PG COPY error: invalid input syntax for integer. But that question did involve integer values, and the lack thereof in an empty quoted string being interpreted as a NULL. In my case here we have no integer values in the data; the only integer is the primary key SERIAL column with a DEFAULT generated value (not in the data being imported).
I also found the Question, PostgreSQL ERROR: invalid input syntax for integer. But it seems irrelevant.
Try specifying the columns . . . without the primary key:
COPY customer_ (first_name_ text, last_name_ text, phone_ text, email_ text)
FROM '/home/parallels/Downloads/customer_.csv'
CSV
HEADER
;
Without the column list, it is looking for a value for id_.
The import data file’s first row of column names are not used for mapping to the table columns. The HEADER flag merely tells Postgres to skip over that first line, as documented:
HEADER
Specifies that… on input, the first line is ignored. …
COPY table_name FROM 'C:\path\file.csv' DELIMITERS ',' CSV header;
This wasn't the OP's problem, but posting because this is one of the top results when I google the error message.
I was trying to import a .csv file with no header. Adding a header to the file and changing COPY ... CSV to COPY ... CSV HEADER in the sql command fixed the problem for me.
I have a table that is defined like this:
CREATE TABLE session_requests
(
id character varying(255) NOT NULL,
authorization_enc character varying(255),
auto_close integer,
date_created character varying(255) DEFAULT '1970-01-01 01:00:00'::character varying,,
....
)
I'm trying to do
alter table session_requests alter column date_created type timestamp using date_created::timestamp;
the error that I'm getting is
ERROR: default for column "date_created" cannot be cast automatically to type timestamp
Anyone has any suggestions?
Do it in one transaction. You can even do it in a single statement:
ALTER TABLE session_requests
ALTER date_created DROP DEFAULT
,ALTER date_created type timestamp USING date_created::timestamp
,ALTER date_created SET DEFAULT '1970-01-01 01:00:00'::timestamp;
SQL Fiddle.
Aside: character varying(255) is almost always a bad (pointless) choice in Postgres. More:
Refactor foreign key to fields