I'm fairly new to postgres. I am trying to copy over a file from my computer to a postgres server. I first initialize the table with
CREATE TABLE coredb (
id text, lng numeric(6,4), lat numeric(6,4),
score1 numeric(5,4), score2 numeric(5,4));
And my CSV looks like this:
ID lng lat score1 score2
1 -72.298 43.218 0.561 0.894
2 -72.298 43.218 0.472 0.970
3 -72.285 43.250 0.322 0.959
4 -72.285 43.250 0.370 0.934
5 -72.325 43.173 0.099 0.976
6 -72.325 43.173 0.099 0.985
However, when I try to copy the CSV over, I get the following error
COPY coredb FROM '/home/usr/Documents/filefordb.csv' DELIMITER ',' CSV;
ERROR: invalid input syntax for type numeric: "lng"
CONTEXT: COPY nhcore, line 1, column lng: "lng"
Oddly enough the csv imports just fine when I set the CREATE TABLE parameters to text for all the columns. Could someone explain why this is happening? I am using psql 9.4.1
You have to use HEADER true to tell COPY to skip the header line.
Related
With ref to other related question; with following configuration I able to insert data into Redshift -
COPY "hits" FROM 's3://your-bucket/your_folder/'
CREDENTIALS 'aws_access_key_id=<AWS_ACCESS_KEY_ID>;aws_secret_access_key=<AWS_SECRET_ACCESS_KEY>'
FORMAT as JSON 's3://your-bucket/config/jsonpaths'
TIMEFORMAT as 'epochmillisecs';
It is converting '1528207694599' into '2018-06-05 14:08:14' , but I'm expecting '2018-06-05 14:08:14.599' .
Any luck? Thanks in advance.
Seems like you may be doing something wrong at somewhere, I would like to show you a systamatic steps, to prove copy populates data properly including milliseconds.
create table sales(
salesid integer not null,
category varchar(10),
update_at timestamp);
data file data.json's content
[1,"Sports1","1528207694599"]
[2,"Sports2","1528207694456"]
Mapping file json_path.json's content
{
"jsonpaths": [
"$[0]",
"$[1]",
"$[2]"
]
}
Then copy command,
COPY sales FROM 's3://s3-path/to/data/data.json' CREDENTIALS 'aws_access_key_id=**********;aws_secret_access_key=*******' FORMAT as JSON 's3://s3-path/to/mapping/json_path.json' TIMEFORMAT as 'epochmillisecs';
Output:
COPY sales FROM .............
INFO: Load into table 'sales' completed, 2 record(s) loaded successfully.
COPY;
$Select * from Sales;
salesid | category | update_at
---------+----------+-------------------------
2 | Sports2 | 2018-06-05 14:08:14.456
1 | Sports1 | 2018-06-05 14:08:14.599
(2 rows)
As you could see, update_at has value including milliseconds.
Hope it helps.
I have a file,
ID,DNS,R_D,R_A
1,123456,2014/11/17,10
2,987654,2016/05/20,30
3,434343,2017/08/01,20
that I'm trying to load to oracle using External Tables. I have to skip the header row and also load the date column.
This is my query:
DECLARE
FILENAME VARCHAR2(400);
BEGIN
FILENAME := 'actual_data.txt';
EXECUTE IMMEDIATE 'CREATE TABLE EXT_TMP (
ID NUMBER(25),
DNS VARCHAR2(20),
R_D DATE,
R_A NUMBER(25)
)
ORGANIZATION EXTERNAL (
TYPE ORACLE_LOADER
DEFAULT DIRECTORY USER_DIR
ACCESS PARAMETERS (
RECORDS DELIMITED BY NEWLINE
FIELDS TERMINATED BY '',''
MISSING FIELD VALUES ARE NULL
SKIP 1
(
"ID",
"DNS",
"R_D" date "dd-mon-yy",
"RECHARGE_AMOUNT"
)
)
LOCATION (''' || FILENAME || ''')
)
PARALLEL 5
REJECT LIMIT UNLIMITED';
END;
I get following exception:
ERROR at line 1:
ORA-29913: error in executing ODCIEXTTABLEOPEN callout
ORA-29400: data cartridge error
KUP-00554: error encountered while parsing access parameters
KUP-01005: syntax error: found "skip": expecting one of: "column, exit, (,
reject"
KUP-01007: at line 4 column 5
ORA-06512: at "SYS.ORACLE_LOADER", line 19
I'm using sqlplus.
Could some oracle veterans please help me out and tell me what I'm doing wrong here? I'm very new to oracle.
You don't want to create any kind of tables (including external ones) in PL/SQL; not that it is impossible, but it is opposite of the best practices.
Have a look at my attempt, based on information you provided - works OK.
SQL> alter session set nls_date_format = 'dd.mm.yyyy';
Session altered.
SQL> create table ext_tmp
2 (
3 id number,
4 dns varchar2(20),
5 r_d date,
6 r_a number
7 )
8 organization external
9 (
10 type oracle_loader
11 default directory kcdba_dpdir
12 access parameters
13 (
14 records delimited by newline
15 skip 1
16 fields terminated by ',' lrtrim
17 missing field values are null
18 (
19 id,
20 dns,
21 r_d date 'yyyy/mm/dd',
22 r_a
23 )
24 )
25 location ('actual_data.txt')
26 )
27 parallel 5
28 reject limit unlimited;
Table created.
SQL> select * from ext_tmp;
ID DNS R_D R_A
---------- -------------------- ---------- ----------
1 123456 17.11.2014 10
2 987654 20.05.2016 30
3 434343 01.08.2017 20
SQL>
In my case skip 1 didn't work even with placing it between records delimited by newline and fields terminated by ',' lrtrim until I used load when. Now skip 1 works with the following access parameters:
access parameters (
records delimited by newline
load when (someField != BLANK)
skip 1
fields terminated by '','' lrtrim
missing field values are null
reject rows with all null fields
)
I would like to import a csv file, which has multiple occurrences of missing values. I recoded them into NULL and tried to import the file as. I suppose that my attributes which include the NULLS are character values. However transforming them to numeric is bit complicated. Therefore I would like to import all of my table as:
\copy player_allstar FROM '/Users/Desktop/Rdaten/Data/player_allstar.csv' DELIMITER ';' CSV WITH NULL AS 'NULL' ';' HEADER
There must be a syntax error. But I tried different combinations and always get:
ERROR: syntax error at or near "WITH NULL"
LINE 1: COPY player_allstar FROM STDIN DELIMITER ';' CSV WITH NULL ...
I also tried:
\copy player_allstar FROM '/Users/Desktop/Rdaten/Data/player_allstar.csv' WITH(FORMAT CSV, DELIMITER ';', NULL 'NULL', HEADER);
and get:
ERROR: invalid input syntax for integer: "NULL"
CONTEXT: COPY player_allstar, line 2, column dreb: "NULL"
I suppose it is caused by preprocessing with R. The Table came with NAs so I change them to:
data[data==NA] <- "NULL"
I`m not aware of a different way chaning to NULL. I think this causes strings. Is there a different way to preprocess and keep the NAs(as NULLS in postgres of course)?
Sample:
pts dreb oreb reb asts stl
11 NULL NULL 8 3 NULL
4 5 3 8 2 1
3 NULL NULL 1 1 NULL
data type is integer
Given /tmp/sample.csv:
pts;dreb;oreb;reb;asts;stl
11;NULL;NULL;8;3;NULL
4;5;3;8;2;1
3;NULL;NULL;1;1;NULL
then with a table like:
CREATE TABLE player_allstar (pts integer, dreb integer, oreb integer, reb integer, asts integer, stl integer);
it works for me:
\copy player_allstar FROM '/tmp/sample.csv' WITH (FORMAT CSV, DELIMITER ';', NULL 'NULL', HEADER);
Your syntax is fine, the problem seem to be in the formatting of your data. Using your syntax I was able to load data with NULLs successfully:
mydb=# create table test(a int, b text);
CREATE TABLE
mydb=# \copy test from stdin WITH(FORMAT CSV, DELIMITER ';', NULL 'NULL', HEADER);
Enter data to be copied followed by a newline.
End with a backslash and a period on a line by itself.
>> col a header;col b header
>> 1;one
>> NULL;NULL
>> 3;NULL
>> NULL;four
>> \.
mydb=# select * from test;
a | b
---+------
1 | one
|
3 |
| four
(4 rows)
mydb=# select * from test where a is null;
a | b
---+------
|
| four
(2 rows)
In your case you can substitute to NULL 'NA' in the copy command, if the original value is 'NA'.
You should make sure that there's no spaces around your data values. For example, if your NULL is represented as NA in your data and fields are delimited with semicolon:
1;NA <-- good
1 ; NA <-- bad
1<tab>NA <-- bad
etc.
I am trying to execute following bteq command on linux environment but couldn't load data properly into Teradata DB server. Can someone please advise me to resolve the below issue that I am facing while loading.
BTEQ Command used :
.SET width 64000;
.SET session transaction btet;
.logmech ldap
.logon XXXXXXX/XXXXXXXX,********;
DATABASE corecm;
.PACK 1000
.IMPORT VARTEXT '~' FILE=/v/global/user/application_event_bus_evt
.REPEAT *
USING(APPLICATION_EVENT_ID CHAR(24),BUS_EVT_ID CHAR(24),BUS_EVT_VID BIGINT,BUS_EVT_RESTATE_IN SMALLINT)
insert into corecm.application_event_bus_evt (APPLICATION_EVENT_ID
, BUS_EVT_ID
, BUS_EVT_VID
, BUS_EVT_RESTATE_IN
)
values
( COALESCE(:APPLICATION_EVENT_ID,1)
, COALESCE(:BUS_EVT_ID,1)
, COALESCE(:BUS_EVT_VID,1)
, COALESCE(:BUS_EVT_RESTATE_IN,1)
) ;
.LOGOFF;
.EXIT;
SAMPLE INPUT FILE DELIMITTER "~" [ /v/global/user/application_event_bus_evt ] :
Ckn3gMxLEeOgIQBQVgErYA==~g+GDDtlaY3n7BdUrYshDFA==~1~1
CL1kEcxLEeOgIQBQVgErYA==~qoKoiuGDbClpcGt/z6RKGw==~1~1
oYIVcMxKEeOgIQBQVgErYA==~mfmQiwl7yAteevzJfilMvA==~1~1
5N7ME5bM4xGhM7exj3ykUw==~yFM2FZbM4xGhM7exj3ykUw==~1~0
JLBH4JfM4xGDH9s5+Ds/8w==~doZ/7pfM4xGDH9s5+Ds/8w==~1~0
fGvpoMxKEeOgIQBQVgErYA==~mQUQIK2mY6WIPcszfp5BTQ==~1~1
Table Definition :
CREATE MULTISET TABLE CORECM.APPLICATION_EVENT_BUS_EVT ,NO FALLBACK ,
NO BEFORE JOURNAL,
NO AFTER JOURNAL,
CHECKSUM = DEFAULT,
DEFAULT MERGEBLOCKRATIO
(
APPLICATION_EVENT_ID CHAR(26) CHARACTER SET LATIN NOT CASESPECIFIC NOT NULL,
BUS_EVT_ID CHAR(26) CHARACTER SET LATIN NOT CASESPECIFIC NOT NULL,
BUS_EVT_VID BIGINT NOT NULL,
BUS_EVT_RESTATE_IN SMALLINT)
UNIQUE PRIMARY INDEX ( APPLICATION_EVENT_ID ,BUS_EVT_ID ,BUS_EVT_VID )
INDEX APPLICATION_EVENT_BUS_EVT_IDX1 ( APPLICATION_EVENT_ID )
INDEX APPLICATION_EVENT_BUS_EVT_IDX2 ( BUS_EVT_ID ,BUS_EVT_VID );
Results set in DB server as,
APPLICATION_EVENT_ID BUS_EVT_ID BUS_EVT_VID BUS_EVT_RESTATE_IN
1 Ckn3gMxLEeOgIQBQVgErYA == g+GDDtlaY3n7BdUrYshD 85,849,873,219,141,958 12,544
2 CL1kEcxLEeOgIQBQVgErYA == qoKoiuGDbClpcGt/z6RK 85,849,873,219,155,783 12,544
3 oYIVcMxKEeOgIQBQVgErYA == mfmQiwl7yAteevzJfilM 85,849,873,219,142,006 12,544
4 5N7ME5bM4xGhM7exj3ykUw == JAf0GpbM4xGhM7exj3yk 85,849,873,219,155,797 12,288
5 JLBH4JfM4xGDH9s5+Ds/8w == Du6T7pfM4xGDH9s5+Ds/ 85,849,873,219,155,768 12,288
6 fGvpoMxKEeOgIQBQVgErYA == mQUQIK2mY6WIPcszfp5B 85,849,873,219,146,068 12,544
If we look at the Data, we can see two issues as,
First two column data length is 24 CHARACTERS ( as per input file ), but the issue is that it been shifted two characters in next column.
Column BUS_EVT_VID and BUS_EVT_RESTATE_IN has wrong data 85,849,873,219,141,958 and 12,544 instead of 1 and 1 respectively (this may be because first two column data got shifted)
I tried following options to resolve the above issue but couldn't resolve the issue,
Modified the Table Definition, i.e. changed datatype to
CHAR(28),CHAR(24),CHAR(26)
Modified the Table Definition column
datatypes to VARCHAR(24), VARCHAR(26)
Modified BTEQ command, i.e. altered datatype in below line,
USING(APPLICATION_EVENT_ID CHAR(24),BUS_EVT_ID CHAR(24),BUS_EVT_VID BIGINT,BUS_EVT_RESTATE_IN SMALLINT)
Thanks in advance.
When you define VARTEXT all input columns must be defined as VARCHAR, but you used CHAR and INT.
This should work, VARCHAR length based on the definition of your target table:
USING(
APPLICATION_EVENT_ID VARCHAR(26),
BUS_EVT_ID VARCHAR(26),
BUS_EVT_VID VARCHAR(19),
BUS_EVT_RESTATE_IN VARCHAR(6)
)
I am new to this postGIS concepts.. I have an postgres table in that i have some 10000 datas.Now i want to update some 100 datas in that table. I have this 100 datas in CSV file. SO i used the following query:
CREATE TEMP TABLE tmp_x AS SELECT * FROM xxxxx LIMIT 0;
UPDATE xxxxxx
SET latitude = tmp_x.latitude
USING tmp_x
WHERE xxxxxxxx.id = tmp_x.id;
But its showing error:
ERROR: syntax error at or near "USING"
LINE 3: USING tmp_x
What wrong with above query.Help me to solve this..Thanks in advance..
Just a little syntax error. Should be "from" not "using".
UPDATE xxxxxx
SET latitude = tmp_x.latitude
FROM tmp_x
WHERE xxxxxxxx.id = tmp_x.id;