I am a newbie to this technology but need to get the work going for my project.
Goal:
I have a large dataset in .txt format space separated, needs to be imported to postgresql database. One of the columns is timestamp with microsecond precision.
Time column data in the .txt looks like 07:54:13.345782 which HH:MM:ss.SSSSSS format.
I am working on a postgresql installed on MAC. Using the terminal to enter commands.
I created the table
CREATE TABLE Shib003_data2 (
cdate DATE NOT NULL,
ctime TIMESTAMP NOT NULL,
location VARCHAR (50) NOT NULL,
status VARCHAR (50) NOT NULL
);
The ctime column refuses to copy data from my .txt file because of microsecond time precision.
I used
CREATE TABLE Shib003_data2 (
cdate DATE NOT NULL,
ctime TIMESTAMP(6) NOT NULL,
location VARCHAR (50) NOT NULL,
status VARCHAR (50) NOT NULL
);
The above command doesn't go through!
Any idea how to resolve?
I can't have VARCHAR or anything else as this time is critical information for my data processing.
The issue is you are trying to INSERT a time into a timestamp and I'm guessing getting 'invalid input syntax for type timestamp: "07:54:13.345782"' per:
create table ts_precision(fld_1 timestamp, fld_2 time);
insert into ts_precision(fld_1) values('07:54:13.345782');
ERROR: invalid input syntax for type timestamp: "07:54:13.345782"
LINE 1: insert into ts_precision(fld_1) values('07:54:13.345782');
insert into ts_precision(fld_2) values('07:54:13.345782');
select * from ts_precision ;
fld_1 | fld_2
-------+-----------------
| 07:54:13.345782
So use a time field. The other option is to add a date portion to the time field.
Related
,0,1,2,3,4,5,6
0,1,1,1,76.0,2.99,2005-05-25 11:30:37.000,2019-04-11 18:11:50
1,2,1,1,573.0,0.99,2005-05-28 10:35:23.000,2019-04-11 18:11:50
2,3,1,1,1185.0,5.99,2005-06-15 00:54:12.000,2019-04-11 18:11:50
3,4,1,2,1422.0,0.99,2005-06-15 18:02:53.000,2019-04-11 18:11:50
4,5,1,2,1476.0,9.99,2005-06-15 21:08:46.000,2019-04-11 18:11:50
5,6,1,1,1725.0,4.99,2005-06-16 15:18:57.000,2019-04-11 18:11:50
Hello, I want to know how to insert this comma separated doc to postgresql.
document details are shown above.
I know timestamp need to insert like this '2019-04-11 18:11:50' . but I don't want to add '' for all of the timestamp values.
What I want to insert datatype in order, details are below.
integer, integer, smallint, smallint, integer, numeric, integer,
timestamp without time zone, timestamp without time zone
please let me know..
How can I import TIMESTAMP into POSTGRES without seconds? Coming from large CSV
https://www.youtube.com/watch?app=desktop&v=6Jf7eTkIaR4
I created simple table with a simple function, to insert some logs for the elapsed semester:
CREATE TABLE log_elapsedsemester(
sy char(9) NOT NULL,
sem char(1) NOT NULL,
date_recorded TIMESTAMP NOT NULL,
recordedby varchar(255)
);
CREATE OR REPLACE FUNCTION addelapsedsemester(p_sy char,p_sem char,p_date_recorded
TIMESTAMP,p_recordedby varchar)
returns void
AS
$$
BEGIN
insert into log_elapsedsemester (sy,sem,date_recorded,recordedby) values
(p_sy,p_sem,p_date_recorded,p_recordedby);
END
$$
LANGUAGE plpgsql;
But evertime I use
select addelapsedsemester('2019-2020','1',now(),'sample#gmail.com');
I get the error:
No function matches the given name and argument types. You might need to add explicit type casts.
If I use a simple INSERT with no function it inserts successfully:
insert into log_elapsedsemester(sy,sem,date_recorded,recordedby) values ('2020-
2021','1',now(),'sample#gmail.com');
I'm using PostgreSQL 9.5 with pgadmin III.
You need to cast to timestamp explicitly. Like:
SELECT addelapsedsemester('2019-2020', '1', now()::timestamp,'sample#gmail.com');
Or use LOCALTIMESTAMP instead of now()::timestamp (equivalent).
The function now() returns type timestamp with time zone (timestamptz), while your function takes timestamp without time zone (timestamp). The now() function produces a typed value (unlike the other untyped literals), where Postgres is more hesitant to coerce it to a different type. Function type resolution does not succeed.
The same type coercion still works for the bare INSERT command because (quoting the manual):
If the expression for any column is not of the correct data type, automatic type conversion will be attempted.
Be aware that the cast from timestamptz to timestamp depends on the current timezone setting of the session. You may want to be more explicit. Like now() AT TIME ZONE 'Europe/London'. Or use timestamptz to begin with. Then your original call without cast just works. See:
Now() without timezone
Also, you most probably do not want to use the type char, which is misleading short syntax for character(1). Use text or varchar instead. See:
Any downsides of using data type "text" for storing strings?
This table definition would make more sense:
CREATE TABLE log_elapsedsemester(
sy varchar(9) NOT NULL
, sem integer NOT NULL
, date_recorded timestamptz NOT NULL
, recordedby text
);
Or even:
sy integer NOT NULL -- 2019 can stand for '2019-2020'
Function parameters would match the column type.
I am importing a csv file into a Postgres Table. The file has the following format:
2019/12/13, 14:56:02, 3172.50, 3174.25, 3172.50, 3172.50, 1, 1, 1, 0
The table is defined as:
CREATE TABLE tablename (
date date,
time time,
v1 numeric,
v2 numeric,
v3 numeric,
v4 numeric,
v5 integer,
v6 integer,
v6 integer,
v7 integer,
PRIMARY KEY(date, time)
);
There is an issue with the time field. In some cases, milliseconds are added for precision:
14:56:02.1
14:56:02.9
14:56:02.10
Unfortunately, Postgres seems to drop the trailing zero, which causes it to mark below two values as duplicates:
14:56:02.1
14:56:02.10
ERROR: duplicate key value violates unique constraint "tablename_pkey"
DETAIL: Key (date, "time")=(2019-12-13, 14:56:02.1) already exists.
CONTEXT: COPY input_file, line 1584
Is there a way to instruct psql not to drop trailing zeroes? I tried time(4) to enforce 4 digit precision, with no difference.
Thanks!
Postgres is not doing anything wrong here. It took me a moment to realize that the issue is with the data.
.1 and .10 are equal. In the data, the timestamp was used creatively, i.e. in this case .1 means "1st record within this second" and .10 means "10th record within this second", so the millisecond component didn't make sense from timestamp's point of view.
I'm trying to get current UTC time, and insert it into PostgreSQL timestamp. But it's not working properly.
I am using the following command:
INSERT INTO public.rt_block_height
VALUES(to_timestamp('2018-09-09 00:36:00.778653', 'yyyy-mm-dd hh24:mi:ss.MS.US'), 83.7)
However, when I check the result, it looks like this:
tutorial=# select * from rt_block_height;
time | block_height
-------------------------+--------------
2018-09-09 00:48:58.653 | 83.7
(1 row)
I don't know what's causing this mismatch.
FYI, here my source code for table:
CREATE TABLE IF NOT EXISTS public.rt_BLOCK_HEIGHT
(
"time" timestamp without time zone,
BLOCK_HEIGHT double precision
)
WITH (
OIDS = FALSE
)
TABLESPACE pg_default;
ALTER TABLE public.rt_BLOCK_HEIGHT
OWNER to postgres;
SELECT create_hypertable('rt_BLOCK_HEIGHT', 'time');
There is a logical error in the format string as you should not use MS and US at the same time. However, you do not need the function at all, just cast the string to timestamp:
INSERT INTO public.rt_block_height
VALUES('2018-09-09 00:36:00.778653'::timestamp, 83.7)
From the documentation:
to_timestamp and to_date exist to handle input formats that cannot be converted by simple casting. For most standard date/time formats, simply casting the source string to the required data type works, and is much easier.
create table Department
(Dname varchar(255) NOT NULL, Dnumber int NOT NULL PRIMARY KEY, Mgr_SSN char(9) NOT NULL, Mgr_start_Date DATE);
insert into Department values('HR', '1', '11001', '2012-04-05 10:15:00');
I am getting the error "not valid month".
Should we define date format when we create the table?
I am using Oracle11g.
When you have a DATE column, you should always insert a DATE, not a VARCHAR2. Relying on implicit casting to correctly convert the string is a bad idea-- it is very easy for different sessions to have different NLS settings and, thus, to do the implicit conversion differently (either resulting in a different DATE or an error). The easiest way to do that is to use the to_date function.
insert into Department( dname,
dnumber,
mgr_ssn,
mgr_start_date )
values('HR',
1,
'11001',
to_date( '2012-04-05 10:15:00', 'yyyy-mm-dd hh24:mi:ss') );
I also modified the statement to list the columns, which is generally a good practice since it ensures that you don't have to look up the physical order of columns in the table every time and since it allows the INSERT statement to work in the future if you add new columns to the table. Since dnumber is a NUMBER, I also changed the INSERT statement to insert a number rather than inserting a string (again, don't rely on implicit conversion if there is no need to do so). I did not correct the apparent bug that you have a CHAR(9) column representing a social security number for which you are inserting a 5 character string.