Timezone and POSIXct handling in RpostgreSQL - postgresql

I'm having an issue with datetime handling in RPostgreSQL. Specifically it relates to POSIXct objects with a UTC timezone being automatically adjusted to daylight saving during upload to a postgres database. A simple example:
library(RPostgreSQL)
example = data.frame(date=as.POSIXct('2016-08-14 15:50:00',tz='UTC'))
con = dbConnect(dbDriver("PostgreSQL"),
dbname="mydb",
host="localhost",
port="5432",
user="me",
password="password")
dbWriteTable(con,name=c('myschema','mytable'),example,overwrite=T)
example2 = dbReadTable(con,name=c('myschema','mytable'))
dbDisconnect(con)
example2 # 2016-08-14 14:50:00
In this case the time is exported as 15:50 but read back in as 14:50, suggesting that British Summer Time daylight saving has been applied. I've tried adjusting my system settings to UTC, setting the timezone in R to UTC using Sys.setenv(TZ='UTC') and setting the timezone in Postgres to UTC using SET timezone TO 'UTC', all to no avail.
Does anybody know where in the process the conversion is likely to be happening and where dbWriteTable is taking its timezone from? Are there any suggestions on other settings that might need adjusting?

I also get strange issues with RPostgreSQL (with UTC somehow being UTC -4:00). But things seem fine using RPostgres.
Note that the time zone displayed in R is in local time. If you go into PostgreSQL (say, psql) after running the R code and SET TIME ZONE 'GMT';, you see that the 2016-08-14 16:50:00 displayed in R is actually stored in the database as 2016-08-14 15:50:00 UTC. In other words, 2016-08-14 16:50:00 displayed in R is correct for rubbish_alt in my example.
crsp=# SET TIME ZONE 'GMT';
SET
crsp=# SELECT * FROM rubbish;
row.names | date
-----------+------------------------
1 | 2016-08-14 19:50:00+00
(1 row)
crsp=# SELECT * FROM rubbish_alt;
date
------------------------
2016-08-14 15:50:00+00
(1 row)
crsp=# \d rubbish
Table "public.rubbish"
Column | Type | Modifiers
-----------+--------------------------+-----------
row.names | text |
date | timestamp with time zone |
crsp=# \d rubbish_alt
Table "public.rubbish_alt"
Column | Type | Modifiers
--------+--------------------------+-----------
date | timestamp with time zone |
R code (note using Sys.setenv(PGHOST="myhost", PGDATABASE="mydb"), etc, elsewhere makes this reprex()-generated code run for anyone):
Sys.setenv(TZ='Europe/London')
# With RPostgreSQL ----
library(RPostgreSQL)
#> Loading required package: DBI
example <- data.frame(date=as.POSIXct('2016-08-14 15:50:00', tz='UTC'))
con = dbConnect(PostgreSQL())
dbWriteTable(con, 'rubbish', example, overwrite=TRUE)
#> [1] TRUE
example2 <- dbReadTable(con, name="rubbish")
dbDisconnect(con)
#> [1] TRUE
example2
#> date
#> 1 2016-08-14 20:50:00
# With RPostgres ----
library(RPostgres)
example <- data.frame(date=as.POSIXct('2016-08-14 15:50:00', tz='UTC'))
con = dbConnect(Postgres())
dbWriteTable(con, 'rubbish_alt', example, overwrite=TRUE)
example2 <- dbReadTable(con, name="rubbish_alt")
dbDisconnect(con)
example2
#> date
#> 1 2016-08-14 16:50:00
example2$date[1]
#> [1] "2016-08-14 16:50:00 BST"

Related

Create a new Date format

I have a dataset containing a date field in the format of MM/dd/yyyy, my goal is to create a table with the same date format but with timestamp format (at the end, there should be an option to execute date functions on that, if it is int or string date function will not work.)
Things I tried:
my column name: as_of_date
1) cast(unix_timestamp(as_of_date, "MM/dd/yyyy") as timestamp)
i/p -> 01/03/2006, o/p ->2006-01-03 00:00:00
problem - I do not want extra zeros in the output. substr is not working on the date function
2) If i keep the value as string, date functions does't work.
day('01/03/2006')
input: '01/03/2006' , output:null (but expected 3)
Can you please help me a date format that already existing or help me to create a new date format for my logic.
Try with this once
use unix_timestamp function to match your input date format then use from_unixtime function to change the output format then cast to date type.
hive> select date(from_unixtime(unix_timestamp("03/06/2018", "MM/dd/yyyy"),"yyyy-MM-dd"));
+-------------+--+
| _c0 |
+-------------+--+
| 2018-03-06 |
+-------------+--+
In Impala:
hive> select from_unixtime(unix_timestamp("03/06/2018", "MM/dd/yyyy"),"yyyy-MM-dd");
+-------------+--+
| _c0 |
+-------------+--+
| 2018-03-06 |
+-------------+--+

Strange date format in database

I have some very strange looking 18 character alphanumeric datetimes in a SQL database, they seem to be using Hexadecimal?
I can find out what the dates are through the application which uses them, but I was looking for a way to convert them via a query. Do you know how I would convert these with TSQL?
000B3E4Bh01F2D947h - 29/05/2018 09:04:52
000B3E0Dh03A16C1Eh - 23/05/2018 10:22:26
000B3E4Eh0248C3D8h - 01/06/2018 10:38:43
000B3E4Eh0249B449h - 01/06/2018 10:39:44
I assume the date and time are separated as below, but I don't know to convert the individual parts if anyone can help with this? Thanks!!
000B3E4Eh (date) - 0249B449h (time)
(The dates are in dd/mm/yyyy format)
Your hex values are separated as you have assumed (with the h used as a delimiter) and represent integer values to add to a baseline date and time value.
Using your 000B3E54h0221CBFEh - 07/06/2018 09:56:09 value this translates as:
Date portion: 000B3E54
Integer Value: 736852
Time portion: 0221CBFE
Integer Value: 35769342
These integer values are then added as days to the date 0001/01/00 (which SQL Server can't handle, hence the +/-1 below) and milliseconds to 00:00:00 respectively, which you can see working in this script:
select convert(int, 0x000B3E54) as DateIntValue
,dateadd(day,convert(int, 0x000B3E54)-1,cast('00010101' as datetime2)) as DateValue
,convert(int, 0x0221CBFE) as TimeIntValue
,cast(dateadd(millisecond,convert(int, 0x0221CBFE),cast('19000101' as datetime2)) as time) as TimeValue
,cast(datediff(day,cast('00010101' as datetime2),'20180607')+1 as binary(4)) as DateHexValue
,cast(datediff(millisecond,cast('20180607' as date),cast('2018-06-07 09:56:09.342' as datetime2)) as binary(4)) as TimeHexValue
Which outputs:
+--------------+-----------------------------+--------------+------------------+--------------+--------------+
| DateIntValue | DateValue | TimeIntValue | TimeValue | DateHexValue | TimeHexValue |
+--------------+-----------------------------+--------------+------------------+--------------+--------------+
| 736852 | 2018-06-07 00:00:00.0000000 | 35769342 | 09:56:09.3420000 | 0x000B3E54 | 0x0221CBFE |
+--------------+-----------------------------+--------------+------------------+--------------+--------------+
Note the judicious use of datetime2 values to ensure the right amount of milliseconds are output/returned, as SQL Server datetime is only accurate to the nearest 3 milliseconds.

Producing date from year and month values in PostgreSQL

Hello I'm having two problems with converting a concatenated date value into an actual date.
I've tired looking here to convert the concatenated value with to_char(DATE ...) but I keep getting odd dates. I think it is because my month does not have a zero padding in front of it.
This is my base query:
SELECT
expiry_month,
expiry_year,
to_date(CONCAT(expiry_year, expiry_month), 'YYYY/MM'),
FROM thisTable
Here is an example of the data output:
expiry_month expiry_year concatvalues
9 2018 20189-01-01
1 2019 20191-01-01
5 2016 20165-01-01
3 2019 20193-01-01
10 2017 201710-01-01
2 2020 20202-01-01
I think I need to LPAD() my month value to get the correct date parsed. E.g. 01 not 1, and 05 not 5.
However when I try to LPAD the month values it does not work. I've tried:
lpad(to_char(expiry_month),2,'0'),
I get this error 'HINT: No function matches the given name and argument types. You might need to add explicit type casts.'
Which I don't understand because lpad is a function. Any suggestion on how to use LPAD()?
Thank you for the advice.
EDIT 1
I've tried to update the to_date() function with this code:
to_date(CONCAT(payment_cards.expiry_year || ' - ' || payment_cards.expiry_month || ' - 01'), 'YYYY-MM-01') and now it is throwing a different error:
ERROR: invalid value "- " for "MM" DETAIL: Value must be an integer.
I'm still thinking I need to pad the month date?
There's a '/' missing:
SELECT
expiry_month,
expiry_year,
to_date(CONCAT(expiry_year, '/', expiry_month), 'YYYY/MM') AS the_start_of_year_month
FROM thisTable ;
will produce:
expiry_month | expiry_year | the_start_of_year_month
-----------: | ----------: | :----------------------
9 | 2018 | 2018-09-01
1 | 2019 | 2019-01-01
5 | 2016 | 2016-05-01
3 | 2019 | 2019-03-01
10 | 2017 | 2017-10-01
2 | 2020 | 2020-02-01
The date format is specifying '/' and it wasn't there, so, the whole text was taken as the year, and the month and day were taken as 1/1. CONCAT('2018','9') was just returning '20189' (which is a valid year).
dbfiddle here
Use:
make_date(year int, month int, day int)
like:
make_date(expiry_year, expiry_month, 1)
Postgresql documentation

RR MILLENNIUM equivalent in Postgres

Is there a built in function in PostgreSQL 9.5 version to calculate the appropriate century/millenium?
When I use birth_date::TIMESTAMP from a table, sometimes it prefix 19 and sometimes it prefix 20. Below example
Input:
28JUN80
25APR48
Output:
"1980-06-28 00:00:00"
"2048-04-25 00:00:00"
I also have records in the table with birth_date holding values like "07APR1963" which gets computed appropriately as "1963-04-07 00:00:00".
I need use CASE statement when the length is 7 characters, then prefix with 19 millennium and when its 9 characters, just load it as it is.
https://en.wikipedia.org/wiki/Unix_time Unix epoch is
beginning (00:00:00 1 January 1970)
So if you don't specify the century, but just last YY it will be 20th century from 00:00:00 1 January and 21st century before YY equal 70. If you want it to guess the 20th century either append year as you do, or specify CC, eg:
t=> select
to_timestamp('1JAN70', 'ddmonYY')
, to_timestamp('31DEC69', 'ddmonyy')
, to_timestamp('31DEC69 20', 'ddmonyy cc');
to_timestamp | to_timestamp | to_timestamp
------------------------+------------------------+------------------------
1970-01-01 00:00:00+00 | 2069-12-31 00:00:00+00 | 1969-12-31 00:00:00+00
(1 row)
https://www.postgresql.org/docs/current/static/functions-formatting.html
In conversions from string to timestamp or date, the CC (century)
field is ignored if there is a YYY, YYYY or Y,YYY field. If CC is used
with YY or Y then the year is computed as the year in the specified
century. If the century is specified but the year is not, the first
year of the century is assumed.
update
So in your case you should do smth like:
vao=# create table arasu (member_birth_date character(9)); insert into arasu values ('28JUN80'),('25APR48');
CREATE TABLE
INSERT 0 2
vao=# select to_timestamp(member_birth_date||' 20', 'ddmonYY cc') from arasu;
to_timestamp
------------------------
1980-06-28 00:00:00+03
1948-04-25 00:00:00+03
(2 rows)

to_char returning wrong date postgresql

So in my database i have a table with startimestamp that is "Timestamp with time zone" type
but what i want to display the timestamp without the time zone so i thought this would work
Select to_char("StartTimestamp",'YYYY/MM/DD HH24:MM:SS')from "Samples" where "ID" = 20
and a i get
"2013/08/02 14:08:04"
but the when i do it without the to_char and just call the timestamp for the same id like this
select "StartTimestamp" from "Samples" where "ID"=20
i get this which is the correct one
"2013-08-02 14:31:04-07"
I'm i missing something the to_char statement? Thanks
Change MM to MI in the minutes place
select
to_char(now(),'YYYY/MM/DD HH24:MM:SS'),
to_char(now(),'YYYY/MM/DD HH24:MI:SS');
to_char | to_char
---------------------+---------------------
2013/08/21 15:08:00 | 2013/08/21 15:51:00