In the datawarehouse we need to capture date and time value with timezone as received from source application DB. But Postgresql is converting those values into native timezone every time. Is there a way to maintain source provided timestamp with timezone value in Postgresql database as is?
E.g. from source DB if we get the timezone value as "01/20/2010 11:15:33.000000 -06:00" , want to store this value as it is in warehouse which is at different timezone. Is there any way to do that in Postgresql?
There is no native way to store the source time zone, all timestamps with time zone are stored as UTC. You can extract the time zone and store it in a separate column, and then use a view to provide the data at the time zone you want.
Related
The entities in my application have a lot of Instant fields. I don't care about time zones, everything is in UTC. I am using Postgresql 13.2. Hibernate 5 maps those fields in the CREATE TABLE statements to timestamp, which Postgresql interprets as "timestamp without time zone". This is the desired behavior.
However, after upgrading the app to use Hibernate 6, the fields in the CREATE TABLE statements are now "timestamp(6) with time zone". Liquibase then generates a diff with lot of false changes.
Is there a way to tell Hibernate 6 to continue mapping Instant fields to timestamp (without time zone)?
I tried setting the timezone of the postgresql server to "Etc/UTC", it did not help. Neither did setting
spring.jpa.properties.hibernate.jdbc.time_zone=UTC
in application.properties.
First: the correct data type to use to represent a timestamp in UTC in Postgres is timestamp with time zone, which does not actually store any time zone information, it just stores timestamps normalized to UTC. From the Postgres docs:
For timestamp with time zone, the internally stored value is always in UTC (Universal Coordinated Time...). An input value that has an explicit time zone specified is converted to UTC using the appropriate offset for that time zone. If no time zone is stated in the input string, then it is assumed to be in the time zone indicated by the system's TimeZone parameter, and is converted to UTC using the offset for the timezone zone.
On the other hand, the SQL type timestamp is more like a LocalDateTime in Java, it is not by nature a UTC datetime.
Second: if you don't like the SQL column type that Hibernate uses by default, then you can of course change it, either in the JPA standard way:
#Column(columnDefinition="timestamp")
or in the IMO much better native Hibernate way:
#JdbcTypeCode(TIMESTAMP)
or:
#JdbcType(TimestampJdbcType.class)
There is even an (incubating) global setting defined by AvailableSettings.PREFERRED_INSTANT_JDBC_TYPE:
hibernate.type.preferred_instant_jdbc_type=TIMESTAMP
However, I do not recommend you use any of these settings, since, as I said, the correct SQL type to use on Postgres is the one Hibernate uses by default.
Let's say that I need to create 'chat-like' system. Of course I need to deal with dates somehow.
I read about it a lot and I have some knowledge but I really don't know how to use it.
I would like to store message date in UTC (postgres 12)
Each user should be able to select his time zone and this time zone should be saved into database (standard approach)
When message is retrieved from database I need to convert message date into valid local date based on user selected timezone.
This is really all I need to do and here problem starts:
In postgres date is stored with offset f.e 2020-05-01 00:00:00+02, but I want to store timezone in another table, not here
How can I store user timezone? I should use names like "EST5EDT" or use time offsets as integer?
Where can I find list of all timezones to present user? (Each global website f.e. facebook has list of timezones with offsets, where can I find list of all valid timezones?)
How can I select date with user appropriate timezone? f.e.
SELECT convert_to_user_date("Date", "timezonename??")
FROM "Messages"
Is this correct way to achieve my goal?
These days you don't have to resort to UTC. You can store full timestamps with time zone. This way e.g. you won't lose DST status at the moment the timestamp was recorded in the database.
https://www.postgresql.org/docs/current/datatype-datetime.html
You can easily select the timestamp stored with any time zone shifted to target user's time zone (assuming it's stored somewhere in user preferences). The syntax is SELECT ... AT TIME ZONE
https://www.postgresql.org/docs/current/functions-datetime.html#FUNCTIONS-DATETIME-ZONECONVERT
You are very close to a workable setup.
First, use timestamp or timestamp without timezone column data type to store those UTC date/time stamps.
Second, store your users' preferred time zones in varchar(63) columns in the form Asia/Kolkata or America/Halifax.
Third, use postgresql's built in view pg_timezone_names to get a list of valid time zones. You can use it to populate a pulldown list of choices in your user-settings screen.
If you have time for some real excellence in your user-settings screen, you can suggest time zone settings you guess from the users' ip adresses and allow them to change them if your guess was wrong. Read this. How to get Time Zone through IP Address in PHP
Then, when your application starts using postgresql on behalf of any user, look up that user's chosen time zone in your users table, and use it in this SQL command. SET TIME ZONE 'America/Halifax'; (whatever the user's choice is).
Then when you retrieve your time stamps, they will be rendered in the user's local time, and when you store the they'll be in UTC.
The 'toobz are full of advice about this. Here's something that might be useful. How to get Time Zone through IP Address in PHP
Use the data type timestamp with time zone. Don't be worried by the name — that data type really represents an absolute point of time and does not store time zone information.
The only thing you have to do is to set the timezone parameter correctly for the time zone of the client connection, then the value will be represented correctly in that time zone. PostgreSQL does all the work for you.
If you don't like the string representation (e.g., you are disturbed by the time zone offset displayed), use to_char to format the output the way you like:
CREATE TABLE dates (x timestamp with time zone NOT NULL);
SET timezone = 'Europe/Vienna';
INSERT INTO dates VALUES ('2020-06-01 03:00:00');
SET timezone = 'Asia/Kolkata';
SELECT to_char(x, 'YYYY-MM-DD HH24:MI:SS') FROM dates;
to_char
---------------------
2020-06-01 06:30:00
(1 row)
I have a table that I am using to store iso dates with timezones. I realize that dates should "always" be stored as utc but I have an exception to that rule. The timestamps aren't in any way related to the server they are running on. I want to be able to store an iso date like this:
2016-03-06T01:15:52-06:00
And regardless of the time zone of the server or anything else I want the timestamp returned as:
2016-03-06T01:15:52-06:00
Currently if I insert an iso date it automatically converts it to whatever the server timezone is. My above date gets converted to:
2016-03-06 07:15:52+00 (server is utc)
The only thing I can think of is storing the timezone offset in a separate column, storing my date as utc and then converting using the offset column, horribly messy. Surely there is a way to store my date in one column and get it out the way it was originally created?
Your proposed solution is correct. Or more precisely, it is one of several correct implementations. Any of the following would work:
Store the UTC timestamp in one field, store the offset in another.
Store the local timestamp in one field, store the offset in another.
Store the local date in one field, and store a time with time zone in another. (though time with time zone is generally discouraged...)
Store the UTC timestamps in one field and the local timestamp in another.
The easiest by far is the first one, which you already proposed.
I'd avoid against storing timestamps in text fields, as they tend not to be very efficiently searchable.
Also note - if you're coming from a SQL Server background, you might recall its datetimeoffset type, which stores the local datetime and offset in the field, and uses the UTC equivalent during indexing. It's common to think that Postgres and MySQL's timestamp with time zone would have the same behavior, but they don't. They simply use the session time zone to convert to/from UTC. SQL Server has no concept of a session time zone, and thus the discrepancy.
Be sure to read this part of the Postgres docs.
PostgreSQL 9.3 / postgresql-9.3-1100-jdbc41.jar
I have a table with a column of type timestamp without time zone, this generates my Object with the applicable java.util.Timestamp property.
What I'm seeing, during insert, is jOOQ's binding process converting a java.util.Timestamp into a date with local timezone offset.
eg for a unix timestamp 1421109419 (13 Jan 2015 00:36:59 GMT) the property is set with new Timestamp(1421109419 * 1000).
from the jOOQ logger I see:
2015-01-13 14:14:31,482 DEBUG [http-bio-8002-exec-4] org.jooq.tools.LoggerListener#debug:255 - -> with bind values : insert into "foo"."bar" ("start_date") values (timestamp '2015-01-13 13:36:59.0') returning "foo"."bar"."id"
2015-01-13 14:14:31,483 TRACE [http-bio-8002-exec-4] org.jooq.impl.DefaultBinding#trace:179 - Binding variable 3 : 2015-01-13 13:36:59.0 (class java.sql.Timestamp)
and sure enough in the record is the value "2015-01-13 13:36:59".
The software is running on a machine in NZDT which explains the +13 offset.
Given the time is being supplied in a TimeZone agnostic container (Timestamp) I would have expected that to be honoured when creating the insert statement.
How can I have jOOQ create timestamps NOT in local time?
Unfortunately you have a few things working against you:
The PostgreSQL JDBC driver sets the timezone to your JVM timezone in the Postgres session. So even if your Database Server is running in UTC a TIMESTAMP field will be inserted using the time zone of your JVM. When you insert or query data the database server will always use the JVM time zone.
You are using TIMESTAMP instead of TIMESTAMPTZ. The description of these types do not reflect their actually usage. TIMESTAMPTZ actually means time zone agnostic. Whatever value you insert it will be adjusted to UTC using the session timezone.
Because of these two issues, if you have two different JVMs -- one using Los Angeles time and the other using New York time -- whenever you write a TIMESTAMP with one JVM it will be a different "UTC time" in the other JVM. TIMESTAMP takes the adjusted value and just uses it as given. If you change your TIMESTAMP columns to be TIMESTAMPTZ then the same time in both JVMs will always be the same UTC time.
If you look at the Postgres JDBC Driver's ConnectionFactoryImpl#openConnectionImp you can see where it sets your local JVM's time zone as the time zone for the database server's session zone.
So the only sane way to deal with this is to only ever use TIMESTAMPTZ instead of TIMESTAMP. Here's some more information on this:
PostgreSQL/JDBC and TIMESTAMP vs. TIMESTAMPTZ
http://justatheory.com/computers/databases/postgresql/use-timestamptz.html
The following (very nasty) code works for me:
eventsRecord.setCreatedOn(new Timestamp(System.currentTimeMillis()
- TimeZone.getDefault().getOffset(new Date().getTime())));
Alas jOOQ simply uses the local timezone when saving into PostgreSQL "timestamp without timezone" or MySQL "datetime" fields. The source code evidence for this travesty is here, it does not specify the timezone nor have any facility for the user to override this functionality and specify a timezone. This renders usage of this very basic datatype from jOOQ completely useless, with a multitude of clients all with different timezones writing data to the same field without recording their timezone nor normalizing the data to UTC.
JDBC provides an extra three-argument setTimestamp where the user can specify what timezone is desired (UTC is basically the only value that makes sense). However jOOQ "abstracts" away from JDBC and does not offer this facility.
We have an application that fetches data from a source and that source present the data with a timestamp in UTC. When our application saves that data to Postgres, it stores that timestamp in a timestamp column without time zone. The default on postgres in our shop is set to our local time, Mountain Time. So that means, I think, that postgres assumes that timestamp is mountain time. How can I query that column so that my result set thinks it's UTC and not the local time zone?
More cleary stated, I need to perform some offsets on that timestamp (moving it to, say EST) and so the math of doing that is different if the resultset thinks it's UTC than my local time
The Answer by Kouber Saparev is mostly correct, though incorrect about storing a time zone.
Wrong data type in Postgres
a timestamp in UTC. When our application saves that data to Postgres, it stores that timestamp in a timestamp column without time zone.
As noted in his Answer, you are using the wrong data type in your Postgres database. When tracking moments, you must use a column of type TIMESTAMP WITH TIME ZONE. When supplying an input during an insert or update, any accompanying info about time zone or offset-from-UTC is used to adjust into UTC. The accompanying zone/offset is then discarded. If you need to remember the original zone/offset, you will need to define a second column and store that info there yourself.
The other type in Postgres, and the SQL standard, is TIMESTAMP WITHOUT TIME ZONE. This type purposely lacks any concept of time zone or offset-from-UTC. So this type cannot represent moments, cannot store points on the timeline. It stores values that represent potential moments along a range of about 26-27 hours, the range of various time zones around the globe. Use this type only when you mean a date with time-of-day everywhere or anywhere, but not specifically somewhere. Also used when you mean appointments far enough out in the future that we run the risk of politicians changing the offset used in any of the time zones we care about.
Always specify time zone
default on postgres in our shop is set to our local time, Mountain Time
Never depend on the current default time zone of your host OS, the database server, or your tools such as the Java Virtual Machine. Always specify the desired/expected time zone in your code.
Tip: Generally best to work in UTC for data storage, data exchange, and most of your business logic. Adjust from UTC to a time zone only for presentation to the user or where business rules require.
As explained above, Postgres always stores date-time values either in UTC or with no zone/offset at all. Beware: Tools used between you and Postgres may apply a time zone to the UTC value retrieved from the database. While well-intentioned, this anti-feature creates the illusion that the time zone was stored when in fact only UTC was stored in TIMESTAMP WITH TIME ZONE or no zone/offset at all in TIMESTAMP WITHOUT TIME ZONE.
Be aware that any zone information accompanying input to a column of TIMESTAMP WITHOUT TIME ZONE is simply ignored, the date and time-of-day taken as-is and stored.
I need to perform some offsets on that timestamp (moving it to, say EST)
Generally best to use your database just for storage, query, and retrieval of data. For massaging the data like adjusting time zone, do such work in your application. For example, in Java use the industry-leading java.time classes, in .NET the Noda Time project (a port of the predecessor of java.time, the Joda-Time project).
Example code in Java using JDBC 4.2 or later.
LocalDateTime
For a value in a column of TIMESTAMP WITHOUT TIME ZONE we use the corresponding type in Java, LocalDateTime, lacking any concept of time zone or offset-from-UTC.
LocalDateTime ldt = myResultSet.getObject( … , LocalDateTime.class ) ; // Retrieve value from database.
String output = ldt.toString() ; // Generate text representing this date-with-time value in standard ISO 8601 format.
2018-01-23T01:23:45.123
If you know for certain that this date and time was meant for UTC but was incorrectly stored without any zone/offset info, you can apply a zone or offset to repair the damage.
OffsetDateTime odt = ldt.atOffset( ZoneOffset.UTC ); // Apply an offset-from-UTC to a `LocalDateTime` lacking such information. Determines a moment.
OffsetDateTime
For a value in a column of TIMESTAMP WITH TIME ZONE we use the corresponding type in Java, OffsetDateTime (or Instant), representing a moment in UTC.
OffsetDateTime odt = myResultSet.getObject( … , OffsetDateTime.class ) ; // Retrieve value from database.
String output = odt.toString() ; // Generate text representing this date-with-time value in standard ISO 8601 format. A `Z` on the end indicates UTC, pronounced “Zulu”.
2018-01-23T01:23:45.123Z
ZonedDateTime
To see that OffsetDateTime value set in UTC through the lens of the wall-clock time used by the people of regions within the mid-west of North America, specify a time zone such as America/Edmonton or America/Denver.
Specify a proper time zone name in the format of continent/region, such as America/Montreal, Africa/Casablanca, or Pacific/Auckland. Never use the 2-4 letter abbreviation such as EST or IST as they are not true time zones, not standardized, and not even unique(!).
ZoneId z = ZoneId.of( "America/Denver" ) ;
ZonedDateTime zdt = odt.atZoneSameInstant( z ) ;
See this code run live at IdeOne.com. We see the same moment but with a different wall-clock time.
2018-01-22T18:23:45.123-07:00[America/Denver]
Beware of tools & middleware injecting a time zone
Unfortunately, many tools and middleware will volunteer to apply some default time zone to a moment retrieved from the database. While well-intentioned, this creates the illusion of the zone having been a part of the stored data when in fact the time zone was added after storage, upon retrieval. This anti-feature creates much confusion. I wish all the tools were clear and truthful by reporting the moment in UTC, as it was stored.
If you use Java, with JDBC 4.2 and later, you can exchange java.time (JSR 310) (tutorial) objects with the database and avoid this time zone injection.
There are two data types handling timestamps in PostgreSQL - timestamp, and timestamptz (timestamp with time zone). The latter stores the time zone along with the timestamp itself.
If you are using just a timestamp without time zone, then there is no way for the result set to think whether the timestamp is UTC or not. It is just a timestamp. It is up to the client application to interpret it and give it some time zone meaning.
On the contrary, if you use timestamptz, then PostgreSQL knows the time zone of that timestamp, and then it can calculate time zone offsets properly for you.
db=# select now();
now
-------------------------------
2014-12-04 19:27:06.044703+02
(1 row)
db=# select timezone('est', now());
timezone
----------------------------
2014-12-04 12:27:06.044703
(1 row)
So, back on the problem posed. You need to make sure that first the data is imported properly and then - when needed, it is returned and displayed properly to the end user. You have two options:
Continue using timestamp
In that case both the writing app and the reading app need to know that all the timestamps in the database are UTC and calculate offsets accordingly.
Switch to timestamptz
Then the only thing that the apps need to know is their own time zone, they just have to declare it after connecting to PostgreSQL and leave the rest to the database.
For example, let's connect as a writing app and declare our time zone as UTC.
db=# create table x (data timestamptz);
CREATE TABLE
db=# set timezone='utc';
SET
db=# insert into x values (now());
INSERT 0 1
db=# select * from x;
data
-------------------------------
2014-12-04 20:02:08.692329+00
(1 row)
Now, let's say a reading app connects and is in the EST time zone.
db=# set timezone='est';
SET
db=# select * from x;
data
-------------------------------
2014-12-04 15:02:08.692329-05
(1 row)
Changing the client time zone setting changes the way all the timestamps are returned, but that's the case only if you use timestamptz - timestamp with time zone. If you cannot switch to this data type, then the application will have to take care of all this magic.