In a scala program, I receive from client side a specific date for instance:
2013-10-20T23:59:59.999Z
and I really want to keep this date when saving into DB and not convert to local, so this line:
debug("--sql timestamp: " + new Timestamp(reading.timestamp.getMillis()))
is printing out: 2013-10-21 02:59:59.999(I am in Romania).
Is there any way I can ignore timezone?
This is Timestamp.toString() behavior. java.sql.Timestamp extends java.util.Date and in its toString() method it uses, in particular, super.getHours(), which, according to javadoc, returns hours interpreted in local timezone - exactly as you observe.
However, internally Timestamp still holds correct timestamp value. There may be problems with storing it to the database, though. See this answer.
2013-10-20T23:59:59.999Z and 2013-10-21 02:59:59.999 are actually the same time: 2013-10-20T23:59:59.999Z is in the UTC time zone (Z), whereas the second one is relative, and expressed as your local time zone (UTC+3 then in Romania).
In PostgreSQL, you should store your timestamps as TIMESTAMP WITH TIME ZONE (TIMESTAMPTZ) in your database to handle this. You'll always be able to print it out later in the time zone you choose then (e.g. UTC). (You might be interested in this recent question to understand why the storage type matters.)
If you want to print out the timestamp in the UTC/Z time zone again, new DateTime(millis, DateTimeZone.UTC) should help (with Joda Time).
Related
The entities in my application have a lot of Instant fields. I don't care about time zones, everything is in UTC. I am using Postgresql 13.2. Hibernate 5 maps those fields in the CREATE TABLE statements to timestamp, which Postgresql interprets as "timestamp without time zone". This is the desired behavior.
However, after upgrading the app to use Hibernate 6, the fields in the CREATE TABLE statements are now "timestamp(6) with time zone". Liquibase then generates a diff with lot of false changes.
Is there a way to tell Hibernate 6 to continue mapping Instant fields to timestamp (without time zone)?
I tried setting the timezone of the postgresql server to "Etc/UTC", it did not help. Neither did setting
spring.jpa.properties.hibernate.jdbc.time_zone=UTC
in application.properties.
First: the correct data type to use to represent a timestamp in UTC in Postgres is timestamp with time zone, which does not actually store any time zone information, it just stores timestamps normalized to UTC. From the Postgres docs:
For timestamp with time zone, the internally stored value is always in UTC (Universal Coordinated Time...). An input value that has an explicit time zone specified is converted to UTC using the appropriate offset for that time zone. If no time zone is stated in the input string, then it is assumed to be in the time zone indicated by the system's TimeZone parameter, and is converted to UTC using the offset for the timezone zone.
On the other hand, the SQL type timestamp is more like a LocalDateTime in Java, it is not by nature a UTC datetime.
Second: if you don't like the SQL column type that Hibernate uses by default, then you can of course change it, either in the JPA standard way:
#Column(columnDefinition="timestamp")
or in the IMO much better native Hibernate way:
#JdbcTypeCode(TIMESTAMP)
or:
#JdbcType(TimestampJdbcType.class)
There is even an (incubating) global setting defined by AvailableSettings.PREFERRED_INSTANT_JDBC_TYPE:
hibernate.type.preferred_instant_jdbc_type=TIMESTAMP
However, I do not recommend you use any of these settings, since, as I said, the correct SQL type to use on Postgres is the one Hibernate uses by default.
I created my PostgreSQL table with a column:
updated_at timestamp(0) with time zone
I'm using clojure.java-time but I can't create a string (or should be an object?) to set the current date-time. I tried with:
(time/format "yyyy-MM-dd HH:ss" (time/local-date-time))
and with that string I tried:
(db/update-answer! {:updated_at "2019-12-25 14:08", :id 102, :answer "Foo"}
but the JDBC tells me that the field "updated_at" doesn't have a valid type. As far as I understand the JDBC is still using the old java API for handle the dates and time zones and not the new in JDK 8. So, in summary I don't know how to create a string or an object valid for Postgresql time-stamp column using clojure.java-time.
Someone advises me about extending the JDBC protocol, I'm reading about it but for now I'm just looking for a way to create a valid object and finish this update.
UPDATED
After followed this page I could give format to Instant with:
(ns zentaur.hiccup.helpers-view
(:require [clojure.tools.logging :as log]
[java-time :as jt])
(:import [java.time ZoneId]))
(defn format-date [date]
(log/info (str ">>> DATE >>>>> " date "und type >>> " (type date)))
(jt/format "yyyy-MM-dd HH:mm" (.atZone date (ZoneId/systemDefault))))
(Java syntax, not Clojure)
tl;dr
Use objects, not Strings. Use java.time classes in Java, never the legacy date-time classes.
myPreparedStatement
.setObject(
… ,
OffsetDateTime.now()
)
Even better would be a trigger to do this automatically.
in summary I don't know how to create … an object valid for Postgresql time-stamp column using clojure.java-time.
Call OffsetDateTime.now() to get a java.time.OffsetDateTime object holding the current moment.
Use a database column of type TIMESTAMP WITH TIME ZONE rather than TIMESTAMP (short for TIMESTAMP WITHOUT TIME ZONE).
Moments
set the current date-time
If you want to track moments, specific points on the timeline, you are using the wrong data type.
You are using TIMESTAMP which is short for TIMESTAMP WITHOUT TIME ZONE. This type lacks any concept of time zone or offset-from-UTC. So this type cannot track moments. I suggest you avoid the short version of this type name in the future, to make your SQL clear. See Postgres doc.
You should be defining your column as TIMESTAMP WITH TIME ZONE. Postgres handles this type by always saving the moment as seen in UTC. Any provided time zone or offset in the input is used to adjust to UTC. Likewise, when retrieved, your value is always in UTC, an offset of zero hours-minutes-seconds.
Beware of middleware and tools that dynamically apply some time zone to the retrieved value. This clouds matters, creating the illusion that the value was stored in that time zone. The value was actually stored in UTC, always in UTC in Postgres.
Smart objects, not dumb strings
but I can't create a string (or should be an object?) to set the current date-time.
Don’t.
You should be exchanging objects between Java and Postgres, rather than mere strings. This avoids the time-zone injection problem mentioned above.
As of JDBC 4.2, you can exchange modern java.time objects with the database. Never use the legacy types such as Calendar, GregorianCalendar, java.util.Date, java.sql.Date, java.sql.Timestamp, and so on. These classes were terribly flawed, designed by people who did not understand date-time handling. They were supplanted as of the adoption of JSR 310.
Capture the current moment as an OffsetDateTime. Your JDBC driver might support Instant & ZonedDateTime but those types are optional, as seen in graphic table above. In contrast, JDBC requires support for OffsetDateTime.
OffsetDateTime odt = OffsetDateTime.now( ZoneOffset.UTC ) ;
myPreparedStatement.setObject( … , odt ) ;
The explicit use of ZoneOffset.UTC in code above is not strictly required. If omitted, your JVM’s current default offset will be applied implicitly. Your JDBC driver and/or Postgres will adjust to UTC as discussed earlier above. My own preference for the sake of debugging/logging is to specify UTC so I can see the value as it will eventually be stored in the database.
And retrieval.
OffsetDateTime odt = myResultSet.getObject( … , OffsetDateTime.class ) ;
Default value
If you are simply recording the moment whenever a row is updated, no need to do that in your SQL. I suggest you write a trigger to be called whenever a row in that table is updated. Then you are guaranteed the value will be written no matter the mechanism by which the row was updated. And less to worry about when writing your SQL statements.
I have a table that I am using to store iso dates with timezones. I realize that dates should "always" be stored as utc but I have an exception to that rule. The timestamps aren't in any way related to the server they are running on. I want to be able to store an iso date like this:
2016-03-06T01:15:52-06:00
And regardless of the time zone of the server or anything else I want the timestamp returned as:
2016-03-06T01:15:52-06:00
Currently if I insert an iso date it automatically converts it to whatever the server timezone is. My above date gets converted to:
2016-03-06 07:15:52+00 (server is utc)
The only thing I can think of is storing the timezone offset in a separate column, storing my date as utc and then converting using the offset column, horribly messy. Surely there is a way to store my date in one column and get it out the way it was originally created?
Your proposed solution is correct. Or more precisely, it is one of several correct implementations. Any of the following would work:
Store the UTC timestamp in one field, store the offset in another.
Store the local timestamp in one field, store the offset in another.
Store the local date in one field, and store a time with time zone in another. (though time with time zone is generally discouraged...)
Store the UTC timestamps in one field and the local timestamp in another.
The easiest by far is the first one, which you already proposed.
I'd avoid against storing timestamps in text fields, as they tend not to be very efficiently searchable.
Also note - if you're coming from a SQL Server background, you might recall its datetimeoffset type, which stores the local datetime and offset in the field, and uses the UTC equivalent during indexing. It's common to think that Postgres and MySQL's timestamp with time zone would have the same behavior, but they don't. They simply use the session time zone to convert to/from UTC. SQL Server has no concept of a session time zone, and thus the discrepancy.
Be sure to read this part of the Postgres docs.
We have an application that fetches data from a source and that source present the data with a timestamp in UTC. When our application saves that data to Postgres, it stores that timestamp in a timestamp column without time zone. The default on postgres in our shop is set to our local time, Mountain Time. So that means, I think, that postgres assumes that timestamp is mountain time. How can I query that column so that my result set thinks it's UTC and not the local time zone?
More cleary stated, I need to perform some offsets on that timestamp (moving it to, say EST) and so the math of doing that is different if the resultset thinks it's UTC than my local time
The Answer by Kouber Saparev is mostly correct, though incorrect about storing a time zone.
Wrong data type in Postgres
a timestamp in UTC. When our application saves that data to Postgres, it stores that timestamp in a timestamp column without time zone.
As noted in his Answer, you are using the wrong data type in your Postgres database. When tracking moments, you must use a column of type TIMESTAMP WITH TIME ZONE. When supplying an input during an insert or update, any accompanying info about time zone or offset-from-UTC is used to adjust into UTC. The accompanying zone/offset is then discarded. If you need to remember the original zone/offset, you will need to define a second column and store that info there yourself.
The other type in Postgres, and the SQL standard, is TIMESTAMP WITHOUT TIME ZONE. This type purposely lacks any concept of time zone or offset-from-UTC. So this type cannot represent moments, cannot store points on the timeline. It stores values that represent potential moments along a range of about 26-27 hours, the range of various time zones around the globe. Use this type only when you mean a date with time-of-day everywhere or anywhere, but not specifically somewhere. Also used when you mean appointments far enough out in the future that we run the risk of politicians changing the offset used in any of the time zones we care about.
Always specify time zone
default on postgres in our shop is set to our local time, Mountain Time
Never depend on the current default time zone of your host OS, the database server, or your tools such as the Java Virtual Machine. Always specify the desired/expected time zone in your code.
Tip: Generally best to work in UTC for data storage, data exchange, and most of your business logic. Adjust from UTC to a time zone only for presentation to the user or where business rules require.
As explained above, Postgres always stores date-time values either in UTC or with no zone/offset at all. Beware: Tools used between you and Postgres may apply a time zone to the UTC value retrieved from the database. While well-intentioned, this anti-feature creates the illusion that the time zone was stored when in fact only UTC was stored in TIMESTAMP WITH TIME ZONE or no zone/offset at all in TIMESTAMP WITHOUT TIME ZONE.
Be aware that any zone information accompanying input to a column of TIMESTAMP WITHOUT TIME ZONE is simply ignored, the date and time-of-day taken as-is and stored.
I need to perform some offsets on that timestamp (moving it to, say EST)
Generally best to use your database just for storage, query, and retrieval of data. For massaging the data like adjusting time zone, do such work in your application. For example, in Java use the industry-leading java.time classes, in .NET the Noda Time project (a port of the predecessor of java.time, the Joda-Time project).
Example code in Java using JDBC 4.2 or later.
LocalDateTime
For a value in a column of TIMESTAMP WITHOUT TIME ZONE we use the corresponding type in Java, LocalDateTime, lacking any concept of time zone or offset-from-UTC.
LocalDateTime ldt = myResultSet.getObject( … , LocalDateTime.class ) ; // Retrieve value from database.
String output = ldt.toString() ; // Generate text representing this date-with-time value in standard ISO 8601 format.
2018-01-23T01:23:45.123
If you know for certain that this date and time was meant for UTC but was incorrectly stored without any zone/offset info, you can apply a zone or offset to repair the damage.
OffsetDateTime odt = ldt.atOffset( ZoneOffset.UTC ); // Apply an offset-from-UTC to a `LocalDateTime` lacking such information. Determines a moment.
OffsetDateTime
For a value in a column of TIMESTAMP WITH TIME ZONE we use the corresponding type in Java, OffsetDateTime (or Instant), representing a moment in UTC.
OffsetDateTime odt = myResultSet.getObject( … , OffsetDateTime.class ) ; // Retrieve value from database.
String output = odt.toString() ; // Generate text representing this date-with-time value in standard ISO 8601 format. A `Z` on the end indicates UTC, pronounced “Zulu”.
2018-01-23T01:23:45.123Z
ZonedDateTime
To see that OffsetDateTime value set in UTC through the lens of the wall-clock time used by the people of regions within the mid-west of North America, specify a time zone such as America/Edmonton or America/Denver.
Specify a proper time zone name in the format of continent/region, such as America/Montreal, Africa/Casablanca, or Pacific/Auckland. Never use the 2-4 letter abbreviation such as EST or IST as they are not true time zones, not standardized, and not even unique(!).
ZoneId z = ZoneId.of( "America/Denver" ) ;
ZonedDateTime zdt = odt.atZoneSameInstant( z ) ;
See this code run live at IdeOne.com. We see the same moment but with a different wall-clock time.
2018-01-22T18:23:45.123-07:00[America/Denver]
Beware of tools & middleware injecting a time zone
Unfortunately, many tools and middleware will volunteer to apply some default time zone to a moment retrieved from the database. While well-intentioned, this creates the illusion of the zone having been a part of the stored data when in fact the time zone was added after storage, upon retrieval. This anti-feature creates much confusion. I wish all the tools were clear and truthful by reporting the moment in UTC, as it was stored.
If you use Java, with JDBC 4.2 and later, you can exchange java.time (JSR 310) (tutorial) objects with the database and avoid this time zone injection.
There are two data types handling timestamps in PostgreSQL - timestamp, and timestamptz (timestamp with time zone). The latter stores the time zone along with the timestamp itself.
If you are using just a timestamp without time zone, then there is no way for the result set to think whether the timestamp is UTC or not. It is just a timestamp. It is up to the client application to interpret it and give it some time zone meaning.
On the contrary, if you use timestamptz, then PostgreSQL knows the time zone of that timestamp, and then it can calculate time zone offsets properly for you.
db=# select now();
now
-------------------------------
2014-12-04 19:27:06.044703+02
(1 row)
db=# select timezone('est', now());
timezone
----------------------------
2014-12-04 12:27:06.044703
(1 row)
So, back on the problem posed. You need to make sure that first the data is imported properly and then - when needed, it is returned and displayed properly to the end user. You have two options:
Continue using timestamp
In that case both the writing app and the reading app need to know that all the timestamps in the database are UTC and calculate offsets accordingly.
Switch to timestamptz
Then the only thing that the apps need to know is their own time zone, they just have to declare it after connecting to PostgreSQL and leave the rest to the database.
For example, let's connect as a writing app and declare our time zone as UTC.
db=# create table x (data timestamptz);
CREATE TABLE
db=# set timezone='utc';
SET
db=# insert into x values (now());
INSERT 0 1
db=# select * from x;
data
-------------------------------
2014-12-04 20:02:08.692329+00
(1 row)
Now, let's say a reading app connects and is in the EST time zone.
db=# set timezone='est';
SET
db=# select * from x;
data
-------------------------------
2014-12-04 15:02:08.692329-05
(1 row)
Changing the client time zone setting changes the way all the timestamps are returned, but that's the case only if you use timestamptz - timestamp with time zone. If you cannot switch to this data type, then the application will have to take care of all this magic.
In our app we're storing datetimes that belong to many different timezones.
We decided to use the Joda LocalDateTime type - so that the user always gets literally whatever they entered in the first place. This is exactly what we need.
Internally we know which timezone the user belongs to - so when they enter a datetime we do a check like this:
dateTimeZone.isLocalDateTimeGap(localDateTime)
If that datetime does not exist in their timezone (it's in the daylight-savings gap) we display an error message that the date is not correct, thus preventing incorrect datetimes from being stored in the DB.
For storing we're using a timestamp column. Problems start when the user-entered datetime exists in their timezone but does not exist in the database timezone (Europe/Berlin). E.g. when I store LocalDateTime 2015-03-29 02:30:00 from the Europe/London timezone (this is valid - in London the gap is between 01:00 and 02:00), PostgreSQL shifts the hour by 1 and saves it as 2015-03-29 03:30:00.
What to do? Is there a way to tell PostgreSQL not do anything regarding timezones and just store datetimes literally as Joda represents them? (other than storing them as strings ;))
In PostgreSQL 7.3 and higher, timestamp is equivalent to timestamp without time zone. That data type is not time zone aware. It stores only a date and time. If you are finding it shifted, then it might be related to the code or tools you are using to store or retrieve the data.
Note that before version 7.3, timestamp was equivalent to timestamp with timezone. This is mentioned in the first note-box in the documentation here.
Postgres offers two date-time types per the SQL standard. The standard barely touches on the topic unfortunately, so the behavior described here is specific to Postgres. Other databases may behave differently.
TIMESTAMP WITHOUT TIME ZONEStores just a date and a time-of-day. Any time zone or offset-from-UTC passed is ignored.
TIMESTAMP WITH TIME ZONEFirst adjusts the passed date+time using its passed zone/offset to get a value in UTC. The passed zone/offset is then discarded after the adjustment is made; if needed, you must store that original zone/offset information in a separate column yourself.
Be aware that TIMESTAMP WITHOUT TIME ZONE does not represent an actual moment, does not store a point on the timeline. Without the context of a zone or offset, it has no real meaning. It represents a range of possible moments over a span of about 26-27 hours. Good for problems such as storing a appointment far enough out in the future that the time zone rules may be changed before its arrival. Also good for problems such as “Christmas starts after midnight on December 25 this year”, where you mean a different moment in time in each zone with each zone westward arriving later and later in succession.
When recording actual moments, specific points on the timeline, use TIMESTAMP WITH TIME ZONE.
The modern approach in Java uses the java.time classes rather than either the Joda-Time library or the troublesome old legacy date-time classes bundled with the earliest versions of Java.
TIMESTAMP WITHOUT TIME ZONE
For TIMESTAMP WITHOUT TIME ZONE, the equivalent class in java.time is LocalDateTime for a date and time-of-day without any offset or zone.
As others pointed out, some tools may dynamically apply a time zone to the retrieved value in a misguided and confusing albeit well-intentioned anti-feature. The following Java code will retrieve your true date-time value sans zone/offset.
Requires a JDBC driver compliant with JDBC 4.2 or later to directly work with java.time types.
LocalDateTime ldt = myResultSet.getObject( … , LocalDateTime.class ) ; // Retrieving a `TIMESTAMP WITHOUT TIME ZONE` value.
To insert/update database:
myPreparedStatement.setObject( … , ldt ) ; // Inserting/updating a `TIMESTAMP WITHOUT TIME ZONE` column.
TIMESTAMP WITH TIME ZONE
Your discussion of time zones suggests you are concerned with actual moments on the timeline. So you should absolutely be using TIMESTAMP WITH TIME ZONE instead of TIMESTAMP WITHOUT TIME ZONE. You should not be messing about with Daylight Saving Time (DST) gaps and such. Let java.time and Postgres do that work for you, with much better code already written and tested.
To retrieve:
Instant instant = myResultSet.getObject( … , Instant.class ) ; // Retrieving a `TIMESTAMP WITH TIME ZONE` value in UTC.
ZonedDateTime zdt = instant.atZone( ZoneId.of( "Africa/Tunis" ) ) ; // Adjusting from a UTC value to a specific time zone.
To insert/update database:
myPreparedStatement.setObject( … , zdt ) ; // Inserting/updating a `TIMESTAMP WITH TIME ZONE` column.
To retrieve from database:
Instant instant = myResultSet.getObject( … , Instant.class ) ;
E.g. when I store LocalDateTime 2015-03-29 02:30:00 from the Europe/London timezone
No, no, no. Do not work this way. You are misusing the types of both Java and Postgres.
If the user entered 2015-03-29 02:30:00 intended to represent a moment in Europe/London time zone, then parse as a LocalDateTime and immediately apply a ZoneId to get a ZonedDateTime.
To parse, replace the SPACE in the middle with a T to comply with ISO 8601 standard formatting used by default in the java.time classes.
String input = "2015-03-29 02:30:00".replace( " " , "T" ) ;
LocalDateTime ldt = LocalDateTime.parse( input ) ;
ZoneId z = ZoneId.of( "Europe/London" ) ;
ZonedDateTime zdt = ldt.atZone( z ) ;
To see that same moment in UTC, extract a Instant. The Instant class represents a moment on the timeline in UTC with a resolution of nanoseconds (up to nine (9) digits of a decimal fraction).
Instant instant = zdt.toInstant() ;
Pass the instant via JDBC for storage in the database in a TIMESTAMP WITH TIME ZONE.
myPreparedStatement.setObject( … , instant ) ;
Use objects, not strings
Note that all my code here is using java.time objects to exchange data with the database. Always use these objects rather than mere strings for exchanging date-time values.
About java.time
The java.time framework is built into Java 8 and later. These classes supplant the troublesome old legacy date-time classes such as java.util.Date, Calendar, & SimpleDateFormat.
The Joda-Time project, now in maintenance mode, advises migration to the java.time classes.
To learn more, see the Oracle Tutorial. And search Stack Overflow for many examples and explanations. Specification is JSR 310.
Where to obtain the java.time classes?
Java SE 8, Java SE 9, and later
Built-in.
Part of the standard Java API with a bundled implementation.
Java 9 adds some minor features and fixes.
Java SE 6 and Java SE 7
Much of the java.time functionality is back-ported to Java 6 & 7 in ThreeTen-Backport.
Android
The ThreeTenABP project adapts ThreeTen-Backport (mentioned above) for Android specifically.
See How to use ThreeTenABP….
The ThreeTen-Extra project extends java.time with additional classes. This project is a proving ground for possible future additions to java.time. You may find some useful classes here such as Interval, YearWeek, YearQuarter, and more.