jooq UTC localdatetime changes between save and load - postgresql

My coworker created a postgresql database using jooq. Since then, we created objects with fields and the value of LocalDateTime.now(ZoneOffset.UTC). When those get saved to this database and read again later a few hours changed in our data object:
public class PlannedInvoice
{
private UUID accountId;
private LocalDateTime billingTime;
}
The save method looks similar to this:
LocalDateTime now = LocalDateTime.now(ZoneOffset.UTC);
UUID accountId = UUID.randomUUID();
PlannedinvoiceRecord record = plannedInvoiceService.create();
record.setAccountid(accountId.toString());
record.setBillingtime(now.atOffset(ZoneOffset.UTC));
record.store();
And the read method like this:
return dsl.selectFrom(PLANNEDINVOICE)
.where(PLANNEDINVOICE.ACCOUNTID.eq(accountId.toString()))
.fetchOneInto(PlannedInvoice.class);
The database uses timestamp with time zone currently, but I would also be happy to replace it with actual LocalDateTime to avoid these problems altogether(JOOQ supports this)!
When we save a value of LocalDateTime.of(2020, Month.AUGUST, 13, 0, 0), it will be 2020-08-12 20:00:00-04 in the database. This still seems correct.
Reading the value from the database seems where things go wrong. After the read method the value of billingTime is 2020-08-12 20:00:00. It seems to me like the time zone gets ignored by fetchOneInto when reconstructing the data object.
So why is there a conversion when saving UTC values and why is there no conversion when reading those back from the database? This seems very counterintuitive to me. I would prefer to avoid any time zone conversions at all.

What worked for me was to create a temporary read object with OffsetDateTime and then converting it using withOffsetSameInstant(ZoneOffset.UTC).toLocalDateTime(). It was rather easy in the end to fix it. It was just counterintuitive from the start, that the db and/or jooq would convert the data to other time zones.
This is the new object:
public class PlannedInvoiceWithOffset
{
private UUID accountId;
private OffsetDateTime billingTime;
}
A new constructor for creating the desired data object and adjusting the time zone to UTC:
public PlannedInvoice(PlannedInvoiceWithOffset tempObject)
{
this.accountId = tempObject.getAccountId();
this.billingTime = tempObject.getBillingTime().withOffsetSameInstant(ZoneOffset.UTC).toLocalDateTime();
}
Now my read method looks like this:
public PlannedInvoice findByAccountId(UUID accountId)
{
PlannedInvoiceWithOffset temp = dsl.selectFrom(PLANNEDINVOICE)
.where(PLANNEDINVOICE.ACCOUNTID.eq(accountId.toString()))
.fetchOneInto(PlannedInvoiceWithOffset.class);
return new PlannedInvoice(temp);
}

Related

JPA Projection spring boot; repository mapping entity instant to do date

I have an entity that has a date modelled as an Instant
I have an DO object that has a date modelled as a Date
when i do the conversion myself in the constructor of the DO it works:
public class DO {
private Date someTimePoint;
public DO(Instant CreatedAt) {
this.someTimePoint = Date.from(CreatedAt);
}
}
and my repo works call:
List<DO> findBySomeField(UUID someField);
Give result.
However: The DO is generated and I do not have access to it, so the constructor is actually:
public DO(Date CreatedAt) {
So the question is:
Is there a way to have the conversion from Instant to Date done on the fly by Spring using the Projection methodology?
Reading https://docs.spring.io/spring-data/jpa/docs/current/reference/html/#projections
doesn't give me any clues...

Instant vs ZonedDateTime

I just do not quite understand which one of those two I should use for the following example:
We have an OfferEntity which has a member availableDay which is the date at which the offer is available.
Now, the table will look something like this:
CREATE TABLE IF NOT EXISTS offer (
created timestamp with time zone NOT NULL DEFAULT NOW(),
id BIGSERIAL PRIMARY KEY,
available timestamp with time zone
);
From the PostgreSQL docs we know that:
For timestamp with time zone, the internally stored value is always in UTC (Universal Coordinated Time, traditionally known as Greenwich Mean Time, GMT). An input value that has an explicit time zone specified is converted to UTC using the appropriate offset for that time zone. If no time zone is stated in the input string, then it is assumed to be in the time zone indicated by the system's TimeZone parameter, and is converted to UTC using the offset for the timezone zone.
Which means I should be fine when it comes to persisting any date/time information.
But what does this mean for my OfferEntity and the REST endpoints I define in OfferController?
#Entity
#Table(name = "offer")
public class OfferEntity {
#Column(name = "available", nullable = false)
private ZonedDateTime availableDay;
}
vs
#Entity
#Table(name = "offer")
public class OfferEntity {
#Column(name = "available", nullable = false)
private Instant availableDay;
}
From what I understood - this should not make a difference. PostgreSQL stores everything as UTC anyway so I should be able to take Instant or ZonedDateTime right? Write something -> UTC. Read it again -> still UTC.
Even the client won't be able to tell the difference:
#RequestMapping(value = "/hello", method = RequestMethod.GET)
public Object hello() {
class Hello {
public Instant instant = Instant.now();
public ZonedDateTime zonedDateTime = ZonedDateTime.now();
public ZonedDateTime viennaTime = ZonedDateTime.now(ZoneId.of("GMT+2"));
public LocalDateTime localDateTime = LocalDateTime.now();
}
return new Hello();
}
Will return:
{
"instant": "2018-10-07T15:30:08.579Z",
"zonedDateTime": "2018-10-07T15:30:08.579Z",
"viennaTime": "2018-10-07T17:30:08.579+02:00",
"localDateTime": "2018-10-07T15:30:08.579",
}
But there must be a crucial difference which I am apparently not seeing.
There are two differences I can make out. It seems to be that Spring has no problem with converting "2018-10-07T15:30:08.579Z" to an Instant object, but fails to do so if I change the type to ZonedDateTime. At least out of the box.
#RequestMapping("/places/{placeId}/offers", method = RequestMethod.GET)
public List<OfferDto> getOffers(
#PathVariable(name = "placeId") Long placeId,
#RequestParam(name = "date") ZonedDateTime date) {
return this.offerService.getOffers(placeId, date);
}
The other difference is the fact that if I use Instant I am forcing my clients to convert all their date/time strings to UTC first. So any client will have to myDate.toUTCString() first. ZonedDateTime would take anything as long as it has a time zone set but why would we care?
So which of the two is the better choice and why would I chose one over the other?
The answer in the following link explains it better than I ever could. The answer goes into all the different date/time classes in Java, as well as their relation to sql types.
What's the difference between Instant and LocalDateTime?
A short summary:
The classes Instant and ZonedDateTime (as well as OffsetDateTime)
represent the same thing: a moment in time. The difference is that
ZonedDateTime and OffsetDateTime offer extra context and functionality
about timezones or time offsets, whereas Instant has no timezone or
offset specified. This can lead to differences especially when Daylight Saving Time is involved. For instance, take the following snippet of code:
ZonedDateTime z1 = zonedDateTime.of(LocalDateTime.of(2019,10,26,6,0,0),ZoneId.of("Europe/Amsterdam"));
Instant i1 = z1.plus(1,ChronoUnit.DAYS).toInstant();
Instant i2 = z1.toInstant().plus(1,ChronoUnit.DAYS);
System.out.println(i1);
System.out.println(i2);
The result will be this:
2019-10-27T05:00:00Z
2019-10-27T04:00:00Z
The difference stems from the fact that in the Amsterdam timezone, the 27th of October has an extra hour. When we convert to Instant the timezone information is lost, so adding a day will add only 24 hours.
LocalDateTime is a different beast alltogether. It represents a date
and time without timezone information. It does not represent a
moment in Time. It is useful for writing things such as "Christmas
morning starts at december 25th 00:00:00". This is true regardless of
timezone, and as such a ZonedDateTime or Instant would not be appropriate.

How do I insert Postgres "infinity" into a Timestamp field with JOOQ?

I have a column defined like this:
expiry timestamp(0) without time zone not null
With Postgres, I can issue SQL like:
insert into my_table(expiry) values ('infinity')
I've been digging through the JOOQ doco, but couldn't find any example of dealing with this.
Can I do that with JOOQ - and what would it look like?
Additionally, is it possible using an UpdatableRecord? Is there some kind of infinity "flag" instance of Timestamp I can use?
Ok, found a way to do it directly.
MyRecord r = db.insertInto(
MY_RECORD,
MY_RECORD.ID,
MY_RECORD.CREATED,
MY_RECORD.EXPIRY
).values(
val(id),
currentTimestamp(),
val("infinity").cast(Timestamp.class)
).returning().fetchOne();
But that feels more like a workaround than the right way to do it. Casting a string to a timestamp seems a little bit round-about to me, so I wrote a CustomField to make using it and querying easier:
public class TimestampLiteral extends CustomField<Timestamp> {
public static final TimestampLiteral INFINITY =
new TimestampLiteral("'infinity'");
public static final TimestampLiteral NEGATIVE_INFINITY =
new TimestampLiteral("'-infinity'");
public static final TimestampLiteral TODAY =
new TimestampLiteral("'today'");
private String literalValue;
public TimestampLiteral(String literalValue){
super("timestamp_literal", SQLDataType.TIMESTAMP);
this.literalValue = literalValue;
}
#Override
public void accept(Context<?> context){
context.visit(delegate(context.configuration()));
}
private QueryPart delegate(Configuration configuration){
switch( configuration.dialect().family() ){
case POSTGRES:
return DSL.field(literalValue);
default:
throw new UnsupportedOperationException(
"Dialect not supported because I don't know how/if this works in other databases.");
}
}
}
Then the query is:
MyRecord r = db.insertInto(
MY_RECORD,
MY_RECORD.ID,
MY_RECORD.CREATED,
MY_RECORD.EXPIRY
).values(
val(id),
TimestampLiteral.TODAY,
TimestampLiteral.INFINITY
).returning().fetchOne();
Don't know if this is necessarily the "right" way to do this, but it seems to work for the moment.
Still interested to hear if there's a way to do this with an UpdatableRecord.
I create a java.sql.Timestamp passing org.postgresql.PGStatement.DATE_POSITIVE_INFINITY to its constructor.
create.insertInto(
MY_RECORD,
MY_RECORD.ID,
MY_RECORD.CREATED,
MY_RECORD.EXPIRY
).values(
1,
new Timestamp(System.currentTimeMillis()),
new Timestamp(PGStatement.DATE_POSITIVE_INFINITY)
).execute();

INET Nordic FIX protocols extending to nanosecond granularity timestamps

All INET Nordic FIX protocols will be enhanced by extending to nanosecond granularity timestamps on 16.oktober 2015 (see notification and section 3.1.1 in the spec).
The timestamps will look like this: 20150924-10:35:20.840117690
quickfix currently rejects messages that contain fields with this new format with the error: Incorrect data format for value
Are there any plans to support this new format? Or maybe some workaround?
You can first try modifying your data dictionary. For example if you are using fix42.xml that comes with QuickFIX, you can change the affected timestamp fields from type='UTCTIMESTAMP' to type='STRING'.
If that isn't enough, you should instead write a patch against QuickFIX in C++, which should be somewhat straightforward once you know where to patch it, which I think is UtcTimeStampConvertor, around here: https://github.com/quickfix/quickfix/blob/master/src/C%2B%2B/FieldConvertors.h#L564
I think you need to add a case 27: above case 21: near the top, because your format has six extra digits. It looks like the rest of the function doesn't care about the total field length.
Of course if you want to actually inspect the sub-millisecond precision part of these timestamps, you'll need to do more.
No plans in QF/n, but only because this is the first I've heard of this.
I'll need to write some tests to see what the repercussions are. It may be that the time/date parser just truncates the extra nano places when it converts the string to a DateTime.
I've opened an issue: https://github.com/connamara/quickfixn/issues/352
This change is as far as I know kind of breaking the fix protocol definition of timestamps but that's another story.
There is a static class in QuickFixn called DateTimeConverter under QuickFix/Fields/Converters.
To get this to work correctly you would need to add format strings in lines in that class.
Add "yyyyMMdd-HH:mm:ss.fffffff" to DATE_TIME_FORMATS and "HH:mm:ss.fffffff" to TIME_ONLY_FORMATS so that it would look like this.
/// <summary>
/// Convert DateTime to/from String
/// </summary>
public static class DateTimeConverter
{
public const string DATE_TIME_FORMAT_WITH_MILLISECONDS = "{0:yyyyMMdd-HH:mm:ss.fff}";
public const string DATE_TIME_FORMAT_WITHOUT_MILLISECONDS = "{0:yyyyMMdd-HH:mm:ss}";
public const string DATE_ONLY_FORMAT = "{0:yyyyMMdd}";
public const string TIME_ONLY_FORMAT_WITH_MILLISECONDS = "{0:HH:mm:ss.fff}";
public const string TIME_ONLY_FORMAT_WITHOUT_MILLISECONDS = "{0:HH:mm:ss}";
public static string[] DATE_TIME_FORMATS = { "yyyyMMdd-HH:mm:ss.fffffff", "yyyyMMdd-HH:mm:ss.fff", "yyyyMMdd-HH:mm:ss" };
public static string[] DATE_ONLY_FORMATS = { "yyyyMMdd" };
public static string[] TIME_ONLY_FORMATS = { "HH:mm:ss.fffffff", "HH:mm:ss.fff", "HH:mm:ss" };
public static DateTimeStyles DATE_TIME_STYLES = DateTimeStyles.AssumeUniversal | DateTimeStyles.AdjustToUniversal;
public static CultureInfo DATE_TIME_CULTURE_INFO = CultureInfo.InvariantCulture;

Why can't nlog read the current date

I'm using Nlog to write some logging to a textfile. Partial nlog.config:
<target name="file" xsi:type="File" fileName="${basedir}/MBWRunner_log.txt"
layout="${date} (${level}): ${message}
Exception: ${exception:format=Method, ToString}"/>
Lines in the logfile look like this:
0001-01-01 00:00:00 (Trace): MBWRunner started
As you can see the date and time are all 0. I have tested {longdate} and {date:format=yyyyMMddHHmmss} with the same result.
The application is a console app, run from an elevated commandline.
Any clues?
[EDIT] I have tested this on 2 machine's within the organisation with the same result. Please help!
Code used:
static Logger _logger = LogManager.GetCurrentClassLogger();
public static void Log(string message, LogLevel priority)
{
LogEventInfo eventinfo = new LogEventInfo(); ;
eventinfo.Message = message;
eventinfo.Level = priority;
Log(eventinfo);
}
static void Log(LogEventInfo logentry)
{
_logger.Log(logentry);
}
UPDATE:
#edosoft I think the problem is your use of the default constructor for LogEventInfo. If you look at the source for LogEventInfo here
https://github.com/NLog/NLog/blob/master/src/NLog/LogEventInfo.cs
You will see that using the default constructor does not populate the .TimeStamp field, so the field will probably just default to the default value for DateTime, which I assume is DateTime.MinValue. You should use one of the other constructors or one of the Create methods. Since you are setting only the Message and Level fields, I would suggest either:
var logEvent = new LogEventInfo(priority, "", message); //Second param is logger name.
Or
var logEvent = LogEventInfo.Create(priority, "", message);
From the NLog source for DateLayoutRenderer (from here) we can see that the date value that gets written as part of the logging stream is calculated like this:
protected override void Append(StringBuilder builder, LogEventInfo logEvent)
{
var ts = logEvent.TimeStamp;
if (this.UniversalTime)
{
ts = ts.ToUniversalTime();
}
builder.Append(ts.ToString(this.Format, this.Culture));
}
What is happening here is that the DateLayoutRenderer is getting the TimeStamp value from the LogEventInfo object (NLog creates one of these each time you use the Logger.Trace, Logger.Debug, Logger.Info, etc methods. You can also create LogEventInfo objects yourself and log them with the Logger.Log method).
By default, when a LogEventInfo object is created, its TimeStamp field is set like this (from the source for LogEventInfo here) (note the use of CurrentTimeGetter.Now):
public LogEventInfo(LogLevel level, string loggerName, IFormatProvider formatProvider, [Localizable(false)] string message, object[] parameters, Exception exception)
{
this.TimeStamp = CurrentTimeGetter.Now;
this.Level = level;
this.LoggerName = loggerName;
this.Message = message;
this.Parameters = parameters;
this.FormatProvider = formatProvider;
this.Exception = exception;
this.SequenceID = Interlocked.Increment(ref globalSequenceId);
if (NeedToPreformatMessage(parameters))
{
this.CalcFormattedMessage();
}
}
The TimeStamp field is set in the LogEventInfo constructor using the TimeSource.Current.Now property, whose implementation can be seen here.
(UPDATE - At some point NLog changed from using CurrentTimeGetter to a more generic approach of having a TimeSource object that has several flavors (one of which, CachedTimeSource, is essentially the same as CurrentTimeGetter)).
To save the trouble of navigating the link, here is the source for CachedTimeSource:
public abstract class CachedTimeSource : TimeSource
{
private int lastTicks = -1;
private DateTime lastTime = DateTime.MinValue;
/// <summary>
/// Gets raw uncached time from derived time source.
/// </summary>
protected abstract DateTime FreshTime { get; }
/// <summary>
/// Gets current time cached for one system tick (15.6 milliseconds).
/// </summary>
public override DateTime Time
{
get
{
int tickCount = Environment.TickCount;
if (tickCount == lastTicks)
return lastTime;
else
{
DateTime time = FreshTime;
lastTicks = tickCount;
lastTime = time;
return time;
}
}
}
}
The purpose of this class is to use a relatively cheap operation (Environment.Ticks) to limit access to a relatively expensive operation (DateTime.Now). If the value of Ticks does not change from call to call (from one logged message to the next), then the value of DateTime.Now retrieved the this time will be the same as the value of DateTime.Now retrieved this time, so just use the last retrieved value.
With all of this code in play (and with Date/Time logging apparently working for most other people), one possible explanation of your problem is that you are using the Logger.Log method to log your messages and you are building the LogEventInfo objects yourself. By default, if you just new a LogEventInfo object, the automatic setting of the TimeStamp property should work fine. It is only dependent on Environment.Ticks, DateTime.Now, and the logic that reuses the last DateTime.Now value, if appropriate.
Is it possible that you are creating a LogEventInfo object and then setting its TimeStamp property to DateTime.MinValue? I ask because the date that is being logged is DateTime.MinValue.
The only other explanation that I can think of would be if Environment.Ticks returns -1 for some reason. If it did, then CurrentTimeGetter would always return the initial value of the lastDateTime private member variable. I can't imagine a scenario where Environment.Ticks would return -1.