how to handle user timezone for daylight savings in quartz for cron Triggers? - quartz-scheduler

My service api takes in startDate for quartz Job and day of Month for the job to executed.
Internally, I convert this to cron expression and save in quartz.
For example, a user in PST submits a job request today (Nov 3 2017) as follows.
{
"start": "2017-11-03T18:00:00-07:00",
"dayOfMonth" : 15
}
Here the user wants to schedule a job that fires on 15th of each month at 6 PM, starting from 2017-11-03. so the first-day quartz will fire will be 2017-11-15.
This is how the above request gets converted to cron expression 0 0 18 15 * ? *, which is correct.
Here is how, QRTZ_CRON_TRIGGERS table looks like.
As you notice, time_zone_id is saved as GMT-07:00, but once daylight savings kick-in on Nov 5, it must be GMT-08:00. or else my quartz job will fire one hour earlier. In fact, when I query for nextFireTime, I do get 1510794000000 which is indeed Wednesday November 15, 2017 17:00:00 (pm) in time zone America/Los Angeles (PST)
how do we handle this time_zone_id issue?
P.S: I am using cronTrigger which does not have the notion of preserveHourOfDayAcrossDaylightSavings that is provided by CalendarIntervalTrigger.

Do not use offset to represent the timezone. Rather you can ask the user to pass in timezone like "America/Los_Angeles". Then you can use http://www.quartz-scheduler.org/api/2.2.1/org/quartz/CronScheduleBuilder.html#inTimeZone(java.util.TimeZone) to create trigger with proper timezone.
inTimeZone(TimeZone.getTimeZone("USER_ENTERED_VALUE")
Finally when you look at QRTZ_CRON_TRIGGERS table, the value for TIME_ZONE_ID will be America/Los_Angeles

You can use ZonedDateTime from java8/java8+, In that way you will not need to explicitly convert the given time into server specific time and it also takes care of daylight saving timezones:
protected static Trigger createCronTrigger(String triggerName, ZonedDateTime startTime, String cronExpression, int misFireInstruction, ZoneId timeZone) {
Trigger trigger = TriggerBuilder.newTrigger()
.withIdentity(triggerName)
.startAt(Date.from(startTime.toInstant())).withSchedule(CronScheduleBuilder.cronSchedule(cronExpression).inTimeZone(TimeZone.getTimeZone(timeZone)).withMisfireHandlingInstructionDoNothing())
.build();
return trigger;
}

Related

utcNow in Power Automate is using incorrect day (`dd`)

I'm running into a timezone issue when running my Flows on a schedule - and I can't seem to find a solution for it.
My timezone is currently +10:00 from UTC - Sydney, Australia.
For simplicities sake, I have a reoccurrence trigger running at 06:00 (6am local time) and 12:00 (12pm local time).
When I run my Actions - anything that needs a StartTime and an EndTime - I pass in the following parameters:
StartTime
Endtime
formatDateTime(utcNow(), 'yyyy-MM-dd"T"22:00:00.0000000Z')
formatDateTime( addDays(utcNow(), 1), 'yyyy-MM-dd"T"08:00:00.0000000Z')
The 22:00:00.0000000Z and 08:00:00.0000000Z above translate directly to 8am and 6pm respectively - essentially the entire work day hours.
Issue
When the trigger runs on the first occurrence, the StartTime and the EndTime are outputted to these values:
Tigger time - local
UTC date time
Start time UTC parameter
End time UTC parameter
2022-09-27 06:00
2022-09-26 20:00
2022-09-26T22:00:00.000000Z
2022-09-27T08:00:00.000000Z
When the trigger runs on the second occurrence, the StartTime and the EndTime are outputted to these values:
Tigger time - local
UTC date time
Start time UTC parameter
End time UTC parameter
2022-09-27 12:00
2022-09-27 02:00
2022-09-27T22:00:00.000000Z
2022-09-28T08:00:00.000000Z
Because the UTC date time has ticked over past midnight, the date value (dd) is now the next day due to the addDays(utcNow(), 1) expression.
This causes issues on the scripting of the Flow as when limiting the StartTime and EndTime of other actions, depending on when the Trigger runs is either looking at the local time's "Today" events or "Tomorrow" events.
For example, if I schedule an Out of Office / Automatic reply - and I schedule it to run on the 06:00 trigger, it will set my Automatic Reply to today - local time - 8am to 6pm. However, when it runs at the second trigger, 12:00 it will set my Automatic Reply to tomorrow - local time - 8am to 6pm.
Though this is not how I am using it, you can see it affects what actions do and perform.
Is there a way to ensure that it is always working of the current local date (dd) regardless if the UTC time has ticked over?
Have you tried using the addHours function instead of hard coding the hours in the formatdatetime function?
For the 6 am run I would try for the start time
formatDateTime(addHours(utcNow(), 2), 'yyyy-MM-ddTHH:mm:ss.0000000Z')
And for the end time
formatDateTime(addHours(utcNow(), 12), 'yyyy-MM-ddTHH:mm:ss.0000000Z')
You could also use a check to see which occurence run it is an combine that with the addhours approach.
equals(utcNow('HH'), '20')
It looks like, you want to do your Start time and End time calculations based on your LOCAL date, but you are doing the calculation based on the UTC date and thus get the wrong day, if UTC ticked over.
I guess the solution would be, to do your Start and End time calculations based on the LOCAL time instead of utcNow() and then transform the resulting LOCAL Start and End times into UTC times to be used as Start Time and End Time parameters.
Something to the effect of:
convertFromUtc(utcNow(),'+10:00','yyyy-MM-dd"T"06:00:00.000000')
to get the local start time of the current day and then from there
convertToUtc(localStartTime,'+10:00')
to get the UTC start time with the proper day.
And the same again for the End Time.
Though the other solutions seem logical in their workings, they dont account for daylight savings and other date time issues.
For the solution I ended up hard coding some of the data and calculations:
Action: "Convert time zone"
Base time: trigger time
Format string: Universal sortable date/time pattern [u]
Source: +10:00
Destination: UTC
Action: "Compose"
Expression: formatDateTime( body('Convert_time_zone'), 'dd')
Then for the start time UTC:
Action: Initialise variable
Name: Output Start Date Time
Type: String
Value: yyyy-MM-[Output from Compose*]"T"22:00:00.0000000Z')
* in that [Output from Compose] I made it an expression so I could subtract 1 day, using: sub( int(outputs('Compose')), 1)
This way I could always get my local date (dd) then manually shift the date, and set it as well as the time for start time.
Then I would use that local date for the End date time, and manually set the time.
Not the cleanest solution, but it works 100% of the time regardless of daylight savings or any time conversions. It is however not transferrable to other users without them editing the info.

How to fix incorrect Date due to timezone in Core Data from using Calendar.current.startOfDay?

I have erroneously used Calendar.current.startOfDay(for: Date()) to populate a date attribute in Core Data. This means that when users cross different timezones I may have different dates unintentially stored in the date attribute field e.g.
Timezone 1 - 25th 23:00
Timezone 2 - 25th 22:00
Timezone 3 - 26th 05:00
I need to update the Calendar to use UTC Timezone but I need to also perform a migration so that the existing entries in Core Data read like this…
Result:
Timezone 1 - 26th 00:00
Timezone 2 - 26th 00:00
Timezone 3 - 26th 00:00
What are the steps to perform this migration. If I do a UTC startOfDay on it Timezone 1 would get 25th 00:00 instead of 26th 0:00 which is what it should be. Is it possible to accurately update existing entries?
Edit:
For some context I need a reliable way to get all the entries for the 26th for example. I used startOfDay to store the date as it meant I could query by it too and have the relevant entry returned (at any moment in time get the startOfDay and it will give me the entries for the whole day). For historical dates I can do the same - let's say the user has navigated back 2 days I can take startOfDay and subtract 2 days using Calendar.current.date(byAdding: .day, value: -2, to: date) and query for that.
So now the timezone breaks the above logic but is there some way to fix this? If I loop through the entries I can figure out the date it was supposed to be for and perhaps change the attribute to a string - e.g. 26-05-2021 or start to store day, month, year instead and query that.
From reading your answer Duncan I don't think I want to use UTC calendar as it would start to store the entry against the incorrect date from the users perspective dependent on their timezone e.g. user moves to next day and utc is still on previous.
Edit 2:
In a migration I will take the date that is stored and map it to new day, month and year properties storing those instead by getting them from Calendar.current.dateComponents([.day, .month, .year], from: date). Then instead of query by date I will query by day month and year of the Calendar.current where the user is. The side effect here is there is potential the user adds something for today (27th) changes timezone and sees 26th data but I don't think it can be avoided and the old data will then show as intended.
If you took the current time and used Calendar.current.startOfDay(for: Date()) to calculate midnight in the user's local time zone, you have a loss of information. You don't know what time of day the operation was performed. If you saved the time of day in the local time zone in another field, you could reconstruct a Date in UTC.
It isn't clear that what you did was wrong. The day, month, and year is only meaningful in a specific time zone. I am in the Washington DC metro area. We are in daylight savings time (EDT). It is currently 20:56 on the 26th of May. However, it's 1:56 AM on the 27th of May in London, 2:57 AM in Munich, and 3:57 AM in Tel Aviv. All at the exact same instant in time. In UTC it is 0:57 AM on the 27th of May.
Most people think of the calendar date in their local time zone. That is their frame of reference. If you ask me the date right now, I'll tell you it's the evening of the 26th of May. Unless I know you are in a different time zone, that's the "right" answer to me.
If I start out at midnight on a given day in my time zone, calling Calendar.current.startOfDay(for: Date()) each hour, I'll get midnight that day for all 24 hours in my local time zone. For the first 20 hours of the day, that would be the same result I would get if I created a Calendar in UTC and made the same call. However, at 20:00 EDT, I would start getting the next calendar day if I made the same query in UTC.
If you don't know what time of day you made the call to Calendar.current.startOfDay(for: Date()), there is no foolproof to figure out the day/month year in UTC at the instant you made the call. It depends on the time of day in the local timezone, and that timezone's offset from UTC.
Consider this code:
var calendarUTC = Calendar(identifier: .gregorian)
if let utcTimeZone = TimeZone(identifier: "UTC") {
print("Valid time zone")
calendarUTC.timeZone = utcTimeZone
}
print ("Start of day in UTC is \(calendarUTC.startOfDay(for: Date()))")
print ("Start of day in local time zone is \(Calendar.current.startOfDay(for: Date()))")
That outputs:
Start of day in UTC is 2021-05-27 00:00:00 +0000
Start of day in local time zone is 2021-05-26 04:00:00 +0000
That's because right now, which is 20:56 on 26 May in my time zone, it's 0:56 on 27 May in UTC. So if I ask the UTC calendar for the start of day for now (Date()) I get midnight on 27 May, in UTC.
If I ask the same question of my local calendar, I get midnight on 26 may in my time zone, which is 4:00 AM on 26 May in UTC.
If I ran the same code this morning at 8:00 AM in my time zone, I would have gotten the output:
Start of day in UTC is 2021-05-26 00:00:00 +0000
Start of day in local time zone is 2021-05-26 04:00:00 +0000
(Since at 8:00 AM on 26 May in EDT is also 26 May in UTC.)
It's tricky and not 100% reliable and only works if you know that all days were created using startOfDay. But first you need to decide what you want. Say one date was created at 10pm in the New York, and one at exactly the same moment in London, at 4am the next day. What day do you want to be stored?
If your date stored is 25th, 10pm, then you know it was created in a timezone where the day started at 10pm UTC. You are lucky, there are only two time zones that would have created this, one without DST, one with DST. So you know it happened in one of these two time zones, within 24 hours.
Unfortunately, time zones cover 26 hours. Fortunately, only some islands in the Pacific Ocean have same time and different dates (+13 and -11 hours). For these places, you cannot possibly know which date is correct, but very few people would be affected.

PostgreSQL: Can't get timezones with DST to work properly

I'm trying to figure out how Postgres handles DST in combination with intervals. Specifically I want to allow my users to create events with recurring dates - e.g. everyday at 16:00 local time.
For what I'm doing I need to store the first date in the user's local time, and then add a number of days to it, without changing the time of day in the user's local time. I was hoping that timestamptz with a full timezone name (so it knows when to apply DST?) combined with simple 1 day intervals would do the job - but it fails at my simple example:
Germany uses CET (+1:00) and switches to CEST (+2:00) at 2:00 in the morning on March 28.
Tunesia uses CET all year.
Thus I expected, that using a timestamptz on March 27 and adding 1 day to it, I'd see a different utc-offset in Berlin, and no change in Tunis - but they both changed the offset equally, as if Tunis was using DST:
select
'2021-03-27 16:00:00 Africa/Tunis'::timestamptz as "tunis_before_dst",
'2021-03-27 16:00:00 Africa/Tunis'::timestamptz + INTERVAL '1 day' as "tunis_after_dst",
'2021-03-27 16:00:00 Europe/Berlin'::timestamptz as "berlin_before_dst",
'2021-03-27 16:00:00 Europe/Berlin'::timestamptz + INTERVAL '1 day' as "berlin_after_dst"
results in:
tunis_before_dst: '2021-03-27 16:00:00+01'
tunis_after_dst: '2021-03-28 16:00:00+02'
berlin_before_dst: '2021-03-27 16:00:00+01'
berlin_after_dst: '2021-03-28 16:00:00+02'
Looking through pg_timezone_names, I can see that my Postgres instance is aware of Africa/Tunis not having DST - so I'm wondering why it's changing the UTC offset for it then.
I guess it's obvious that timezones and DST are very confusing to me, but am I doing something wrong handling them or is timezonetz not what I think it is?
Rant first. The concept of DST is breathtaking nonsense. Even the name is obvious BS. "Daylight Saving Time". No daylight has been saved. I can't believe the EU still did not manage to get rid of it, even though the overwhelming majority wants it gone, and it's been consensus to scrap it for a while now.
With that out of my system, the primary misunderstanding is this: you assume that the data type timestamp with time zone would store time zone information. It does not. Becomes obvious here:
Thus I expected, [...] I'd see a different utc-offset in Berlin, and no change in
Tunis - but they both changed the offset equally
The time zone offset you see in the output is the offset determined by the current timezone setting of your session. Time zone serves as input / output modifier / decorator. Postgres always stores UTC time internally. And no time zone information whatsoever.
The type name is a bit deceiving there. It has been known to fool the best:
Time zone storage in data type "timestamp with time zone"
Once you've grasped that concept, the rest should become obvious.
To preserve the local time (wall clock time of day), use the data type timestamp without time zone (timestamp), or even just time (never use the broken timetz), and store time zone information additionally - ideally the time zone name ('Europe/Berlin' like you have it), not a time zone abbreviation or a numeric offset.
timestamp with time zone (timestamptz) is the right choice to store unique points in time, independent of any time zones. The time zone offset is just an input modifier. Both of the following literals result in the same timestamptz value exactly, because both time zones happen to apply the same offset at this time of the year:
'2021-03-27 16:00:00 Africa/Tunis'::timestamptz
'2021-03-27 16:00:00 Europe/Berlin'::timestamptz
But these differ by one hour, because the German offset has changed according to the local DST regime:
'2021-03-28 16:00:00 Africa/Tunis'::timestamptz
'2021-03-28 16:00:00 Europe/Berlin'::timestamptz
Related:
Ignoring time zones altogether in Rails and PostgreSQL
Preserve timezone in PostgreSQL timestamptz type

Postgres: query schedule given day, time_of_day, and time zone name

I'm writing a "send me messages at this time" app. I'm storing recurrence information in this manner:
Schedules
----------
days_of_week: [3, 4, 5]
hours_of_day: [8, 13, 22]
time_zone: "Pacific Time (US & Canada)"
Works fine in displaying, but I need to write a frequent cron job that grabs all schedules for "right now (utc)". So, if the the cron job is running at 09:00 UTC Monday, I need to grab all schedules where
Monday is in days_of_week (where days_of_week #> ARRAY[1])
hours_of_day is at 09:00 UTC. This is given hours_of_day is stored as an array of integers, but we are also storing the user's time_zone.
So the user may say: "deliver me a message at 9am Monday" (which we store as [9]), but that means 9am in their time zone.
Questions:
Any way to query all schedules given these parameters?
If not, is there a better way to structure the data to ensure easier querying through Postgres? The schema is flexible.
Thanks in advance!
Postgres has superb facilities for working with timezones, and I've written something very similar to what you're asking about here using the AT TIME ZONE construct. In addition to your fields, I use a last_scheduled_at to flag when a schedule was last "executed"--i.e., when the last successful cron job ran for that schedule to avoid double-scheduling, and a deleted_at for logical deletion of schedules.
My schema for schedules was similar, except that I had only a single hour. I stored days in an array, like you, and the timezone as text. The fields in my schedules table are dows, hour, and timezone.
This was the query:
SELECT
s.*
FROM
schedules s
WHERE
ARRAY[extract(dow from timestamptz (now() at time zone timezone))] && dows
AND hour = extract(hour from timestamptz (now() at time zone timezone))
AND (s.last_scheduled_at IS NULL
OR s.last_scheduled_at < (now() - interval '12 hours'))
AND s.deleted_at IS NULL
LIMIT
1000
I use && (overlaps) rather than #> (contains), but either works. You'll probably also want the limit so you can process the work in batches (keep running this and you're done for hour X if you get zero results; make sure you're done well before the hour is up). You'll also probably want to pass the timestamp as a parameter to this query--I've inlined it here as now() to simplify things, but passing the time as a parameter makes testing this a lot easier.
Note also that Postgres can be picky with time zone names and abbreviations and its behavior with daylight saving time can be counterintuitive: e.g., Pacific Standard Time and Pacific Daylight Time are treated as two distinct time zones (for the purposes of AT TIME ZONE):
maciek=# select now() at time zone 'pst';
timezone
----------------------------
2015-10-09 23:14:51.856813
(1 row)
maciek=# select now() at time zone 'pdt';
timezone
----------------------------
2015-10-10 00:14:54.402524
(1 row)
That is, Daylight Saving Time is always there, whether you are currently observing it or not. If you're letting people enter the time zone directly, it's good to either reject these or automatically coerce these to 'America/Los_Angeles' (or whatever time zone they happen to map to), which will handle these conversions for you automatically according to the time zone rules your Postgres version has (make sure you update to point releases promptly if accuracy is critical here for areas that have frequent time zone changes). The list of time zone names used by Postgres can be found in the Olson database. The Postgres tables pg_timezone_names
and pg_timezone_abbrevs may also be of interest.

JPA Date and Timezone confusion

I'm having a problem reading dates from a database using JPA. I'm using EclipseLink and PostgreSQL
I've populated my database from a CSV file, witch had dates as strings (in this format: 6/30/2009-23:59:56). I used the following snipet to convert it to a Date object:
public static Date parseDate(String s){
DateFormat formatter = new SimpleDateFormat("d/M/yyyy-k:m:s");
try {
return new Date( ((java.util.Date)formatter.parse(s)).getTime() );
} catch (ParseException ex) {
Logger.getLogger(Type.class.getName()).log(Level.SEVERE, null, ex);
return null;
}
}
The date is correctly converted and stored in the database as expected. Here is how i map the Date object to the database:
#Column(name="data_ts", nullable=false)
#Temporal(TemporalType.TIMESTAMP)
private Date dataTs;
The problem seems to happen when i try to read the Date from the database to use it in a chart(Highcharts). The record with that same timestamp above get read as:
Mon Jun 06 23:59:56 BRT 2011 and it's timestamp as 1307415596000
Note that it is in Brazilian Time(+3h), so the timestamp (that is calculated from GMT) is 3 hours shifted. Once ploted, the timestamp turns to point to 07/06/2011 02:59:56
Here's an example:
List<TimedataEnt> timeData = currentWellsite.getTimeData();
String debug = timeData.get(timeData.size()-1).getDataTs().toString() + ">>>" + timeData.get(timeData.size()-1).getDataTs().getTime();
where currentWellsite is and JPA Entity, and getDataTs() returns a java.util.Date object.
The string turns out to be "Tue Jun 30 23:59:56 BRT 2009>>>1246417196000"
How do I tell JPA not to convert the timestamp read from the database?
As said, Date and Timestamps have no timezone consideration. It seems that the issue is caused because Java considers that the time it reads from the database is the current default timezone.
So, if the database contais 2011-04-04 14:00:00 and my current timezone is +3, assigning that to java Date will generate 2011-04-04 14:00:00 BRT time(+3), with a timestamp shifted 3 hours (since timestamps are caclulated from UTC).
Solved the issue by getting an calculated timestamp:
long ts = myDate().getTime() + TimeZone.getDefault().getRawOffset();
It's important to say that getRawOffset() does not take Daylight Saving periods in consideration. For that, use getOffset()
Your date is 6/30/2009-23:59:56. I read that as 30 june 2009, 23:59:56. So the format to parse it should be M/d/yyyy-HH:mm:ss or M/d/yyyy-kk:mm:ss (depending on if your hours go from 1 to 24 or from 0 to 23). But definitely not d/M/yyyy-k:m:s: the month comes before the day.
Also, a Timestamp doesn't have any time zone. It's a universal instant in time. It's only when you display its value that the timezone is important, because then you have to choose which time zone to use to display the time. Use a DateFormat with the appropriate timezone set to display your timestamp.
Your issue seems to be that you are storing your Timestamp (which does not have a timezone) into a java.util.Date (which has a timezone offset).
If you want control over how the timezone is set, then store your Timestamp as a java.sql.Timestamp, or use your own #Converter.
In general Calendar should be used in Java instead of java.util.Date, which is for the most part deprecated. Calendar also has a Timezone, so you may have similar issues.