The date/time strings we're sending over to pub/sub look like this:
2018-07-18T17:30:08Z
I created a data flow job to insert these into Big Query and it failed at insert.
Stripping out the "Z" at the end like this was successful:
2018-07-18T17:30:08
The problem is that Big Query seems to be interpreting this as a local time, and not UTC.
I've tried both of these ways to insert the time zone:
2018-07-18T17:30:08+00:00
2018-07-18T17:30:08+0000
Both are rejected.
What's the correct way to do this, or is there some other way I can force Big Query to interpret these times as UTC?
Related
I'm trying to find a way to tell Postgres to truncate all datetime columns so that they are displayed and filtered by seconds (ignoring milliseconds).
I'm aware of the
date_trunc('second', my_date_field)
method, but do not want to do that for all datetime fields in every select and where clause that mentions them. Dates in the where clause need to also capture records with the granularity of seconds.
Ideally, I'd avoid stripping milliseconds from the data when it is stored. But then again, maybe this is the best way. I'd really like to avoid that data migration.
I can imagine Postgres having some kind of runtime configuration like this:
SET DATE_TRUNC 'seconds';
similar to how timezones are configured, but of course that doesn't work and I'm unable to find anything else in the docs. Do I need to write my own Postgres extension? Did someone already write this?
CURRENT SITUATION:
I have a table of wildfire incidents with a timestamp with time zone (timestamptz) field to track when the observation occurred.
Everything in the data creation process is in UTC: the incoming data from the source, the app server that inserts the data, the insert python code (appends a "Z" to the time), and the database server are all in UTC.
The incidents' geographic extent spans several time zones in the US, Canada, and Mexico.
PROBLEM:
I've been querying on a day's worth of data in UTC time, but need to extract out data relative to local time. The midnight to midnight range will be different in each time zone.
My use case now is one day, but I was asked to consider arbitrary time ranges. E.g.: find all incidents in the hottest part of the day (say 10:00 to 18:00) local time.
This table is quite large and I have an index on the timestamptz field right now. Any changes I make will need to work with an index.
Account for daylight saving time.
I have a method to get the time zone for each record, so I don't need help with that.
I created a test table for this with a timestamptz field ts_with and a varchar field for the time zone tz. The following query returns what I want, so I feel like I'm making progress.
SELECT
name, test_tz.ts_with, test_tz.tz,
TIMEZONE(test_tz.tz, test_tz.ts_with) as timezone_with
FROM fire_info.test_tz
WHERE TIMEZONE(test_tz.tz, test_tz.ts_with) BETWEEN
'2018-08-07 00:00:00' AND '2018-08-07 23:59:59';
QUESTIONS:
Will this use my index? I'm thinking the timezone function will avoid it. Any solution for that? I'm considering adding another condition to the where clause that selects on timestamptz buffered by a day on either side. That would use the index and then the timezone function isn't sorting through too much data (~6k records per day during fire season). Would PG figure that out?
The timezone function is giving me DST offsets (e.g.: Denver is currently UTC-06). I assume I'll get standard time after DST ends. If I run a query in December for data in August, will it apply standard time or DST?
thanks!!!
The way you wrote the query, it cannot use an index on ts_with.
To use an index, the condition would have to be of the form ts_with <operator> <constant>, and there is no way to rewrite the query in that fashion.
So you should create a second index on timezone(test_tz.tz, test_tz.ts_with).
Here is my problem:
I want to calculate how long ago a record was updated in a DB.
The DB is in PostgreSQL, the update_time field is populated by a trigger that uses CURRENT_TIMESTAMP(2). The field is inflated to a DateTime object by DBIx::Class. I get the current time in my code using DateTime->now()
My problem is that when I retrieve the field value, it's off by 1 h (ie it's 1h ahead of DateTime->now()). I am in the CET time zone, so 1h ahead of UTC currently.
The right way to solve the problem is likely at the DB level. I have tried to replace CURRENT_TIMESTAMP with LOCALTIMESTAMP, to no avail.
I think actually a more robust solution (ie one that doesn't rely on getting the DB right) would be to get the current time stamp from the DB itself. I really just need the epoch, since that's what I use to compute the difference.
So the question is: is there a simple way to do this: get the current time from the DB using DBIx::Class?
A different way to get the DB and DateTime to agree on what the current time is would also be OK!
You can use dbh_do from your DBIx::Class::Storage to run arbitrary queries. With that, just SELECT the CURRENT_TIMESTAMP.
my ( $timestamp ) = $schema->storage->dbh_do(
sub {
my ($storage, $dbh) = #_;
$dbh->selectrow_array("SELECT CURRENT_TIMESTAMP");
},
);
I always recommend to do all date/time related things on the app server and not rely on the database server(s). Essentially that means to not use a trigger but pass the datetime on insert/update and make it mandatory (NOT NULL).
Besides that you should store datetimes in UTC and convert to your local or other required timezone in your code.
Your issue likely happens because of an incorrect or missing timezone configuartion in which case DateTime defaults to its floating timezone.
I have a table that I am using to store iso dates with timezones. I realize that dates should "always" be stored as utc but I have an exception to that rule. The timestamps aren't in any way related to the server they are running on. I want to be able to store an iso date like this:
2016-03-06T01:15:52-06:00
And regardless of the time zone of the server or anything else I want the timestamp returned as:
2016-03-06T01:15:52-06:00
Currently if I insert an iso date it automatically converts it to whatever the server timezone is. My above date gets converted to:
2016-03-06 07:15:52+00 (server is utc)
The only thing I can think of is storing the timezone offset in a separate column, storing my date as utc and then converting using the offset column, horribly messy. Surely there is a way to store my date in one column and get it out the way it was originally created?
Your proposed solution is correct. Or more precisely, it is one of several correct implementations. Any of the following would work:
Store the UTC timestamp in one field, store the offset in another.
Store the local timestamp in one field, store the offset in another.
Store the local date in one field, and store a time with time zone in another. (though time with time zone is generally discouraged...)
Store the UTC timestamps in one field and the local timestamp in another.
The easiest by far is the first one, which you already proposed.
I'd avoid against storing timestamps in text fields, as they tend not to be very efficiently searchable.
Also note - if you're coming from a SQL Server background, you might recall its datetimeoffset type, which stores the local datetime and offset in the field, and uses the UTC equivalent during indexing. It's common to think that Postgres and MySQL's timestamp with time zone would have the same behavior, but they don't. They simply use the session time zone to convert to/from UTC. SQL Server has no concept of a session time zone, and thus the discrepancy.
Be sure to read this part of the Postgres docs.
I have an annoying issue with Rails/Active Record and dates that I'm trying to figure out. We're using date fields for composite keys, which I am turning into timestamps to make it easier for use as URL parameters. This is working fine, but I have inconsistencies when trying to lookup records after converting the timestamp back into a DateTime. When the object is serialised, the ID sent back looks something like 1401810373.197,63 where the first number is the timestamp with milliseconds, and the second value is the original ID that rails normally uses.
When a request is received with this ID, the timestamp is parsed using the following
... get timestamp from input ...
Time.at(Rational(timestamp)).utc.strftime('%Y-%m-%d %H:%M:%S.%3N')
This works as expected, and the queries produced using this also work as expected. The issue lies in that the date time produced here, is slightly different to the one on the original object. They're out by something like 1ms which I'm assuming is due to the loss of precision when using to_f to get the timestamp.
I did a quick test in a console with the following code to replicate it
Model.all.each do |m|
puts Time.at(Rational("#{m.to_param.split(',').first}")).utc.strftime('%Y-%m-%d %H:%M:%S.%3N') == m.created_at.utc.strftime('%Y-%m-%d %H:%M:%S.%3N')
end
The output of this shows multiple true and false values, so something is definitely going wrong in the conversion.
Currently, the to_param method simply converts the created_at field using to_f. I've tried changing this to "%.6f" % m.created_at.to_f but this hasn't changed anything.
Is there anything I can do to prevent this difference in times, as it's causing an array of issues.
This is currently running on Postgres, where the created_at column is a timestamp(3) column. We're using Rails 4.1 with jRuby 1.7.12
Ended up dropping the accuracy entirely. Now the database has a type of timestamp(0) and rails has been modified to not provide the milliseconds. Seems to be working :)