I am working with Metabase, to query a PostgreSQL database.
The database is storing records with timestamps, which are in UTC.
How can I query the database for, for example all records between 1.sept and 7 sept. CEST time?
Currently I have to manually convert, and query '08.31.2022 22:00:00' to get records from midnight 1.sept CEST.
Is there a way to solve this by using timestamptz or something similar?
Related
Postgresql timestamp issue: timestamp is adding 1 hour to only one date - the date of day light saving
When i store 2019-11-03T08:00:00Z in database column which is timestamp, when i read timestamp back returns me this 2019-11-03T09:00:00Z. I understand that day light saving came into effect on Nov-03-2019 but it should not affect my date . All other dates like 2019-11-04T08:00:00Z works
Written to DB:
endedAt=2019-11-03T08:00:00Z
Read from DB:
endedAt=2019-11-03T09:00:00Z
For all other dates, its perfectly fine.
CURRENT SITUATION:
I have a table of wildfire incidents with a timestamp with time zone (timestamptz) field to track when the observation occurred.
Everything in the data creation process is in UTC: the incoming data from the source, the app server that inserts the data, the insert python code (appends a "Z" to the time), and the database server are all in UTC.
The incidents' geographic extent spans several time zones in the US, Canada, and Mexico.
PROBLEM:
I've been querying on a day's worth of data in UTC time, but need to extract out data relative to local time. The midnight to midnight range will be different in each time zone.
My use case now is one day, but I was asked to consider arbitrary time ranges. E.g.: find all incidents in the hottest part of the day (say 10:00 to 18:00) local time.
This table is quite large and I have an index on the timestamptz field right now. Any changes I make will need to work with an index.
Account for daylight saving time.
I have a method to get the time zone for each record, so I don't need help with that.
I created a test table for this with a timestamptz field ts_with and a varchar field for the time zone tz. The following query returns what I want, so I feel like I'm making progress.
SELECT
name, test_tz.ts_with, test_tz.tz,
TIMEZONE(test_tz.tz, test_tz.ts_with) as timezone_with
FROM fire_info.test_tz
WHERE TIMEZONE(test_tz.tz, test_tz.ts_with) BETWEEN
'2018-08-07 00:00:00' AND '2018-08-07 23:59:59';
QUESTIONS:
Will this use my index? I'm thinking the timezone function will avoid it. Any solution for that? I'm considering adding another condition to the where clause that selects on timestamptz buffered by a day on either side. That would use the index and then the timezone function isn't sorting through too much data (~6k records per day during fire season). Would PG figure that out?
The timezone function is giving me DST offsets (e.g.: Denver is currently UTC-06). I assume I'll get standard time after DST ends. If I run a query in December for data in August, will it apply standard time or DST?
thanks!!!
The way you wrote the query, it cannot use an index on ts_with.
To use an index, the condition would have to be of the form ts_with <operator> <constant>, and there is no way to rewrite the query in that fashion.
So you should create a second index on timezone(test_tz.tz, test_tz.ts_with).
First, I have read a lot about date types in Postgres using timezone or not. It seems that managing dates (timestamps) with timezone is the best to do in 99% of cases.
Currently I have built a sync engine using a Rails app set to UTC time by default (very important) and MariaDB managing dates (so UTC dates are written in the database but without seeing anything except the date: 2017-04-19 17:45:09). The devices that syncs to the server engine use local SQLite database with dates in UTC too. So for the same document, I can see the same date for a created_at field in the SQLite field and in the MariaDB field. It's working well.
Note that users never have to set a custom timezone in the server app and in the device apps. I want all UTC.
As I plan to migrate to PostgreSQL for many reasons, I'm quite sure I need to use Timestamp without timezone too, but I have some doubts after all readings.
What are your suggestions about this use case ?
EDIT:
So I made 3 tests using pgloader:
Test1: MariaDB => PostgreSQL using computer default timezone
Dates are the same but as you can see, sometimes there is +01, sometimes +02 ???
Test2: MariaDB => PostgreSQL using timezone after setting timezone to UTC in Postgres
Here there is +00, logical for UTC, but my dates were changed.
Test3: MariaDB => PostgreSQL using timestamp WITHOUT timezone
All the same there.
"With Timezone" simply tells postgres to convert from local time to UTC before storing the timestamp, and to convert back to local time when querying. The underlying data is stored in UTC and no timezone information is stored in the actual data, it's simply a flag the DB uses to determine how to convert the data for human use.
If your client connection is setting the PG timezone to UTC upon DB connection, there will be pretty much no difference. Conventional wisdom says to always use timestamps with timezone, as it'll make queries in other systems (postgresql client) a lot more flexible and easy to use in the future.
try with one of these in your config file
SET timezone= 'UTC'
SET client_timezone TO '+00:00'
my full settings were:
LOAD DATABASE
FROM mysql://usr#localhost/kk
INTO postgresql://usr:pwd#localhost/kk
WITH include drop,
create tables,
create indexes,
reset sequences,
workers = 8, concurrency = 1,
multiple readers per thread, rows per range = 50000
SET PostgreSQL PARAMETERS
maintenance_work_mem to '512MB',
work_mem to '64MB',
timezone to 'UTC'
SET MySQL PARAMETERS
net_read_timeout = '220',
net_write_timeout = '220'
SET timezone= 'UTC'
SET client_timezone TO '+00:00'
ALTER SCHEMA 'kk' RENAME TO 'public'
;
Redshift only support TIMESTAMP format, I want to dump some data that is originally stored in Postgres in time without time zone format. My data looks like this: 15:30:00
When I COPY my data into Redshift, it says Invalid timestamp format or value [YYYY-MM-DD HH24:MI:SS], error code 1206.
My current workaround is to create that column in Redshift as CHAR(8)
Thanks
There's no other solution that to use the char(8).
Redshift does not support timezones.
Workaround would be to parse data to UTC and store time difference in separate column. You will be able to calculate local time by adding/substracting hours to the UTC data.
How can I express 24 hour time as 12 hour time when using timestamp without timezone in PostgreSQL 9.1?
More specifically, I inserted order data into a database and I need to extract it back in its original state.
Original data: 06/10/2013 12:43:30
Data in DB: 2013-06-10 04:43:30+00
If I query order data like:SELECT order_date AT TIME ZONE('EDT') ...
I return 2013-06-10 00:43:30
I need the 00 in the time to show as 12. How can I do this?
User to_char():
SELECT to_char('2013-06-10 00:43:30'::timestamp, 'DD/MM/YYYY HH12:MI:SS PM')
I added PM to make the output unambiguous. Consider available patterns in the manual.