I just started with PostgreSQL. Since we use a lot of Unixtimestamps I figured it would be best to just try a little example. So I made a table test with a field ID and a field timestamp of type timestamp without time zone using phpPgAdmin. There is valid data.
Now I just wanted to retrieve the data using a Unixtimestamp. And I run into really odd behavior.
I made this little test which selects data by just converting the first best timestamp into a unixtimestamp and back into a timestamp:
This will return the timestamp in the raw and unix value for the first entry
SELECT timestamp, extract(epoch FROM timestamp)
FROM test
WHERE id IN (SELECT id FROM test LIMIT 1);
Ok, that worked as intended.
This does only convert therefore it should return one entry
SELECT timestamp, extract(epoch FROM timestamp)
FROM test
WHERE timestamp IN (SELECT TO_TIMESTAMP(extract(epoch FROM timestamp))
FROM test LIMIT 1);
Nothing is returned!
I just can't wrap my head around this behavior. If I convert a timestamp into something else and then convert it back it should be the same timestamp, right?
I just want a way to cast timestamps into unixtime and back without the chance that information is lost.
Please note: this is just a test! I do not want to use this code. I just want to check if the SQL code behaves as expected!
Also, I can not do anything about the unixtimestamps! I just want a way to cast them properly!
The difference of not using timestamp type without time zone vs with:
select '09/12/2020 11:16 PDT'::timestamp, to_timestamp(extract(epoch from '09/12/2020 11:16 PDT'::timestamp));
timestamp | to_timestamp
---------------------+-------------------------
09/12/2020 11:16:00 | 09/12/2020 04:16:00 PDT
select '09/12/2020 11:16 PDT'::timestamptz, to_timestamp(extract(epoch from '09/12/2020 11:16 PDT'::timestamptz));
timestamptz | to_timestamp
-------------------------+-------------------------
09/12/2020 11:16:00 PDT | 09/12/2020 11:16:00 PDT
UPDATE. For more detail see. the important part is:
In a literal that has been determined to be timestamp without time zone, PostgreSQL will silently ignore any time zone indication. That is, the resulting value is derived from the date/time fields in the input value, and is not adjusted for time zone.
For timestamp with time zone, the internally stored value is always in UTC (Universal Coordinated Time, traditionally known as Greenwich Mean Time, GMT). An input value that has an explicit time zone specified is converted to UTC using the appropriate offset for that time zone. If no time zone is stated in the input string, then it is assumed to be in the time zone indicated by the system's TimeZone parameter, and is converted to UTC using the offset for the timezone zone.
When a timestamp with time zone value is output, it is always converted from UTC to the current timezone zone, and displayed as local time in that zone. To see the time in another time zone, either change timezone or use the AT TIME ZONE construct (see Section 9.9.3).
This means:
select '09/12/2020 11:16 PDT'::timestamp;
timestamp
---------------------
09/12/2020 11:16:00
select '09/12/2020 11:16'::timestamp at time zone 'UTC';
timezone
-------------------------
09/12/2020 04:16:00 PDT
select '09/12/2020 11:16 PDT'::timestamp at time zone 'UTC';
timezone
-------------------------
09/12/2020 04:16:00 PDT
select '09/12/2020 11:16 PDT'::timestamptz;
timestamptz
-------------------------
09/12/2020 11:16:00 PDT
select '09/12/2020 11:16 PDT '::timestamptz at time zone 'UTC';
timezone
---------------------
09/12/2020 18:16:00
So in the first case the time zone is ignored and the time is taken to be 09/12/2020 11:16 . In the second and third cases the timestamp is taken as being at UTC and the displayed value is rotated to my time zone PDT.In the third case I'm using timestamptz so the timestamp is correctly tagged with the time zone. This means when I ask to display at UTC it does the correct thing.
As documented in the manual to_timestamp() returns a timestamp WITH time zone, so obviously the comparison with a timestamp WITHOUT time zone won't work.
You will need to convert the result from to_timestamp() back to a timestamp without time zone, by telling Postgres which time zone it should take:
SELECT "timestamp",
extract(epoch FROM "timestamp"),
cast("timestamp" as timestamp with time zone),
TO_TIMESTAMP(extract(epoch FROM "timestamp")) at time zone 'UTC'
FROM test
where "timestamp" in (select to_timesatmp(extract(epoch FROM "timestamp")) at time zone 'UTC'
from test
limit 1)
Which is essentially the same as:
SELECT "timestamp",
extract(epoch FROM "timestamp"),
cast("timestamp" as timestamp with time zone),
TO_TIMESTAMP(extract(epoch FROM "timestamp")) at time zone 'UTC'
FROM test
where "timestamp" in (select "timestamp"
from test
limit 1)
But the whole converting back and forth between a number and a proper timestamp is useless - and error prone if you mix different data types (as you have just discovered). The best thing is to keep everything as a timestamp or better as a timestamptz.
For a short introduction why timestamptz (with time zone)` is in general a better choice, you might want to read this
I uploaded csv data into table in postgres. One of the columns is a character varying column with an epoch filling. The data going in is as following:
Example record: 1507656308.
The epoch character varying column is transformed to a timestamp column as follows:
select
TIMESTAMP WITH TIME ZONE 'epoch' + timestamp::integer * INTERVAL '1 second' as timestamp
from
table
The result appeared good and I continued with it. I live in the Netherlands and here we have the winter-summertime change twice a year. Apparently my data does not differentiate.
I tried to figure out my current timezone. Which I did like this:
SELECT EXTRACT(TIMEZONE FROM timestamp) FROM table;
Resulting in:
3,600.0000
When I try to figure out my general time zone I get this:
show timezone
= Europe/Berlin
How do I get my timestamps to recognize the summer vs wintertime? Or what did I do wrong?
Of all the creative ways to convert an epoch value (why is it stored as string?) to a timestamp with time zone, this is one of the more interesting one.
The simple and boring way is
SET timezone = 'Europe/Berlin';
SELECT to_timestamp(1507656308);
to_timestamp
------------------------
2017-10-10 19:25:08+02
(1 row)
The time zone offset for that is two hours, because on October 10, daylight savings time was in effect (Central European Summer Time).
Indeed:
SELECT extract(timezone FROM to_timestamp(1507656308));
date_part
-----------
7200
(1 row)
So everything is working as expected, right?
I am trying to understand the timestamps and timezones in Postgre. I think I got it, until I red this article. Focus on the "Converting Between Timezones" part. It has two examples.
(Consider the default timezone configuration to be UTC.)
Example 1
db=# SELECT timezone('US/Pacific', '2016-01-01 00:00'); outputs 2015-12-31 16:00:00
According to the article and what I understand, because the '2016-01-01 00:00' part of the timezone function is just a string, it is silently converted to the default UTC. So from '2016-01-01 00:00' UTC it is then converted to US/Pacific as asked by the timezone function, that is 2015-12-31 16:00:00.
Example 2
db=# SELECT timezone('US/Pacific', '2016-01-01 00:00'::timestamp); outputs 2016-01-01 08:00:00+00
Excuse me, I dont see why and the explanation there does not help. Ok, the '2016-01-01 00:00'::timestamp part of the timezone function is no longer a string, but an actual timestamp. In what timezone? If it is UTC, the output would have to be the same as the Example 1. So it is automatically converted to US/Pacific? Then the output is in UTC? But why? I asked for a US/Pacific in my timezone not a UTC.
Please explain how the timezone behaves when gets a timestamp and gets asked to transform it. Thank you.
Let me explain the two examples:
In both we assume a timezone UTC (i.e. SET timezone TO UTC).
db=# SELECT timezone('US/Pacific', '2016-01-01 00:00');
timezone
---------------------
2015-12-31 16:00:00
(1 row)
This is equivalent to SELECT timezone('US/Pacific', '2016-01-01 00:00'::timestamptz), i.e. Postgres implicitly converted the string to a timestamptz.
We know that the timezone function converts back and forth between timestamp and timestamptz:
Since we are giving it a timestamptz as input, it'll output a timestamp. In other words, it is converting the absolute point in time 2016-01-01 00:00Z to a wall time in US/Pacific, i.e. what the clock in Los Angeles showed at that absolute point in time.
In example 2 we are doing the opposite, namely taking a timestamp and converting it to a timestamptz. In other words, we are asking: what was the absolute point in time when the clock in Los Angeles showed 2016-01-01 00:00?
You mention:
Ok, the '2016-01-01 00:00'::timestamp part of the timezone function is no longer a string, but an actual timestamp. In what timezone?
'2016-01-01 00:00'::timestamp is a timestamp, i.e. a wall time. It doesn't have a notion of timezone.
I think you might not have fully understood the difference between timestamp and timestamptz, which is key here. Just think of them as wall time, i.e. the time that showed somewhere in the world on a clock hanging on the wall, and absolute time, i.e. the absolute time in our universe.
The examples you make in your own answer are not quite accurate.
SELECT ts FROM (VALUES
(timestamptz '2012-03-05 17:00:00+0') -- outputs 2012-03-05 17:00:00+00 --1
,(timestamptz '2012-03-05 18:00:00+1') -- outputs 2012-03-05 17:00:00+00 --2
,(timestamp '2012-03-05 18:00:00+1') -- outputs 2012-03-05 18:00:00+00 --3
,(timestamp '2012-03-05 11:00:00' AT TIME ZONE '+6') -- outputs 2012-03-05 17:00:00+00 --4
,(timestamp '2012-03-05 17:00:00' AT TIME ZONE 'UTC') -- outputs 2012-03-05 17:00:00+00 --5
,(timestamp '2012-03-05 17:00:00'::timestamp) -- outputs 2012-03-05 17:00:00+00 --6
,(timestamp '2012-03-05 17:00:00'::timestamptz) -- outputs 2012-03-05 17:00:00+00 --7
) t(ts);
The problem with your example is that you're constructing one data set with a single column. Since a column can only have one type, each row (or single value in this case) is being converted to the same type, namely timestamptz, even though some values were calculated as timestamp (e.g. value 3). Thus, you have an additional implicit conversion here.
Let's split the example into separate queries and see what is going on:
Example 1
db=# SELECT timestamptz '2012-03-05 17:00:00+0';
timestamptz
------------------------
2012-03-05 17:00:00+00
As you might already know, timestamptz '2012-03-05 17:00:00+0' and '2012-03-05 17:00:00+0'::timestamptz are equivalent (I prefer the latter). Thus, just to use the same syntax as in the article, I'll rewrite:
db=# SELECT '2012-03-05 17:00:00+0'::timestamptz;
timestamptz
------------------------
2012-03-05 17:00:00+00
Now, what's going on here? Well, less than in your original explanation. The string is simply parsed as a timestamptz. When the result gets printed, it uses the currently set timezone config to convert it back to a human readable representation of the underlying data structure, i.e. 2012-03-05 17:00:00+00.
Let's change the timezone config and see what happens:
db=# SET timezone TO 'Europe/Berlin';
SET
db=# SELECT '2012-03-05 17:00:00+0'::timestamptz;
timestamptz
------------------------
2012-03-05 18:00:00+01
The only thing that changed is how the timestamptz gets printed on screen, namely using the Europe/Berlin timezone.
Example 2
db=# SELECT timestamptz '2012-03-05 18:00:00+1';
timestamptz
------------------------
2012-03-05 17:00:00+00
(1 row)
Again, just parsing the date.
Example 3
db=# SELECT timestamp '2012-03-05 18:00:00+1';
timestamp
---------------------
2012-03-05 18:00:00
(1 row)
This is the same as '2012-03-05 18:00:00+1'::timestamp. What happens here is that the timezone offset is simply ignored because you're asking for a timestamp.
Example 4
db=# SELECT timestamp '2012-03-05 11:00:00' AT TIME ZONE '+6';
timezone
------------------------
2012-03-05 17:00:00+00
(1 row)
Let's rewrite to be simpler:
db=# SELECT timezone('+6', '2012-03-05 11:00:00'::timestamp);
timezone
------------------------
2012-03-05 17:00:00+00
(1 row)
This is asking: what was the absolute time when the clock on the wall in the timezone with an offset of +6 hours was showing 2012-03-05 11:00:00?
Example 5
db=# SELECT timestamp '2012-03-05 17:00:00' AT TIME ZONE 'UTC';
timezone
------------------------
2012-03-05 17:00:00+00
(1 row)
Let's rewrite:
db=# SELECT timezone('UTC', '2012-03-05 17:00:00'::timestamp);
timezone
------------------------
2012-03-05 17:00:00+00
(1 row)
This is asking: what was the absolute time when the clock on the wall in the timezone UTC was showing 2012-03-05 17:00:00?
Example 6
db=# SELECT timestamp '2012-03-05 17:00:00'::timestamp;
timestamp
---------------------
2012-03-05 17:00:00
(1 row)
Here you're casting twice to timestamp, which makes no difference. Let's simplify:
db=# SELECT '2012-03-05 17:00:00'::timestamp;
timestamp
---------------------
2012-03-05 17:00:00
(1 row)
That's clear I think.
Example 7
db=# SELECT timestamp '2012-03-05 17:00:00'::timestamptz;
timestamptz
------------------------
2012-03-05 17:00:00+00
(1 row)
Let's rewrite:
db=# SELECT ('2012-03-05 17:00:00'::timestamp)::timestamptz;
timestamptz
------------------------
2012-03-05 17:00:00+00
(1 row)
You're first parsing the string as a timestamp and then converting it to a timestamptz using the currently set timezone. If we change the timezone, we get something else because Postgres assumes that timezone when converting a timestamp (or a string lacking timezone information) to timestamptz:
db=# SET timezone TO 'Europe/Berlin';
SET
db=# SELECT ('2012-03-05 17:00:00'::timestamp)::timestamptz;
timestamptz
------------------------
2012-03-05 17:00:00+01
(1 row)
This absolute time, expressed in UTC, is 2012-03-05 16:00:00+00, thus different from the original example.
I hope this clarifies things. Again, understanding the difference between timestamp and timestamptz is key. Think of wall time versus absolute time.
Here is what I understand. Please bare with me.My default timezone, defined in the postgresql.conf is UTC. Check this code
SELECT ts FROM (VALUES
(timestamptz '2012-03-05 17:00:00+0') -- outputs 2012-03-05 17:00:00+00 --1
,(timestamptz '2012-03-05 18:00:00+1') -- outputs 2012-03-05 17:00:00+00 --2
,(timestamp '2012-03-05 18:00:00+1') -- outputs 2012-03-05 18:00:00+00 --3
,(timestamp '2012-03-05 11:00:00' AT TIME ZONE '+6') -- outputs 2012-03-05 17:00:00+00 --4
,(timestamp '2012-03-05 17:00:00' AT TIME ZONE 'UTC') -- outputs 2012-03-05 17:00:00+00 --5
,(timestamp '2012-03-05 17:00:00'::timestamp) -- outputs 2012-03-05 17:00:00+00 --6
,(timestamp '2012-03-05 17:00:00'::timestamptz) -- outputs 2012-03-05 17:00:00+00 --7
) t(ts);
Now, pretend this is Postgre talking :
There is special timezone defined for the output. So I will output everything in the default UTC. Lets go.
1 (timestamptz '2012-03-05 17:00:00+0')This is time-aware data, the offset is 0, so its UTC. The default is also UTC. I will save it as is (no need to convert) and output 2012-03-05 17:00:00+00 because UTC input to UTC save to UTC output.
2 (timestamptz '2012-03-05 18:00:00+1')Also time-aware data, the offset is +1, so its not UTC. Offset by minus 1 to convert it to UTC, so I can save it as UTC, that is the default. Output 2012-03-05 17:00:00+00 because not-UTC input to UTC save to UTC output.
3 (timestamp '2012-03-05 18:00:00+1')Time-unaware data. Ignore the offset, assume this is the default UTC and save it as is. Output 2012-03-05 18:00:00+00 because, I-dont-know-I-dont-care-I-will-pretend-this-is-my-default-UTC-input to UTC save to UTC output.
4 (timestamp '2012-03-05 11:00:00' AT TIME ZONE '+6')Again time-unaware data. Ignore the offset, if any. Then convert it to the given AT TIME ZONE '+6' offset so I can treat it as a complete time-unaware data. So my final data is 2012-03-05 17:00:00+00. But this is still not time-aware data. So, I will assume this is my default UTC and save it as is. Output 2012-03-05 17:00:00+00 because, I-dont-know-I-dont-care-I-will-pretend-this-is-my-default-UTC-input to UTC save to UTC output.
5 (timestamp '2012-03-05 17:00:00' AT TIME ZONE 'UTC')Like the previous data, time-unaware data. I will ignore the offset, if any. Then I will convert it to the given AT TIME ZONE 'UTC', so no actual conversion, because there is no actual offset (UTC offset is 0). So my final data is 2012-03-05 17:00:00. But this is still not time-aware data. So, I will assume this is my default UTC and save it as is. Output 2012-03-05 17:00:00 because, I-dont-know-I-dont-care-I-will-pretend-this-is-my-default-UTC-input to UTC save to UTC output
6 (timestamp '2012-03-05 17:00:00'::timestamp)This is time-unaware data , converted to time-unaware data again. So, like 4, I will ignore any offset , if any. There is no AT TIME ZONE either, so no conversions. My final time-unaware data is '2012-03-05 17:00:00'. I will assume this is my default UTC and save it as is. Output 2012-03-05 17:00:00+00 because, I-dont-know-I-dont-care-I-will-pretend-this-is-my-default-UTC-input to UTC save to UTC output
7 (timestamp '2012-03-05 17:00:00'::timestamptz)This is time-unaware data , converted to time-aware data. But there is no offset, conversion, nothing. So, this is UTC. So, I will save it as is. Output 2012-03-05 17:00:00+00 because UTC input to UTC save to UTC output.
(Hope the above will help anyone, in general)
NOW! Regarding the article
Example 1
SELECT timezone('US/Pacific', '2016-01-01 00:00');
Time-unaware data, but I can convert it to time-aware. According to the article, since there is no time zone information, it can be parsed in the default UTC timezone.
So, time-aware, UTC data, save it as is, but convert it to US/Pacific before output it. This is why article says "We get the wall time in California for 2016-01-01 00:00 UTC. " Output is 2015-12-31 16:00:00, that is the wall time in California, for the '2016-01-01 00:00' UTC input.
Article also says "Note that we passed the timestamp as a string, which was implicitly cast to a timestamptz". This could be written as SELECT timezone('US/Pacific', '2016-01-01 00:00'::timestamptz); and still output 2015-12-31 16:00:00. Time-aware data, no offset, so its offset 0, so its UTC. UTC is also the default, so just save it as is. Convert it to US/Pacific before outputting it. This is why it outputs 2015-12-31 16:00:00 again.
Since "timezone(zone, timestamp) is equivalent to the SQL-conforming construct timestamp AT TIME ZONE zone", according to the article, then
SELECT timezone('US/Pacific', '2016-01-01 00:00');
SELECT timezone('US/Pacific', '2016-01-01 00:00'::timestamptz);
timestamptz '2016-01-01 00:00' at time zone 'US/Pacific'
timestamptz '2016-01-01 00:00+00' at time zone 'US/Pacific'
are all the same
Time-aware data (or make it be time-aware), no offset, save it as UTC, output it converted, as US/Pacific.
Example 2
SELECT timezone('US/Pacific', '2016-01-01 00:00'::timestamp);
Time-unaware data. Can I convert it to the UTC default, like in Example 1? No, because is converted to time-unaware (::timestamp part). There is nothing I can do. It is time-aware data.
I will ignore the offset, if any. Unlike 4 above, there is no offset defined, no AT TIME ZONE '+ or -X'. So, to get the UTC I will convert the '2016-01-01 00:00' back to UTC, according to the US/Pacific. Add 8 hours to go from Pacific to UTC. My UTC now is 2016-01-01 08:00:00+00. Save it as is. Output 2016-01-01 08:00:00+00 because, I-dont-know-I-dont-care-I-will-pretend-this-is-my-default-UTC-input to UTC save to UTC output
Again, according to the article "timezone(zone, timestamp) is equivalent to the SQL-conforming construct timestamp AT TIME ZONE zone", so
SELECT timezone('US/Pacific', '2016-01-01 00:00'::timestamp);
timestamp '2016-01-01 00:00' at time zone 'US/Pacific'
timestamp '2016-01-01 00:00+00' at time zone 'US/Pacific'
are all the same
Time-unaware data, ignore offset, convert back to UTC, this is UTC, save as UTC output as UTC.
Thanks
I am trying to process input data putting timestamp when save it
tsSrc timestamp with time zone;
...
tsSrc := strTelegram.rte_data[ iPos ];-- this input data datetime
-- string e.g.'2015/12/13 21:35:26.000'
...
insert into telegram(
tld_id,
ddt_num, tld_src_timestamp,
tld_dst_timestamp, tld_year, tld_month,
tld_day, tld_hour, tld_min,
tld_sec, tld_data
) values(
uuId,
strTelegram.rte_type,
tsSrc,
strTelegram.rte_dst_timestamp,
extract(year from tsSrc), extract(month from tsSrc),
extract(day from tsSrc), extract(hour from tsSrc),
extract(minute from tsSrc), extract(second from tsSrc),
strTelegram.rte_data
);
But I have got unexpected result, tsSrc saved as 2015-12-13 20:35:26+03 i.e. has hour -1 shift, at the same time extract(hour from tsSrc) returns right value and saved as 21. What I am doing wrong?
Timezone is set as 'MSK-3' in postgresql.conf, select now() returns right datetime, postgresql 9.3.
You need to understand handling of timestamp (timestamp without time zone) and timestamptz (timestamp with time zone) and how each interacts with the timezone setting of your current session.
To explain the "difference" you observe we would need to know the exact table definition and the timezone setting of the session saving the row, as well as the timezone setting of the session displaying the row.
For example, if you take the timestamp literal '2015-12-13 21:35:26' (use ISO format to avoid additional complication with the input format!) and save it to a timestamptz column in a session with time zone offset +2 and later select the same row in a session with with time zone offset +3, then you get what you see:
SELECT '2015-12-13 21:35:26'::timestamp AT TIME ZONE '+2' AT TIME ZONE '+3';
Result:
'2015-12-13 20:35:26'
In other words: the timestamptz value '2015-12-13 20:35:26+03' is exactly the same (same point in time) as '2015-12-13 21:35:26+02', only the display has been adapted to your time zone setting. When you treat the timestamptz value according to the clock on the wall in your corner of the world (like you do with extract(hour from tsSrc)), you get different results depending where you are currently (the timezone setting of your session).
Detailed explanation:
Ignoring timezones altogether in Rails and PostgreSQL
I'm in the Time Zone Europe/Berlin (+02), Postgresql is running at UTC (+00).
I need to get the timestamp with time zone at the latest local midnight date (The start of day date of the current day in my time zone).
So my end result would be something like this if we have 2013-03-03 14:00:00+02
2013-03-03 22:00:00+00
2013-03-04 00:00:00+02 // the same
I tried to get this date with
SELECT TIMESTAMP 'today' AT TIME ZONE 'Europe/Berlin'
Unfortunately this yields the wrong date (the previous day midnight) during 00:00 and 02:00 as the UTC time is stil at the previous day and today seems to use utc to calculate the rest.
If we have 2013-03-03 00:05 at Europe/Berlin this will return
2013-05-01 22:00:00+00
If I want to have the correct date I need to use
SELECT date_trunc('day', now() AT TIME ZONE 'Europe/Berlin') AT TIME ZONE 'Europe/Berlin';
2013-05-02 22:00:00+00
which is correct, but quite ugly.
Is there a cleaner variant of this command?
Use timestamptz. The tz at the end meaning with time zone:
SELECT TIMESTAMPTZ 'today' AT TIME ZONE 'Europe/Berlin'
Or if you like it more explicit:
SELECT TIMESTAMP with time zone 'today' AT TIME ZONE 'Europe/Berlin'
Wrap it in a function:
create function midnight() returns timestamptz as $$
select date_trunc('day', now() AT TIME ZONE 'Europe/Berlin') AT TIME ZONE 'Europe/Berlin';
$$ language sql;
Based on Erwin's answer to a related question, this was the simplest and fastest way I figured out how to do it:
SELECT timezone('Europe/Berlin', now()::date::timestamp) AS local_midnight_in_utc;
The key is the cast to a timestamp, which removes the time zone from the date.
You can test your sample time of '2013-03-03 00:05' with this:
SELECT timezone('Europe/Berlin', '2013-03-03 00:05'::date::timestamp) AS midnight;
and it returns
2013-03-02 23:00:00+00
According to explain analyze, this is about 3x as fast as the datetrunc version. A runtime of .017ms vs 0.006ms based on a best of 5 runs.