I'm trying to convert a timestamp string to a timestamp which is in the ISO8601 format (-isch) in a PostgreSQL DB. Although I am almost there, it's just not what it should be.
The string is as follows:
2022-06-22T02:22:11.310682187Z
Using various sources on to_timestamp, I was able to run the following command:
to_timestamp(timestampstring,'yyyy-MM-dd"T"HH24:MI:SS.MSUS"Z"')
Which gave me the following result:
2022-06-22 02:22:11.992 +0200
Which is almost what it should be, where it not for the fact that it has added a +2 timezone where the string is in UTC and the milli/micro seconds are not what the should be.
Some things that I should mention, I know that the string is a bit odd (there is no need for that precision in the seconds), but this is what I get from the API I'm calling (which is not my own). I've tried using a format with only the milli of micro seconds but to_timestamp will throw an error in that situation. I've tried to indicate that the Z is actually the UTC timezone, but to no avail. Also, I realise that I can also parse this by taking a substring of the time but giving that the format is a known format, that feels a bit like giving up. Any help on how I could get this to parse properly.
This one works for me:
SELECT '2022-06-22T02:22:11.310682187Z'::timestamptz at time zone 'UTC';
Result: 2022-06-22 02:22:11.310682
Related
If I run this query:
SELECT ('2021-11-02 08:00:00+00' AT TIME ZONE 'UCT') at TIME ZONE 'UCT+2';
It gives this result:
2021-11-02 10:00:00+00
Which seems correct to me...
However, my timezone data comes in string format, for example: "Africa/Johannesburg", which I know is on the UCT+2 timezone.
But running this query:
SELECT ('2021-11-02 08:00:00+00' AT TIME ZONE 'UCT') at TIME ZONE 'Africa/Johannesburg';
Gives this result:
2021-11-02 06:00:00+00
Which is the opposite of what I expect to happen. I know I can write code to convert the strings to a UCT value but I don't understand why the string values seems to do the opposite of using UCT values. Could someone please explain why this happens?
Wow, there's lots to unpack here.
The main thing is that (to mis-quote Inigo Montoya) I do not think that UTC+2 means what you think it means.
One important thing here is the precise format.
You say you get this result:
2021-11-02 06:00:00+00
But What I get is:
2021-11-02 06:00:00
And that's really important because what I get doesn't include that +00. And that's because it's still 08:00 in the +00 time zone.
The AT TIME ZONE syntax says something like "What would a local at this place see the time being at this specific UTC time?". And someone in Johannesburg would see it as 10:00. Someone in London (in winter time) would see it as 06:00. But it's the same instant in time.
So the output of AT TIME ZONE doesn't usually have that time zone offset from UTC (+00) included in it.
Your conversions with two AT TIME ZONE clauses are redundant. Because it knows it's at that time zone, and hence what the UCT time is. You'll get the same result with one AT TIME ZONE clause (try it).
The reason why you get different answers appears to be because PostgreSQL does not interpret UCT+2 as the TimeZone that is 2 hours ahead of UCT. It interprets that as you specifying a POSIX time zone definition, which could also include daylight savings time rules etc.
This page: https://www.postgresql.org/docs/current/datatype-datetime.html says PostgreSQL will accept time zones specified in 3 ways:
Full name - e.g. 'Africa/Johannesburg'
Abbreviation - e.g. 'SAST'
Posix format, e.g. 'CET-1CEST,M3.5.0,M10.5.0/3' (This is, apparently, Paris)
Actually, it doesn't include the Posix format example, that's on this page: https://www.postgresql.org/docs/current/datetime-posix-timezone-specs.html
Basically, the posix format is a way of specifying the time zone rules.
Something like
[abbreviation][offset][daylight savings abbrev][DST offset][dst rules]
And so saying 'UCT+2' is probably "bad form" because you're trying to change the meaning of the abbreviation 'UTC' which has a standard usage.
You can verify that it isn't using the "UCT" part of your time zone to refer to the actual UCT time zone. If you do this:
SELECT ('2021-11-02 08:00:00+00' AT TIME ZONE 'BOB')
PostgreSQL will say it doesn't recognise time zone "BOB".
But if you do
SELECT ('2021-11-02 08:00:00+00' AT TIME ZONE 'BOB+2')
It'll happily give you 06:00:00. In fact, if you like, you can stick (almost) anything in there instead of "UCT". It doesn't get used in the calculation.
Note that the posix page I linked says
POSIX time zone specifications are inadequate to deal with the
complexity of real-world time zone history, but there are sometimes
reasons to use them.
Finally, 'UCT' is perfectly valid but it's doing my head in. I've only seen it as 'UTC', so TIL.
I want to convert the date from former to later format
2020-04-14T14:56:43
TO
2020-04-14 14:56:43 UTC
Basically how to convert the DATETIME into TIMESTAMP IN Dataprep?
Why would you want to use TIMESTAMP instead of a DATETIME?
because it's smaller in size? (4 bytes vs 8 bytes)
Are you using MYSQL? I understand that there it's converted to UTC on write, and then converted back to the server's timezone on retrieval. (which isn't happening for DATETIME)
Or in general, working with timezones?
anyway, you can just remove the T in the middle between the date and the time by brushing over it, and replacing it with an empty space.
And if you desire to add a timezone (which would be.. +00), you can do that aswell. Trifacta will recognize that as a valid date type.
IF it's a unix-timestamp you're interested in - you can do that aswell with the function "unixtime()".
you can see all the tokens with which to construct and change your datetype here:
https://docs.trifacta.com/display/SS/Datetime+Data+Type
also, the community-website has a lot of dateformats-related questions.
https://community.trifacta.com/s/
In pySpark need to convert forex trading date time as everyday 16pm as day+1, so I did below code and works well, however when I tried to truncate hour/minute/second and keep up to date only, no matter what I do system always converts to Sydney time (I am in Sydney Australia).
I searched google and know I need to set spark session to GMT, however I am using a system called Palantir, it is different from normal pySpark code editor, when I dump code spark.conf.set("spark.sql.session.timeZone", "UTC") it always gave me error. Is there anyway can round to day in London zone without setting spark timezone? Thank you
.withColumn('test_trade_date', F.from_utc_timestamp(F.from_unixtime(F.unix_timestamp(F.col('trade_date_time'))+8*60*60), 'LONDON'))\
You can write a pandas_udf to convert the datetime to whatever timezone you want.
I have to convert IST(India) to EET(Finland) timing using perl or shell ...
Means i have to subtract 3 hours,30 minutes from a given specific (ISD)time (not from the current time).
Time is in this format: YYYY-MM-DD HH:MM:SS
For ex: IST: 2016-01-01 02:30:00
Then after subtracting 3hours and 30 minutes ,I should get,
EET: 2015-12-31 23:00:00
The thing is after subtracting,if required the date,month and year should also change.
Can i do this using perl? Can anyone help me on this?
I'm not going to give you the actual code as you haven't demonstrated that you have made any effort to solve this yourself.
But the way to do this is to use a real Date/Time handling library. In Perl, that probably means DateTime. You can use DateTime::Format::Strptime to generate a DateTime object from a string.
In summary, your approach should be:
Parse your string into a DateTime object (being careful to ensure that the parsing object knows that the time zone is ISD (Icelandic time, I assume [Update: or, more likely, Indian]).
Convert the time zone in your parsed object to EET.
Use the parsed object's strftime method to produce the output time in the correct output.
Update: And I'll just add the standard advice about handling dates and times. You should always transmit and store dates and times in UTC. Local time zones should only every be displayed to users.
In a scala program, I receive from client side a specific date for instance:
2013-10-20T23:59:59.999Z
and I really want to keep this date when saving into DB and not convert to local, so this line:
debug("--sql timestamp: " + new Timestamp(reading.timestamp.getMillis()))
is printing out: 2013-10-21 02:59:59.999(I am in Romania).
Is there any way I can ignore timezone?
This is Timestamp.toString() behavior. java.sql.Timestamp extends java.util.Date and in its toString() method it uses, in particular, super.getHours(), which, according to javadoc, returns hours interpreted in local timezone - exactly as you observe.
However, internally Timestamp still holds correct timestamp value. There may be problems with storing it to the database, though. See this answer.
2013-10-20T23:59:59.999Z and 2013-10-21 02:59:59.999 are actually the same time: 2013-10-20T23:59:59.999Z is in the UTC time zone (Z), whereas the second one is relative, and expressed as your local time zone (UTC+3 then in Romania).
In PostgreSQL, you should store your timestamps as TIMESTAMP WITH TIME ZONE (TIMESTAMPTZ) in your database to handle this. You'll always be able to print it out later in the time zone you choose then (e.g. UTC). (You might be interested in this recent question to understand why the storage type matters.)
If you want to print out the timestamp in the UTC/Z time zone again, new DateTime(millis, DateTimeZone.UTC) should help (with Joda Time).