In pySpark need to convert forex trading date time as everyday 16pm as day+1, so I did below code and works well, however when I tried to truncate hour/minute/second and keep up to date only, no matter what I do system always converts to Sydney time (I am in Sydney Australia).
I searched google and know I need to set spark session to GMT, however I am using a system called Palantir, it is different from normal pySpark code editor, when I dump code spark.conf.set("spark.sql.session.timeZone", "UTC") it always gave me error. Is there anyway can round to day in London zone without setting spark timezone? Thank you
.withColumn('test_trade_date', F.from_utc_timestamp(F.from_unixtime(F.unix_timestamp(F.col('trade_date_time'))+8*60*60), 'LONDON'))\
You can write a pandas_udf to convert the datetime to whatever timezone you want.
Related
I'm trying to convert a timestamp string to a timestamp which is in the ISO8601 format (-isch) in a PostgreSQL DB. Although I am almost there, it's just not what it should be.
The string is as follows:
2022-06-22T02:22:11.310682187Z
Using various sources on to_timestamp, I was able to run the following command:
to_timestamp(timestampstring,'yyyy-MM-dd"T"HH24:MI:SS.MSUS"Z"')
Which gave me the following result:
2022-06-22 02:22:11.992 +0200
Which is almost what it should be, where it not for the fact that it has added a +2 timezone where the string is in UTC and the milli/micro seconds are not what the should be.
Some things that I should mention, I know that the string is a bit odd (there is no need for that precision in the seconds), but this is what I get from the API I'm calling (which is not my own). I've tried using a format with only the milli of micro seconds but to_timestamp will throw an error in that situation. I've tried to indicate that the Z is actually the UTC timezone, but to no avail. Also, I realise that I can also parse this by taking a substring of the time but giving that the format is a known format, that feels a bit like giving up. Any help on how I could get this to parse properly.
This one works for me:
SELECT '2022-06-22T02:22:11.310682187Z'::timestamptz at time zone 'UTC';
Result: 2022-06-22 02:22:11.310682
I save my date as a local date, but when I read it back, it treats it as if it was a UTC date so it slips by several hours.
The dates are passed in as strings in the form '2020-03-05 09:05:23' as query parameters but when they are retrieved they might look like '2020-03-04 10:05:23' because I am 13 hours ahead of Greenwich.
For MariaDB (or MySQL):
Use DATETIME as a picture of the clock on the wall.
Use TIMESTAMP to adjust to the system's timezone.
Set the system's timezone according to where it lives in the world.
I have to convert IST(India) to EET(Finland) timing using perl or shell ...
Means i have to subtract 3 hours,30 minutes from a given specific (ISD)time (not from the current time).
Time is in this format: YYYY-MM-DD HH:MM:SS
For ex: IST: 2016-01-01 02:30:00
Then after subtracting 3hours and 30 minutes ,I should get,
EET: 2015-12-31 23:00:00
The thing is after subtracting,if required the date,month and year should also change.
Can i do this using perl? Can anyone help me on this?
I'm not going to give you the actual code as you haven't demonstrated that you have made any effort to solve this yourself.
But the way to do this is to use a real Date/Time handling library. In Perl, that probably means DateTime. You can use DateTime::Format::Strptime to generate a DateTime object from a string.
In summary, your approach should be:
Parse your string into a DateTime object (being careful to ensure that the parsing object knows that the time zone is ISD (Icelandic time, I assume [Update: or, more likely, Indian]).
Convert the time zone in your parsed object to EET.
Use the parsed object's strftime method to produce the output time in the correct output.
Update: And I'll just add the standard advice about handling dates and times. You should always transmit and store dates and times in UTC. Local time zones should only every be displayed to users.
In a scala program, I receive from client side a specific date for instance:
2013-10-20T23:59:59.999Z
and I really want to keep this date when saving into DB and not convert to local, so this line:
debug("--sql timestamp: " + new Timestamp(reading.timestamp.getMillis()))
is printing out: 2013-10-21 02:59:59.999(I am in Romania).
Is there any way I can ignore timezone?
This is Timestamp.toString() behavior. java.sql.Timestamp extends java.util.Date and in its toString() method it uses, in particular, super.getHours(), which, according to javadoc, returns hours interpreted in local timezone - exactly as you observe.
However, internally Timestamp still holds correct timestamp value. There may be problems with storing it to the database, though. See this answer.
2013-10-20T23:59:59.999Z and 2013-10-21 02:59:59.999 are actually the same time: 2013-10-20T23:59:59.999Z is in the UTC time zone (Z), whereas the second one is relative, and expressed as your local time zone (UTC+3 then in Romania).
In PostgreSQL, you should store your timestamps as TIMESTAMP WITH TIME ZONE (TIMESTAMPTZ) in your database to handle this. You'll always be able to print it out later in the time zone you choose then (e.g. UTC). (You might be interested in this recent question to understand why the storage type matters.)
If you want to print out the timestamp in the UTC/Z time zone again, new DateTime(millis, DateTimeZone.UTC) should help (with Joda Time).
I have a requirement to disregard records from the DB that are more than 10 minutes old. However, the DB server is present in a different time zone than the app server. I tried to leverage the time zone details from the timestamp column value but it seems that they do not store the time zone details in that column value (bad design?). However, i have found a way to get this information for the DB instance using a query:
select dbtimezone from dual.
However, most of the implementations in java support time zones via names and not offset information. I need to be able to translate this offset exactly to a timezone (EST etc) so that i may not miss any DST related time in my calculations. like so:
TimeZone dbZone = TimeZone.getTimeZone(10000); // offset is +10000
Calendar cal = Calendar.getInstance(dbZone);
cal.setTime(new Date());
DateFormat df = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss");
df.setTimeZone(dbZone);
Date parsedDate = df.parse(df.format(cal.getTime()));
The plan is to convert the present client/app time to the DB specific timezone time and perform the difference between the two.
This cannot be done in a query due to some restraints. Please do not ask me to write a query to get latest records etc. Must be done in Java.
Any tips?
I am guessing it might get you in the right direction. You can try the following so you know the offset from EST and can do the calculation accordingly:
SELECT TZ_OFFSET('US/Eastern') FROM DUAL;
So a return of -3:00 would mean it is PST time. Maybe you've already tried this?