How can I cast string with offset after space into pyspark timestamp? - pyspark

Question

I have a DF like this:
df = spark.createDataFrame(
["2003-01-01 02:00:00.0 -8:00"],
"string"
).toDF('ts')
df.collect()
[Row(ts='2003-01-01 02:00:00.0 -8:00')]
An I'm trying to make a timestamp type out of my ts but I just can't seem to make it work.
I tried many variants:
df = df.withColumn('cast', to_timestamp('ts', 'yyyy-MM-dd HH:mm:ss.S Z'))
df.collect()
[Row(ts='2003-01-01 02:00:00.0 -8:00', cast=None)]
df = df.withColumn('cast', to_timestamp('ts', 'yyyy-MM-dd HH:mm:ss.S X'))
df.collect()
[Row(ts='2003-01-01 02:00:00.0 -8:00', cast=None)]
df = df.withColumn('cast', to_timestamp('ts', 'yyyy-MM-dd HH:mm:ss.S x'))
df.collect()
[Row(ts='2003-01-01 02:00:00.0 -8:00', cast=None)]
df = df.withColumn('cast', to_timestamp('ts'))
df.collect()
[Row(ts='2003-01-01 02:00:00.0 -8:00', cast=None)]
But it does not work. It is frustrating especially since just removing the space before the offset works even without specifying the format..
df = spark.createDataFrame(
["2003-01-01 02:00:00.0-8:00"],
"string"
).toDF('ts')
df = df.withColumn('cast', to_timestamp('ts'))
df.collect()
[Row(ts='2003-01-01 02:00:00.0-8:00', cast=datetime.datetime(2003, 1, 1, 11, 0))]

Answer 1

You need to use ZZZZZ instead of Z. Also works with x and X:
Spark SQL manual says:
Five letters outputs the hour and minute and optional second, with a colon, such as +01:30:15.
Full text with other options explained:
One letter outputs just the hour, such as +01, unless the minute is non-zero in which case the minute is also output, such as +0130. Two letters outputs the hour and minute, without a colon, such as +0130. Three letters outputs the hour and minute, with a colon, such as +01:30. Four letters outputs the hour and minute and optional second, without a colon, such as +013015. Five letters outputs the hour and minute and optional second, with a colon, such as +01:30:15. Six or more letters will fail.
Spark SQL example:
SELECT to_timestamp('2003-01-01 02:00:00.1 -08:00', 'yyyy-MM-dd HH:mm:ss.S ZZZZZ')
returns
2003-01-01T10:00:00.100+0000