Add Hours, minutes and seconds to Spark dataframe - pyspark

Question

Is there a Spark SQL function to add Hours, Minutes and Seconds to existing timestamp column.
For example:
+----------+-------------------+-------------------+
| dt| txn_dt| txn_dt_tm|
+----------+-------------------+-------------------+
|2008-08-15|2008-08-15 00:00:00|2008-08-15 05:00:00|
+----------+-------------------+-------------------+
I need add 23 hours 59 minutes and 59 seconds to txn_dt column.
Output:
+----------+-------------------+-------------------+
| dt| txn_dt| txn_dt_tm|
+----------+-------------------+-------------------+
|2008-08-15|2008-08-15 23:59:59|2008-08-15 05:00:00|
+----------+-------------------+-------------------+
Update:
I was able to get it using INTERVAL, but not sure this is efficient way of doing it.
df.select((F.col("txn_dt") + F.expr("INTERVAL 23 HOURS") + F.expr("INTERVAL 59 MINUTES") + F.expr("INTERVAL 59 SECONDS")).alias("txn_dt_tm"))

Answer 1

You need to customize the udf function, such as:
import org.apache.spark.sql.functions._
val timeUdf = udf{(time: java.sql.Timestamp) => new java.sql.Timestamp(time.getTime + 24*60*60*1000 - 1000)}
df.withColumn("dt", timeUdf(df("dt"))).show()
and the result:
+--------------------+---+
| dt| id|
+--------------------+---+
|2008-08-15 23:59:...| 1|
+--------------------+---+
i hope this will help you.