Spark Scala - Null when trying to extract time from timestamp - scala

Question

I'm trying to extract the time from a timestamp using the below code, but it is returning a null value instead of the time. I have already filtered my dataset to have the records I need, so I can ignore AM/PM that comes from the input column.
I did some reading and it seems using date_format should work in this circumstance.
Any thoughts?
Current Output:
+----------------------+----------------------+---------+------------+------------+
|tpep_pickup_datetime |tpep_dropoff_datetime |timestamp|total_amount|pickupWindow|
+----------------------+----------------------+---------+------------+------------+
|05/18/2018 09:56:20 PM|05/18/2018 10:50:38 PM|35780 |52.87 |null |
|05/18/2018 10:52:49 PM|05/18/2018 11:08:47 PM|39169 |14.76 |null |
|05/18/2018 09:01:22 PM|05/18/2018 09:05:36 PM|32482 |6.3 |null |
|05/18/2018 09:00:29 PM|05/18/2018 09:05:31 PM|32429 |7.56 |null |
+----------------------+----------------------+---------+------------+------------+
Current Code:
val taxiSub = spark.read.format("csv").option("header", true).option("inferSchema", true).load("/user/zeppelin/taxi/TaxiSubset.csv") //read Data
taxiSub.createOrReplaceTempView("taxiSub") //Create View
val stamp = taxiSub.withColumn("timestamp", unix_timestamp($"tpep_pickup_datetime", "MM/dd/yyyy hh:mm:ss")) //create timestamp
val h = hour(unix_timestamp($"tpep_pickup_datetime","MM/dd/yyyy hh:mm:ss").cast("timestamp"))
val subset= stamp.withColumn("hour",h).filter("hour BETWEEN 9 AND 10").where($"tpep_pickup_datetime".contains("PM")).filter($"total_amount" < 200.00) //filter records between 9pm and 11pm and < 200 total amount
val myData = subset.withColumn("tmp",to_timestamp(col("tpep_pickup_datetime"),"MM/dd/yyyy HH:mm:ss")).//add new timestamp type field
withColumn("timestamp", unix_timestamp(concat_ws(":",hour(col("tmp")),minute(col("tmp")),second(col("tmp"))),"hh:mm:ss")). //extract hour,minute and convert to epoch timestamp value
drop("tmp").select("tpep_pickup_datetime","tpep_dropoff_datetime","timestamp","total_amount")
val testing = myData.withColumn("pickupWindow",date_format($"tpep_pickup_datetime","hh:mm:ss"))
testing.show(false)

Answer 1

.dateformat() expects col value in yyyy-MM-dd [hh|HH]:mm:ss format but the input data having MM/dd/yyyy..etc.
first we need to convert tpep_pickup_datetime to timestamp using to_timestamp function then apply date_format to extract hh:mm:ss.
Example:
df.show(false)
//+----------------------+
//|tpep_pickup_datetime |
//+----------------------+
//|05/18/2018 09:56:20 PM|
//|05/18/2018 10:52:49 PM|
//+----------------------+
//to get 24hr format HH value
df.withColumn("pickupWindow",date_format(to_timestamp(col("tpep_pickup_datetime"),"MM/dd/yyyy hh:mm:ss a"),"HH:mm:ss")).
show()
//using from_unixtime,unix_timestamp
df.withColumn("pickupWindow",from_unixtime(unix_timestamp(col("tpep_pickup_datetime"),"MM/dd/yyyy hh:mm:ss a"),"HH:mm:ss")).show()
//+--------------------+------------+
//|tpep_pickup_datetime|pickupWindow|
//+--------------------+------------+
//|05/18/2018 09:56:...| 21:56:20|
//|05/18/2018 10:52:...| 22:52:49|
//+--------------------+------------+
//to get 12hr format hh value
df.withColumn("pickupWindow",date_format(to_timestamp(col("tpep_pickup_datetime"),"MM/dd/yyyy hh:mm:ss a"),"hh:mm:ss")).
show()
//Or using unix_timestamp,from_unixtime
df.withColumn("pickupWindow",from_unixtime(unix_timestamp(col("tpep_pickup_datetime"),"MM/dd/yyyy hh:mm:ss a"),"hh:mm:ss")).show()
//+--------------------+------------+
//|tpep_pickup_datetime|pickupWindow|
//+--------------------+------------+
//|05/18/2018 09:56:...| 09:56:20|
//|05/18/2018 10:52:...| 10:52:49|
//+--------------------+------------+