Spark (scala) change date in datetime column - scala

Question

pyspark change day in datetime column
I'm trying to do something similar to the answer above. I'm getting
value replace is not a member of java.sql.Timestamp
val changeDay = udf((date:java.sql.Timestamp) => {
val day = 1
date.replace(day=day)
})
val df2 = df1.withColumn("newDateTime", changeDay($"datetime"))
What I can't figure out is what functions are available for this java.sql.Timestamp object. When I google it, it almost seems like the answers are not related to the same type.

Answer 1

You can convert Timestamp to java.time's LocalDateTime and change its day value via withDayOfMonth(day), as shown below:
import java.sql.Timestamp
import org.apache.spark.sql.functions._
import spark.implicits._
val df = Seq(
(1, Timestamp.valueOf("2019-03-07 12:30:00")),
(2, Timestamp.valueOf("2019-04-08 09:00:00"))
).toDF("id", "ts")
def changeDay(day: Int) = udf{ (ts: Timestamp) =>
import java.time.LocalDateTime
val changedTS = ts.toLocalDateTime.withDayOfMonth(day)
Timestamp.valueOf(changedTS)
}
df.withColumn("newTS", changeDay(1)($"ts")).show
// +---+-------------------+-------------------+
// | id| ts| newTS|
// +---+-------------------+-------------------+
// | 1|2019-03-07 12:30:00|2019-03-01 12:30:00|
// | 2|2019-04-08 09:00:00|2019-04-01 09:00:00|
// +---+-------------------+-------------------+

Answer 2

so this probably isn't the best way to do this but here is one way
val DateTimeString = date.toString()
val DTtime = DateTimeString.split(" ")(1)
DTday + " " + DTtime