How to extract year from a date string? - scala

Question

I'm using spark 2.1.2.
I'm working with datetime data, and would like to get the year from a dt string using spark sql functions.
The code I use is as follows:
import org.apache.spark.sql.functions._
import org.apache.spark.sql._
import org.apache.spark.sql.types._
val spark: SparkSession = SparkSession.builder().
appName("myapp").master("local").getOrCreate()
case class Person(id: Int, date: String)
import spark.implicits._
val mydf: DataFrame = Seq(Person(1,"9/16/13")).toDF()
val select_df: DataFrame = mydf.select(unix_timestamp(mydf("date"),"MM/dd/yy").cast(TimestampType))
select_df.select(year($"date")).show()
I expect the year of the date as 13 in the example above.
Actual: org.apache.spark.sql.AnalysisException: cannot resolve 'date' given input columns: [CAST(unix_timestamp(date, MM/dd/yy) AS TIMESTAMP)];;
'Project [year('date) AS year(date)#11]

Answer 1

case class Person(id: Int, date: String)
val mydf = Seq(Person(1,"9/16/13")).toDF
val solution = mydf.withColumn("year", year(to_timestamp($"date", "MM/dd/yy")))
scala> solution.show
+---+-------+----+
| id| date|year|
+---+-------+----+
| 1|9/16/13|2013|
+---+-------+----+
It looks like year does not give you two digits but four for years. I'm leaving the string truncation as a home exercise for you :)
Actual: org.apache.spark.sql.AnalysisException: cannot resolve 'date' given input columns: [CAST(unix_timestamp(date, MM/dd/yy) AS TIMESTAMP)];; 'Project [year('date) AS year(date)#11]
The reason of the exception is that you want to access the "old" date column (in select(year($"date"))) that's no longer available after select (select(unix_timestamp(mydf("date"),"MM/dd/yy").cast(TimestampType)).
You could use alias or as to change the weird-looking auto-generated name into something else like date again, and that would work.