read files from current date minus 90 days in spark - scala

Question

I am reading all one by one files which is stored in a directory structure as YY=18/MM=12/DD=10 and need to read only current date minus 60 days. Files will be created created for every day and possibility is also that some day files wont create. so, for that day folder will not create.
I am reading all files which is stored in a directory structure as YY/MM/DD.
I am writing below code but its not working.
var datecalculate = {
var days = 0
do{
val start = DateTime.now
var start1 = DateTime.now.minusDays(days)
days = days + 1
var start2 = start1.toString
datecalculatenow(start2) }
while (days <= 90)
}
def datecalculatenow(start2:String):String={
var YY:String = start2.toString.substring(0,4)
var MM:String = start2.toString.substring(5,7)
var DD:String = start2.toString.substring(8,10)
var datepath = "YYYY=" + YY +"/MM=" +MM +"/DD=" +DD
var datepath1 = datepath.toString
org.apache.spark.sql.SparkSession.read.option("delimiter","|").
option("header","true").option("inferSchema","true").
csv("/Table/Files" + datepath1 )
}
I expect to read every files from current date minus 60 days, which has directory structure as YY/MM/DD

Answer 1

With spark sql you can use the following in a select statement to subtract 90 days;
date_sub(CAST(current_timestamp() as DATE), 90)

Answer 2

As it's possible to generate a dataframe from a list of path , why are you not first generating list of path. Here is the simple and concise way to read data from multiple paths:
val paths = (0 until 90).map(days => {
val tmpDate = DateTime.now.minusDays(days).toString()
val year = tmpDate.substring(0,4)
val month = tmpDate.substring(5,7)
val opdate = tmpDate.toString.substring(8,10)
(s"basepath/YY=$year/MM=$month/DD=$opdate")
}).toList
val df = spark.read.
option("delimiter", "|").
option("header", "true").
option("inferSchema","true")
.csv(paths:_*)
While generating paths, you can filter out the paths that do not exist. I've used some of your codes with some modifications. I've not tested in my local setup but the idea is same. Hopefully it'll help you.