SCALA multiple nested case statement in map RDD operations - scala

Question

I have an RDD with 7 string elements
linesData: org.apache.spark.rdd.RDD[(String, String, String, String, String, String, String)]
I need to figure out for each record the following 2 items: Status and CSrange.
the logic is something like:
case when a(6)=0
if Date(a(5)) > Date(a(4)) then
if Date(a(5)) - Date(a(4)) > 60 days then
staus = '60+'
else
status = 'Curr'
endif
else
status = 'Curr'
end
when (a(6) >=1 and a(6) <=3 ) then
Status = 'In-Forcl'
when (a(6) >=4 and a(6) <=8)
Status = 'Forclosed'
else
Status = 'Unknown'
end case
case when (a(1) <640 and a(1) >0 ) then CSrange = '<640'
when (a(1) <650 and a(1)> 579 then CSrange = '640-649'
when (a(1) <660 and a(1)> 619 then CSrange = '650-659'
when (a(1) <680 and a(1)> 639 then CSrange = '640-649'
when (a(1) >789 then CSrange = '680+'
else
CSRange ='Unknown'
end case
At this point, I would like to write the data out to disk, with probably 9 elements insteatd of 7. (later I will need to do calculate the rate of each of the statuses above by various elements).
My first problems are:
1. how to do date arithmetics? as I need to stay at the RDD level (no data frames).
2. I don't know how to do the CASE statements in SCALA.
SAMPLE DATA:
(2017_7_0555,794,Scott,CORNERSTONE,8/1/2017,8/1/2017,0)
(2017_7_0557,682,Hennepin,LAKE AREA MT,9/1/2017,8/1/2017,0)
(2017_7_0565,754,Ramsey,GUARANTEED R,6/1/2017,8/1/2017,0)
(2017_7_0570,645,Hennepin,FAIRWAY INDE,2/1/2015,8/1/2017,5)
(2017_7_0574,732,Wright,GUARANTEED R,7/1/2017,8/1/2017,0)
(2017_7_0575,789,Hennepin,GUARANTEED R,8/1/2017,8/1/2017,0)
(2017_7_0577,662,Hennepin,MIDCOUNTRY,8/1/2017,8/1/2017,0)
(2017_7_4550,642,Mower,WELLS FARGO,5/1/2017,8/1/2017,0)
(2017_7_4574,689,Hennepin,RIVER CITY,8/1/2017,8/1/2017,0)
(2017_7_4584,662,Hennepin,WELLS FARGO,8/1/2017,8/1/2017,0)
(2017_7_4600,719,Ramsey,PHH HOME LOA,5/1/2017,8/1/2017,0)

Answer 1

I would create a case class of seven fields and map RDD[(,...)] to RDD[MyClass] and cast to appropriate types. I recommend you JodaTime date library for dates. This will make your code more descriptive.
Then in a map extract status and range into two functions:
myRDD.map(myInstance => (getStatus(myInstance), getRange(myInstance)))
def getStatus(myInstance: MyClass) : String = {
val (_,_,_,_,date4,date5,field6,_) = MyClass.unapply(myInstance).get
field6 match {
case 0 => {
if(date5.isAfter(date4) {
if(date4.plusDays(60).isAfter(date5)){
"60+"
} else {
"Curr"
}
}
}
case x if (x >= 1 && x <= 3) => "Forclosed"
....
}
}
Note:
I haven't tested the code.
Rename variables from the example.
I show you an example about how to manage JodaTime dates, and scala pattern matching. You have to finish the function and define the other one.