How could I filter into Join RDD just to select some fields of the table in Apache Spark? - scala

Question

I'm trying to filter two fields of two datasets in csv files. I've already applied an INNER JOIN for dataset1.csv and dataset2.csv.
This is my starting code:
case class Customer(
Customer_ID:Int,
Customer_Name:String,
Account_Number:Double,
Marital_Status:String,
Age:Int,
Contact:Double,
Location:String,
Monthly_Income_USD:Double,
Yearly_Balance_USD:Double,
Job_type:String,
Credit_Card:String
)
case class HouseLoan(
Customer_ID:Int,
Account_Number:Double,
House_Loan_Amount_USD:Double,
Total_Installment:Int,
Installment_Pending:Int,
Loan_Defaulter:String
)
val custRDD = sc.textFile("dataset1.csv")
.map(_.split(","))
.map(r => (
r(0),
Customer( r(0).toInt, r(1), r(2).toDouble, r(3), r(4).toInt, r(5).toDouble, r(6), r(7).toDouble, r(8).toDouble, r(9), r(10))
)
)
val houseRDD = sc.textFile("dataset2.csv")
.map(_.split(","))
.map(r => (
r(0),
HouseLoan(r(0).toInt,r(1).toDouble,r(2).toDouble,r(3).toInt,r(4).toInt,r(5))
)
)
val joinTab = custRDD.join(houseRDD)
joinTab.collect().foreach(println)
Until here everything is okay, the result is show in the image.
Now, I need:
the field of Customer_ID (the key of both tables)
the Job_Type
for those who are:
"Doctor" and
House_Loan_Amoun_USD is more than 1000000
using join.
I tried something like
val joinTab = custRDD.filter{record => (record.split(",")(9) == "Doctor").join(houseRDD)filter{record => (record.split(",")(2) > 100000}
but it is obviously wrong and I'm still noob for Apache Spark.
Note: I can't use spark sql because I'm learning this topic in my university (Spark Core - RDD) so I must do it with RDD join.