Processing multiple files separately in single spark submit job - scala

Question

I have following directory structure:
/data/modelA
/data/modelB
/data/modelC
..
Each of these files have data in format (id,score), I have to do following for them separately-
1) group by scores and sort the scores in descending(DF_1: score,count)
2) from DF_1 compute the cumulative frequency for each sorted group of score (DF_2: score, count, cumFreq)
3) from DF_2 select cumulative frequencies that lie between 5-10 (DF_3: score, cumFreq)
4) from DF_3 select minimum score(DF_4: score)
5) from file select all id which have score greater than score in DF_4 and save
I am able to do this by reading the directory as wholeTextFile and creating a common dataframe for all the models, then use group by on model.
I want to do -
val scores_file = sc.wholeTextFiles("/data/*/")
val scores = scores_file.map{ line =>
//step 1
//step 2
//step 3
//step 4
//step 5 : save as line._1
}
This will help dealing with each file separately, and avoid group by.

Answer 1

Assuming that your models are discrete values and you know then you can define all the model into a list
val model = List("modelA", "modelB", "modelC", ... )
you can have the following approach:
model.forEach( model => {
val scoresPerModel = sc.textFile(model);
scoresPerModel.map { line =>
// business logic here
}
})
If the you don't know the model prior to computing the business logic that you have to read using the Hadoop file system API and extract the models from there.
private val fs = {
val conf = new org.apache.hadoop.conf.Configuration()
FileSystem.get(conf)
}
fs.listFiles(new Path(hdfsPath))