Filename usage with Spark RDD - scala

Question

I am getting a filename in my RDD when i use :
val file=sc.wholeTextFiles("file:///c://samples//finalout.txt",0)
However, right after I flatten it, I loose the first tuple, how do I make sure that filename is carried forward to my map function?
My code:
val res= file.flatMap{e=>e._2.split("\n")}.map{line => line.split(",")}.map(elem => {
...I want to use filename here
}

Answer 1

A slightly different approach. It should be all possible with just RDD, but I use a DF to RDD approach. It's a bit late in the evening.
import org.apache.spark.sql.functions.input_file_name
val inputPath: String = "/FileStore/tables/sample_text.txt" //does work, also for directory of course
val rdd = spark.read.text(inputPath)
.select(input_file_name, $"value")
.as[(String, String)]
.rdd
val rdd2 = rdd.map(line => (line._1, line._2.split("\n")))
//rdd2.collect
val rdd3 = rdd2.map(line => (line._1, line._2.flatMap(x => x.split(" ")).map(x => (x,1))))
rdd3.collect
returns in this case:
res61: Array[(String, Array[(String, Int)])] = Array((dbfs:/FileStore/tables/sample_text.txt,Array((Hi,1), (how,1), (are,1), (you,1), (today,1), (ZZZ,1))), (dbfs:/FileStore/tables/sample_text.txt,Array((I,1), (am,1), (fine,1))), (dbfs:/FileStore/tables/sample_text.txt,Array((I,1), (am,1), (also,1), (tired,1))), (dbfs:/FileStore/tables/sample_text.txt,Array((You,1), (look,1), (good,1))), (dbfs:/FileStore/tables/sample_text.txt,Array((Can,1), (I,1), (stay,1), (with,1), (you?,1))), (dbfs:/FileStore/tables/sample_text.txt,Array((Bob,1), (will,1), (pop,1), (in,1), (later,1), (ZZZ,1))), (dbfs:/FileStore/tables/sample_text.txt,Array((Oh,1), (really?,1), (Nice,,1), (cool,1))))
You can modify accordingly, stripping out this and that. line._1 is your file_name. For RDDs use case statement approach.
UPD
Using your approach - and correcting it - as you cannot do what you want, the following random code as not sure what u r wishing to do:
val file = sc.wholeTextFiles("/FileStore/tables/sample_text.txt",0)
val res = file.map(line => (line._1, line._2.split("\n").flatMap(x => x.split(" ")) )).map(elem => {(elem._1,elem._2, "This is also possible") })
res.collect
returns:
res: org.apache.spark.rdd.RDD[(String, Array[String], String)] =
MapPartitionsRDD[69] at map at command-1038477736158514:15
res9: Array[(String, Array[String], String)] =
Array((dbfs:/FileStore/tables/sample_text.txt,Array(Hi, how, are, you, today, "ZZZ
", I, am, "fine
", I, am, also, "tired
", You, look, "good
", Can, I, stay, with, "you?
", Bob, will, pop, in, later, "ZZZ
", Oh, really?, Nice,, "cool
"),This is also possible))
Your approach is not possible with flatMap, need to follow as per above.