how to apply pattern matching on the file name using spark(scala) - scala

Question

I am monitoring a directory in hdfs and if a file is put into it i want to retrieve the name of the file entering the directory and apply pattern matching so as to sort them based on names so far i have been able to retrieve the file path.which gives me the file name but i don't know how to proceed further with pattern matching
import org.apache.hadoop.mapreduce.lib.input.{FileSplit, TextInputFormat}
import org.apache.spark.rdd.{NewHadoopRDD}
import org.apache.spark.{SparkConf, SparkContext}
import org.apache.hadoop.io.LongWritable
import org.apache.hadoop.io.Text
object path {
def main(args: Array[String]) {
val conf = new SparkConf().setAppName("Simple Application").setMaster("local")
val sc = new SparkContext(conf)
val fc = classOf[TextInputFormat]
val kc = classOf[LongWritable]
val vc = classOf[Text]
val path :String = "/home/hduser/Desktop/foldername"
val text = sc.newAPIHadoopFile(path, fc ,kc, vc, sc.hadoopConfiguration)
println("++++++++++++++ "+text)
val linesWithFileNames = text.asInstanceOf[NewHadoopRDD[LongWritable, Text]].mapPartitionsWithInputSplit((inputSplit, iterator) => {
val file = inputSplit.asInstanceOf[FileSplit]
println(">>>>>>>>>>>>>>>> "+file.getPath)
iterator.map(tup => (file.getPath, tup._2))
}
)
linesWithFileNames.collect()
}
}
this gives me the path like
/home/hduser/Desktop/folder_name/XYZ_123_pqr_abc_FILENAME
and i want to apply pattern matching based on the file name start....here it will based on XYZ.
Any help will be appreciated.

Answer 1

You are not specific enough, i think. From context of your question i assume XYZ are digits and you intend to sort files by that numbers, if that's the case consider the following example:
val matchStart = """^(\d{3})""".r.unanchored
val number = file match {
case matchStart(number, _*) => number.toInt
case _ => ???//error handling, not necessarily unimplemented error
}
Now having extracted the number from the begining of the file name you can perform sorting etc.
Edit
Following information in comment answer would be similar:
val matchStart = """^([a-zA-Z1-9]{3})""".r.unanchored
val prefix = file match {
case matchStart(a, _*) => a
case _ => ???//error handling, not necessarily unimplemented error
}
Having the prefix extracted you can do anything you want with it.