Create RDD in scala and spark - scala

Question

def main(args: Array[String]): Unit = {
val spark = SparkSession
.builder()
.master("local")
.appName("SparkAndHive")
.config("spark.sql.warehouse.dir", "/tmp/spark-warehouse 2")
.enableHiveSupport()
.getOrCreate()
GeoSparkSQLRegistrator.registerAll(spark.sqlContext)
val sparkConf: SparkConf = new SparkConf().setAppName("Spark RDD foreach Example").setMaster("local[2]").set("spark.executor.memory", "2g")
def displayFiles(files: Array[File], a: util.List[String], b: util.List[String]): Unit = {
for (filename <- files) { // If a sub directory is found,
if (filename.isDirectory) if (filename.getName.contains("fire")) {
rds.add(filename.getAbsolutePath)
println(filename.getAbsolutePath)
}
else if (filename.getName.contains("water")){
rdd.add(filename.getAbsolutePath)
println(filename.getAbsolutePath)
}
else {
displayFiles(filename.listFiles, a, b)
}
}
}
val files = new File("C://folder").listFiles
val list1 = new util.ArrayList[String]
val list2 = new util.ArrayList[String]
displayFiles(files, list1, list2)
val a= Seq(list1)
println(a)
val b= Seq(list2)
println(b)
val rdd1 = spark.sparkContext.parallelize(Seq(a))
rdd1.foreach(println))
val rdd2 = spark.sparkContext.parallelize(Seq(b))
rdd2.foreach(println))
I printed the list of subdirectory paths that ends with _fire and _water.
Then I created a list to store the paths that ends with _fire in one list and _water in another list.
I have created the RDD's for all the directories that are stored in both lists using a foreach loop.
When I am declaring a variable for the foreach loop and printing it, it is showing an empty list
Question: How to all the RDD's into single RDD i.e., one for _fire and another for _water?

Answer 1

You can create them more directly. The issue in your code is that displayFiles isn't actually returning anything, or modifying list1 or list2. Therefore, those lists will be empty, and so will a and b.
Instead, you can try something like:
val sc = spark.sparkContext
val basePath = "C://folder/"
val rddWater = sc.textFile(basePath + "*_water")
val rddFire = sc.textFile(basePath + "*_fire")
The above will get just all the contents of all the files matching the globs/paths matching the wildcards. Or, if you need to also figure out the file paths corresponding to each record, you can use sc.wholeTextFiles.
val rddWater = sc.wholeTextFiles("*_water")
val rddFire = sc.wholeTextFiles("*_fire")
// inspect contents using rddWater.collect() or rddFire.collect()
This site has more examples: https://sparkbyexamples.com/apache-spark-rdd/spark-read-multiple-text-files-into-a-single-rdd/