Scala - List Variable outside of foreach loop [closed] - scala

Question

Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 4 years ago.
Improve this question
The code below isn't giving me the desired output. I am getting the output of finallist as individual characters separated by commas; I was expecting lists with two values only (filename, sizeofcolumn).
val pathurl="adl://*****.azuredatalakestore.net/<folder>/<sub_folder>"
val filelist=dbutils.fs.ls(pathurl)
val newdf = df.select("path").rdd.map(r => r(0)).collect.toList
var finallist = scala.collection.mutable.ListBuffer.empty[Any]
newdf.foreach(f => {
val MasterPq = spark.read.option("header","true").option("inferSchema","true").parquet(f.toString())
val size = MasterPq.columns.length
val mergedlist = List(f.toString(), size.toString())
mergedlist.map((x => {finallist = finallist ++ x}))
})
println(finallist)

Answer 1

The bug in your code is that you're using the ++ method to append values to your list. This method is used to append two list.
scala> List(1, 2) ++ List(3, 4)
res0: List[Int] = List(1, 2, 3, 4)
In scala strings are viewn as a list of characters, so your appending each individual character to your list.
scala> List(1, 2) ++ "Hello"
res3: List[AnyVal] = List(1, 2, H, e, l, l, o)
Since you're using a mutable list, you can append values with the '+=' method. If you just want to get your code working, than the following should be enough, but it is not a good solution.
// mergedlist.map((x => {finallist = finallist ++ x}))
mergedlist.map((x => finallist += x}))
You're probably new to scala, coming from a imperative language like Java. Scala collections do not work as you're known from such programming languages. Scala's collections are immutable by default. Instead of modifying collections, you're using using functions such as map to build new lists based on the old list.
The map function is one of the most used functions on lists. It takes an anonymous function as parameter that takes one element and transforms it to another value. This function is applied onto all methods of the list thereby build a new list. Here's an example:
scala> val list = List(1, 2, 3).map(i => i * 2)
list: List[Int] = List(2, 4, 6)
In this example, a function that multiplies integers by two is applied onto each element in the list. The results are put into the new list. Maybe this illustration helps to comprehend the process:
List(1, 2, 3)
| | |
* 2 * 2 * 2
↓ ↓ ↓
List(2, 4, 6)
We could use the map function to solve your task.
We can use it to map each element in the newdf list into a tuple with the corresponding (filename, filesize).
val finallist = newdf.map { f =>
val masterPq = spark.read.option("header","true").option("inferSchema","true").parquet(f.toString())
val size = masterPq.columns.length
(f.toString(), size.toString())
}
I think this code is shorter, simpler, easier to read and just way more beautiful. I will definitely recommend you to learn more about Scala's collections and immutable collections in general. Once you understand them, you'll just love them!