How do I determine the optimal number of threads in Spark application? - scala

Question

In my Scala/Spark application, I am trying correctly to use multiprocessing. As you can see from the code below, the number of threads is equal to the number of elements in the storage array. I tested the current code and it works. But as you can see there are only 2 elements in the storage array. It seems to me that if there are a large number of elements in the array, problems will occur. In my case, I don't know how many elements there will be in the array in the future. Perhaps I should limit the number of threads and start new threads only when the previous ones are processed.
Question: How do I determine the optimal number of threads?
Main.app:
import org.apache.spark.sql.DataFrame
import utils.CustomThread
object MainApp {
def main(args: Array[String]): Unit = {
// Create the main DataFrame with all information.
var baseDF: DataFrame = spark.read.option("delimiter", "|").csv("/path_to_the_files/")
// Cache the main DataFrame.
baseDF.persist(StorageLevel.MEMORY_AND_DISK)
// The first time DataFrame is computed in an action, it will be kept in memory on the nodes.
baseDF.count()
// Create arrays with the different identifiers
var array1 = Array("6fefc487-bd57-4fa2-808a-3845703b83d0", "9baba76b-07c2-48ec-a153-6cfb8b138ecf")
var array2 = Array("ab654369-77f5-478c-94e5-ee2755ae8571", "3b43e0a6-deba-4919-a2cc-9d450e28e0fe")
var storage = Array(array1, array2)
// Check if the main DataFrame is empty or not.
if (baseDF.head(1).nonEmpty) {
for (item <- storage) {
val thread = new Thread(new CustomThread(baseDF, item))
thread.start()
}
}
}
}
CustomThread.scala:
package utils
import org.apache.spark.sql.DataFrame
class CustomThread(baseDF: DataFrame, item: Array[String]) extends Runnable {
override def run(): Unit = {
val df = baseDF.filter(col("col1").isin(item:_*))
println("Count: " + df.count())
}
}
I use such configurations:
spark.serializer: org.apache.spark.serializer.KryoSerializer
spark.kryoserializer.buffer.max.mb: 1024
spark.executor.memory: 2g
spark.sql.autoBroadcastJoinThreshold: -1
spark.sql.files.ignoreCorruptFiles: true
spark.driver.memory: 30g
spark.driver.maxResultSize: 20g
spark.executor.cores: 1
spark.cores.max: 48
spark.scheduler.mode: FAIR

Answer 1

What do you want to achieve with your multi-threading? In a spark application, you should not worry about the number of threads etc. What your code does is the launch parallel jobs (your multi-threading is only on the driver), for the executors, there is no difference.
In my experience, I only use parallel job launches if I have several jobs which are small or skewed, such as the cluster resources are not fully utilized. If I do that, I use scalas parallel collections.
To answer your question : The optimal number if threads is probably 1
EDIT: I would suggest to rewrite your code completely with the goal to have all results in a new dataframe, this is better than implementing complicated multi-threading:
// testcase
val baseDf = Seq(
"6fefc487-bd57-4fa2-808a-3845703b83d0",
"9baba76b-07c2-48ec-a153-6cfb8b138ecf",
"ab654369-77f5-478c-94e5-ee2755ae8571",
"dummy"
).toDF("col1")
var array1 = Seq("6fefc487-bd57-4fa2-808a-3845703b83d0", "9baba76b-07c2-48ec-a153-6cfb8b138ecf")
var array2 = Seq("ab654369-77f5-478c-94e5-ee2755ae8571", "3b43e0a6-deba-4919-a2cc-9d450e28e0fe")
var storage = Seq(array1, array2)
broadcast(storage.toDF("storage"))
.join(baseDf,array_contains($"storage",$"col1"),"left")
.groupBy($"storage").agg(count($"col1").as("count"))
.show()
gives :
+--------------------+-----+
| storage|count|
+--------------------+-----+
|[ab654369-77f5-47...| 1|
|[6fefc487-bd57-4f...| 2|
+--------------------+-----+