How do I efficiently debug a long-running Spark application? - scala

Question

I have a Spark application which cleans and prepares a data set, and then applies a K-means clustering algorithm onto this set. Afterwards, some metrics of the resulting clusters are calculated.
Naturally, calculating the K-means vector clusters is a task that has a long execution time. When debugging the calculation for the cluster metrics I cannot iterate quickly on my code due to the clusters being calculated on each execution. How do I solve this?
Ideas I have are:
writing a unit test for the metric calculation methods. But it would be cumbersome to mock cluster data as
Serialise computed K-means vector cluster data to disk
Any help is appreciated
Code for reference:
def main(args: Array[String]): Unit = {
// -- start of long running execution
val lines = sc.textFile("src/main/resources/stackoverflow/stackoverflow.csv")
val raw = rawPostings(lines)
val grouped = groupedPostings(raw)
val scored = scoredPostings(grouped)
val vectors = vectorPostings(scored)
// assert(vectors.count() == 2121822, "Incorrect number of vectors: " + vectors.count())
val means = kmeans(sampleVectors(vectors), vectors, debug = true)
// -- end of long running execution
val results = clusterResults(means, vectors) // < -- this method operates on the result of previous ops
printResults(results)
}
Implementation of clusterResults:
def clusterResults(means: Array[(Int, Int)], vectors: RDD[(LangIndex, HighScore)]): Array[(String, Double, Int, Int)] = {
// -- Note that means is quite intensive to compute
val closest = vectors.map(p => (findClosest(p, means), p)) // -- (Int, (LangIndex, HighScore))
val closestGrouped = closest.groupByKey() // -- (Int, Iter((LangIndex, HighScore))
vectors.take(3).foreach(println)
val median = closestGrouped.mapValues { vs =>
// #todo: what does groupBy(identity) do?
// Predef.idintity is equivalent ot x => x
val langId: Int = vs.map(_._1).groupBy(identity).maxBy(_._2.size)._1 // most common language in the cluster
val langLabel: String = langs(langId / langSpread)
val langPercent: Double = vs.map(_._1).count(_.equals(langId)) / vs.size // percent of the questions in the most common language (= number of questions in most common lang divided by total questions)
val clusterSize: Int = vs.size
val medianScore: Int = vs.map(_._2).
(langLabel, langPercent, clusterSize, medianScore)
}
median.collect().map(_._2).sortBy(_._4)
}