Spark calculate the percentage of appearances of words - scala

Question

I've a PairRDD like this (word, wordCount). Now, I need to calculate for each word the percentage of appearances on the total of the words count, obtaining a final PairRDD like this (word, (wordCount, percentage)).
I tried with:
val pairs = .... .cache() // (word, wordCount)
val wordsCount = pairs.map(_._2).reduce(_+_)
pairs.map{
case (word, count) => (word, (count, BigDecimal(count/wordsCount.toDouble * 100).setScale(3, BigDecimal.RoundingMode.HALF_UP).toDouble))
}
But it does not seem very efficient (i'm a beginner). Is there a better way to do it?

Answer 1

When calculating wordsCount, map(_._2).reduce(_ + _) is more idiomatically expressed by the aggregate method:
val pairs = .... .cache() // (word, wordCount)
val wordsCount = pairs.aggregate(0)((sum, pair) => sum + pair._2, _ + _)
The first argument is the initial total count, which should be zero. The next two arguments are in a separate argument list, and are executed for each member of your pair RDD. The first of this second set of arguments ((sum, pair) => sum + pair._2, termed the sequence operator) adds each value in a partition to the current sum value for that partition; while the second (_ + _, termed the combination operator) combines counts from different partitions. The main benefit is that this operation is performed in parallel, by each partition, across the data, and guarantees no expensive repartitioning (a.k.a. shuffling) of that data. (Repartitioning occurs when data needs to be transferred between nodes in the cluster, and is very slow due to network communication.)
While aggregate does not affect partitioning of the data, you should note that, if your data is partitioned, a map operation will remove the partitioning scheme - because of the possibility that the keys in the pair RDD might get changed by the transformation. This might resulting in subsequent repartitioning if you perform any further transformations on the result of a map transformation. That said, a map operation followed by a reduce shouldn't result in repartitioning, but might be slightly less efficient because of the extra step (but not hugely so).
If you're concerned that the total word count might overflow an Int type, you could use a BigInt instead:
val wordsCount = pairs.aggregate(BigInt(0))((sum, pair) => sum + pair._2, _ + _)
As for producing a pair RDD with the word counts and percentages as the value, you should use mapValues instead of map. Again mapValues will retain any existing partitioning scheme (because it guarantees that the keys will not get changed by the transformation), while map will remove it. Furthermore, mapValues is simpler in that you do not need to process key values:
pairs.mapValues(c => (c, c * 100.0 / wordsCount))
That should provide sufficient accuracy for your purposes. I would only worry about rounding, and using BigDecimal, when retrieving and outputting values. However, if you really do need that, your code would look like this:
pairs.mapValues{c =>
(c, BigDecimal(c * 100.0 / wordsCount).setScale(3, BigDecimal.RoundingMode.HALF_UP).toDouble))
}