Improve performance for function which counts common words - scala

Question

I have this program, that uses Apache Spark to calculate the frequency of words.
I create an RDD with the key/value pairs(word=key, frequency=value). The dataset is distributed over worker nodes. The function frequentWordCount is executed at regular intervals. It selects strings from the files.
which are then converted into key-value-pairs and connected to the wordDataset-RDD. The words with a frequency of >50, are counted.
I was told that this approach is not performant. Can somebody tell me why and how I could improve this?
val sc = new SparkContext(...)
var wordDataset:RDD[(String, Int)] = sc.sequenceFile[String, Int](“…”).persist()
def frequentWordCount(fileName:String):Long = {
val words = sc.sequenceFile[String](fileName)
val joined = wordDataset.join(words.map(x=>(x,1)))
joined.filter(x=>x._1._2>50).count
}

Answer 1

Approximately how many frequent words will you have? For a lot of reasonable tasks, I think it should be unexpectedly small - small enough to fit into each individual machine's memory. IIRC, words tend to obey a power law distribution, so there shouldn't be that many "common" words. In that case, broadcasting a set of frequent words could be much faster than joining:
val sc = new SparkContext(...)
var commonWords: BroadCast[Set[String]] = sc.broadcast(sc.sequenceFile[String, Int](“…”).filter(_._2 > 50).collect().toSet)
def frequentWordCount(fileName:String):Long = {
val words = sc.sequenceFile[String](fileName)
words.filter(commonWords.value.contains).count
}
If you are calling frequentWordCount multiple times, it probably is also better to do it in just one RDD operation where your words are associated with a filename and then grouped and counted or something... specifics depend on how it's used.

Answer 2

If the number of common words is small enough to fit into an in-memory Set, then what the other answer suggests (except, you need to map(_._1) there after filter.
Otherwise, the two things you could improve are (1) filter before join, you want to throw out extra data as soon as you can rather than unnecessarily scanning over it multiple times, and (2) as a general rule, you always want to join the larger dataset to the smaller one, not the other way around.
sc.sequenceFile[String](fileName)
.keyBy(identity)
.join(wordDataset.filter(_._2 > 50))
.count