Looking for ways to optimize the program [Scala/Spark] - scala

Question

I have an RDD which look the following:
( (tag_1, set_1), (tag_2, set_2) ) , ... , ( (tag_M, set_M), (tag_L, set_L) ), ...
And for each pair from the RDD I'm going to compute the expression
for k=0,..,3 and to find the sum: p(0)+...p(3). For each pair of pairs n_1 is length of the set in the first pair and n_2 is length of the set in the second pair.
For now I wrote the following:
val N = 1000
pairRDD.map({
case ((t1,l1), (t2,l2)) => (t1,t2, {
val n_1 = l1.size
val n_2 = l2.size
val vals = (0 to 3).map(k => {
val P1 = (0 to (n_2-k-1))
.map(j => 1 - n_1/(N-j.toDouble))
.foldLeft(1.0)(_*_)
val P2 = (0 to (k-1))
.map(j => (n_1-j.toDouble)*(n_2-j.toDouble)/(N-n_2+k.toDouble-j.toDouble)/(k.toDouble-j.toDouble) )
.foldLeft(1.0)(_*_)
P1*P2
})
vals.sum.toDouble
})
})
The problem is it seems to work really slow and I hope there are some features of scala/spark that I don't know about and that could reduce the time of execution there.
Edit:
1) In the first place I have a csv-file with 2 columns: tag and message_id. For each tag I'm finding messages where it could be found and creating pairs like I described above (tagIdsZipped). The code is here
2) Then I want to compute the expression for each pair and write it down to file.
Actually, I also would like to filter the result, but it would be even longer, so I'm even not trying for now.
3) No, actually I dont, but the problems happened, when I tried to use this code, previously I did the following:
val tagPairsWithMeasure: RDD[(Tag, Tag, Measure)] = tagIdsZipped.map({
case ((t1,l1), (t2,l2)) => (t1,t2, {
val numer = l1.intersect(l2).size
val denom = Math.sqrt(l1.size)*Math.sqrt(l2.size)
numer.toDouble / denom
})
})
and everything worked fine. (see 4) )
4) In the file I described in 1) there are about 25million rows (~1.2 GB). I'm computing in on Xeon E5-2673 #2.4GHz and 32 GB RAM. It took about 1.5h to execute the code with the function I described in 3). I see, that there are more operations now, but it took about 3hours and only about 25% of the task was done. The main problem is I will have to work with about 3 times more data, but I can't even do it on a 'smaller' one.
Thank you in advance!

Answer 1

As has been mentioned there is not much to improve about Spark.
The biggest issue I can see here is using range.map.
(0 to (n_2-k-1)) creates a Range object.
Calling map on it creates a Vector allocating much memory.
The most simple solution is to work with iterators since foldLeft is a streaming-friendly function:
(0 to (n_2-k-1)).iterator instead of (0 to (n_2-k-1))
It also probably makes sense to try rewriting it imperatively using vars, loops and arrays since since computation inside a loop is extremely cheap. But it is a weapon of the last chance.

Answer 2

have you tried using dataframes?
maybe you can create a dataframe with a schema like this:
tagIdsDF
+-----------------------------+
|tag1 | set1 |tag2 | set2 |
+-----------------------------+
|tag_1 |set_1 |tag_2 |set_2 |
|... |
|tag_M |set_M |tag_L |set_L |
+-----------------------------+
and define a UDF to compute the sum:
val pFun = udf((l1:Seq[Double], l2:Seq[Double]) => {
val n_1 = l1.size
val n_2 = l2.size
val vals = (0 to 3).map(k => {
val P1 = (0 to (n_2-k-1))
.map(j => 1 - n_1/(N-j.toDouble))
.foldLeft(1.0)(_*_)
val P2 = (0 to (k-1))
.map(j => (n_1-j.toDouble)*(n_2-j.toDouble)/(N-n_2+k.toDouble-j.toDouble)/(k.toDouble-j.toDouble) )
.foldLeft(1.0)(_*_)
P1*P2
})
vals.sum.toDouble
})
notice that you don't need to pass tag_1/tag_2 because this information is on the resulting dataframe, then you can call it like this:
val tagWithMeasureDF = tagIdsDF.withColumn("measure", pFun($"set1", $"set2"))
and you get this df:
tagWithMeasureDF
+-----------------------------+---------+
|tag1 | set1 |tag2 | set2 | measure |
+---------------------------------------+
|tag_1 |set_1 |tag_2 |set_2 | m1 |
|... ... ...|
|tag_M |set_M |tag_L |set_L | mn |
+---------------------------------------+
doing something like this maybe helps you to achieve the desired performance.
Hope this helps you and if it works tell me!