Set similarity join using Spark - scala

Question

I have two text files each line is in the form of (id, sequence of numbers).
I have a threshold value as well.
File 1 looks like below where, in the first line, 0 is the id and rest is
a sequence of numbers.
0 1 4 5 6
1 2 3 6
2 4 5 6
Similarly I have file 2 with following contents.
0 1 4 6
1 2 5 6
2 3 5
I have to find those lines which have similarity value greater or equal to a threshold. Similarity value can be calculated as the intersection of two lines divided by the union of two lines. For example line id- 0 of file1 has seq 1,4,5,6 and line id-0 of file2 has seq 1,4,6. They have intersection size = 3 and union size = 4. Therefore their similarity will be 3/4 = 0.75 which is greater than the threshold.
I have written python code to do this task and trying to convert it to Scala.
with open("file1.txt") as f1:
content1 = f1.readlines()
content1 = [x.strip() for x in content1]
with open("file2.txt") as f2:
content2 = f2.readlines()
content2 = [x.strip() for x in content2]
threshold = 0.5
final_index_list_with_similarity = []
for i in range(len(content1)):
for j in range(len(content2)):
index_content1 = content1[i][0]
index_content2 = content2[j][0]
s = set(content1[i][1:])
t = set(content2[j][1:])
intersect = s & t
intersect_size = len(intersect) - 1
union_size = len(s) + len(t) - intersect_size - 2 #substracting two because I am getting two extra space.
similarity = intersect_size/union_size
if similarity >= threshold:
final_index_list_with_similarity.append((index_content1, index_content2, similarity))
print(final_index_list_with_similarity)
Output : [('0', '0', 0.75), ('1', '1', 0.5), ('2', '0', 0.5), ('2', '1', 0.5)]
What I have tried till now in scala is something looks like this.
val inputFile1 = args(0)
val inputFile2 = args(1)
val threshold = args(2).toDouble
val ouputFolder = args(3)
val conf = new SparkConf().setAppName("SetSimJoin").setMaster("local")
val sc = new SparkContext(conf)
val lines1 = sc.textFile(inputFile1).flatMap(line => line.split("\n"))
val lines2 = sc.textFile(inputFile2).flatMap(line => line.split("\n"))
val elements1 = lines1.map { x => x.drop(x.split(" ")(0).length.toInt + 1) }.flatMap { x => x.split(" ") }.map { x => (x, 1) }.reduceByKey(_+_)
val elements2 = lines2.map { x => x.drop(x.split(" ")(0).length.toInt + 1) }.flatMap { x => x.split(" ") }.map { x => (x, 1) }.reduceByKey(_+_)
This gives me the frequency of every number in the entire file.
Any help or guidance will be much appreciated.

Answer 1

Both files can be joined as RDD, and then formula applied: "intersection size/union size":
val lines1 = sparkContext.textFile("inputFile1")
val lines2 = sparkContext.textFile("inputFile2")
val rdd1 = lines1.map(_.split(" ")).map(v => (v(0), v.tail))
val rdd2 = lines2.map(_.split(" ")).map(v => (v(0), v.tail))
val result = rdd1.join(rdd2).map(r => (
r._1,
r._2._1.intersect(r._2._2).size * 1.0 /
r._2._1.union(r._2._2).distinct.size
)
)
result.foreach(println)
Output is:
(1,0.5)
(0,0.75)
(2,0.25)