How to efficeintly self-join an RDD of 1.7 million lines? - scala

Question

I am new to Spark/Big data, and this totally has me stumped.
I have an RDD of around 1.7 million lines, of the form
SetA = (1,7,9,12,.....)
They are all integers, with one value per line.
I need to do a self-join on this RDD, so that I get pairs of elements satisfying certain conditions.
For example, like this:
result = ((1, 7), (1, 9), (12, 1),......)
The result is supposed to have around 24 million lines.
I tried doing a Cartesian, followed by several filters, like this:
setA.cartesian(setA).filter(''do something).filter('do something more')
This works fine for small datasets and I get what is required, but for the huge dataset of 1.7 million lines, the job never completes even after waiting for several hours. The target time for completion is around 30 minutes.
The console keeps showing lines like this continuously (the RDD setA is cached):
2021-12-02 08:36:17,758 INFO storage.BlockManager: Found block rdd_2_0 locally
2021-12-02 08:36:18,926 INFO storage.BlockManager: Found block rdd_2_0 locally
2021-12-02 08:36:19,892 INFO storage.BlockManager: Found block rdd_2_0 locally
2021-12-02 08:36:20,848 INFO storage.BlockManager: Found block rdd_2_0 locally
2021-12-02 08:36:21,666 INFO storage.BlockManager: Found block rdd_2_0 locally
2021-12-02 08:36:22,532 INFO storage.BlockManager: Found block rdd_2_0 locally
2021-12-02 08:36:23,495 INFO storage.BlockManager: Found block rdd_2_0 locally
2021-12-02 08:36:24,463 INFO storage.BlockManager: Found block rdd_2_0 locally
2021-12-02 08:36:25,438 INFO storage.BlockManager: Found block rdd_2_0 locally
2021-12-02 08:36:26,270 INFO storage.BlockManager: Found block rdd_2_0 locally
2021-12-02 08:36:27,153 INFO storage.BlockManager: Found block rdd_2_0 locally
2021-12-02 08:36:28,121 INFO storage.BlockManager: Found block rdd_2_0 locally
2021-12-02 08:36:29,121 INFO storage.BlockManager: Found block rdd_2_0 locally
2021-12-02 08:36:30,095 INFO storage.BlockManager: Found block rdd_2_0 locally
2021-12-02 08:36:31,209 INFO storage.BlockManager: Found block rdd_2_0 locally
2021-12-02 08:36:32,199 INFO storage.BlockManager: Found block rdd_2_0 locally
2021-12-02 08:36:33,033 INFO storage.BlockManager: Found block rdd_2_0 locally
When I just read the input and write it to a textfile, the job completes without issues in less than a minute.
I am pretty sure I am missing something basic here. Is the Cartesian product being performed first, and then the filters are being applied? I thought that RDD transformations aren't evaluated until the last step.
Is there a better way of doing a conditional self-join without using Cartesian?