Many to Many join with fact and dim table in Spark 3? - scala

Question

I have two data frames one with 10M records and the other with 100K records.
Let's say the schema of the first data frame
create table fact_table
(
id string,
dim_id_list ARRAY<string>
);
where dim_id_list is an array with a length of 1-200.
and the fact table
create table dim_table
(
id string,
tag string
)
where 80K+ ids can have the same tag
example random data
fact_table
id | dim_id_list
----------------------
1 | [a, b]
2 | [a, b, c]
3 | [a, b, c, d]
4 | [a, b, c, d, e]
5 | [a, b, c, d, e, f]
....
dim table
id | tag
----------------------
a | john
b | foo
c | foo
d | foo
f | foo
g | bar
h | random
i | spark
.......
What I want to do is add unique tags to the fact table, as shown in the output
output:
id | dim_id_list | tag
---------------------------------
1 | [a, b] | john
1 | [a, b] | foo
2 | [a, b, c] | john
2 | [a, b, c] | foo
3 | [a, b, c, d] | john
3 | [a, b, c, d] | foo
4 | [a, b, c, d, e] | john
4 | [a, b, c, d, e] | foo
4 | [a, b, c, d, e] | random
5 | [a, b, c, d, e, f] | john
5 | [a, b, c, d, e, f] | foo
5 | [a, b, c, d, e, f] | random
5 | [a, b, c, d, e, f] | bar
.....
basically a left join but with unique tags
the query I wrote for this is
select fact_table.id, dim_id_list, tag
from fact_table
left join (select collect_list(id) as id_list,
tag
from dim_table
group by tag) as dim_table
on arrays_overlap(fact_table.dim_id_list, dim_table.id_list);
spark scala equivalent:
val dimDf_agg = dimDf
.groupBy("tag")
.agg(collect_set("id")
.as("id_list"))
val join_df = fact_df(broadcast(dimDf_agg),
arrays_overlap(fact_df("dim_id_list"),
dimDf_agg("id_list")), "left")
.drop("id_list")
but here it becomes so complex to execute as the complexity of array_overlap is very high.
Is there a more optimized approach that I can follow?
Something like doing broadcast join (left) with the hash of id_list and dim_id_list and just taking the first 'foo' or something else?

Answer 1

What you can do is explode the array (so as to generate one row per item in dim_id_list) and then join over that value. Then you can use distinct to remove the duplicates.
// simply generating the data for reproducibility
val fact_df = Seq(
1 -> Seq("a","b"),
2 -> Seq("a","b","c"),
3 -> Seq("a","b","c","d"),
4 -> Seq("a","b","c","d","e"),
5 -> Seq("a","b","c","d","e","f")
).toDF("id", "dim_id_list")
val dim_df = Seq(
"a" -> "john",
"b" -> "foo",
"c" -> "foo",
"d" -> "foo",
"f" -> "foo",
"g" -> "bar",
"h" -> "random",
"i" -> "spark"
).toDF("id", "tag")
And the solution:
fact_df
.withColumn("id_tag", explode('dim_id_list))
.join(dim_df.select('id as "id_tag", 'tag), Seq("id_tag"))
.drop("id_tag")
.distinct
.orderBy("id")
.show
+---+------------------+----+
| id| dim_id_list| tag|
+---+------------------+----+
| 1| [a, b]|john|
| 1| [a, b]| foo|
| 2| [a, b, c]|john|
| 2| [a, b, c]| foo|
| 3| [a, b, c, d]|john|
| 3| [a, b, c, d]| foo|
| 4| [a, b, c, d, e]| foo|
| 4| [a, b, c, d, e]|john|
| 5|[a, b, c, d, e, f]|john|
| 5|[a, b, c, d, e, f]| foo|
+---+------------------+----+