Is using multiple spark windows a bad thing? - scala

Question

Recently, i started working on spark windows and trying to understand what happens under the hood in spark executors when applying these windowing functions.
My question is, since every window must be created using partitionBy function which means shuffling data across the cluster, is it normal to use multiple windows?
For example i have this dataframe :
cli date item1 item2
--------- ---------- ------- -------
1234567 20191030 A D
1234567 20191030 B D
1234567 20191029 A E
1234567 20191026 A F
7456123 20191026 C D
7456123 20191025 C F
The aim here is to calculate the frequency of each item for each client for every date based on history.
For example the client 1234567 at 20191030 used 4 item_1 from 20191030 and backwards so the frequency of A will be 3/4 and B is 1/4.
I chose to calculate these frequencies for each day by using windows since it calculates a value for each row, but I need to use three windows :
// This will give me the number of items used by a client
// in that day and all history.
val lByCliWindow = Window.partitionBy(col("cli"))
// This will give me how many times a client used this exact item_1 A in
// that day and back in history (here my history is 120 days)
val lByCliItem1Window = Window
.partitionBy(col("cli"), col("item_1"))
.orderBy(to_timestamp(col("date"), "yyyyMMdd").cast(LongType))
.rangeBetween(-86400*120,0)
// This will give me how many times a client used this exact item_3 F in
// that day and back in history (here my history is 120 days)
val lByCliItem2Window = Window
.partitionBy(col("cli"), col("item_2"))
.orderBy(to_timestamp(col("date"), "yyyyMMdd").cast(LongType))
.rangeBetween(-86400*120,0)
The expected output is :
cli date frequency_item1 frequency_item2
--------- ---------- ------------------------- --------------------------------
1234567 20191030 Map(A -> 3/4, B -> 1/4) Map(D -> 2/4, E -> 1/4, F 1/4)
1234567 20191029 Map(A -> 2/2) Map(E -> 1/2, F -> 1/2)
1234567 20191026 Map(A -> 1/1) Map(F -> 1/1)
7456123 20191026 Map(C -> 2/2) Map(D -> 1/2, F -> 1/2)
7456123 20191025 Map(C -> 1/1) Map(F -> 1/1)
When i do explain() on this approach I can see so many exchange plans of hashpartitioning etc and that is very expectable since we are doing a partitionBy every time.
Giving that I have almost 30 variables this means 30 times partitioning of data. (This is a lot of shuffling)
What I want to understand is this approach normal? Will spark work on this partitioning on parallel (create multiple windows on the same time therefore partitioning the dataframe in multiple different ways at the same time) or sequentially?
Can we use multiple windows? what is more costly the groupBy shuffle or partitionBy windows shuffle?
Thank you for your replies and don't hesitate to propose different approach for calculating frequencies using windows.

Answer 1

I have a solution that involves only one window. I'll explain with comments.
// The columns you are interested in
val items = df.columns.filter(_ startsWith "item")
// collect_list aggregation. It avoid duplicates. We will group by cli and date.
val aggs = items.map(c => collect_list(col(c)) as c)
// A window over "cli" and ordered by date.
val win = Window.partitionBy("cli").orderBy("date")
// A UDF that computes the frequencies you want
// It takes as input a seq of seq because of the first aggregation we do
val compute_freqs = udf((s : Seq[Seq[String]]) => {
val flat_s = s.flatten
val total = flat_s.size
flat_s.groupBy(identity).mapValues(_.size.toDouble / total)
})
// for each item, we collect the values over the window, and compute the frequency
val frequency_columns = items
.map(item => compute_freqs(collect_list(col(item)) over win)
.alias(s"frequency_$item"))
// Then we use everything
val result = df
.groupBy("cli", "date")
.agg(aggs.head, aggs.tail : _*)
.select((Seq("cli", "date").map(col) ++ frequency_columns) :_*)
.orderBy($"cli", $"date" desc)
And here is the result:
scala> result.show(false)
+-------+--------+----------------------+--------------------------------+
|cli |date |frequency_item1 |frequency_item2 |
+-------+--------+----------------------+--------------------------------+
|1234567|20191030|[A -> 0.75, B -> 0.25]|[D -> 0.5, F -> 0.25, E -> 0.25]|
|1234567|20191029|[A -> 1.0] |[F -> 0.5, E -> 0.5] |
|1234567|20191026|[A -> 1.0] |[F -> 1.0] |
|7456123|20191026|[C -> 1.0] |[D -> 0.5, F -> 0.5] |
|7456123|20191025|[C -> 1.0] |[F -> 1.0] |
+-------+--------+----------------------+--------------------------------+