How to do yearly comparison in spark scala - scala

Question

I have dataframe which contains columns like Month and Qty as you can see in below table:
| Month | Fruit | Qty |
| -------- | ------ | ------ |
| 2021-01 | orange | 5223 |
| 2021-02 | orange | 23 |
| ...... | ..... | ..... |
| 2022-01 | orange | 2342 |
| 2022-02 | orange | 37667 |
I need to do sum of the Qty group by the Fruit. My output DF will be below table:
| Year | Fruit | sum_of_qty_This_year | sum_of_qty_previous_year |
| ---- | -------- | --------------------- | -------------------------- |
| 2022 | orange | 29384 | 34534 |
| 2021 | orange | 34534 | 93584 |
but there is a catch here, consider below table.
| current year | jan | feb | mar | apr | may | jun | jul | aug | sep | oct | nov | dec |
| --------------------------------------------------------------------------------------------------------|
| previous year | jan | feb | | apr | may | jun | jul | aug | | oct | nov | dec |
as you can see the data for mar and sep is missing in previous year. So when we calculate sum of current year, Qty should exclude the missing months. and this should be done for each year

Answer 1

import org.apache.spark.sql.SparkSession
import org.apache.spark.sql.functions.{col, sum}
import spark.implicits._
val df1 = Seq(
("2021-01", "orange", 5223),
("2021-02", "orange", 23),
("2022-01", "orange", 2342),
("2022-02", "orange", 37667),
("2022-03", "orange", 50000)
).toDF("Month", "Fruit", "Qty")
val currentYear = 2022
val priorYear = 2021
val currentYearDF = df1
.filter(col("Month").substr(1, 4) === currentYear)
val priorYearDF = df1
.filter(col("Month").substr(1, 4) === priorYear)
.withColumnRenamed("Month", "MonthP")
.withColumnRenamed("Fruit", "FruitP")
.withColumnRenamed("Qty", "QtyP")
val resDF = priorYearDF
.join(
currentYearDF,
priorYearDF
.col("FruitP") === currentYearDF.col("Fruit") && priorYearDF
.col("MonthP")
.substr(6, 2) === currentYearDF.col("Month").substr(6, 2)
)
.select(
currentYearDF.col("Fruit").as("Fruit"),
currentYearDF.col("Qty").as("CurrentYearQty"),
priorYearDF.col("QtyP").as("PriorYearQty")
)
.groupBy("Fruit")
.agg(
sum("CurrentYearQty").as("sum_of_qty_This_year"),
sum("PriorYearQty").as("sum_of_qty_previous_year")
)
resDF.show(false)
// +------+--------------------+------------------------+
// |Fruit |sum_of_qty_This_year|sum_of_qty_previous_year|
// +------+--------------------+------------------------+
// |orange|40009 |5246 |
// +------+--------------------+------------------------+