Pyspark - Find new, left and existing sales - pyspark

Question

I have a dataframe like this
and I want output like this
I need to aggregate the sales for each year band like this as below. For example for 2018-2019,
New_sales = sum of all sales of 2019 (which is the later year in 2018-2019) where the id didn't exists in 2018 but exists in 2019
Existing_sales = sum of sales of 2018 where the id is there in 2018 and 2019 subtract sum of sales of 2019
Existing_sales = 50+75 (sales of 2018) - (20+50) (sales of 2019) = 125-70 = 55
Left_sales = sum of all sales of 2018 (the earlier year in 2018-2019) where the id exists in 2018 but not in 2019
How do I achieve that?

Answer 1

I've added some more dummy data to include 2020 year for this example.
# +---+---+----------+
# | id|amt| dt|
# +---+---+----------+
# | 1| 50|2018-12-31|
# | 2|100|2018-12-31|
# | 3| 75|2018-12-31|
# | 1| 20|2019-12-31|
# | 3| 50|2019-12-31|
# | 5| 25|2019-12-31|
# | 1| 70|2020-12-31|
# | 2|150|2020-12-31|
# | 5|125|2020-12-31|
# +---+---+----------+
You can extract the year from the dates and pivot them with the sales values under it. This would make the calculations manageable by reducing the number of rows - Number of years would be less, so calculations would be optimized.
yr_sales_sdf = data_sdf. \
fillna(0, subset=['amt']). \
withColumn('yr', func.year('dt')). \
groupBy('id'). \
pivot('yr'). \
agg(func.first('amt'))
# +---+----+----+----+
# | id|2018|2019|2020|
# +---+----+----+----+
# | 5|null| 25| 125|
# | 1| 50| 20| 70|
# | 3| 75| 50|null|
# | 2| 100|null| 150|
# +---+----+----+----+
We'll need the years as a list - This can be extracted from the pivoted dataframe's columns as done in yrs list - and a case when function that will calculate if the ID is existing, left or new - I've created sale_cat_cond function (and it is purely pyspark so automatically optimized) that takes in two consecutive years as columns and checks the condition to generate the category and its sales as a struct. Structs are very helpful to track in these cases where the same condition is checked for more than one required values.
yrs = [k for k in yr_sales_sdf.columns if k[0:2] == '20']
sale_cat_cond = lambda frstCol, scndCol: (
func.when(func.col(frstCol).isNull() & func.col(scndCol).isNotNull(),
func.struct(func.lit(frstCol+'-'+scndCol).alias('year_band'),
func.lit('new_sales').alias('salecat'),
func.col(scndCol).alias('sale')
)
).
when(func.col(frstCol).isNotNull() & func.col(scndCol).isNull(),
func.struct(func.lit(frstCol+'-'+scndCol).alias('year_band'),
func.lit('left_sales').alias('salecat'),
func.col(frstCol).alias('sale')
)
).
otherwise(func.struct(func.lit(frstCol+'-'+scndCol).alias('year_band'),
func.lit('existing_sales').alias('salecat'),
(func.col(frstCol)-func.col(scndCol)).alias('sale')
)
)
)
Run the sale_cat_cond on all year columns, using a list comprehension, to calculate sales categories and their sales. This creates additional columns for all year bands.
yr_salecat_sdf = yr_sales_sdf. \
select('*',
*[sale_cat_cond(yrs[i], yrs[i+1]).alias(yrs[i]+'_'+yrs[i+1]+'_salecat') for i in range(len(yrs) - 1)]
)
# +---+----+----+----+-------------------------------+---------------------------------+
# |id |2018|2019|2020|2018_2019_salecat |2019_2020_salecat |
# +---+----+----+----+-------------------------------+---------------------------------+
# |5 |null|25 |125 |{2018-2019, new_sales, 25} |{2019-2020, existing_sales, -100}|
# |1 |50 |20 |70 |{2018-2019, existing_sales, 30}|{2019-2020, existing_sales, -50} |
# |3 |75 |50 |null|{2018-2019, existing_sales, 25}|{2019-2020, left_sales, 50} |
# |2 |100 |null|150 |{2018-2019, left_sales, 100} |{2019-2020, new_sales, 150} |
# +---+----+----+----+-------------------------------+---------------------------------+
The only thing left is to pivot-sum the sales categories and their sales. To do this, first collate all year band category structs into an array - This will make it easy to explode and pivot per ID (using SQL's inline function).
yr_salecat_sdf. \
withColumn('salecat_struct_arr',
func.array(*[k for k in yr_salecat_sdf.columns if '_salecat' in k])
). \
selectExpr('id', 'inline(salecat_struct_arr)'). \
groupBy('year_band'). \
pivot('salecat'). \
agg(func.sum('sale')). \
show()
# +---------+--------------+----------+---------+
# |year_band|existing_sales|left_sales|new_sales|
# +---------+--------------+----------+---------+
# |2019-2020| -150| 50| 150|
# |2018-2019| 55| 100| 25|
# +---------+--------------+----------+---------+
Additional details - schemas for all dataframes
data_sdf
# root
# |-- id: long (nullable = true)
# |-- amt: long (nullable = true)
# |-- dt: date (nullable = true)
yr_sales_sdf
# root
# |-- id: long (nullable = true)
# |-- 2018: long (nullable = true)
# |-- 2019: long (nullable = true)
# |-- 2020: long (nullable = true)
yr_salecat_sdf
# root
# |-- id: long (nullable = true)
# |-- 2018: long (nullable = true)
# |-- 2019: long (nullable = true)
# |-- 2020: long (nullable = true)
# |-- 2018-2019_salecat: struct (nullable = false)
# | |-- year_band: string (nullable = false)
# | |-- salecat: string (nullable = false)
# | |-- sale: long (nullable = true)
# |-- 2019-2020_salecat: struct (nullable = false)
# | |-- year_band: string (nullable = false)
# | |-- salecat: string (nullable = false)
# | |-- sale: long (nullable = true)
final result
# root
# |-- year_band: string (nullable = false)
# |-- existing_sales: long (nullable = true)
# |-- left_sales: long (nullable = true)
# |-- new_sales: long (nullable = true)

Answer 2

#Sparc
here is the solution . do let me know if you have questions around this.
--Approach--
You can use create two df based on year(date) and then
do inner join ---> to find the existing sales
df_2018 left_anti with df_2019--> give left_sales
df_2019 left_anti with df_2018 ---> give new sales.
combines these three by union , boom you get the result.
kindly upvote if you like my approach.
Solution:-
from pyspark.sql import Window
import pyspark.sql.functions as F
schema=["id","date_val","sales"]
data =[("1","2018-12-31","50"),
("2","2018-12-31","100"),
("3","2018-12-31","75"),
("1","2019-12-31","20"),
("3","2019-12-31","50"),
("5","2019-12-31","25")]
date_range=["2018","2019"]
df=spark.createDataFrame(data,schema)
df= df1.withColumn("date_val",F.col("date_val").cast("date"))\
.withColumn("year",F.year(F.col("date_val")).cast("string"))\
.withColumn("year_bands", F.lit(date_range[0]+"-"+date_range[1]))
filter_cond_2018 = (F.col("year") == "2018")
df_2018=df.filter(filter_cond_2018)
df_2019 = df.filter(~filter_cond_2018)
df_left_sales = df_2018.join(df_2019,["id"],"left_anti")\
.groupBy(["year","year_bands"]).agg(F.sum(F.col("sales")).alias("Left_Sales"))
df_new_sales=df_2019.join(df_2018,["id"],"left_anti")\
.groupBy(["year","year_bands"]).agg(F.sum(F.col("sales")).alias("New_Sales"))
df_ext_sales_2018 = df_2018.join(df_2019,["id"],"inner").select(df_2018["*"])\
.groupBy(["year","year_bands"]).agg(F.sum(F.col("sales")).alias("Existing_Sale_{}".format(date_range[0])))
df_ext_sales_2019 = df_2019.join(df_2018,["id"],"inner").select(df_2019["*"])\
.groupBy(["year","year_bands"]).agg(F.sum(F.col("sales")).alias("Existing_Sale_{}".format(date_range[1])))
df_agg = df_left_sales.join(df_new_sales,["year_bands"])\
.join(df_ext_sales_2018,["year_bands"])\
.join(df_ext_sales_2019,["year_bands"])
df_agg_fnl =df_agg\
.withColumn("Existing_Sales", F.col("Existing_Sale_{}".format(date_range[0]))-F.col("Existing_Sale_{}".format(date_range[1])))\
.select(["year_bands","Left_Sales","New_Sales","Existing_Sales"])
df_agg_fnl.show(10,0)
Generic Solution :
from pyspark.sql import functions as F ,DataFrame
schema=["id","date_val","sales"]
data =[("1","2018-12-31","50"),
("2","2018-12-31","100"),
("3","2018-12-31","75"),
("1","2019-12-31","20"),
("3","2019-12-31","50"),
("5","2019-12-31","25"),
("6","2020-12-31","25"),
("5","2020-12-31","10"),
("7","2020-12-31","25")]
date_range=["2018","2019"]
df=spark.createDataFrame(data,schema)
df = df.withColumn("year",F.split(F.col('date_val'), '-').getItem(0))
year_bands=df.select("year").distinct().toPandas()["year"].tolist()
def calculate_agg_data(df,start_year,end_year):
df_start_year=df.filter(F.col("year").isin([start_year]))
df_end_year=df.filter(F.col("year").isin([end_year]))
df_left_sales = df_start_year.join(df_end_year,["id"],"left_anti")\ .groupBy(["year","year_bands"]).agg(F.sum(F.col("sales")).alias("Left_Sales"))
df_new_sales=df_end_year.join(df_start_year,["id"],"left_anti")\
.groupBy(["year","year_bands"]).agg(F.sum(F.col("sales")).alias("New_Sales"))
df_start_year_ext_sales = df_start_year.join(df_end_year,["id"],"inner").select(df_start_year["*"])\
.groupBy(["year","year_bands"]).agg(F.sum(F.col("sales")).alias("Existing_Sale_{}".format(start_year)))
df_end_year_ext_sales = df_end_year.join(df_start_year,["id"],"inner").select(df_end_year["*"])\
.groupBy(["year","year_bands"]).agg(F.sum(F.col("sales")).alias("Existing_Sale_{}".format(end_year)))
# final agg
df_agg = df_left_sales.join(df_new_sales,["year_bands"])\
.join(df_start_year_ext_sales,["year_bands"])\
.join(df_end_year_ext_sales,["year_bands"])
df_agg_fnl =df_agg\
.withColumn("Existing_Sales", F.col("Existing_Sale_{}".format(start_year))-F.col("Existing_Sale_{}".format(end_year)))\
.select(["year_bands","Left_Sales","New_Sales","Existing_Sales"])
return df_agg_fnl
df_lst=[]
for index in range(len(year_bands)-1):
start_year=year_bands[index]
end_year=year_bands[index+1]
df=df.withColumn("year_bands", F.lit(start_year+"-"+end_year))
df_flt =df.filter(F.col("year").isin([start_year,end_year]))
df_agg =calculate_agg_data(df_flt,start_year,end_year)
df_lst.append(df_agg)
df_fnl = reduce(DataFrame.unionByName,df_lst)
df_fnl.show(10,0)
Alternative for reduce
df_fnl=df_lst[0]
for index in range(1,len(df_lst)):
if len(df_lst)>=1:
df_fnl=df_fnl.unionByName(df_lst[index])
df_fnl.show(40,0)
Kindly upvote of you like my solution and Approach .
()

Answer 3

So based on your previous post we have a dataframe df which is
+---+----------+----+------------+---------+---------+---------+---------+
|id |date |year|year_lst |2017-2018|2018-2019|2019-2020|2020-2021|
+---+----------+----+------------+---------+---------+---------+---------+
|1 |31/12/2017|2017|[2017, 2018]|Existing |Left |null |null |
|2 |31/12/2017|2017|[2017] |Left |null |null |null |
|3 |31/12/2017|2017|[2017, 2018]|Existing |Left |null |null |
|1 |31/12/2018|2018|[2017, 2018]|Existing |Left |null |null |
|3 |31/12/2018|2018|[2017, 2018]|Existing |Left |null |null |
|5 |31/12/2018|2018|[2018] |New |Left |null |null |
+---+----------+----+------------+---------+---------+---------+---------+
As you mentioned that you have a sales column now which you need to do the aggregation, assume when you use df.columns, your output should look like this:
['id', 'date', 'year', 'sales', 'year_lst', '2017-2018', '2018-2019', '2019-2020', '2020-2021']
To achieve your goal, what you can do is:
from pyspark.sql.functions import func
# loop from 2017-2018 column
for idx, year_year in enumerate(df.columns[4:]):
new_df = df.filter(func.col('year')==year_year.split('-')[0])\
.select('id',
'sales',
func.lit(year_year).alias('year_year'),
func.col(year_year).alias('status'))
if idx == 0:
output = new_df
else:
output = output.unionAll(new_df)
The remain is just the aggregation:
output.groupby('year_year').pivot('status').agg(func.sum('sales')).fillna(0).orderBy(func.col('year_year')).show()