Splitting annual data into quarters in pyspark df - pyspark

Question

I have a df that looks like below. I need to break down annual data into quarters, so for each company for each year create a new row with quarter date and new EV (simply divide annual value by 4). Any suggestions how to do it?
+----------+------+---+
| date|entity| EV|
+----------+------+---+
|2018-12-31| x| 40|
|2019-12-31| x| 80|
|2018-12-31| y|120|
+----------+------+---+
Expected output:
+----------+------+---+
| date|entity| EV|
+----------+------+---+
|2018-03-31| x| 10|
|2018-06-30| x| 10|
|2018-09-30| x| 10|
|2018-12-31| x| 10|
|2019-03-31| x| 20|
|2019-06-30| x| 20|
|2019-09-30| x| 20|
|2019-12-31| x| 20|
|2018-03-31| y| 30|
|2018-06-30| y| 30|
|2018-09-30| y| 30|
|2018-12-31| y| 30|
+----------+------+---+

Answer 1

Here's one way to do it using arrays and transform.
data_sdf. \
withColumn('qtr_dt_suffix',
func.array(func.lit('03-31'), func.lit('06-30'), func.lit('09-30'), func.lit('12-31'))
). \
withColumn('qtr_dts',
func.transform('qtr_dt_suffix', lambda x: func.concat(func.year('dt'), func.lit('-'), x).cast('date'))
). \
select(func.explode('qtr_dts').alias('qtr_dt'), 'entity', (func.col('ev') / 4).alias('ev')). \
show()
# +----------+------+----+
# |qtr_dt |entity|ev |
# +----------+------+----+
# |2018-03-31|x |10.0|
# |2018-06-30|x |10.0|
# |2018-09-30|x |10.0|
# |2018-12-31|x |10.0|
# |2019-03-31|x |20.0|
# |2019-06-30|x |20.0|
# |2019-09-30|x |20.0|
# |2019-12-31|x |20.0|
# |2018-03-31|y |30.0|
# |2018-06-30|y |30.0|
# |2018-09-30|y |30.0|
# |2018-12-31|y |30.0|
# +----------+------+----+
Idea is to create an array containing all the quarter ending months and their end dates - [03-31, 06-30, 09-30, 12-31]. Use transform on this array to create dates for that year - [2018-03-31, 2018-06-30, 2018-09-30, 2018-12-31]. Explode this resulting array to create rows for each quarter dates.
If transform is not available in your spark version, you can use transform in expr.
data_sdf. \
withColumn('qtr_dt_suffix',
func.array(func.lit('03-31'), func.lit('06-30'), func.lit('09-30'), func.lit('12-31'))
). \
withColumn('qtr_dts',
func.expr('transform(qtr_dt_suffix, x -> cast(concat(year(dt), "-", x) as date))')
). \
show(truncate=False)