PySpark join columns after pivot - pyspark

Question

For the following example DataFrame:
df = spark.createDataFrame(
[
('2017-01-01', 'A', 1),
('2017-01-01', 'B', 2),
('2017-01-01', 'C', 3),
('2017-01-02', 'A', 4),
('2017-01-02', 'B', 5),
('2017-01-02', 'C', 6),
('2017-01-03', 'A', 7),
('2017-01-03', 'B', 8),
('2017-01-03', 'C', 9),
],
('date', 'type', 'value')
)
I would like to convert it to have the columns equal to all unique "types" (A, B, and C).
Currently, I have found this code works closest to what I would like to achieve:
df.groupby("date", "type").pivot("type").sum().orderBy("date").show()
+----------+----+----+----+----+
| date|type| A| B| C|
+----------+----+----+----+----+
|2017-01-01| C|null|null| 3|
|2017-01-01| A| 1|null|null|
|2017-01-01| B|null| 2|null|
|2017-01-02| B|null| 5|null|
|2017-01-02| C|null|null| 6|
|2017-01-02| A| 4|null|null|
|2017-01-03| A| 7|null|null|
|2017-01-03| C|null|null| 9|
|2017-01-03| B|null| 8|null|
+----------+----+----+----+----+
The issue is that I still have too many rows (containing all "null").
What I would like to get is:
+----------+---+---+---+
| date| A| B| C|
+----------+---+---+---+
|2017-01-01| 1| 2| 3|
|2017-01-02| 4| 5| 6|
|2017-01-03| 7| 8| 9|
+----------+---+---+---+
Aka, I would like something that has similar functionality to pandas.DataFrame.unstack().
If anyone has any tips on how I can achieve this in PySpark that would be great.

Answer 1

You need to do another group by "date" column then select max values from A,B,C.
Example:
df.groupby("date", "type").pivot("type").sum().orderBy("date").groupBy("date").agg(max(col("A")).alias("A"),max(col("B")).
#+----------+---+---+---+
#| date| A| B| c|
#+----------+---+---+---+
#|2017-01-01| 1| 2| 3|
#|2017-01-02| 4| 5| 6|
#|2017-01-03| 7| 8| 9|
#+----------+---+---+---+
# dynamic way
aggregate = ["A","B","C"]
funs=[max]
exprs=[f(col(c)).alias(c) for f in funs for c in aggregate]
df.groupby("date", "type").pivot("type").sum().orderBy("date").groupBy("date").agg(*exprs).show()
#+----------+---+---+---+
#| date| A| B| c|
#+----------+---+---+---+
#|2017-01-01| 1| 2| 3|
#|2017-01-02| 4| 5| 6|
#|2017-01-03| 7| 8| 9|
#+----------+---+---+---+