Count Rows where a date is within a date range for each ID - pyspark

Question

We have a dataframe (of multiple million rows) consisting of:
Id
start date
end date
date
For each Row we take the date Variable and want to count how many rows for each id exist, where this date lies between start date and end date.
This value then should be included in a new Column ("sum_of_rows").
Here is the table we expect (with sum_of_rows the to creating variable):
+---+----------+----------+----------+-----------+
| Id| start| end| date|sum_of_rows|
+---+----------+----------+----------+-----------+
| A|2008-01-02|2010-01-01|2009-01-01| 2|
| A|2005-01-02|2012-01-01| null| null|
| A|2013-01-02|2015-01-01|2014-01-01| 1|
| B|2002-01-02|2019-01-01|2003-01-01| 1|
| B|2015-01-02|2017-01-01|2016-01-01| 2|
+---+----------+----------+----------+-----------+
Example:
We look at the first Row. Take the date "2009-01-01" and want to look
in all rows where ID is the ID of the Row (so A here) and count
in how many Rows the date "2009-01-01" is within start and end (True for row 1 and 2 in this example).
Code for the original table:
table = spark.createDataFrame(
[
["A", '2008-01-02', '2010-01-01', '2009-01-01'],
["A", '2005-01-02', '2012-01-01', None],
["A", '2013-01-02', '2015-01-01', '2014-01-01'],
["B", '2002-01-02', '2019-01-01', '2003-01-01'],
["B", '2015-01-02', '2017-01-01', '2016-01-01']
],
("Id", "start", "end", "date")
)

Answer 1

This code works but creates a "product" join which is not recommended with big volumes of data.
table2 = table.select(
F.col("id"),
F.col("start").alias("s"),
F.col("end").alias("e"),
)
table3 = table.join(
table2, on="id"
)
table3 = table3.withColumn(
"one",
F.when(
F.col("date").between(F.col("s"),F.col("e")),
1
).otherwise(0)
)
table3.groupBy(
"Id",
"start",
"end",
"date"
).agg(F.sum("one").alias("sum_of_rows")).show()
+---+----------+----------+----------+-----------+
| Id| start| end| date|sum_of_rows|
+---+----------+----------+----------+-----------+
| B|2002-01-02|2019-01-01|2003-01-01| 1|
| B|2015-01-02|2017-01-01|2016-01-01| 2|
| A|2008-01-02|2010-01-01|2009-01-01| 2|
| A|2013-01-02|2015-01-01|2014-01-01| 1|
| A|2005-01-02|2012-01-01| null| 0|
+---+----------+----------+----------+-----------+