six months record of every account number in pyspark - pyspark

Question

I have tried rank function but it gives results in numbers which goes beyond 180 days.
This is the result I am getting but I do not want this result this is wrong it is giving transaction beyond 180 days
window = Window.partitionBy(df3['acctno']).orderBy(df3['trans_date'])
df3.select('*', rank().over(window).alias('rank')) \
.filter(col('rank') <= 180) \
.show(500)
{ |year|month|day|date| txnrefid| acc|branch|channel|rank|
+----+-----+---+-----------+-----------+--------------+----------+-----------+----+
|2020| 2| 6| 2020-02-06| 1234abcd6| 2074-556-1111| 6666| CBS| 1|
|2020| 2| 7| 2020-02-07| 1234abcd7| 2074-556-1111| 6666| CBS| 2|
|2020| 2| 8| 2020-02-08| 1234abcd8| 2074-556-1111| 6666| CBS| 3|
|2020| 2| 9| 2020-02-09| 1234abcd9| 2074-556-1111| 6666| CBS| 4|
But I want like this
{|year|month|day|date| txnrefid| acc|branch|channel|rank|
|2020| 2| 6| 2020-02-06| 1234abcd6| 2074-556-1111| 6666| CBS| 1|
|2020| 2| 7| 2020-02-07| 1234abcd7| 2074-556-1111| 6666| CBS| 2|
|2020| 2| 8| 2020-02-08| 1234abcd8| 2074-556-1111| 6666| CBS| 3|
|2020| 2| 9| 2020-02-09| 1234abcd9| 2074-556-1111| 6666|
}

Answer 1

As you edited your question, here is a new answer that use a different approach.
The idea is to get for each account number the min date, compute the limite date (min date + 180 days) then remove all the lines that are older.
df.count() # I used your sample data, so 60 lines
> 60
w = Window.partitionBy(df["acctno"])
df = df.withColumn("min_date", F.min(F.col("trans_date").cast("date")).over(w))
df = df.where(
F.col("trans_date").cast("date")
<= F.date_add( # Use F.date_add to add days or F.add_months to add month.
F.col("min_date"), 180
)
).drop("min_date")
df.count() # Final dataframe limited to 180 days, nothing older than 2020-08-04
> 54

Answer 2

if you want the first 6 month, then you should use fields "year" and "month", not "trans_date".
something like window = Window.partitionBy(df3['acctno']).orderBy(df3['year'], df3['month']) should give you better results.
Then you filter on rank <= 6
df3.select("*", dense_rank().over(window).alias("rank")).filter(col("rank") <= 6).show(500)
EDIT: You need to use a dense_rank