Keep only modified rows in Pyspark - pyspark

Question

I need to clean a dataset filtering only modified rows (compared to the previous one) based on certain fields (in the example below we only consider cities and sports, for each id), keeping only the first occurrence.
If a row goes back to a previous state (but not for the immediately preceding), I still want to keep it.
Input df1
id
city
sport
date
abc
london
football
2022-02-11
abc
paris
football
2022-02-12
abc
paris
football
2022-02-13
abc
paris
football
2022-02-14
abc
paris
football
2022-02-15
abc
london
football
2022-02-16
abc
paris
football
2022-02-17
def
paris
volley
2022-02-10
def
paris
volley
2022-02-11
ghi
manchester
basketball
2022-02-09
Output DESIDERED
id
city
sport
date
abc
london
football
2022-02-11
abc
paris
football
2022-02-12
abc
london
football
2022-02-16
abc
paris
football
2022-02-17
def
paris
volley
2022-02-10
ghi
manchester
basketball
2022-02-09

Answer 1

I would simply use a lag function to compare over a hash :
from pyspark.sql import functions as F, Window
output_df = (
df.withColumn("hash", F.hash(F.col("city"), F.col("sport")))
.withColumn(
"prev_hash", F.lag("hash").over(Window.partitionBy("id").orderBy("date"))
)
.where(~F.col("hash").eqNullSafe(F.col("prev_hash")))
.drop("hash", "prev_hash")
)
output_df.show()
+---+----------+----------+----------+
| id| city| sport| date|
+---+----------+----------+----------+
|abc| london| football|2022-02-11|
|abc| paris| football|2022-02-12|
|abc| london| football|2022-02-16|
|abc| paris| football|2022-02-17|
|def| paris| volley|2022-02-10|
|ghi|manchester|basketball|2022-02-09|
+---+----------+----------+----------+

Answer 2

Though following solution works for the given data, there are 2 things:
Spark's architecture is not suitable for serial processing like this.
As I pointed out in the comment, you must have a key attribute or combination of attributes which can bring your data back in order if it gets fragmented. A slight change in partitioning and fragmentation can change the results.
The logic is:
Shift "city" and "sport" row by one index.
Compare with this row's "city" and "sport" with these shifted values. If you see a difference, then that is a new row. For similar rows, there will be no difference. For this we use Spark's Window util and a "dummy_serial_key".
Filter the data which matches above condition.
You can feel free to add more columns as per your data design:
from pyspark.sql.window import Window
df = spark.createDataFrame(data=[["abc","london","football","2022-02-11"],["abc","paris","football","2022-02-12"],["abc","paris","football","2022-02-13"],["abc","paris","football","2022-02-14"],["abc","paris","football","2022-02-15"],["abc","london","football","2022-02-16"],["abc","paris","football","2022-02-17"],["def","paris","volley","2022-02-10"],["def","paris","volley","2022-02-11"],["ghi","manchester","basketball","2022-02-09"]], schema=["id","city","sport","date"])
df = df.withColumn("date", F.to_date("date", format="yyyy-MM-dd"))
df = df.withColumn("dummy_serial_key", F.lit(0))
dummy_w = Window.partitionBy("dummy_serial_key").orderBy("dummy_serial_key")
df = df.withColumn("city_prev", F.lag("city", offset=1).over(dummy_w))
df = df.withColumn("sport_prev", F.lag("sport", offset=1).over(dummy_w))
df = df.filter(
(F.col("city_prev").isNull())
| (F.col("sport_prev").isNull())
| (F.col("city") != F.col("city_prev"))
| (F.col("sport") != F.col("sport_prev"))
)
df = df.drop("dummy_serial_key", "city_prev", "sport_prev")
+---+----------+----------+----------+
| id| city| sport| date|
+---+----------+----------+----------+
|abc| london| football|2022-02-11|
|abc| paris| football|2022-02-12|
|abc| london| football|2022-02-16|
|abc| paris| football|2022-02-17|
|def| paris| volley|2022-02-10|
|ghi|manchester|basketball|2022-02-09|
+---+----------+----------+----------+