Identifying changes in large amounts of data using pyspark - pyspark

Question

I have a very large amount of data (about a billion rows) of a DATE column and a RESULT column.
The values in the RESULT column are predominantly the name but every now and then there would be a significant deviation in the value. I want to only identify the dates where there was a large deviation.
So from an input dataframe as such:
+----------+------+
| DATE|RESULT|
+----------+------+
|2020-06-24| 4.2|
|2020-05-17| 4.5|
|2020-05-11| 4.5|
|2020-07-30| 4.2|
|2020-07-30| 4.2|
|2020-06-29| 4.2|
|2020-06-29| 4.2|
|2020-03-04| 4.5|
|2020-06-01| 4.2|
|2020-06-27| 4.2|
|2020-06-29| 4.2|
|2020-06-29| 4.2|
|2020-04-17| 4.5|
|2020-04-17| 4.5|
|2020-01-04| 4.5|
|2020-02-29| 4.5|
|2020-07-07| 4.2|
|2020-05-07| 4.5|
|2020-06-09| 4.2|
|2020-06-22| 4.2|
+----------+------+
I would expect an output of:
+----------+------+
| DATE|RESULT|
+----------+------+
|2020-05-11| 4.5|
|2020-07-30| 4.2|
|2020-06-29| 4.2|
|2020-04-17| 4.5|
|2020-02-29| 4.5|
|2020-07-07| 4.2|
|2020-05-07| 4.5|
|2020-06-09| 4.2|
+----------+------+
I tried using the window and lag functions, but it is forcing the entire dataset into a single node, and therefore loses the advantage of using distributed computing.
I came across a suggestion in StackOverflow to use the Median and Mean Absolute Deviation (MAD) and defining a threshold to identify the records with abnormal shifts, but I could not find a MAD statistic function in the pyspark.sql.functions library.
Does anyone have any better ideas? I would greatly appreciate it.
I am coding in pyspark, but if the solution is in spark/scala that's fine too.
Thank You

Answer 1

You may find this link useful for calculation of MAD https://www.advancinganalytics.co.uk/blog/2020/9/2/identifying-outliers-in-spark-30
Adding relevant content from that link below:
MAD=Median(|xi-xm|)
where xm is median of the data set and xi is value in data set.
MAD is median of the difference between each value and median of the entire dataset
consider a df with columns 'category', 'data_col'
‘percentile()’ expects a column and an array of percentiles to calculate (for median we can provide ‘array(0.5)’ as 50% value is median) and will return an array of results.
MADdf = df.groupby('category')\
.agg(F.expr('percentile(data_col, array(0.5))')[0]\
.alias('data_col_median'))\
.join(df, "category", "left")\
.withColumn("data_col_difference_median", F.abs(F.col('data_col')-F.col('data_col_median')))\
.groupby('category', 'data_col_median')\
.agg(F.expr('percentile(data_col_difference_median, array(0.5))')[0]\
.alias('median_absolute_difference'))