Update the dataframe with specific conditions - pyspark

Question

I have dataframe as below:
RankNumber Value Dept Number
5 200 5
4 200 5
3 205 5
2 198 5
1 197 5
5 200 6
4 202 6
3 205 6
2 198 6
1 194 6
I would like to update some of the cells from Value Column from dataframe. If the current "Value" is greater than previous value then it should get updated to previous value. If the "Value" is same or less than previous value then it should skip. It has been grouped by dept number.
I am trying to do this on pyspark, but cant find a way to accomplish that. Can someone please help??
Expected Results from dataframe is as below:
RankNumber Value Dept Number
5 200 5
4 200 5
3 200 5 (record updated)
2 198 5
1 197 5
5 200 6
4 200 6 (record updated)
3 200 6 (record updated)
2 198 6
1 194 6

Answer 1

I believe your 8th row will get updated as '3 202 6 (record updated)' instead of
'3 200 6 (record updated)'. since it's previous value was '202' and current value '205' is greater than previous '202'.
from pyspark.sql.window import Window
import pyspark.sql.functions as F
w=Window.partitionBy("DeptNumber").orderBy(desc("RankNumber"))
df = df.withColumn('previous_value',F.coalesce(F.lag(df['value'],1).over(w),df['value']))
Below code will get the previous value if Value is greater than previous value.
newdf = df.select(df.RankNumber,df.DeptNumber,df.Value,df.previous_value,when( df.Value<=df.previous_value, df.Value).otherwise(df.previous_value).alias('newValue'))
>>> newdf.show()
+----------+----------+-----+--------------+--------+
|RankNumber|DeptNumber|Value|previous_value|newValue|
+----------+----------+-----+--------------+--------+
| 5| 6| 200| 200| 200|
| 4| 6| 202| 200| 200|
| 3| 6| 205| 202| 202|
| 2| 6| 198| 205| 198|
| 1| 6| 194| 198| 194|
| 5| 5| 200| 200| 200|
| 4| 5| 200| 200| 200|
| 3| 5| 205| 200| 200|
| 2| 5| 198| 205| 198|
| 1| 5| 197| 198| 197|
+----------+----------+-----+--------------+--------+
Below code will get the minimum of previous value as a new value.
from pyspark.sql.window import Window
import pyspark.sql.functions as F
from pyspark.sql.functions import desc,when,lit
w=Window.partitionBy("DeptNumber").orderBy(desc("RankNumber"))
df = df.withColumn('previous_value',F.coalesce(F.lag(df['value'],1).over(w),df['value']))
newdf = df.select(df.RankNumber,df.DeptNumber,df.Value,df.previous_value,when( df.Value<=df.previous_value, df.Value) \
.when(F.lag(df['previous_value'],1).over(w)<=df.previous_value, F.first(df.previous_value).over(w)) \
.otherwise(df.previous_value).alias('newValue'))
>>> newdf.show()
+----------+----------+-----+--------------+--------+
|RankNumber|DeptNumber|Value|previous_value|newValue|
+----------+----------+-----+--------------+--------+
| 5| 6| 200| 200| 200|
| 4| 6| 202| 200| 200|
| 3| 6| 205| 202| 200|
| 2| 6| 198| 205| 198|
| 1| 6| 194| 198| 194|
| 5| 5| 200| 200| 200|
| 4| 5| 200| 200| 200|
| 3| 5| 205| 200| 200|
| 2| 5| 198| 205| 198|
| 1| 5| 197| 198| 197|
+----------+----------+-----+--------------+--------+
If you are looking a lowest value which is just above the previous value of that group then you need to changed a code like this.
newdf = df.select(df.RankNumber,df.DeptNumber,df.Value,df.previous_value,when( df.Value<=df.previous_value, df.Value) \
.when(F.lag(df['previous_value'],1).over(w)<=df.previous_value, F.lag(df['previous_value'],1).over(w)) \
.otherwise(df.previous_value).alias('newValue'))
This will results:
>>> newdf.show()
+----------+----------+-----+--------------+--------+
|RankNumber|DeptNumber|Value|previous_value|newValue|
+----------+----------+-----+--------------+--------+
| 5| Dept2| 100| 100| 100|
| 4| Dept2| 102| 100| 100|
| 3| Dept2| 105| 102| 100|
| 2| Dept2| 198| 105| 102|
| 1| Dept2| 194| 198| 194|
| 5| Dept1| 200| 200| 200|
| 4| Dept1| 202| 200| 200|
| 3| Dept1| 205| 202| 200|
| 2| Dept1| 198| 205| 198|
| 1| Dept1| 194| 198| 194|
+----------+----------+-----+--------------+--------+
Update:
Now creating a new dataframe as mentioned in the comment section below:
listOfTuples = [(5, 200, "Dept1"), (4, 202, "Dept1"), (3, 205, "Dept1"), (2, 198, "Dept1"), (1, 194, "Dept1") , (5, 100, "Dept2"), (4, 102, "Dept2"), (3, 105, "Dept2"), (2, 198, "Dept2"), (1, 194, "Dept2") ]
df = spark.createDataFrame(listOfTuples , ["RankNumber", "Value", "DeptNumber"])
>>> df.show()
+----------+-----+----------+
|RankNumber|Value|DeptNumber|
+----------+-----+----------+
| 5| 200| Dept1|
| 4| 202| Dept1|
| 3| 205| Dept1|
| 2| 198| Dept1|
| 1| 194| Dept1|
| 5| 100| Dept2|
| 4| 102| Dept2|
| 3| 105| Dept2|
| 2| 198| Dept2|
| 1| 194| Dept2|
+----------+-----+----------+
I believe your intention is to look in a range between current and preceding row and pick the lowest value if first condition get satisfied. ie: value is greater than previous value.
w1=Window.partitionBy("DeptNumber").orderBy(desc("RankNumber"))
w2=Window.partitionBy("DeptNumber").orderBy(desc("RankNumber")).rowsBetween(Window.unboundedPreceding, Window.currentRow)
df = df.withColumn('previous_value',F.coalesce(F.lag(df['value'],1).over(w1),df['value']))
here's your code:
newdf = df.select(df.RankNumber,df.DeptNumber,df.Value,df.previous_value,when( df.Value<=df.previous_value, df.Value) \
.otherwise(F.min(df.previous_value).over(w2)).alias('newValue'))
>>> newdf.show()
+----------+----------+-----+--------------+--------+
|RankNumber|DeptNumber|Value|previous_value|newValue|
+----------+----------+-----+--------------+--------+
| 5| Dept2| 100| 100| 100|
| 4| Dept2| 102| 100| 100|
| 3| Dept2| 105| 102| 100|
| 2| Dept2| 198| 105| 100|
| 1| Dept2| 194| 198| 194|
| 5| Dept1| 200| 200| 200|
| 4| Dept1| 202| 200| 200|
| 3| Dept1| 205| 202| 200|
| 2| Dept1| 198| 205| 198|
| 1| Dept1| 194| 198| 194|
+----------+----------+-----+--------------+--------+