Fix Dates in Pyspark DataFrame - set to minimum value - pyspark

Question

I have a data frame with a timestamp field - RECEIPTDATEREQUESTED:timestamp
For some reason, there are dates that are less than 1900-01-01. I don't want these, what I want to do, is for every value in the column of the dataframe where the RECEIPTDATEREQUESTED<'1900-01-01 00:00:00' then set the timestamp to either 1900-01-01 or null.
I've tried a few ways to do this, but it seems some more simple must exist. I thought something like this might work, but
import datetime
def testdate(date_value):
oldest = datetime.datetime.strptime('1900-01-01 00:00:00', '%Y-%m-%d')
try:
if (date_value < oldest):
return oldest
else:
return date_value
except ValueError:
return oldest
udf_testdate = udf(lambda x:testdate(x),TimestampType())
bdf = olddf.withColumn("RECEIPTDATEREQUESTED",udf_testdate(col("RECEIPTDATEREQUESTED")))

Answer 1

You can use conditional evaluation using when and otherwise to set RECEIPTDATEREQUESTED to either null or 1900-01-01 00:00:00 whenerver the value is < '1900-01-01 00:00:00'.
from pyspark.sql import functions as F
data = [("1000-01-01 00:00:00",),
("1899-12-31 23:59:59",),
("1900-01-01 00:00:00",),
("1901-01-01 00:00:00",)]
df = spark.createDataFrame(data, ("RECEIPTDATEREQUESTED",))\
.withColumn("RECEIPTDATEREQUESTED", F.to_timestamp(F.col("RECEIPTDATEREQUESTED")))
# Fill null
df.withColumn("RECEIPTDATEREQUESTED",
F.when(F.col("RECEIPTDATEREQUESTED") < "1900-01-01 00:00:00", F.lit(None))
.otherwise(F.col("RECEIPTDATEREQUESTED")))\
.show(200, False)
# Fill default value
df.withColumn("RECEIPTDATEREQUESTED",
F.when(F.col("RECEIPTDATEREQUESTED") < "1900-01-01 00:00:00", F.lit("1900-01-01 00:00:00").cast("timestamp"))
.otherwise(F.col("RECEIPTDATEREQUESTED")))\
.show(200, False)
Output
Fill null
+--------------------+
|RECEIPTDATEREQUESTED|
+--------------------+
|null |
|null |
|1900-01-01 00:00:00 |
|1901-01-01 00:00:00 |
+--------------------+
Fill 1900-01-01 00:00:00
+--------------------+
|RECEIPTDATEREQUESTED|
+--------------------+
|1900-01-01 00:00:00 |
|1900-01-01 00:00:00 |
|1900-01-01 00:00:00 |
|1901-01-01 00:00:00 |
+--------------------+