data frame date field with null values to SQL database - pyspark

Question

I have a pyspark script that processes a dataframe containing a date & time, called startdate, for which I use the following code to convert it into a timestamp format:
## Convert startdate to timestamp
df = df.withColumn('startdate', regexp_replace('startdate', 'T', ' ')) \
.withColumn("startdate",expr("substring(startdate, 1, length(startdate)-1)")) \
.withColumn('startdate',to_timestamp(col('startdate'), 'yyyy-MM-dd HH:mm:ss'))
I'm creating a type 2 slowly changing dimension, so for every ID, I'm taking the start date and "lagging it" to create an end date. For this, I run the following code:
# Get end date
df = df.withColumn("rank",
dense_rank().over(Window.partitionBy('id').orderBy(desc("startdate"))))
partition = Window().partitionBy().orderBy(col('id'))
df = df.withColumn("enddate",when(df.rank == 1,lit(None)).otherwise(lag("startdate").over(partition)))
This all works fine and the script outputs to parquet files in data lake Gen 2, and when I display my output I can see the right output. But the next step that I'm doing which is that I have an ADF pipeline that copies data from parquet files into an Azure SQL database fails because it isn't turning the 'undefined' (or the output of the lit(none) part of the script) values into NULL in the database.
My question is, what do I need to do in my script above, where I'm currently using lit(none) to get an output that will be compatible to turn into a NULL 'value' in the SQL database as part of my pipeline?
The column in my database is of datetime2 type, and is nullable. The startdate column is working fine, but it will never be NULL / empty, which is why I've concluded the issue here is with the 'empty' values.

Answer 1

The question was asked in error as something else was obstructing my pipeline, but to answer the question, of how to produce an "empty" value in parquet files, that becomes a NULL value in a SQL database, the following values can be used:
lit(None)
None
So the following line in my code was actually working:
df = df.withColumn("enddate",when(df.rank == 1,lit(None)).otherwise(lag("startdate").over(partition)))