Increase the date based on the length of Name column in PySpark dataframe - date

Question

I'm trying to add new columns based on the input Name and Date columns as below:
Input:
+------+-----------+
|Name |Date |
+------+-----------+
|PETER |1986-May-29|
+------+-----------+
Expected Output:
+---------+-----------+
|Character| New_Date|
+---------+-----------+
| P|1986-May-29|
| E|1986-May-30|
| T|1986-May-31|
| E|1986-Jun-01|
| R|1986-Jun-02|
+---------+-----------+
df_withchars = df.withColumn("Character", F.explode(F.split('Name','')))\
.filter(F.col('Character') != '')
df_withchars.withColumn('New_Date', (lambda x: F.date_add(x['Date'], 1) for i in range(len(x[0])))).show()
Tried the above code and throwing NameError: name 'x' is not defined

Answer 1

You can use split to create an array from the column and then posexplode to explode this array.
posexplode is similar to explode function but it returns one additional column - the position/index of an item. That means it will give you the number from 0 to 4 in your particular example. You can add this number to the date using date_add function.
Before we start lets import relevant functions:
from pyspark.sql.types import StructField, StructType, DateType, StringType
from pyspark.sql.functions import split, col, date_add, posexplode
from datetime import date
Then create a data frame:
sample_data = [('PETER', date.today())]
sample_schema = StructType([
StructField('Name', StringType(), True),
StructField('Date', DateType(), True),
])
df = spark.createDataFrame(data=sample_data, schema=sample_schema)
Final step is to use split and posexplode functions to get the index of each character, and then add the index to date.
df \
.withColumn('SplittedName', split('Name', "(?!$)")) \
.select('Name', 'Date', posexplode('SplittedName')) \
.withColumn('NewDate', date_add('Date', col('pos'))).show()
Note that in the above code, I've used regex pattern (?!$) (reference). You can use empty string "" as well. However, it will return an additional empty item in the resulting array in SplittedName column, which will requires adding another step just to remove this empty, not relevant item.
The result:
EDIT:
To get the resulting date in the desired format, you just need to use date_format (don't forget to import the function first) as follows:
df \
.withColumn('SplittedName', split('Name', "(?!$)")) \
.select('Name', 'Date', posexplode('SplittedName')) \
.withColumn('NewDate', date_format(date_add('Date', col('pos')), "yyyy-MMM-dd")).show()
New result: