Pyspark using withColumn to add a derived column to a dataframe - pyspark

Question

I currently have a dataframe which has the following schema;
Year: integer (nullable = true)
Month: integer (nullable = true)
Day: integer (nullable = true)
Hour: integer (nullable = true)
Minute: integer (nullable = true)
Second: integer (nullable = true)
I want to basically add an additional column to my dataframe which uses the above date components to construct a datetime type column. I am currently attempting this using the following;
df = df.withColumn("DeptDateTime",getDate(df['Year'], df['Month'], df['Day'], df['Hour'], df['Minute'], df['Second']))
I'm struggling with writing the function getDate as I want to check the length of Year (currently an Integer) & if it's 2 digits (i.e. 16) then prefix "20" to make "2016" etc. This needs to be done for each of the date components to essentially construct a date time in the following format: yyyy-mm-dd hh:mm:ss
Any help would be appreciated.

Answer 1

Convert to date
First, let's create some sample dataset.
df_pd = pd.DataFrame([[16, 3, 15],
[2016, 4, 3]],
columns=['Year', 'Month', 'Day'])
df = spark.createDataFrame(df_pd)
Then you can write udf function to work around your question.
from pyspark.sql import functions as func
from pyspark.sql.types import *
def get_date(year, month, day):
year = str(year)
month = str(month)
day = str(day)
if len(str(year)) == 2:
year = '20' + year
return year + '-' + month + '-' + day
udf_get_date = func.udf(get_date, returnType=StringType())
Now, we can apply the our UDF function to 3 columns and use .cast(DateType()) again so that you have the right format
df = df.withColumn('date', udf_get_date('Year', 'Month', 'Day').cast(DateType()))
Output
+----+-----+---+----------+
|Year|Month|Day| date|
+----+-----+---+----------+
| 16| 3| 15|2016-03-15|
|2016| 4| 3|2016-04-03|
+----+-----+---+----------+
Convert to date-time format
This is very similar, I put some variant here where you can use datetime also.
import pandas as pd
import datetime
df_pd = pd.DataFrame([[16, 3, 15, 10, 34, 14],
[2016, 4, 3, 23, 8, 12]],
columns=['Year', 'Month', 'Day', 'Hour', 'Minute', 'Second'])
df = spark.createDataFrame(df_pd)
def get_date(year, month, day, hour, minute, second):
year = str(year)
if len(str(year)) == 2:
year = '20' + year
return str(datetime.datetime(int(year), month, day, hour, minute, second))
udf_get_date = func.udf(get_date, returnType=StringType())
df = df.withColumn('date', udf_get_date('Year', 'Month', 'Day', 'Hour', 'Minute', 'Second').cast(TimestampType()))
Output
+----+-----+---+----+------+------+--------------------+
|Year|Month|Day|Hour|Minute|Second| date|
+----+-----+---+----+------+------+--------------------+
| 16| 3| 15| 10| 34| 14|2016-03-15 10:34:...|
|2016| 4| 3| 23| 8| 12|2016-04-03 23:08:...|
+----+-----+---+----+------+------+--------------------+

Answer 2

You can use datetime module to create your format.
The following code worked for me,
from datetime import datetime
def getdate(*args):
dt_str = '-'.join(map(str,args[:3])) + ' ' + ':'.join(map(str,args[3:]))
yr_len = len(str(args[0]))
if yr_len == 2:
yr = 'y'
else:
yr = 'Y'
formtd_date = datetime.strptime(dt_str,"%{}-%m-%d %H:%M:%S".format(yr)).strftime("%Y-%m-%d %H:%M:%S")
return formtd_date
Test input :
1.getdate(16,1,2,4,5,6)
2.getdate(2016,1,2,04,5,58)
output :
1.2016-01-02 04:05:06
2.2016-01-02 04:05:58