extracting day, week, hour,date,year in pyspark from a string column - pyspark

Question

I am trying to extract day, week, hour,date,year in pyspark however after using dayofweek it shows null as output.
DF is something like this :
Mailed Date
Wed, 09/29/10 03:52 PM
Tue, 09/21/10 11:51 PM
Tue, 09/21/10 11:51 PM
Tue, 09/21/10 11:51 PM
I am trying to have different column named day day of week month year and hour of day
however after using from pyspark.sql.functions import year, month, dayofweek it shows null as day output column
Code i have used:
df01 = emaildf.withColumn('Day', dayofweek('Mailed_Date')).show(5)
converted into timesetemp:
df01 = vdf.withColumn("Mailed_Date",col("Mailed_Date").cast("Timestamp"))
Output:

Answer 1

Since, the string datetime provided is not in the default format, you'd have to convert the datetime to a readable format using to_timestamp(). Also, you'll need to set the timeParserPolicy to LEGACY, if you're parsing in spark 3.0+, due to the presence of week in the string.
spark.conf.set('spark.sql.legacy.timeParserPolicy', 'LEGACY') # if spark 3.0+
ts_sdf = spark.sparkContext.parallelize([('Wed, 09/29/10 03:52 PM',)]).toDF(['ts_str']). \
withColumn('ts', func.to_timestamp('ts_str', 'EEE, MM/dd/yy hh:mm a')). \
withColumn('year', func.year('ts')). \
withColumn('month', func.month('ts')). \
withColumn('dayofweek', func.dayofweek('ts')). \
withColumn('hour', func.hour('ts')). \
withColumn('minute', func.minute('ts'))
ts_sdf.show(truncate=False)
# +----------------------+-------------------+----+-----+---------+----+------+
# |ts_str |ts |year|month|dayofweek|hour|minute|
# +----------------------+-------------------+----+-----+---------+----+------+
# |Wed, 09/29/10 03:52 PM|2010-09-29 15:52:00|2010|9 |4 |15 |52 |
# +----------------------+-------------------+----+-----+---------+----+------+
ts_sdf.printSchema()
# root
# |-- ts_str: string (nullable = true)
# |-- ts: timestamp (nullable = true)
# |-- year: integer (nullable = true)
# |-- month: integer (nullable = true)
# |-- dayofweek: integer (nullable = true)
# |-- hour: integer (nullable = true)
# |-- minute: integer (nullable = true)