PySpark DateTime Functions returning nulls - pyspark

Question

I am reading in some Chicago Crimes data, and needs to use the built in pyspark datetime functions to create a month and year column. I have followed the documentation and tried several methods with no luck.
I import the following.
from pyspark.sql import SparkSession
from pyspark.sql import functions as F
from pyspark.sql import types
from pyspark.sql.functions import unix_timestamp, from_unixtime
from pyspark.sql.window import Window
My schema shows the Date, and it has string values.
df.printSchema()
root
|-- ID: integer (nullable = true)
|-- Case_Number: string (nullable = true)
|-- Date: string (nullable = true)
|-- Block: string (nullable = true)
|-- IUCR: string (nullable = true)
|-- Primary_Type: string (nullable = true)
|-- Description: string (nullable = true)
|-- Location_Description: string (nullable = true)
|-- Arrest: boolean (nullable = true)
|-- Domestic: boolean (nullable = true)
|-- District: integer (nullable = true)
|-- Community_Area: integer (nullable = true)
|-- FBI_Code: string (nullable = true)
|-- X_Coordinate: integer (nullable = true)
|-- Y_Coordinate: integer (nullable = true)
|-- Year: integer (nullable = true)
|-- Updated_On: string (nullable = true)
|-- Location: string (nullable = true)
See values, there are no nulls in the data column.
df.select('Date').show()
+--------------+
| Date|
+--------------+
|9/5/2015 13:30|
|9/4/2015 11:30|
| 9/1/2018 0:01|
|9/5/2015 12:45|
|9/5/2015 13:00|
|9/5/2015 10:55|
|9/4/2015 18:00|
|9/5/2015 13:00|
|9/5/2015 11:30|
| 5/1/2016 0:25|
|9/5/2015 14:00|
|9/5/2015 11:00|
| 9/5/2015 3:00|
|9/5/2015 12:50|
|9/3/2015 13:00|
|9/5/2015 11:45|
|9/5/2015 13:30|
| 7/8/2015 0:00|
| 9/5/2015 9:55|
|9/5/2015 12:35|
+--------------+
only showing top 20 rows
Then I call the following but get all nulls.
df2 = df.withColumn("Date", unix_timestamp("Date", "MM/dd/yyyy hh:mm"))
df2.select("Date").show()
+----------+
| Date|
+----------+
| null|
|1441384200|
| null|
|1441431900|
| null|
|1441468500|
| null|
| null|
|1441470600|
| null|
| null|
|1441468800|
|1441440000|
|1441432200|
| null|
|1441471500|
| null|
| null|
|1441464900|
|1441431300|
+----------+
only showing top 20 rows
df2 = df.withColumn("Date", df.Date.cast(types.TimestampType()))
df2.select("Date").show()
+----+
|Date|
+----+
|null|
|null|
|null|
|null|
|null|
|null|
|null|
|null|
|null|
|null|
|null|
|null|
|null|
|null|
|null|
|null|
|null|
|null|
|null|
|null|
+----+
only showing top 20 rows
I want to just use the Date column to create Month and Year.
### Get Month from date in pyspark
from pyspark.sql.functions import month, year
#df = df.withColumn("Date", df.Date.cast(types.TimestampType()))
#df = df.withColumn("Date", unix_timestamp("Date", "MM/dd/yyyy"))
df = df.withColumn('Year', year(df['Date']))
df = df.withColumn('Month', month(df['Date']))
In: df.select('Month').distinct().collect()
Out: [Row(Month=None)]

Answer 1

Ok first of all - some reproducable example would have been nice..
import pyspark
from pyspark.sql import SparkSession
spark = SparkSession.builder.appName('stack').getOrCreate()
data = [("9/5/2015 13:30"),("9/4/2015 11:30"), ("9/1/2018 0:01"), ("9/5/2015 12:45"),("9/5/2015 13:00"), ("9/5/2015 10:55"), ("9/4/2015 18:00"), ("9/5/2015 13:00")]
from pyspark.sql.types import *
dummy_df = spark.createDataFrame(data, StringType()).toDF('Datetime')
dummy_df.show()
dummy_df.printSchema()
that will give you the output:
+--------------+
| Datetime|
+--------------+
|9/5/2015 13:30|
|9/4/2015 11:30|
| 9/1/2018 0:01|
|9/5/2015 12:45|
|9/5/2015 13:00|
|9/5/2015 10:55|
|9/4/2015 18:00|
|9/5/2015 13:00|
+--------------+
root
|-- Datetime: string (nullable = true)
Now, as mentioned before be carefull with the format of the date-column and follow the instructions of https://spark.apache.org/docs/latest/sql-ref-datetime-pattern.html
from pyspark.sql.functions import regexp_replace, month, year, col, to_date
#simple regex, watch out for the white space
dummy_df = dummy_df.withColumn('Date', regexp_replace('Datetime', '( \d+:\d+)', ''))
#transform the date-column
dummy_df = dummy_df.withColumn('Date', to_date(col('Date'), "M/d/y"))\
.withColumn('Month', month(col('Date')))\
.withColumn('Year', year(col('Date')))
dummy_df.show()
dummy_df.printSchema()
finally we get:
+--------------+----------+-----+----+
| Datetime| Date|Month|Year|
+--------------+----------+-----+----+
|9/5/2015 13:30|2015-09-05| 9|2015|
|9/4/2015 11:30|2015-09-04| 9|2015|
| 9/1/2018 0:01|2018-09-01| 9|2018|
|9/5/2015 12:45|2015-09-05| 9|2015|
|9/5/2015 13:00|2015-09-05| 9|2015|
|9/5/2015 10:55|2015-09-05| 9|2015|
|9/4/2015 18:00|2015-09-04| 9|2015|
|9/5/2015 13:00|2015-09-05| 9|2015|
+--------------+----------+-----+----+
root
|-- Datetime: string (nullable = true)
|-- Date: date (nullable = true)
|-- Month: integer (nullable = true)
|-- Year: integer (nullable = true)

Answer 2

I often use the timestampFormat option when reading data in pyspark. I combine it with schema injection when reading csv files:
# -*- coding: utf-8 -*-
from pyspark.sql import DataFrame, SparkSession
from pyspark.sql.types import *
def main():
print('Spark read Crimes DataFrames inject schema')
# Create Spark Session
spark: SparkSession = (SparkSession
.builder
#.config("spark.sql.legacy.timeParserPolicy", "LEGACY")
.appName('Spark Course')
.getOrCreate())
# Define schema of the data
schema: StructType = StructType([
StructField('ID', IntegerType(), nullable=False),
StructField('Case_Number', StringType(), nullable=False),
StructField('Date', TimestampType(), nullable=True),
StructField('Block', IntegerType(), nullable=True),
StructField('IUCR', StringType(), nullable=True),
StructField('Primary_Type', StringType(), nullable=True),
StructField('Description', StringType(), nullable=True),
StructField('Location_Description', StringType(), nullable=True),
StructField('Arrest', BooleanType(), nullable=True),
StructField('Domestic', BooleanType(), nullable=True),
StructField('District', IntegerType(), nullable=True),
StructField('Community_Area', IntegerType(), nullable=True),
StructField('FBI_Code', StringType(), nullable=True),
StructField('X_Coordinate', IntegerType(), nullable=True),
StructField('Y_Coordinate', IntegerType(), nullable=True),
StructField('Year', IntegerType(), nullable=True),
StructField('Updated_On', StringType(), nullable=True),
StructField('Location', StringType(), nullable=True)
])
# Read csv injecting schema
crimes: DataFrame = (spark
.read
#.option('timestampFormat', "MM/dd/yyyy kk:mm:ss a") # Use with "spark.sql.legacy.timeParserPolicy", "LEGACY")
.option('timestampFormat', "M/d/y h:m:s a")
.csv('/home/san/Downloads/Crimes_-_2001_to_Present.csv.crdownload', header=True, schema=schema))
crimes.show()
crimes.printSchema()
if __name__ == "__main__":
try:
main()
except Exception as e:
print('Failed to execute process: {}'.format(e))
You can store this code in crimes.py and run it with
/opt/spark/bin/spark-submit crimes.py
Please, pay attention to the new date an timestamp format instroduced in Spark 3: https://spark.apache.org/docs/latest/sql-ref-datetime-pattern.html
You can always use old format configuring spark session with
.config("spark.sql.legacy.timeParserPolicy", "LEGACY")