Reading data from csv returns null values - scala

Question

I am trying to read data from csv using Scala and Spark but
the values of columns are null.
I tried to read data from csv. I also provided a schema
for querying the data easily.
private val myData= sparkSession.read.schema(createDataSchema).csv("data/myData.csv")
def createDataSchema = {
val schema = StructType(
Array(
StructField("data_index",StringType, nullable = false),
StructField("property_a",IntegerType, nullable = false),
StructField("property_b",IntegerType, nullable = false),
//some other columns
)
)
schema
Querying data:
val myProperty= accidentData.select($"property_b")
myProperty.collect()
I expect that the data are returned as a List of certain values
but they are returned as a list containing null values (values are null).
Why?
When I print the schema then nullable is set to true instead of false.
I am using Scala 2.12.9 and Spark 2.4.3.

Answer 1

While loading data from CSV file though schema has been provided as nullable = false, Still Spark overwrites schema as nullable = true, so that null pointer could be avoided during data load.
Let us take an example, let's assume CSV file has two rows with second-row has an empty or null column value.
CSV:
a,1,2
b,,2
If nullable = false, a null pointer exception would be thrown while loading data when an action has called on the data frame, as there is empty/null value to be loaded & there is no default value a Null pointer is thrown. So to avoid it Spark overwrites it as nullable = true.
However, this could be handled by replacing all null with a default value and then re-applying schema.
val df = spark.read.schema(schema).csv("data/myData.csv")
val dfWithDefault = df.withColumn("property_a", when(col("property_a").isNull, 0).otherwise(df.col("property_a")))
val dfNullableFalse = spark.sqlContext.createDataFrame(dfWithDefault.rdd, schema)
dfNullableFalse.show(10)
df.printSchema()
root
|-- data_index: string (nullable = true)
|-- property_a: integer (nullable = true)
|-- property_b: integer (nullable = true)
dfNullableFalse.printSchema()
root
|-- data_index: string (nullable = false)
|-- property_a: integer (nullable = false)
|-- property_b: integer (nullable = false)