why spark to_json() not populating null values? - scala

Question

Can try in spark-shell
case class Employee(id: Int, name: String, department: String, salary: Option[Double])
import org.apache.spark.sql.functions._
import spark.implicits._
case class Employee(id: Int, name: String, department: String, salary: Option[Double])
val data = List(Employee(1, "XYZ", "dep1", Some(1234.0)), Employee(0, null, "unknown", None)).toDS()
data.select($"id", to_json(struct($"id",$"name", $"department", $"salary")).as("json_data")).show(false)
return =>
|id |json_data |
+---+---------------------------------------------------------+
|1 |{"id":1,"name":"XYZ","department":"dep1","salary":1234.0}|
|0 |{"id":0,"department":"unknown"} |
expecting =>
|id |json_data |
+---+------------------------------------------------------------+
|1 |{"id":1,"name":"XYZ","department":"dep1","salary":1234.0} |
|0 |{"id":0,"name": null, "department":"unknown","salary":null} |
null fields(name & salary) also should be populated in resulting json. I don't want to use lit("null") to populate null values

Answer 1

A feature was recently added to preserve null values when generating JSON, and should be available in the upcoming Spark 3.0 release. See SPARK-29444 for details. In 3.0, you'll be able to control this via:
data.select($"id", to_json(struct($"id",$"name", $"department", $"salary"), Map("ignoreNullFields" -> "false")).as("json_data")).show(false)
AFAIK, there are no plans at present to add this to the 2.x branch.

Answer 2

For spark 3.0, you could use Map(ignoreNullFields" -> "false") as an option to to_json method.
For spark 2.0 and less, you can use the below implementation -
def convertIntoJsonWithNullValuesIncluded(df: DataFrame): DataFrame = {
val colnms_n_vals = df.columns.flatMap { c => Array(lit(c), col(c)) }
val jsonDf = df.withColumn("myMap", functions.map(colnms_n_vals: _*)).select(to_json(struct(col("myMap"))).alias("json"))
val cutStringUdf = udf((x: String) => cutString(x))
jsonDf.withColumn("value", cutStringUdf(col("json"))).drop("json")
}
def cutString(s: String): String = {
s.substring(9, -2)
}