How best to handle schema conflicts converting MongoRDD to DataFrame? - mongodb

Question

I'm trying to read some documents from a mongo database and parse the schema in a spark DataFrame. So far I have had success reading from mongo and transforming the resulting mongoRDD into a DataFrame using a schema defined by case classes, but there's a scenario where the mongo collection has a field containing multiple datatypes (array of strings vs. array of nested objects). So far I have been simply parsing the field as a string, then using spark sql's from_json() to parse the nested objects in the new schema, but I am finding that when a field does not conform to the schema, it returns null for all fields in the schema - not simply the field that does not conform. Is there a way to parse this so that only fields not matching the schema will return null?
//creating mongo test data in mongo shell
db.createCollection("testColl")
db.testColl.insertMany([
{ "foo" : ["fooString1", "fooString2"], "bar" : "barString"},
{ "foo" : [{"uid" : "fooString1"}, {"uid" : "fooString2"}], "bar" : "barString"}
])
import com.mongodb.spark.config.ReadConfig
import org.apache.spark.sql.{DataFrame, Row}
import org.apache.spark.sql.catalyst.ScalaReflection
import org.apache.spark.sql.functions._
import com.mongodb.spark.MongoSpark
import org.apache.spark.sql.types.{StringType, StructField, StructType}
//mongo connector and read config
val testConfig = ReadConfig(Map("uri" -> "mongodb://some.mongo.db",
"database" -> "testDB",
"collection" -> "testColl"
))
//Option 1: 'lowest common denominator' case class - works, but leaves the nested struct type value as json that then needs additional parsing
case class stringArray (foo: Option[Seq[String]], bar: Option[String])
val df1 : DataFrame = MongoSpark.load(spark.sparkContext, testConfig).toDF[stringArray]
df1.show()
+--------------------+---------+
| foo| bar|
+--------------------+---------+
|[fooString1, fooS...|barString|
|[{ "uid" : "fooSt...|barString|
+--------------------+---------+
//Option 2: accurate case class - fails with:
//com.mongodb.spark.exceptions.MongoTypeConversionException: Cannot cast STRING into a StructType(StructField(uid,StringType,true)) (value: BsonString{value='fooString1'})
case class fooDoc (uid: Option[String])
case class docArray (foo: Option[Seq[fooDoc]], bar: Option[String])
val df2 : DataFrame = MongoSpark.load(spark.sparkContext, testConfig).toDF[docArray]
//Option 3: map all rows to json string, then use from_json - why does return null for 'bar' in the case of the schema that doesn't fit?
val mrdd = MongoSpark.load(spark.sparkContext, testConfig)
val jsonRDD = mrdd.map(x => Row(x.toJson()))
val simpleSchema = StructType(Seq(StructField("wholeRecordJson", StringType, true)))
val schema = ScalaReflection.schemaFor[docArray].dataType.asInstanceOf[StructType]
val jsonDF = spark.createDataFrame(jsonRDD, simpleSchema)
val df3 = jsonDF.withColumn("parsed",from_json($"wholeRecordJson", schema))
df3.select("parsed.foo", "parsed.bar").show()
+--------------------+---------+
| foo| bar|
+--------------------+---------+
| null| null|
|[[fooString1], [f...|barString|
+--------------------+---------+
//Desired results:
//desired outcome is for only the field not matching the schema (string type of 'foo') is null, but matching columns are populated
+--------------------+---------+
| foo| bar|
+--------------------+---------+
| null|barString|
|[[fooString1], [f...|barString|
+--------------------+---------+

Answer 1

No, there is no easy way to do this as having merge incompatible schema in the same document collection is an anti-pattern, even in Mongo.
There are three main approaches to deal with this:
Fix the data in MongoDB.
Issue a query that "normalizes" the Mongo schema, e.g., drops fields with incompatible types or converts them or renames them, etc.
Issue separate queries to Mongo for documents of a particular schema type. (Mongo has query operators that can filter based on the type of a field.) Then post-process in Spark and, finally, union the data into a single Spark dataset.