How to save the null values from a Dataset to mongodb? - mongodb

Question

I have strict requirement, to save the null values to the Mongodb, as I am aware of the case of nosql where storing null is not recommended but my business requirement have a scenario.
a sample csv file which has a null value
a,b,c,id
,2,3,A
4,4,4,B
code to save csv to mongodb
StructType schema = DataTypes.createStructType(new StructField[] {
DataTypes.createStructField("a", DataTypes.IntegerType, false),
DataTypes.createStructField("b", DataTypes.IntegerType, true),
DataTypes.createStructField("c", DataTypes.IntegerType, true),
DataTypes.createStructField("id", DataTypes.StringType, true),
});
Dataset<Row> g = spark.read()
.format("csv")
.schema(schema)
.option("header", "true")
.option("inferSchema","false")
.load("/home/Documents/SparkLogs/a.csv");
MongoSpark.save(g
.write()
.option("database", "A")
.option("collection","b").mode("overwrite")
)
;
Mongodb Output
{
"_id" : ObjectId("5d663b6bec20c94c990e6d0c"),
"a" : 4,
"b" : 4,
"c" : 4,
"id" : "B"
}
/* 2 */
{
"_id" : ObjectId("5d663b6bec20c94c990e6d0d"),
"b" : 2,
"c" : 3,
"id" : "A"
}
My requirement is to have a 'a' field will null type in it.

Answer 1

Saving as DataSet with MongoSpark will ignore the null value keys defaultly. So my workaround is to convert Dataset to javaPairRDD of BsonObject types.
Code
/** imports ***/
import scala.Tuple2;
import java.beans.Encoder;
import java.util.UUID;
import org.apache.hadoop.conf.Configuration;
import org.apache.spark.api.java.JavaPairRDD;
import org.apache.spark.sql.Encoders;
import org.apache.spark.sql.types.StructField;
import org.apache.spark.sql.types.StructType;
import org.bson.BSONObject;
import org.bson.BasicBSONObject;
import com.mongodb.hadoop.MongoOutputFormat;
/** imports ***/
private static void saveToMongoDB_With_Null(Dataset<Row> ds, Configuration outputConfig,String [] cols) {
JavaPairRDD<Object,BSONObject> document = ds
.toJavaRDD()
.mapToPair(f -> {
BSONObject doc = new BasicBSONObject();
for(String p:cols)
doc.put(p, f.getAs(p));
return new Tuple2<Object, BSONObject>(null, doc);
});
document.saveAsNewAPIHadoopFile(
"file:///this-is-completely-unused"
, Object.class
, BSONObject.class
, MongoOutputFormat.class
, outputConfig);
}
Configuration outputConfig = new Configuration();
outputConfig.set("mongo.output.uri",
"mongodb://192.168.0.19:27017/database.collection");
outputConfig.set("mongo.output.format",
"com.mongodb.hadoop.MongoOutputFormat");
Dataset<Row> g = spark.read()
.format("csv")
.schema(schema)
.option("header", "true")
.option("inferSchema","false")
.load("/home/Documents/SparkLogs/a.csv");
saveToMongoDB_With_Null(g, outputConfig,g.columns());
Needed Maven Dependency
<!-- https://mvnrepository.com/artifact/org.mongodb.mongo-hadoop/mongo-hadoop-core -->
<dependency>
<groupId>org.mongodb.mongo-hadoop</groupId>
<artifactId>mongo-hadoop-core</artifactId>
<version>2.0.2</version>
</dependency>
MongoDB output after workflow
{
"_id" : "a62e9b02-da97-493b-9563-fc19054df60e",
"a" : null,
"b" : 2,
"c" : 3,
"id" : "A"
}
{
"_id" : "fed373a8-e671-44a4-8b85-7c7e2ff59585",
"a" : 4,
"b" : 4,
"c" : 4,
"id" : "B"
}
Downsides
Bringing the high-level api like Dataset to low-level rdds will loose the spark's ability to optimise the queryplans , so trade-off is performance.