Fill SQLContext with data from MongoDB in Spark with Java - mongodb

Question

I want to prepare a Java class that will read an index from MongoDB into an SQLContext in order to get a Dataset to process in Spark.My code is as follows
SparkConf conf = new SparkConf().setAppName("Aggregation").setMaster("local");
conf.set("mongo.job.input.format", "com.mongodb.hadoop.MongoInputFormat");
conf.set("mongo.input.uri", uri);
conf.set("mongo.output.uri", uri);
JavaSparkContext sc = new JavaSparkContext(conf);
SQLContext sqlContext = new SQLContext(sc);
Map<String, String> options = new HashMap<String, String>();
options.put("host", mongoHost +":27017");
options.put("inferSchema", "true");
options.put("database", database);
Dataset df = sqlContext.read().format("com.stratio.datasource.mongodb").options(options).option("collection", "kpi_aggregator").load();
I use the following dependencies in maven
<!-- Spark -->
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_2.11</artifactId>
<version>2.0.0</version>
<exclusions>
<exclusion>
<groupId>org.slf4j</groupId>
<artifactId>log4j</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-sql_2.11</artifactId>
<version>2.0.0</version>
</dependency>
<!-- MongoDB -->
<dependency>
<groupId>org.mongodb.spark</groupId>
<artifactId>mongo-spark-connector_2.11</artifactId>
<version>2.0.0-rc0</version>
</dependency>
<dependency>
<groupId>org.mongodb</groupId>
<artifactId>mongo-java-driver</artifactId>
<version>3.3.0</version>
</dependency>
<dependency>
<groupId>com.stratio.datasource</groupId>
<artifactId>spark-mongodb_2.11</artifactId>
<version>0.12.0</version>
</dependency>
However when I execute the code I get the following exception
Caused by: java.lang.IllegalArgumentException: state should be: w >= 0
at com.mongodb.assertions.Assertions.isTrueArgument(Assertions.java:99)
at com.mongodb.WriteConcern.<init>(WriteConcern.java:316)
at com.mongodb.WriteConcern.<init>(WriteConcern.java:227)
at com.mongodb.casbah.WriteConcern$.<init>(WriteConcern.scala:41)
at com.mongodb.casbah.WriteConcern$.<clinit>(WriteConcern.scala)
... 43 more
Any ideas what the problem could be?

Answer 1

I managed to solve this by removing com.stratio.datasource dependency and changing my code as follows:
SparkConf conf = new SparkConf().setAppName("Aggregation").setMaster("local");
conf.set("mongo.job.input.format", "com.mongodb.hadoop.MongoInputFormat");
conf.set("mongo.input.uri", uri);
conf.set("mongo.output.uri", uri);
conf.set("spark.mongodb.input.partitioner","MongoPaginateBySizePartitioner");
JavaSparkContext sc = new JavaSparkContext(conf);
SQLContext sqlContext = new SQLContext(sc);
Map<String, String> options = new HashMap<String, String>();
options.put("uri", uri);
options.put("database", database);
Dataset df = MongoSpark.read(sqlContext).options(options).option("collection", "kpi_aggregator").load();