Spark with Mongodb is very slow - mongodb

Question

I am running the spark-shell with mongodb connector. But the program was very slow , i think i will don't have the response from program.
My spark-shell command is :
./spark-shell --master spark://spark_host:7077 \
--conf "spark.mongodb.input.uri=mongodb://mongod_user:password#mongod_host:27017/database.collection?readPreference=primaryPreferred" \
--jars /mongodb/lib/mongo-spark-connector_2.10-2.0.0.jar,/mongodb/lib/bson-3.2.2.jar,/mongodb/lib/mongo-java-driver-3.2.2.jar
And my app code is :
import org.apache.spark.SparkConf
import org.apache.spark.SparkContext
import com.mongodb.spark._
import org.bson.Document
import com.mongodb.spark.config.ReadConfig
import org.apache.spark.sql.SparkSession
import com.mongodb.spark.rdd.MongoRDD
val sparkSession = SparkSession.builder().getOrCreate()
val df = MongoSpark.load(sparkSession)
val dataset = df.filter("thisRequestTime > 1499250131596")
dataset.first // will wait to long time
What thing i was missed ? Help me please ~
PS: my spark is standalone model . App dependency is :
<properties>
<encoding>UTF-8</encoding>
<maven.compiler.source>1.8</maven.compiler.source>
<maven.compiler.target>1.8</maven.compiler.target>
<scala.compat.version>2.11</scala.compat.version>
<spark.version>2.1.1</spark.version>
</properties>
<dependencies>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_${scala.compat.version}</artifactId>
<version>${spark.version}</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-sql_${scala.compat.version}</artifactId>
<version>${spark.version}</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.mongodb.spark</groupId>
<artifactId>mongo-spark-connector_${scala.compat.version}</artifactId>
<version>2.0.0</version>
</dependency>
</dependencies>

Answer 1

I've been trapped into this kind of problem for a while, but got it over at last.
I don't know the detail of your Mongodb configuration, but here is my solution for my problem, hope you find it helpful.
My dataset is huge, too. So I configured a sharded cluster for mongodb, that's why make it slow. To solve it, add one piece of conf spark.mongodb.input.partitioner=MongoShardedPartitioner. Otherwise a default partition policy with be put into use, which is not suitable for a sharded mongodb.
You can find more specific information here
good luck!