What is the work logic of the Mongo-Spark Connector? - mongodb

Question

I've been trying to understand how the mongo-spark connector works under the hood, but I'm still not getting the whole work logic behind it.
Details:
I'm trying to use Mongo-Spark to run a spark job to which perform mainly text search against a MongoDB collection.
Spark and MongoDB run on two different clusters
So I created the following Spark-mongo Data frame:
entity_df = sqlContext.read.format("com.mongodb.spark.sql.DefaultSource") \
.option("spark.mongodb.input.database", "WikiData") \
.option("spark.mongodb.input.collection", "entities_slim") \
.load()
entity_df.cache()
df = entity_df.filter(entity_df['id'] == "Q2834810").show()
Does the first instruction mean that the entities_slim collection is being copied from the MongoDB cluster to the spark cluster and represented as a Spark DataFrame?
If yes, does this mean that the connector is just a tool that only read/write data between MongoDB and Spark?
If yes, Is there a way to create spark jobs that run MongoDB quires by the MongoDB engine? something like:
import pymongo
from pyspark import SparkContext
spark_rdd.map(lamda x: entities.find_one( {'id': best} ))
Note that executing statement entity_df.filter(entity_df['id'] == "Q2834810").show() runs much slower than directly querying MongoDB using pymongo

Answer 1

If yes, does this mean that the connector is just a tool that only read/write data between MongoDB and Spark?
To some extent, but it doesn't mean that
the entities_slim collection is being copied from the MongoDB cluster.
Selections (filters) are converted to aggregation pipelines:
When using filters with DataFrames or Spark SQL, the underlying Mongo Connector code constructs an aggregation pipeline to filter the data in MongoDB before sending it to Spark
This type of behavior is in general common for data source API - where projections and selections are pushed down to the source if possible.
So going back to your concern:
Note that executing statement entity_df.filter(entity_df['id'] == "Q2834810").show() runs much slower than directly querying MongoDB using pymongo
This is to be expected. Neither Apache Spark nor MongoDB aggregations pipelines are designed for low latency, single item queries. Both are intended for large scale, batch processing. If you need fast single item access don't use Apache Spark in the first place - this is what you have databases for.
Finally if run
job to which perform mainly text search against a MongoDB collection.
built-in MongoDB text search capabilities (as imperfect as they are) might be a better choice.

Answer 2

Using mongodb and spark connector you can load MongoDB data into spark to leverage sparks range of API's(Spark SQL, Spark streaming, machine learning, and graph APIs) to perform rich aggregations to your mongodb data
This enables you to leverage the spark's capabilities to analyze your data offloading to Spark
This is two-way connector, you can load mongodb data into spark and spark RDD's back to MongoDB
Does the first instruction mean that the entities_slim collection is
being copied from the MongoDB cluster to the spark cluster and
represented as a Spark DataFrame?
Yes
If yes, does this mean that the connector is just a tool that only
read/write data between MongoDB and Spark?
Yes
If yes, Is there a way to create spark jobs that run MongoDB quires by
the MongoDB engine?
You may need to query data from mongodb in mongodb itself? You can process your data into spark and store it back to mongodb