Error connecting from pyspark to mongodb with password - mongodb

Question

I'm running following pyspark code with connection to mongodb
sparkConf = SparkConf().setMaster("local").setAppName("MongoSparkConnectorTour").set("spark.app.id", "MongoSparkConnectorTour")
# If executed via pyspark, sc is already instantiated
sc = SparkContext(conf=sparkConf)
sqlContext = SQLContext(sc)
# create and load dataframe from MongoDB URI
df = sqlContext.read.format("com.mongodb.spark.sql.DefaultSource")\
.option("spark.mongodb.input.uri", config.MONGO_URL_AUTH + "/spark.times")\
.load()
inside Docker image with
CMD [ "spark-submit" \
, "--conf", "spark.mongodb.input.uri=mongodb://root:example#mongodb:27017/spark.times" \
, "--conf", "spark.mongodb.output.uri=mongodb://root:example#mongodb:27017/spark.output" \
, "--packages", "org.mongodb.spark:mongo-spark-connector_2.11:2.4.1" \
, "./spark.py" ]
config.MONGO_URL_AUTH is mongodb://root:example#mongodb:27017
but I'm getting exception on run:
db_1 | 2019-10-09T13:44:34.354+0000 I ACCESS [conn4] Supported SASL mechanisms requested for unknown user 'root#spark'
db_1 | 2019-10-09T13:44:34.378+0000 I ACCESS [conn4] SASL SCRAM-SHA-1 authentication failed for root on spark from client 172.22.0.4:49302 ; UserNotFound: Could not find user "root" for db "spark"
pyspark_1 | Traceback (most recent call last):
pyspark_1 | File "/home/ubuntu/./spark.py", line 35, in <module>
pyspark_1 | .option("spark.mongodb.input.uri", config.MONGO_URL_AUTH + "/spark.times")\
pyspark_1 | File "/home/ubuntu/spark-2.4.4-bin-hadoop2.7/python/pyspark/sql/readwriter.py", line 172, in load
pyspark_1 | return self._df(self._jreader.load())
pyspark_1 | File "/home/ubuntu/spark-2.4.4-bin-hadoop2.7/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 1257, in __call__
pyspark_1 | File "/home/ubuntu/spark-2.4.4-bin-hadoop2.7/python/pyspark/sql/utils.py", line 63, in deco
pyspark_1 | return f(*a, **kw)
pyspark_1 | File "/home/ubuntu/spark-2.4.4-bin-hadoop2.7/python/lib/py4j-0.10.7-src.zip/py4j/protocol.py", line 328, in get_return_value
pyspark_1 | py4j.protocol.Py4JJavaError: An error occurred while calling o34.load.
pyspark_1 | : com.mongodb.MongoSecurityException: Exception authenticating MongoCredential{mechanism=SCRAM-SHA-1, userName='root', source='spark', password=<hidden>, mechanismProperties={}}
pyspark_1 | at com.mongodb.internal.connection.SaslAuthenticator.wrapException(SaslAuthenticator.java:173)
everything in this setup works flawlessly if I don't use user and password in mongodb docker and just connect with mongodb://mongodb:27017 address, and just with using pymongo I can connect with password, something is wrong with my spark to mongodb configuration when password is used and I can't understand what is wrong.
setup for mongodb (part of docker-compose file):
db:
image: mongo
restart: always
networks:
miasnet:
aliases:
- "miasdb"
environment:
MONGO_INITDB_ROOT_USERNAME: root
MONGO_INITDB_ROOT_PASSWORD: example
MONGO_INITDB_DATABASE: spark
ports:
- "27017:27017"
volumes:
- /data/db:/data/db

Answer 1

https://hub.docker.com/_/mongo reads:
MONGO_INITDB_ROOT_USERNAME, MONGO_INITDB_ROOT_PASSWORD
These variables, used in conjunction, create a new user and set that user's password. This user is created in the admin authentication database and given the role of root, which is a "superuser" role.
You don't specify the authentication database, so mongo uses current database by default - spark in your case.
You need to specify "admin" auth database in the connection string:
spark.mongodb.input.uri=mongodb://root:example#mongodb:27017/spark.times?authSource=admin
spark.mongodb.output.uri=mongodb://root:example#mongodb:27017/spark.output?authSource=admin