Connection With Azure to MongoDB - mongodb

Question

I am trying to connect Azure Databricks with MongoDB, but I am getting an error message which I am not able to resolve.
I am getting the following error
com.mongodb.MongoTimeoutException: Timed out after 30000 ms while waiting to connect. Client view of cluster state is {type=UNKNOWN, servers=[{address=sopt-bo-halo-01.northeurope.cloudapp.azure.com:27017, type=UNKNOWN, state=CONNECTING, exception={com.mongodb.MongoSocketWriteException: Exception sending message}, caused by {javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target}, caused by {sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target}, caused by {sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target}}, {address=sopt-bo-halo-03.northeurope.cloudapp.azure.com:27017, type=UNKNOWN, state=CONNECTING, exception={com.mongodb.MongoSocketWriteException: Exception sending message}, caused by {javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target}, caused by {sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target}, caused by {sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target}}, {address=sopt-bo-halo-02.northeurope.cloudapp.azure.com:27017, type=UNKNOWN, state=CONNECTING, exception={com.mongodb.MongoSocketWriteException: Exception sending message}, caused by {javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target}, caused by {sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target}, caused by {sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target}}]
from pyspark.sql import SparkSession
database = "prod-backoffice"
collection = "AmazonRegion"
connectionString = ‘<Username>://<Password> #hostN
/defaultdb?
ssl=true&readPreference=primary&maxIdleTimeMS=60000&connectTimeoutMS=10000&
authSource=DBNAME &authMechanism=SCRAM-SHA-1&
tlsAllowInvalidCertificates=true&tlsAllowInvalidHostnames=true
&sslAllowConnectionsWithoutCertificates=true&sslInvalidHostNameAllowed=true'
spark = SparkSession.builder\
.appName("MongoSparkConnectorIntro")\
.config('spark.mongodb.input.uri',connectionString)\
.config('spark.mongodb.output.uri', connectionString)\
.config('spark.jars.packages', 'org.mongodb.spark:mongo-spark-connector_2.12:3.0.1')\
.getOrCreate()
df = spark.read\
.format("com.mongodb.spark.sql.DefaultSource")\
.option("uri", connectionString)\
.option("database", database)\
.option("collection", collection)\
.load()

Answer 1

Connect MongoDB Atlas with DataBricks
Connection with databricks
Connect and Enable Databricks clusters to the cluster by adding the external IP addresses for the Databricks cluster nodes to the whitelist in Atlas.
Access on MongoDB Network and add the Databrick cluster IP address.
Configure Databricks Cluster with MongoDB Connection URI
Get the MongoDB connection URI. click the cluster you have created in the MongoDB Atlas UI.
Click the Connect button.
Click Connect Your Application.
According to your Databrick MongoDB connector configuration make sure your scala and spark version
Copy the generated connection string. It should look like mongodb+srv://:#Firstdatabase-wlcof.azure.mongodb.net/test?retryWrites=true
Replace password and Firstdatabase name with your password and database name.
Configuration in DataBricks
METHOD 1
In your databricks cluster, select the configuration tab and click the edit button -> under advanced options, Please select the spark configuration tab using the connection string and update the spark config. Follow the below format in the config tab
spark.mongodb.output.uri<connection-string>
spark.mongodb.input.uri<connection-string>
METHOD 2
In python notebook configure settings directly using below code.
from pyspark.sql import SparkSession
database = "<datanase_name>" #your database name
collection = "millionsongs" #your collection name
connectionString= copy your connection string here ('mongodb+srv://user:<password>#cluster0.9rvsi.mongodb.net/<database>?retryWrites=true&w=majority')
spark = SparkSession
.builder
.config('spark.mongodb.input.uri',connectionString)
.config('spark.mongodb.input.uri', connectionString)
.config('spark.jars.packages', 'org.mongodb.spark:mongo-spark-connector_2.12:3.0.1')
.getOrCreate()
#Reading from MongoDB
df = spark.read
.format("com.mongodb.spark.sql.DefaultSource")
.option("uri", connectionString)
.option("database", database)
.option("collection", collection)
.load()