Spark-Snowflake connection failure - scala

Question

I'm trying to connect to Snowflake via Spark and read data. Utilizing below two approaches to achieve the same result:
#Shell Env Variable
export COMM_LIBS=/home/user/libs
Failing to connect to Snowflake==>
spark-shell --driver-memory 5g --num-executors 3 --executor-cores 4 --executor-memory 4g --deploy-mode client --queue queueName --jars ${COMM_LIBS}/snowflake-jdbc-3.13.6.jar,${COMM_LIBS}/spark-snowflake_2.11-2.9.1-spark_2.4.jar
import scala.collection.Map
val SNOWFLAKE_SOURCE_NAME = "net.snowflake.spark.snowflake"
import net.snowflake.spark.snowflake.Utils.SNOWFLAKE_SOURCE_NAME
import net.snowflake.spark.snowflake._
import org.apache.spark.SparkConf
import org.apache.spark.sql.SparkSession
val sparkConf = new SparkConf()
.setMaster("yarn").set("spark.executor.extraJavaOptions","-Djavax.net.ssl.trustStore=/certpath -Djavax.net.ssl.trustStorePassword=password")
.set("spark.driver.extraJavaOptions","-Djavax.net.ssl.trustStore=/certPath -Djavax.net.ssl.trustStorePassword=password")
val spark = new SparkSession.Builder().config(sparkConf).enableHiveSupport().getOrCreate()
val sfOptions = Map(
"sfURL" -> "URL",
"sfAccount" -> "ACCOUNT",
"sfUser" -> "USER",
"sfPassword" -> "PASSWORD",
"sfDatabase" -> "SF_DB",
"sfSchema" -> "SF_SCHEMA",
"sfWarehouse" -> "SF_WAREHOUSE",
"sfRole" -> "SF_ROLE",
"tracing" -> "all"
)
SnowflakeConnectorUtils.disablePushdownSession(spark)
spark.read.format(SNOWFLAKE_SOURCE_NAME)
.options(sfOptions)
.option("query", "select count(1) from table")
.option("autopushdown", "off")
.load().show()
Succeeding in making connection with Snowflake==>
/opt/mapr/spark/spark-2.4.4/bin/spark-shell -v --driver-memory 4g --master yarn --deploy-mode client --num-executors 4 --executor-cores 3 --executor-memory 1g --queue queueName --conf "spark.yarn.executor.memoryOverhead=4096" --jars ${COMM_LIBS}/spark-snowflake_2.11-2.9.1-spark_2.4.jar,${COMM_LIBS}/snowflake-jdbc-3.13.6.jar --conf "spark.executor.extraJavaOptions=-Djavax.net.ssl.trustStore=/certpath -Djavax.net.ssl.trustStorePassword=password" --conf "spark.driver.extraJavaOptions=-Djavax.net.ssl.trustStore=/certpath -Djavax.net.ssl.trustStorePassword=password"
import scala.collection.Map
val SNOWFLAKE_SOURCE_NAME = "net.snowflake.spark.snowflake"
import net.snowflake.spark.snowflake.Utils.SNOWFLAKE_SOURCE_NAME
import net.snowflake.spark.snowflake._
val sfOptions = Map(
"sfURL" -> "URL",
"sfAccount" -> "ACCOUNT",
"sfUser" -> "USER",
"sfPassword" -> "PASSWORD",
"sfDatabase" -> "SF_DB",
"sfSchema" -> "SF_SCHEMA",
"sfWarehouse" -> "SF_WAREHOUSE",
"sfRole" -> "SF_ROLE",
"tracing" -> "all"
)
SnowflakeConnectorUtils.disablePushdownSession(spark)
spark.read.format(SNOWFLAKE_SOURCE_NAME)
.options(sfOptions)
.option("query", "select count(1) from table")
.option("autopushdown", "off")
.load().show()
In this case, integration with Snowflake requires passing truststore certificate, which is what I'm passing to Spark Session but in two different ways. Passing configuration programmatically in the first piece of code, which is failing due to the error Caused by: sun.security.validator.ValidatorException: No trusted certificate found . However, passing the same truststore cert as a configuration parameter to the spark-shell in the second piece of code, renders expected result. Since in both the approaches I'm passing truststore, not sure as to why spark is unable to detect truststore certs in the first version but in the second one. Requesting the community to share some insights and help with this issue.