Dependency issue with spark-sql-kafka with spark-submit - scala

Question

I have written a simple driver class in scala that uses spark-sql-kafka for structured streaming. I have used eclipse+maven to package it into a jar. Relevant part of pom.xml file is as follows:
<dependencies>
<dependency>
<groupId>org.scala-lang</groupId>
<artifactId>scala-library</artifactId>
<version>2.11.8</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_2.11</artifactId>
<version>2.1.1</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-hive_2.11</artifactId>
<version>2.1.1</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>com.databricks</groupId>
<artifactId>spark-csv_2.11</artifactId>
<version>1.5.0</version>
<scope>runtime</scope>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-sql-kafka-0-10_2.11</artifactId>
<version>2.0.2</version>
<scope>provided</scope>
</dependency>
</dependencies>
The, I submit the resulting jar file to spark-submit using following command:
spark-submit --properties-file {path}/kafka-streaming-conf --packages org.apache.spark:spark-sql-kafka-0-10_2.11:2.0.2 --class TestStreamDriver --master yarn {path}/StructuredStreaming-1.0-SNAPSHOT.jar
kafka-streaming-conf is as follows:
spark.executor.extraJavaOptions -Dhttp.proxyHost=proxyName -Dhttp.proxyPort=8080 -Dhttps.proxyHost=proxyName -Dhttps.proxyPort=8080
spark.jars.ivySettings {path}/ivysettings_proxy.xml
ivysettings_proxy.xml file is as follows:
<ivysettings>
<settings defaultResolver="default" />
<credentials host = "proxyName:8080" username = "" passwd = ""/>
<include url="${ivy.default.settings.dir}/ivysettings-public.xml" />
<include url="${ivy.default.settings.dir}/ivysettings-shared.xml" />
<include url="${ivy.default.settings.dir}/ivysettings-local.xml" />
<include url="${ivy.default.settings.dir}/ivysettings-main-chain.xml" />
<include url="${ivy.default.settings.dir}/ivysettings-default-chain.xml"/>
</ivysettings>
I also changed JAVA_OPTS variable by:
export JAVA_OPTS="$JAVA_OPTS -Dhttp.proxyHost=proxyName -Dhttp.proxyPort=8080 -Dhttps.proxyHost=proxyName -Dhttps.proxyPort=8080"
When I run spark-submit with above command, it tries to download from maven repository and other urls and then exists with Connection timed out error.
How can I make spark-submit download dependencies through a proxy?
Thanks.

Answer 1

What worked for me is:
I changed spark-submit properties file to:
spark.driver.extraJavaOptions -Dhttp.proxyHost=proxyName -Dhttp.proxyPort=8080 -Dhttps.proxyHost=proxyName -Dhttps.proxyPort=8080
spark.executor.extraJavaOptions -Dhttp.proxyHost=proxyName -Dhttp.proxyPort=8080 -Dhttps.proxyHost=proxyName -Dhttps.proxyPort=8080
which led to a certificate error.
Then I added certificate for https://repo.maven.apache.org/maven2/
to {path}/jdk1.8.0_144\jre\lib\security\cacerts file. (I used a free program called portecle to add certificates to cacerts file. )
Since I run spark-submit in yarn mode, I had to copy new cacerts file to all nodes with:
pscp.pssh -h cluster-hosts ./cacerts {path}/jdk1.8.0_40/jre/lib/security/