SBT compile failure for Spark 2.2 - scala

Question

I just started using Spark 2.2 on HDP 2.6 and Iam facing issues when trying to do sbt compile
Error
[info] Updated file /home/maria_dev/structuredstreaming/project/build.properties: set sbt.version to 1.3.0
[info] Loading project definition from /home/maria_dev/structuredstreaming/project
[info] Fetching artifacts of
[info] Fetched artifacts of
[error] lmcoursier.internal.shaded.coursier.error.FetchError$DownloadingArtifacts: Error fetching artifacts:
[error] https://repo1.maven.org/maven2/com/squareup/okhttp3/okhttp-urlconnection/3.7.0/okhttp-urlconnection-3.7.0.jar: download error: Caught java.net.UnknownHostException: repo1.maven.org (repo1.maven.org) while downloading https://repo1.maven.org/maven2/com/squareup/okhttp3/okhttp-urlconnection/3.7.0/okhttp-urlconnection-3.7.0.jar
build.sbt file is as below
buid.sbt
scalaVersion := "2.11.8"
resolvers ++= Seq(
"Conjars" at "http://conjars.org/repo",
"Hortonworks Releases" at "http://repo.hortonworks.com/content/groups/public"
)
publishMavenStyle := true
libraryDependencies ++= Seq(
"org.apache.spark" %% "spark-core" % "2.2.0.2.6.3.0-235",
"org.apache.spark" %% "spark-sql" % "2.2.0.2.6.3.0-235",
"org.apache.phoenix" % "phoenix-spark2" % "4.7.0.2.6.3.0-235",
"org.apache.phoenix" % "phoenix-core" % "4.7.0.2.6.3.0-235",
"org.apache.kafka" % "kafka-clients" % "0.10.1.2.6.3.0-235",
"org.apache.spark" %% "spark-streaming" % "2.0.2" % "provided",
"org.apache.spark" %% "spark-streaming-kafka-0-10" % "2.0.2",
"org.apache.spark" %% "spark-sql-kafka-0-10" % "2.0.2" % "provided",
"com.typesafe" % "config" % "1.3.1",
"com.typesafe.play" %% "play-json" % "2.7.2",
"com.solarmosaic.client" %% "mail-client" % "0.1.0",
"org.json4s" %% "json4s-jackson" % "3.2.10",
"org.apache.logging.log4j" % "log4j-api-scala_2.11" % "11.0",
"com.databricks" %% "spark-avro" % "3.2.0",
"org.elasticsearch" %% "elasticsearch-spark-20" % "5.0.0-alpha5",
"io.spray" %% "spray-json" % "1.3.3"
)
retrieveManaged := true
fork in run := true

Answer 1

It looks like coursier is attempting to fetch dependencies from repo1.maven.org which is being blocked. The Scala-Metals people an explanation here. Basically, you have to set a global Coursier config pointing to your corporate proxy server by setting up a mirror.properties file that looks like this:
central.from=https://repo1.maven.org/maven2
central.to=http://mycorporaterepo.com:8080/nexus/content/groups/public
Based on your OS, it will be:
Windows: C:\Users\\AppData\Roaming\Coursier\config\mirror.properties
Linux: ~/.config/coursier/mirror.properties
MacOS: ~/Library/Preferences/Coursier/mirror.properties
You also might need to setup SBT to use a proxy for downloading dependencies. For that, you will need to edit this file:
~/.sbt/repositories
Set it to the following:
[repositories]
local
maven-central: http://mycorporaterepo.com:8080/nexus/content/groups/public
The combination of those two settings should take care of everything you need to do to point SBT to the correct places.