I am trying to set up IntelliJ IDEA with sbt for a Scala project. The external dependencies are specified in my build.sbt and also listed inside the IDE, as can be seen in the screenshot. However, I still get compiler errors that saying the respective symbol cannot be resolved. Can anyone point me into the right direction?
Contents of my build.sbt:
lazy val midas = (project in file("."))
.settings(
name := "test",
mainClass in assembly := Some("core.Service"),
assemblyJarName in assembly := "test.jar",
test in assembly := {},
libraryDependencies ++= Seq(
"org.slf4j" % "slf4j-api" % "1.7.25",
"com.typesafe.akka" %% "akka-actor" % "2.5.13",
"com.typesafe.akka" %% "akka-slf4j" % "2.5.13",
"com.typesafe.akka" %% "akka-remote" % "2.5.13",
"org.scala-lang.modules" %% "scala-xml" % "1.1.0",
"com.typesafe.play" %% "play-json" % "2.6.9",
"com.typesafe.slick" %% "slick" % "3.2.3",
"com.typesafe.scala-logging" %% "scala-logging" % "3.9.0",
"ch.qos.logback" % "logback-classic" % "1.2.3",
"ch.qos.logback" % "logback-core" % "1.2.3",
"com.mchange" % "c3p0" % "0.9.5.2",
"joda-time" % "joda-time" % "2.10",
"org.joda" % "joda-convert" % "2.0.1",
"net.sourceforge.jtds" % "jtds" % "1.3.1"
)
)
Still the dependencies for both, akka and joda-time cannot be resolved inside the IDE. The sbt compile from the command line works fine however.
joda-convert does not include org.joda.time.DateTime - you need "joda-time" % "joda-time" % "2.9.4" or so.
Once you have added this (and fixed your missing Akka import, maybe "com.typesafe.akka" %% "akka-actor" % "2.4.17"), you need to refresh the sbt project.
I press Shift, Shift to bring up "Search Everywhere" and type "sbt". Choose sbt under Tool Windows, then click "Refresh all sbt projects" (blue reload icon) in the sbt window. I close this window because it is not usually useful.
Hope that helps.
Related
I just started using Spark 2.2 on HDP 2.6 and Iam facing issues when trying to do sbt compile
Error
[info] Updated file /home/maria_dev/structuredstreaming/project/build.properties: set sbt.version to 1.3.0
[info] Loading project definition from /home/maria_dev/structuredstreaming/project
[info] Fetching artifacts of
[info] Fetched artifacts of
[error] lmcoursier.internal.shaded.coursier.error.FetchError$DownloadingArtifacts: Error fetching artifacts:
[error] https://repo1.maven.org/maven2/com/squareup/okhttp3/okhttp-urlconnection/3.7.0/okhttp-urlconnection-3.7.0.jar: download error: Caught java.net.UnknownHostException: repo1.maven.org (repo1.maven.org) while downloading https://repo1.maven.org/maven2/com/squareup/okhttp3/okhttp-urlconnection/3.7.0/okhttp-urlconnection-3.7.0.jar
build.sbt file is as below
buid.sbt
scalaVersion := "2.11.8"
resolvers ++= Seq(
"Conjars" at "http://conjars.org/repo",
"Hortonworks Releases" at "http://repo.hortonworks.com/content/groups/public"
)
publishMavenStyle := true
libraryDependencies ++= Seq(
"org.apache.spark" %% "spark-core" % "2.2.0.2.6.3.0-235",
"org.apache.spark" %% "spark-sql" % "2.2.0.2.6.3.0-235",
"org.apache.phoenix" % "phoenix-spark2" % "4.7.0.2.6.3.0-235",
"org.apache.phoenix" % "phoenix-core" % "4.7.0.2.6.3.0-235",
"org.apache.kafka" % "kafka-clients" % "0.10.1.2.6.3.0-235",
"org.apache.spark" %% "spark-streaming" % "2.0.2" % "provided",
"org.apache.spark" %% "spark-streaming-kafka-0-10" % "2.0.2",
"org.apache.spark" %% "spark-sql-kafka-0-10" % "2.0.2" % "provided",
"com.typesafe" % "config" % "1.3.1",
"com.typesafe.play" %% "play-json" % "2.7.2",
"com.solarmosaic.client" %% "mail-client" % "0.1.0",
"org.json4s" %% "json4s-jackson" % "3.2.10",
"org.apache.logging.log4j" % "log4j-api-scala_2.11" % "11.0",
"com.databricks" %% "spark-avro" % "3.2.0",
"org.elasticsearch" %% "elasticsearch-spark-20" % "5.0.0-alpha5",
"io.spray" %% "spray-json" % "1.3.3"
)
retrieveManaged := true
fork in run := true
It looks like coursier is attempting to fetch dependencies from repo1.maven.org which is being blocked. The Scala-Metals people an explanation here. Basically, you have to set a global Coursier config pointing to your corporate proxy server by setting up a mirror.properties file that looks like this:
central.from=https://repo1.maven.org/maven2
central.to=http://mycorporaterepo.com:8080/nexus/content/groups/public
Based on your OS, it will be:
Windows: C:\Users\\AppData\Roaming\Coursier\config\mirror.properties
Linux: ~/.config/coursier/mirror.properties
MacOS: ~/Library/Preferences/Coursier/mirror.properties
You also might need to setup SBT to use a proxy for downloading dependencies. For that, you will need to edit this file:
~/.sbt/repositories
Set it to the following:
[repositories]
local
maven-central: http://mycorporaterepo.com:8080/nexus/content/groups/public
The combination of those two settings should take care of everything you need to do to point SBT to the correct places.
I cannot create a new scala project in IntelliJ. It does not recognize the scala syntax. For example, in the below code
libraryDependencies ++= Seq(
"org.apache.spark" %% "spark-core" % sparkVersion,
"org.apache.spark" %% "spark-sql" % sparkVersion,
)
It does not recognize the word Seq. THis is my assembly file
addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.12.0")
My build.properties is
sbt.version=0.13.16
Currently I have a simple scala project.
Running sbt compile from the command line succeeds
In Intellij doing Build > Rebuild Project the build will succeed.
However there are several issues with syntax highlighting.
Receive types are not being recognized, context.become(<some Receive def>) is failing (this is due to Receive not being recognized), and lots of other issues that are seemingly random.
Current build.sbt:
scalaVersion := "2.12.1"
resolvers += "Typesafe Repository" at
"http://repo.typesafe.com/typesafe/releases/"
libraryDependencies ++= Seq(
"com.typesafe.akka" %% "akka-actor" % "2.5.1",
"com.typesafe.akka" %% "akka-http-core" % "10.0.5",
"com.typesafe.akka" %% "akka-http" % "10.0.5",
"com.typesafe.akka" %% "akka-http-spray-json" % "10.0.6",
"com.typesafe.akka" %% "akka-slf4j" % "2.5.1",
"com.typesafe.akka" %% "akka-stream" % "2.5.1",
"ch.qos.logback" % "logback-classic" % "1.1.7",
"com.typesafe.akka" %% "akka-testkit" % "2.5.1" % "test"
)
I've tried invalidating my cache and restarting, didn't work.
I've updated Intellij. I tried wiping my .idea files and re-importing the project.
Any thoughts on how to resolve this?
I'm trying to build a "hello world"-esque app that uses Spark streaming to stream data from a Kafka broker (this works), filters/processes this data, and pushes it to a (local) web browser using the Scalatra web framework and its supported web sockets functionality from Atmosphere. The Kafka/Spark chunk works independently, and the Scalatra/Atmosphere chunk also works independently. It's when I try to bring the two halves together that I run into issues with library dependencies.
The real question: how do I go about selecting library versions for which Spark will play nice with Scalatra?
A bare bones Scalatra/Atmosphere app works fine as follows:
organization := "com.example"
name := "example app"
version := "0.1.0"
scalaVersion := "2.12.2"
val ScalatraVersion = "2.5.+"
libraryDependencies ++= Seq(
"org.json4s" %% "json4s-jackson" % "3.5.2",
"org.scalatra" %% "scalatra" % ScalatraVersion,
"org.scalatra" %% "scalatra-scalate" % ScalatraVersion,
"org.scalatra" %% "scalatra-specs2" % ScalatraVersion % "test",
"org.scalatra" %% "scalatra-atmosphere" % ScalatraVersion,
"org.eclipse.jetty" % "jetty-webapp" % "9.4.6.v20170531" % "provided",
"javax.servlet" % "javax.servlet-api" % "3.1.0" % "provided"
)
enablePlugins(JettyPlugin)
But if I add new dependencies for Spark and Spark streaming, and knock the Scala version down to 2.11 (required for Spark-Kafka streaming):
organization := "com.example"
name := "example app"
version := "0.1.0"
scalaVersion := "2.11.8"
val ScalatraVersion = "2.5.+"
val SparkVersion = "2.2.0"
libraryDependencies ++= Seq(
"org.json4s" %% "json4s-jackson" % "3.5.2",
"org.scalatra" %% "scalatra" % ScalatraVersion,
"org.scalatra" %% "scalatra-scalate" % ScalatraVersion,
"org.scalatra" %% "scalatra-specs2" % ScalatraVersion % "test",
"org.scalatra" %% "scalatra-atmosphere" % ScalatraVersion,
"org.eclipse.jetty" % "jetty-webapp" % "9.4.6.v20170531" % "provided",
"javax.servlet" % "javax.servlet-api" % "3.1.0" % "provided"
)
libraryDependencies ++= Seq(
"org.apache.spark" %% "spark-core" % SparkVersion,
"org.apache.spark" %% "spark-streaming" % SparkVersion,
"org.apache.spark" %% "spark-streaming-kafka-0-8" % SparkVersion
)
enablePlugins(JettyPlugin)
The code compiles, but I get SBT's eviction warning:
[warn] There may be incompatibilities among your library dependencies.
[warn] Here are some of the libraries that were evicted:
[warn] * org.json4s:json4s-jackson_2.11:3.2.11 -> 3.5.3
[warn] Run 'evicted' to see detailed eviction warnings
Then finally, when Jetty tries to run the web server, it fails with this error:
WARN:oejuc.AbstractLifeCycle:main: FAILED org.eclipse.jetty.annotations.ServletContainerInitializersStarter#53fb3dab: java.lang.NoClassDefFoundError: com/sun/jersey/spi/inject/InjectableProvider
java.lang.NoClassDefFoundError: com/sun/jersey/spi/inject/InjectableProvider
How do I get to the bottom of this? I'm new to the Scala world, and the intricacies of dependencies are blowing my mind.
One way to remove the eviction warning is to add the library dependency with the required version using dependencyOverrides
try to add the following in your SBT file and re-build the application
dependencyOverrides += "org.json4s" % "json4s-jackson_2.11" % "3.5.3"
Check SBT documentation here
I'm using Typesafe Activator's last version (1.2.8 with Scala 2.11.x).
When I add a dependency "org.apache.httpcomponents" %% "httpclient" % "4.4-alpha1" to my project in build.sbt, something like this:
libraryDependencies ++= Seq(
"com.typesafe.akka" %% "akka-actor" % "2.3.4",
"com.typesafe.akka" %% "akka-testkit" % "2.3.4",
"org.scalatest" %% "scalatest" % "2.1.6" % "test",
"junit" % "junit" % "4.11" % "test",
"com.novocode" % "junit-interface" % "0.10" % "test",
"org.apache.httpcomponents" %% "httpclient" % "4.4-alpha1" // my added dependency
)
... and try to update the project ( in activator's cli ) I get an error:
[error] (*:update) sbt.ResolveException: unresolved dependency: org.apache.httpcomponents#httpclient_2.11;4.4-alpha1: not found
I know scala's version aren't binary compatible, but I try to get a pure-java library org.apache.httpcomponent#httpclient! Why does activator add "_2.11" at the end of artifactId and make wrong urls...? How to resolve it?
Change the first %% to a single %. The double character version is for fetching cross-built libraries, and yours isn't.