This is an SBT build file which I have written
name := "HelloSpark"
version := "1.0"
scalaVersion := "2.12.2"
libraryDependencies += "org.apache.spark" % "spark-core_2.11" % "2.1.1"
when I run sbt package I get error message
[error] (*:update) sbt.ResolveException: unresolved dependency: org.apache.spark#spark-core_2.11;2.1.1: not found
[error] Total time: 2 s, completed Oct 29, 2017 1:01:13 AM
Use Scala 2.11
scalaVersion := "2.11.11"
libraryDependencies += "org.apache.spark" %% "spark-core" % "2.1.1"
using scala 2.12 with spark 2.1
Spark does not support Scala 2.12
Just in case, you can find binaries for spark-core here:
https://mvnrepository.com/artifact/org.apache.spark/spark-core_2.11
Related
Installed intellij community edition 2022.3.1. Trying to compile a simple scala spark program and getting "Extracting structure failed: Build status: Error" error.
Below is my build.sbt
ThisBuild / version := "0.1.0-SNAPSHOT"
//ThisBuild / scalaVersion := "2.13.5"
ThisBuild / scalaVersion := "2.10.1"
lazy val root = (project in file("."))
.settings(
name := "untitled"
)
libraryDependencies ++= Seq( "org.apache.spark" % "spark-core_2.11" % "2.3.4")
Any help is most appreciated.
If you use a dependency _2.11 then your Scala version should be 2.11.x.
If your Scala version is 2.13.x (or 2.10.x) then you should use a dependency _2.13 (or _2.10 correspondingly).
https://mvnrepository.com/artifact/org.apache.spark/spark-core
"org.apache.spark" %% ... instead of "org.apache.spark" % ... automatically adds proper suffix _2.13, _2.12, _2.11, _2.10 ...
Current Spark is 3.3.1 (it exists for Scala 2.13.x or 2.12.x), current Scala is 2.13.10 (former are 2.12.17, 2.11.12, 2.10.7).
Former Spark 2.3.4 exists only for Scala 2.11.x, not 2.13.x or 2.10.x.
Version of IntelliJ is now irrelevant.
Try
ThisBuild / scalaVersion := "2.13.10"
libraryDependencies ++= Seq( "org.apache.spark" %% "spark-core" % "3.3.1")
// libraryDependencies ++= Seq( "org.apache.spark" % "spark-core_2.13" % "3.3.1")
I have installed Flink, Scala and sbt
Flink Version: 1.9.1
Scala Version: 2.10.6
Sbt Version: 1.3.7
I made relevant changes in build.sbt.
Compile command is failing
Here is the relevant information.
Any information is greatly appreciated
**Versions Information
[osboxes#osboxes local]$ scala -version
Scala code runner version 2.10.6 -- Copyright 2002-2013, LAMP/EPFL
[osboxes#osboxes local]$ flink --version
Version: 1.9.1, Commit ID: 4d56de8
[osboxes#osboxes readcsvfile]$ sbt -version
sbt version in this project: 1.3.7
sbt script version: 1.3.7
** build.sbt changes
val flinkVersion = "1.9.1"
val flinkDependencies = Seq(
"org.apache.flink" %% "flink-scala" % flinkVersion % "provided",
"org.apache.flink" %% "flink-streaming-scala" % flinkVersion % "provided")
** Compile Errors
sbt:readCsvfile> compile
[info] Updating
[info] Resolved dependencies
[warn]
[warn] Note: Unresolved dependencies path:
[error] stack trace is suppressed; run last update for the full output
[error] (update) sbt.librarymanagement.ResolveException: Error downloading org.apache.flink:flink-streaming-scala_2.13:1.9.1
[error] Not found
[error] Not found
[error] not found: /home/osboxes/.ivy2/local/org.apache.flink/flink-streaming-scala_2.13/1.9.1/ivys/ivy.xml
[error] not found: https://repo1.maven.org/maven2/org/apache/flink/flink-streaming-scala_2.13/1.9.1/flink-streaming-scala_2.13-1.9.1.pom
[error] Error downloading org.apache.flink:flink-scala_2.13:1.9.1
[error] Not found
[error] Not found
[error] not found: /home/osboxes/.ivy2/local/org.apache.flink/flink-scala_2.13/1.9.1/ivys/ivy.xml
[error] not found: https://repo1.maven.org/maven2/org/apache/flink/flink-scala_2.13/1.9.1/flink-scala_2.13-1.9.1.pom
[error] Total time: 4 s, completed Jan 30, 2020 3:59:12 PM
sbt:readCsvfile>
Few points I want to mention here regarding the SBT dependencies issues are:
Please add scalaVersion := "2.12.11" in build.sbt file like this, which includes the Scala version in your SBT dependencies automatically due to this%%.
name := "flink-streaming-demo"
scalaVersion := "2.12.11"
val flinkVersion = "1.10.0"
libraryDependencies += "org.apache.flink" %% "flink-scala" % flinkVersion % "provided"
libraryDependencies += "org.apache.flink" %% "flink-streaming-scala" % flinkVersion % "provided"
If you want Scala version specific SBT dependencies then use % like this:
libraryDependencies += "org.apache.flink" % "flink-scala_2.12" % flinkVersion % "provided"
libraryDependencies += "org.apache.flink" % "flink-streaming-scala_2.12" % flinkVersion % "provided"
In worst case if all these does not work then simply delete or rename these existing .sbt and .ivy2 hidden folder in your system home directory, where your all dependecies and plugins get sotred after downloading from maven central and then refresh/build the SBT project.
SBT dependency format
libraryDependencies += groupID % artifactID % revision % configuration
Meaning of % and %%
%: A method used to construct an Ivy Module ID from the strings you supply.
%%: When used after the groupID, it automatically adds your project’s Scala version (such as _2.12) to the end of the artifact name.
NOTE: To get more details click here.
summing up the comments since perhaps it is a bit hard to know what you should do
In general, if you get an "Unresolved dependencies" error, look at mvnrepository.com, search for your artifact:
https://mvnrepository.com/artifact/org.apache.flink/flink-scala
This tells you (second column) which Scala versions are supported by it. In this case, the library is available for 2.11.x and 2.12.x.
Thus, you have to use a Scala version compatible with that in your build, in build.sbt:
ThisBuild / scalaVersion := "2.12.10"
I'm trying to use log4j2 in scala.
Below is my build.sbt
name := "trial"
version := "0.1"
scalaVersion := "2.13.1"
libraryDependencies += "org.apache.logging.log4j" %% "log4j-api" % "2.12.1"
libraryDependencies += "org.apache.logging.log4j" %% "log4j-core" % "2.12.1"
libraryDependencies += "org.apache.logging.log4j" %% "log4j-api-scala" % "2.12.1"
How to resolve these errors:
[warn] Note: Unresolved dependencies path:
[error] sbt.librarymanagement.ResolveException: Error downloading org.apache.logging.log4j:log4j-api_2.13:2.12.1
sbt.librarymanagement.ResolveException: Error downloading org.apache.logging.log4j:log4j-api_2.13:2.12.1
not found: /Users/username/.ivy2/local/org.apache.logging.log4j/log4j-api_2.13/2.12.1/ivys/ivy.xml
Note :
There is no directory /Users/username/.ivy2/local/
but I can see there is /Users/username/.ivy2/cache/
and plugins log4j-api,core,api-scala are present in cache.
The IDE i'm using is IntelliJ IDEA.
There are multiple problems
You are using Scala 2.13 however the latest build of log4j-api-scala is for Scala 2.12.
The latest version of log4j-api-scala is 11.0 not 2.12.1
log4j-api and log4j-core are Java libraries so we should use single % instead of double%%
I followed this link : http://blog.miz.space/tutorial/2016/08/30/how-to-integrate-spark-intellij-idea-and-scala-install-setup-ubuntu-windows-mac/
When I try to compile my project with Intellij, sbt is complaining about unresolved dependencies
[Warn] ===public: tried [Warn]
https://repol.maven.org/maven2/org/apache/spark/spark-core/2.1.1/spark-core-2.1.1.pom
[Warn] Unresolved dependencies path: org.apache.spark:spark-core:2.1.1
My scala version is 2.12.2 and sparkVersion is 2.1.1
Here is what my build.sbt look like :
name := "test" version := "1.0" scalaVersion := "2.12.2"
val sparkVersion = "2.1.1"
libraryDependencies ++= Seq("org.apache.spark" % "spark-core" & sparkVersion)`
thank you
your last line should be
libraryDependencies += "org.apache.spark" % "spark-core_2.10" % sparkVersion
Or
libraryDependencies ++= Seq("org.apache.spark" % "spark-core_2.10" % sparkVersion)
And its better not to use scalaVersion := "2.12.2" as mentioned by #Nonontb and #Cyrelle. Please reduce the version that spark supports for better performance and to avoid unexpected errors.
the following line is from Spark docs
For the Scala API, Spark 2.1.1 uses Scala 2.11. You will need to use a compatible Scala version (2.11.x).
I'm using spark-cassandra, with the following dependencies:
scalaVersion := "2.11.8"
resolvers += "Spark Packages Repo" at "https://dl.bintray.com/spark-packages/maven"
libraryDependencies ++= Seq(
"org.apache.spark" %% "spark-core" % "2.0.0",
"org.apache.spark" %% "spark-mllib" % "2.0.0",
"datastax" % "spark-cassandra-connector" % "2.0.0-M3"
)
SBT fails to resolve dependencies. What should I change to make it work?
Error:Error while importing SBT project:<br/>...<br/><pre>[info] Resolving org.scala-sbt#run;0.13.8 ...
...
[error] (*:update) sbt.ResolveException: unresolved dependency: datastax#spark-cassandra-connector;2.0.0-M3: not found
[error] (*:ssExtractDependencies) sbt.ResolveException: unresolved dependency: datastax#spark-cassandra-connector;2.0.0-M3: not found
use libraryDependencies += "com.datastax.spark" % "spark-cassandra-connector_2.11" % "2.0.0-M3", as you are using scala 2.11, this should resolve it
maven central repo