I am starting to learn spark sql I am using the following dependencies in sbt. I am getting errors
name := "sparkLearning"
version := "1.0"
scalaVersion := "2.11.8"
val sparkVersion = "1.6.1"
val sqlVersion = "1.3.1"
libraryDependencies ++= Seq(
"org.apache.spark" %% "spark-core" % sparkVersion,
"org.apache.spark" % "spark-sql" % sqlVersion
)
I am getting an error.
Error:Error while importing SBT project:<br/>...<br/><pre>[info] Resolving com.thoughtworks.paranamer#paranamer;2.6 ...
[info] Resolving org.scala-sbt#completion;0.13.15 ...
[info] Resolving org.scala-sbt#control;0.13.15 ...
[info] Resolving org.scala-sbt#sbt;0.13.15 ...
[info] Resolving org.scala-sbt#run;0.13.15 ...
[info] Resolving org.scala-sbt.ivy#ivy;2.3.0-sbt-48dd0744422128446aee9ac31aa356ee203cc9f4 ...
[info] Resolving org.scala-sbt#test-interface;1.0 ...
[info] Resolving com.jcraft#jsch;0.1.50 ...
[info] Resolving org.scala-lang#scala-compiler;2.10.6 ...
[info] Resolving jline#jline;2.14.3 ...
[info] Resolving org.scala-sbt#compiler-ivy-integration;0.13.15 ...
[info] Resolving org.scala-sbt#incremental-compiler;0.13.15 ...
[info] Resolving org.scala-sbt#logic;0.13.15 ...
[info] Resolving org.scala-sbt#main-settings;0.13.15 ...
[trace] Stack trace suppressed: run 'last *:update' for the full output.
[trace] Stack trace suppressed: run 'last *:ssExtractDependencies' for the full output.
[error] (*:update) sbt.ResolveException: unresolved dependency: org.apache.spark#spark-sql;1.3.1: not found
[error] (*:ssExtractDependencies) sbt.ResolveException: unresolved dependency: org.apache.spark#spark-sql;1.3.1: not found
[error] Total time: 15 s, completed 27-Jul-2017 15:29:52
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=384M; support was removed in 8.0
Please let me know how to resolve this.
correct form for you sbt file is
name := "sparkLearning"
version := "1.0"
scalaVersion := "2.11.8"
val sparkVersion = "1.6.1"
libraryDependencies ++= Seq(
"org.apache.spark" % "spark-core_2.10" % sparkVersion,
"org.apache.spark" % "spark-sql_2.10" % sparkVersion
)
I would suggest you to use latest spark versions which should be compatible with scala 2.11.8
name := "sparkLearning"
version := "1.0"
scalaVersion := "2.11.8"
val sparkVersion = "2.2.0"
libraryDependencies ++= Seq(
"org.apache.spark" %% "spark-core" % sparkVersion,
"org.apache.spark" %% "spark-sql" % sparkVersion
)
Related
I'm new in scala world (coming from android world), I have created scala project with play-framework, everything works fine now I need to add database and for that I decided to chose slick, but when I'm trying to add dependency like that
libraryDependencies ++= Seq(
"com.typesafe.slick" %% "slick" % "3.1.0",
"org.slf4j" % "slf4j-nop" % "1.6.4"
)
I'm getting this error log
Error:Error while importing SBT project:<br/>...<br/><pre>[info] Resolving com.typesafe#jse_2.10;1.2.3 ...
[info] Resolving org.scala-sbt#run;0.13.15 ...
[info] Resolving org.scala-sbt.ivy#ivy;2.3.0-sbt-48dd0744422128446aee9ac31aa356ee203cc9f4 ...
[info] Resolving com.typesafe.play#play-exceptions;2.6.5 ...
[info] Resolving org.scala-sbt#test-interface;1.0 ...
[info] Resolving com.jcraft#jsch;0.1.50 ...
[info] Resolving org.scala-lang#scala-compiler;2.10.6 ...
[info] Resolving jline#jline;2.14.3 ...
[info] Resolving org.scala-sbt#compiler-ivy-integration;0.13.15 ...
[info] Resolving org.scala-sbt#incremental-compiler;0.13.15 ...
[info] Resolving org.scala-sbt#logic;0.13.15 ...
[info] Resolving com.typesafe#config;1.3.1 ...
[info] Resolving org.scala-sbt#main-settings;0.13.15 ...
[info] Resolving com.lightbend.play#play-file-watch_2.10;1.0.0 ...
[info] Resolving com.typesafe.play#sbt-plugin;2.6.5 ...
[trace] Stack trace suppressed: run 'last *:ssExtractDependencies' for the full output.
[trace] Stack trace suppressed: run 'last *:update' for the full output.
[error] (*:ssExtractDependencies) sbt.ResolveException: unresolved dependency: com.typesafe.slick#slick_2.12;3.1.0: not found
[error] (*:update) sbt.ResolveException: unresolved dependency: com.typesafe.slick#slick_2.12;3.1.0: not found
[error] Total time: 4 s, completed Sep 28, 2017 12:21:57 PM</pre><br/>See complete log in file:/home/dev-00/.IntelliJIdea2017.1/system/log/sbt.last.log
here is my hole sbt build script
name := """play-scala-starter-example"""
version := "1.0-SNAPSHOT"
lazy val root = (project in file(".")).enablePlugins(PlayScala)
resolvers += Resolver.sonatypeRepo("snapshots")
scalaVersion := "2.12.2"
libraryDependencies ++= Seq(
"com.typesafe.slick" %% "slick" % "3.1.0",
"org.slf4j" % "slf4j-nop" % "1.6.4"
)
libraryDependencies ++= Seq(
jdbc,
"com.typesafe.play" %% "anorm" % "2.5.3"
)
libraryDependencies += guice
libraryDependencies += "org.scalatestplus.play" %% "scalatestplus-play" % "3.0.0" % Test
libraryDependencies += "com.h2database" % "h2" % "1.4.194"
libraryDependencies += jdbc
libraryDependencies += "mysql" % "mysql-connector-java" % "5.1.41"
From what I find on maven, it seems only few versions of slick are compiled against scala 2.12:
3.2.1
3.2.0
2.1.0
see: http://search.maven.org/#search%7Cgav%7C1%7Cg%3A%22com.typesafe.slick%22%20AND%20a%3A%22slick_2.12%22
Either use a more recent version of slick or another scala version (2.11.x).
Add the following line to your build.sbt
resolvers += "typesafe" at "http://repo.typesafe.com/typesafe/releases/"
Here is my build.sbt
resolvers ++= Seq("Apache Development Snapshot Repository" at "https://repository.apache.org/content/repositories/snapshots/",
Resolver.mavenLocal)
name := "flink-wiki-edits"
version := "1.0"
scalaVersion := "2.11.7"
val flinkVersion = "1.1.3"
val flinkDependencies = Seq(
"org.apache.flink" %% "flink-scala" % flinkVersion % "provided",
"org.apache.flink" %% "flink-streaming-scala" % flinkVersion % "provided",
"org.apache.flink" %% "flink-connector-wikiedits"% flinkVersion )
lazy val root = (project in file(".")).
settings(
libraryDependencies ++= flinkDependencies
)
mainClass in assembly := Some("com.ta.WikipediaAnalytics")
// make run command include the provided dependencies
run in Compile <<= Defaults.runTask(fullClasspath in Compile, mainClass in (Compile, run), runner in (Compile, run))
// exclude Scala library from assembly
assemblyOption in assembly := (assemblyOption in assembly).value.copy(includeScala = false)
and the console output snippet of $ sbt assembly
vagrant#vagrant-ubuntu-trusty-64:/workspace/flink-wiki-edits$ sbt assembly
[info] Loading project definition from /workspace/flink-wiki-edits/project
[info] Set current project to flink-wiki-edits (in build file:/workspace/flink-wiki-edits/)
[info] Including: flink-connector-wikiedits_2.11-1.1.3.jar
[info] Including: flink-streaming-java_2.11-1.1.3.jar
[info] Including: flink-core-1.1.3.jar
[info] Including: flink-annotations-1.1.3.jar
[info] Including: jsr305-1.3.9.jar
[info] Including: commons-lang3-3.3.2.jar
[info] Including: slf4j-api-1.7.7.jar
[info] Including: slf4j-log4j12-1.7.7.jar
[info] Including: log4j-1.2.17.jar
[info] Including: force-shading-1.1.3.jar
[info] Including: flink-metrics-core-1.1.3.jar
[info] Including: kryo-2.24.0.jar
[info] Including: minlog-1.2.jar
where I see lot of libraries that are part of the
"org.apache.flink" %% "flink-scala" % flinkVersion % "provided",
"org.apache.flink" %% "flink-streaming-scala" % flinkVersion % "provided"
lineage, I assumed that provided will exclude the libraries while assembling the fat-jar, let me know if I have overlooked something.
This is caused by flink-connector-wikiedits library has dependency flink-streaming-java_2.11, see: flink-streaming-java_2.11 pom:
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-streaming-java_2.11</artifactId>
<version>1.1.3</version>
<scope>compile</scope>
</dependency>
and also flink-streaming-java_2.11 has flink-core ... dependency, see:
http://search.maven.org/#artifactdetails%7Corg.apache.flink%7Cflink-streaming-java_2.11%7C1.1.3%7Cjar
I am having a build problem. Here is my sbt file:
name := "SparkPi"
version := "0.2.15"
scalaVersion := "2.11.8"
// https://mvnrepository.com/artifact/org.apache.spark/spark-core_2.10
libraryDependencies += "org.apache.spark" % "spark-core_2.10" % "2.0.1"
// old:
//libraryDependencies += "org.apache.spark" %% "spark-core" % "2.0.1"
// https://mvnrepository.com/artifact/com.amazonaws/aws-java-sdk
libraryDependencies += "com.amazonaws" % "aws-java-sdk" % "1.0.002"
scalacOptions ++= Seq("-feature")
Here is the full error message I am seeing:
[info] Set current project to SparkPi (in build file:/Users/xxx/prog/yyy/)
[info] Updating {file:/Users/xxx/prog/yyy/}yyy...
[info] Resolving jline#jline;2.12.1 ...
[info] Done updating.
[info] Compiling 2 Scala sources to /Users/xxx/prog/yyy/target/scala-2.11/classes...
[error] /Users/xxx/prog/yyy/src/main/scala/PiSpark.scala:6: object profile is not a member of package com.amazonaws.auth
[error] import com.amazonaws.auth.profile._
[error] ^
[error] /Users/xxx/prog/yyy/src/main/scala/PiSpark.scala:87: not found: type ProfileCredentialsProvider
[error] val creds = new ProfileCredentialsProvider(profile).getCredentials()
[error] ^
[error] two errors found
[error] (compile:compileIncremental) Compilation failed
[error] Total time: 14 s, completed Nov 3, 2016 1:43:34 PM
And here are the imports I am trying to use:
import com.amazonaws.services.s3._
import com.amazonaws.auth.profile._
How do I import com.amazonaws.auth.profile.ProfileCredentialsProvider in Scala?
EDIT
Changed sbt file so spark core version corresponds to Scala version, new contents:
name := "SparkPi"
version := "0.2.15"
scalaVersion := "2.11.8"
// https://mvnrepository.com/artifact/org.apache.spark/spark-core_2.11
libraryDependencies += "org.apache.spark" % "spark-core_2.11" % "2.0.1"
// old:
//libraryDependencies += "org.apache.spark" %% "spark-core" % "2.0.1"
// https://mvnrepository.com/artifact/com.amazonaws/aws-java-sdk
libraryDependencies += "com.amazonaws" % "aws-java-sdk" % "1.0.002"
scalacOptions ++= Seq("-feature")
You are using scalaVersion := "2.11.8" but library dependency has underscore 2.10 spark-core_2.10 which is bad.
libraryDependencies += "org.apache.spark" % "spark-core_2.10" % "2.0.1"
^
change 2.10 to 2.11
libraryDependencies += "org.apache.spark" % "spark-core_2.11" % "2.0.1"
`
After adding
libraryDependencies += "org.scalatest" %% "scalatest" % "3.0.0" % "test"
to build.sbt, and refreshed the project, I got this msg.
SBT project import
[warn] Multiple dependencies with the same organization/name but
different versions. To avoid conflict, pick one version:
[warn] * org.scala-lang.modules:scala-xml_2.11:(1.0.5, 1.0.4)
Changing the above to
libraryDependencies ++= Seq(
"org.scalatest" %% "scalatest" % "3.0.0" % "test"
exclude("org.scala-lang", "scala-reflect")
exclude("org.scala-lang.modules", "scala-xml_2.11")
)
solves the issue. However, instead of excluding scala-xml_2.11 version 1.0.5 from scalatest, I would like to force the scala compiler to use scala-xml_2.11 version 1.0.5 instead of version 1.0.4. (I researched the versions at https://mvnrepository.com.) Thus I tried substituting scalaVersion := "2.11.8" for
libraryDependencies ++= Seq(
"org.scala-lang" % "scala-reflect" % "2.11.8",
"org.scala-lang.modules" % "scala-xml_2.11" % "1.0.5"
)
This however results in
SBT project import
[warn] Binary version (2.11) for dependency
org.scala-lang#scala-reflect;2.11.8
[warn] in default#myproject$sources_javadoc_2.10;1.0 differs
from Scala binary version in project (2.10).
[warn] Binary version (2.11) for dependency
org.scala-lang#scala-library;2.11.8
[warn] in default#myproject$sources_javadoc_2.10;1.0 differs
from Scala binary version in project (2.10).
[warn] Multiple dependencies with the same organization/name but
different versions. To avoid conflict, pick one version:
[warn] * org.scala-lang:scala-library:(2.11.8, 2.10.4)
[warn] * org.scala-lang:scala-reflect:(2.11.8, 2.10.4)
[warn] [FAILED ]
com.artima.supersafe#supersafe_2.10.4;1.1.0!supersafe_2.10.4.jar(src):
(0ms)
[warn] ==== local: tried
[warn]
/home/user/.ivy2/local/com.artima.supersafe/supersafe_2.10.4/1.1.0/srcs/supersafe_2.10.4-sources.jar
[warn] ==== activator-local: tried [warn] /Development/Activator/activator-dist-1.3.10/repository/com.artima.supersafe/supersafe_2.1...
(show balloon)
What am I supposed to do?
Edit: What else I tried and did not work:
scalaVersion := "2.11.8"
libraryDependencies ++= Seq(
"org.scala-lang" % "scala-reflect" % scalaVersion.value,
"org.scala-lang.modules" %% "scala-xml" % "1.0.5"
)
// ScalaTest
libraryDependencies += "org.scalatest" %% "scalatest" % "3.0.0" % "test"
[warn] Multiple dependencies with the same organization/name but
different versions. To avoid conflict, pick one version: [warn] *
org.scala-lang.modules:scala-xml_2.11:(1.0.5, 1.0.4)
scalaVersion := "2.11.8"
libraryDependencies ++= Seq(
"org.scala-lang" % "scala-reflect" % scalaVersion.value,
"org.scala-lang.modules" %% "scala-xml_2.11" % "1.0.5"
)
// ScalaTest
libraryDependencies += "org.scalatest" %% "scalatest" % "3.0.0" % "test"
Error:Error while importing SBT project:...[info]
Resolving org.scala-sbt#run;0.13.8 ... [info] Resolving
org.scala-sbt#task-system;0.13.8 ... [info] Resolving
org.scala-sbt#tasks;0.13.8 ... [info] Resolving
org.scala-sbt#tracking;0.13.8 ... [info] Resolving
org.scala-sbt#cache;0.13.8 ... [info] Resolving
org.scala-sbt#testing;0.13.8 ... [info] Resolving
org.scala-sbt#test-agent;0.13.8 ... [info] Resolving
org.scala-sbt#test-interface;1.0 ... [info] Resolving
org.scala-sbt#main-settings;0.13.8 ... [info] Resolving
org.scala-sbt#apply-macro;0.13.8 ... [info] Resolving
org.scala-sbt#command;0.13.8 ... [info] Resolving
org.scala-sbt#logic;0.13.8 ... [info] Resolving
org.scala-sbt#precompiled-2_8_2;0.13.8 ... [info] Resolving
org.scala-sbt#precompiled-2_9_2;0.13.8 ... [info] Resolving
org.scala-sbt#precompiled-2_9_3;0.13.8 ... [trace] Stack trace
suppressed: run 'last *:update' for the full output. [trace] Stack
trace suppressed: run 'last :ssExtractDependencies' for the full
output. [error] (:update) sbt.ResolveException: unresolved
dependency: org.scala-lang.modules#scala-xml_2.11_2.11;1.0.5: not
found [error] (*:ssExtractDependencies) sbt.ResolveException:
unresolved dependency:
org.scala-lang.modules#scala-xml_2.11_2.11;1.0.5: not found [error]
Total time: 4 s, completed 01.10.2016 17:46:55
scalaVersion := "2.11.8"
libraryDependencies ++= Seq(
"org.scala-lang" % "scala-reflect" % scalaVersion.value,
"org.scala-lang.modules" %% "scala-xml" % "1.0.5"
)
// ScalaTest
libraryDependencies ++= Seq(
"org.scalatest" %% "scalatest" % "3.0.0" % "test"
exclude("org.scala-lang", "scala-reflect")
exclude("org.scala-lang.modules", "scala-xml")
)
[warn] Multiple dependencies with the same organization/name but
different versions. To avoid conflict, pick one version: [warn] *
org.scala-lang.modules:scala-xml_2.11:(1.0.5, 1.0.4)
My build.sbt
name := "MyProject"
version := "0.1.0"
scalaVersion := "2.11.8"
libraryDependencies ++= Seq(
"org.scala-lang" % "scala-reflect" % scalaVersion.value,
"org.scala-lang.modules" %% "scala-xml" % "1.0.5"
)
// ScalaTest
//libraryDependencies += "org.scalactic" %% "scalactic" % "3.0.0"
//libraryDependencies += "org.scalatest" %% "scalatest" % "3.0.0" % "test"
libraryDependencies ++= Seq(
"org.scalatest" %% "scalatest" % "3.0.0" % "test"
exclude("org.scala-lang", "scala-reflect")
exclude("org.scala-lang.modules", "scala-xml_2.11")
)
The way to exclude specific transitive dependency is this:
Run sbt evicted to figure out which of the project dependencies is pulling in outdated library, let's assume the problematic library is: com.typesafe.slick.
Add the following exclude (the parentheses are important):
("com.typesafe" %% "slick" % "3.1.1").exclude("org.scala-lang.modules", "scala-xml_2.11")
Add this normally as you would be listing dependencies.
This will prevent sbt from including any version of scala-xml that was coming in as a transitive dependency of Slick.
Don't substitute; you need both scalaVersion and libraryDependencies.
Though use
scalaVersion := "2.11.8"
libraryDependencies ++= Seq(
"org.scala-lang" % "scala-reflect" % scalaVersion.value,
"org.scala-lang.modules" %% "scala-xml" % "1.0.5"
)
to avoid bugs when you eventually change scalaVersion.
By removing scalaVersion you get default scalaVersion := "2.10.4" (with your version/settings of SBT, at least) but your libraryDependencies still require 2.11.
I'm migrating a Scala application that compiles and runs fine by manually
including jars in the classpath to a SBT build configuration.
My build.sbt is as follows:
name := "hello"
version := "1.0"
scalaVersion := "2.9.2"
libraryDependencies += "org.slf4j" % "slf4j-simple" % "1.6.4"
libraryDependencies += "junit" % "junit" % "4.11"
libraryDependencies += "org.scalatest" % "scalatest_2.10" % "1.9.2"
libraryDependencies += "org.hamcrest" % "hamcrest-all" % "1.3"
libraryDependencies += "ch.qos.logback" % "logback-classic" % "1.0.13"
libraryDependencies += "com.github.scct" % "scct_2.10" % "0.2.1"
libraryDependencies += "org.scala-lang" % "scala-swing" % "2.9.2"
When I compile it I get the following errors:
Loading /usr/share/sbt/bin/sbt-launch-lib.bash
[info] Set current project to hello (in build file:/home/kevin/gitrepos/go-game-msc/)
> compile
[info] Updating {file:/home/kevin/gitrepos/go-game-msc/}go-game-msc...
[info] Resolving org.fusesource.jansi#jansi;1.4 ...
[info] Done updating.
[info] Compiling 25 Scala sources to /home/kevin/gitrepos/go-game-msc/target/scala-2.9.2/classes...
[error] error while loading package, class file needed by package is missing.
[error] reference value <init>$default$2 of object deprecated refers to nonexisting symbol.
[error] error while loading Assertions, class file needed by Assertions is missing.
[error] reference value <init>$default$2 of object deprecated refers to nonexisting symbol.
[error] two errors found
[error] (compile:compile) Compilation failed
[error] Total time: 21 s, completed 09-Mar-2014 12:07:14
I've tried matching up the dependencies with the jar files I am using:
hamcrest-all-1.3.jar
logback-classic-1.0.13.jar
scalaedit-assembly-0.3.7(1).jar
scalatest_2.9.0-1.9.1.jar
slf4j-simple-1.6.4.jar
hamcrest-core-1.3.jar
logback-core-1.0.13.jar
scalaedit-assembly-0.3.7.jar
scct_2.9.2-0.2-SNAPSHOT.jar
junit-4.11.jar
miglayout-4.0.jar
scalariform.jar
slf4j-api-1.7.5.jar
Please advise.
Never mix scala binary versions.
Use always %% (instead of % and _2.x.x):
libraryDependencies +="org.scalatest" %% "scalatest" % "1.9.2"
libraryDependencies +="com.github.scct" %% "scct" % "0.2-SNAPSHOT"