Scala: object profile is not a member of package com.amazonaws.auth - scala

I am having a build problem. Here is my sbt file:
name := "SparkPi"
version := "0.2.15"
scalaVersion := "2.11.8"
// https://mvnrepository.com/artifact/org.apache.spark/spark-core_2.10
libraryDependencies += "org.apache.spark" % "spark-core_2.10" % "2.0.1"
// old:
//libraryDependencies += "org.apache.spark" %% "spark-core" % "2.0.1"
// https://mvnrepository.com/artifact/com.amazonaws/aws-java-sdk
libraryDependencies += "com.amazonaws" % "aws-java-sdk" % "1.0.002"
scalacOptions ++= Seq("-feature")
Here is the full error message I am seeing:
[info] Set current project to SparkPi (in build file:/Users/xxx/prog/yyy/)
[info] Updating {file:/Users/xxx/prog/yyy/}yyy...
[info] Resolving jline#jline;2.12.1 ...
[info] Done updating.
[info] Compiling 2 Scala sources to /Users/xxx/prog/yyy/target/scala-2.11/classes...
[error] /Users/xxx/prog/yyy/src/main/scala/PiSpark.scala:6: object profile is not a member of package com.amazonaws.auth
[error] import com.amazonaws.auth.profile._
[error] ^
[error] /Users/xxx/prog/yyy/src/main/scala/PiSpark.scala:87: not found: type ProfileCredentialsProvider
[error] val creds = new ProfileCredentialsProvider(profile).getCredentials()
[error] ^
[error] two errors found
[error] (compile:compileIncremental) Compilation failed
[error] Total time: 14 s, completed Nov 3, 2016 1:43:34 PM
And here are the imports I am trying to use:
import com.amazonaws.services.s3._
import com.amazonaws.auth.profile._
How do I import com.amazonaws.auth.profile.ProfileCredentialsProvider in Scala?
EDIT
Changed sbt file so spark core version corresponds to Scala version, new contents:
name := "SparkPi"
version := "0.2.15"
scalaVersion := "2.11.8"
// https://mvnrepository.com/artifact/org.apache.spark/spark-core_2.11
libraryDependencies += "org.apache.spark" % "spark-core_2.11" % "2.0.1"
// old:
//libraryDependencies += "org.apache.spark" %% "spark-core" % "2.0.1"
// https://mvnrepository.com/artifact/com.amazonaws/aws-java-sdk
libraryDependencies += "com.amazonaws" % "aws-java-sdk" % "1.0.002"
scalacOptions ++= Seq("-feature")

You are using scalaVersion := "2.11.8" but library dependency has underscore 2.10 spark-core_2.10 which is bad.
libraryDependencies += "org.apache.spark" % "spark-core_2.10" % "2.0.1"
^
change 2.10 to 2.11
libraryDependencies += "org.apache.spark" % "spark-core_2.11" % "2.0.1"
`

Related

What dependencies to use in build.sbt for neo4j spark connector?

I was running scala code in spark-shell using this:
spark-shell --conf spark.neo4j.bolt.password=TestNeo4j --packages neo4j-contrib:neo4j-spark-connector:2.0.0-M2,graphframes:graphframes:0.2.0-spark2.0-s_2.11 -i neo4jsparkCluster.scala
This would run execute just fine on the one spark instance, now I want to clusterize it.
I have a build.sbt file as follows:
name := "Simple Project"
version := "1.0"
scalaVersion := "2.11.8"
libraryDependencies ++= Seq(
"org.apache.spark" %% "spark-core" % "2.2.0",
"org.apache.spark" %% "spark-sql" % "2.2.0"
)
So I am not sure what I need in the libraryDependencies given the libs I am trying to load, I guess the syntax for it?
The errors I get with the sbt package are:
[info] Compiling 2 Scala sources to /Users/shane.thomas/SparkCourse/spark-sbt-builds/target/scala-2.11/classes...
[error] /Users/shane.thomas/SparkCourse/spark-sbt-builds/neo4jSparkCluster.scala:1: object neo4j is not a member of package org
[error] import org.neo4j.spark._
[error] ^
[error] /Users/shane.thomas/SparkCourse/spark-sbt-builds/neo4jSparkCluster.scala:5: object streaming is not a member of package org.apache.spark
[error] import org.apache.spark.streaming._
[error] ^
[error] /Users/shane.thomas/SparkCourse/spark-sbt-builds/neo4jSparkCluster.scala:6: object streaming is not a member of package org.apache.spark
[error] import org.apache.spark.streaming.StreamingContext._
[error] ^
[error] /Users/shane.thomas/SparkCourse/spark-sbt-builds/neo4jSparkCluster.scala:539: not found: value Neo4j
[error] val neo = Neo4j(sc)
[error] ^
[error] four errors found
[error] (compile:compileIncremental) Compilation failed
[error] Total time: 5 s, completed Dec 7, 2017 2:45:00 PM
Try adding the following
resolvers += "Spark Packages Repo" at "http://dl.bintray.com/spark-packages/maven"
libraryDependencies += "neo4j-contrib" % "neo4j-spark-connector" % "2.1.0-M4"
Taken from https://github.com/neo4j-contrib/neo4j-spark-connector under the SBT section
these work fine to me
scalaVersion := "2.12.13"
val spark_version:String = "3.1.0"
resolvers ++= Seq(
"Spark Packages Repo" at "https://dl.bintray.com/spark-packages/maven"
)
libraryDependencies ++= Seq(
"org.apache.spark" %% "spark-core" % spark_version,
"org.apache.spark" %% "spark-sql" % spark_version,
"org.neo4j" % "neo4j-kernel" % "4.2.3",
"neo4j-contrib" % "neo4j-spark-connector" % "2.4.5-M2",
"graphframes" % "graphframes" % "0.8.1-spark3.0-s_2.12",
)

Build.sbt breaks when adding GraphFrames build with scala 2.11

I'm trying to add GraphFrames to my scala spark application, and this was going fine when I added the one based on 2.10. However, as soon as I tried to build it with GraphFrames build with scala 2.11, it breaks.
The problem would be that there are conflicting versions of scala used (2.10 and 2.11). I'm getting the following error:
[error] Modules were resolved with conflicting cross-version suffixes in {file:/E:/Documents/School/LSDE/hadoopcryptoledger/examples/scala-spark-graphx-bitcointransaction/}root:
[error] org.apache.spark:spark-launcher _2.10, _2.11
[error] org.json4s:json4s-ast _2.10, _2.11
[error] org.apache.spark:spark-network-shuffle _2.10, _2.11
[error] com.twitter:chill _2.10, _2.11
[error] org.json4s:json4s-jackson _2.10, _2.11
[error] com.fasterxml.jackson.module:jackson-module-scala _2.10, _2.11
[error] org.json4s:json4s-core _2.10, _2.11
[error] org.apache.spark:spark-unsafe _2.10, _2.11
[error] org.apache.spark:spark-core _2.10, _2.11
[error] org.apache.spark:spark-network-common _2.10, _2.11
However, I can't troubleshoot what causes this.. This is my full build.sbt:
import sbt._
import Keys._
import scala._
lazy val root = (project in file("."))
.settings(
name := "example-hcl-spark-scala-graphx-bitcointransaction",
version := "0.1"
)
.configs( IntegrationTest )
.settings( Defaults.itSettings : _*)
scalacOptions += "-target:jvm-1.7"
crossScalaVersions := Seq("2.11.8")
resolvers += Resolver.mavenLocal
fork := true
jacoco.settings
itJacoco.settings
assemblyJarName in assembly := "example-hcl-spark-scala-graphx-bitcointransaction.jar"
libraryDependencies += "com.github.zuinnote" % "hadoopcryptoledger-fileformat" % "1.0.7" % "compile"
libraryDependencies += "org.apache.spark" %% "spark-core" % "1.5.0" % "provided"
libraryDependencies += "org.apache.spark" %% "spark-graphx" % "1.5.0" % "provided"
libraryDependencies += "org.apache.hadoop" % "hadoop-client" % "2.7.0" % "provided"
libraryDependencies += "javax.servlet" % "javax.servlet-api" % "3.0.1" % "it"
libraryDependencies += "org.apache.hadoop" % "hadoop-common" % "2.7.0" % "it" classifier "" classifier "tests"
libraryDependencies += "org.apache.hadoop" % "hadoop-hdfs" % "2.7.0" % "it" classifier "" classifier "tests"
libraryDependencies += "org.apache.hadoop" % "hadoop-minicluster" % "2.7.0" % "it"
libraryDependencies += "org.apache.spark" % "spark-sql_2.11" % "2.2.0" % "provided"
libraryDependencies += "org.scalatest" %% "scalatest" % "3.0.1" % "test,it"
libraryDependencies += "graphframes" % "graphframes" % "0.5.0-spark2.1-s_2.11"
Can anyone pinpoint which dependency is based on scala 2.10 causing the build to fail?
I found out what the problem was. Apparently, if you use:
libraryDependencies += "org.apache.spark" %% "spark-core" % "1.5.0" % "provided"
It uses the 2.10 version by default. It all worked once I changed the dependencies of spark core and spark graphx to:
libraryDependencies += "org.apache.spark" % "spark-core_2.11" % "2.2.0"
libraryDependencies += "org.apache.spark" % "spark-graphx_2.11" % "2.2.0" % "provided"

How to force Scala to use a different library version?

After adding
libraryDependencies += "org.scalatest" %% "scalatest" % "3.0.0" % "test"
to build.sbt, and refreshed the project, I got this msg.
SBT project import
[warn] Multiple dependencies with the same organization/name but
different versions. To avoid conflict, pick one version:
[warn] * org.scala-lang.modules:scala-xml_2.11:(1.0.5, 1.0.4)
Changing the above to
libraryDependencies ++= Seq(
"org.scalatest" %% "scalatest" % "3.0.0" % "test"
exclude("org.scala-lang", "scala-reflect")
exclude("org.scala-lang.modules", "scala-xml_2.11")
)
solves the issue. However, instead of excluding scala-xml_2.11 version 1.0.5 from scalatest, I would like to force the scala compiler to use scala-xml_2.11 version 1.0.5 instead of version 1.0.4. (I researched the versions at https://mvnrepository.com.) Thus I tried substituting scalaVersion := "2.11.8" for
libraryDependencies ++= Seq(
"org.scala-lang" % "scala-reflect" % "2.11.8",
"org.scala-lang.modules" % "scala-xml_2.11" % "1.0.5"
)
This however results in
SBT project import
[warn] Binary version (2.11) for dependency
org.scala-lang#scala-reflect;2.11.8
[warn] in default#myproject$sources_javadoc_2.10;1.0 differs
from Scala binary version in project (2.10).
[warn] Binary version (2.11) for dependency
org.scala-lang#scala-library;2.11.8
[warn] in default#myproject$sources_javadoc_2.10;1.0 differs
from Scala binary version in project (2.10).
[warn] Multiple dependencies with the same organization/name but
different versions. To avoid conflict, pick one version:
[warn] * org.scala-lang:scala-library:(2.11.8, 2.10.4)
[warn] * org.scala-lang:scala-reflect:(2.11.8, 2.10.4)
[warn] [FAILED ]
com.artima.supersafe#supersafe_2.10.4;1.1.0!supersafe_2.10.4.jar(src):
(0ms)
[warn] ==== local: tried
[warn]
/home/user/.ivy2/local/com.artima.supersafe/supersafe_2.10.4/1.1.0/srcs/supersafe_2.10.4-sources.jar
[warn] ==== activator-local: tried [warn] /Development/Activator/activator-dist-1.3.10/repository/com.artima.supersafe/supersafe_2.1...
(show balloon)
What am I supposed to do?
Edit: What else I tried and did not work:
scalaVersion := "2.11.8"
libraryDependencies ++= Seq(
"org.scala-lang" % "scala-reflect" % scalaVersion.value,
"org.scala-lang.modules" %% "scala-xml" % "1.0.5"
)
// ScalaTest
libraryDependencies += "org.scalatest" %% "scalatest" % "3.0.0" % "test"
[warn] Multiple dependencies with the same organization/name but
different versions. To avoid conflict, pick one version: [warn] *
org.scala-lang.modules:scala-xml_2.11:(1.0.5, 1.0.4)
scalaVersion := "2.11.8"
libraryDependencies ++= Seq(
"org.scala-lang" % "scala-reflect" % scalaVersion.value,
"org.scala-lang.modules" %% "scala-xml_2.11" % "1.0.5"
)
// ScalaTest
libraryDependencies += "org.scalatest" %% "scalatest" % "3.0.0" % "test"
Error:Error while importing SBT project:...[info]
Resolving org.scala-sbt#run;0.13.8 ... [info] Resolving
org.scala-sbt#task-system;0.13.8 ... [info] Resolving
org.scala-sbt#tasks;0.13.8 ... [info] Resolving
org.scala-sbt#tracking;0.13.8 ... [info] Resolving
org.scala-sbt#cache;0.13.8 ... [info] Resolving
org.scala-sbt#testing;0.13.8 ... [info] Resolving
org.scala-sbt#test-agent;0.13.8 ... [info] Resolving
org.scala-sbt#test-interface;1.0 ... [info] Resolving
org.scala-sbt#main-settings;0.13.8 ... [info] Resolving
org.scala-sbt#apply-macro;0.13.8 ... [info] Resolving
org.scala-sbt#command;0.13.8 ... [info] Resolving
org.scala-sbt#logic;0.13.8 ... [info] Resolving
org.scala-sbt#precompiled-2_8_2;0.13.8 ... [info] Resolving
org.scala-sbt#precompiled-2_9_2;0.13.8 ... [info] Resolving
org.scala-sbt#precompiled-2_9_3;0.13.8 ... [trace] Stack trace
suppressed: run 'last *:update' for the full output. [trace] Stack
trace suppressed: run 'last :ssExtractDependencies' for the full
output. [error] (:update) sbt.ResolveException: unresolved
dependency: org.scala-lang.modules#scala-xml_2.11_2.11;1.0.5: not
found [error] (*:ssExtractDependencies) sbt.ResolveException:
unresolved dependency:
org.scala-lang.modules#scala-xml_2.11_2.11;1.0.5: not found [error]
Total time: 4 s, completed 01.10.2016 17:46:55
scalaVersion := "2.11.8"
libraryDependencies ++= Seq(
"org.scala-lang" % "scala-reflect" % scalaVersion.value,
"org.scala-lang.modules" %% "scala-xml" % "1.0.5"
)
// ScalaTest
libraryDependencies ++= Seq(
"org.scalatest" %% "scalatest" % "3.0.0" % "test"
exclude("org.scala-lang", "scala-reflect")
exclude("org.scala-lang.modules", "scala-xml")
)
[warn] Multiple dependencies with the same organization/name but
different versions. To avoid conflict, pick one version: [warn] *
org.scala-lang.modules:scala-xml_2.11:(1.0.5, 1.0.4)
My build.sbt
name := "MyProject"
version := "0.1.0"
scalaVersion := "2.11.8"
libraryDependencies ++= Seq(
"org.scala-lang" % "scala-reflect" % scalaVersion.value,
"org.scala-lang.modules" %% "scala-xml" % "1.0.5"
)
// ScalaTest
//libraryDependencies += "org.scalactic" %% "scalactic" % "3.0.0"
//libraryDependencies += "org.scalatest" %% "scalatest" % "3.0.0" % "test"
libraryDependencies ++= Seq(
"org.scalatest" %% "scalatest" % "3.0.0" % "test"
exclude("org.scala-lang", "scala-reflect")
exclude("org.scala-lang.modules", "scala-xml_2.11")
)
The way to exclude specific transitive dependency is this:
Run sbt evicted to figure out which of the project dependencies is pulling in outdated library, let's assume the problematic library is: com.typesafe.slick.
Add the following exclude (the parentheses are important):
("com.typesafe" %% "slick" % "3.1.1").exclude("org.scala-lang.modules", "scala-xml_2.11")
Add this normally as you would be listing dependencies.
This will prevent sbt from including any version of scala-xml that was coming in as a transitive dependency of Slick.
Don't substitute; you need both scalaVersion and libraryDependencies.
Though use
scalaVersion := "2.11.8"
libraryDependencies ++= Seq(
"org.scala-lang" % "scala-reflect" % scalaVersion.value,
"org.scala-lang.modules" %% "scala-xml" % "1.0.5"
)
to avoid bugs when you eventually change scalaVersion.
By removing scalaVersion you get default scalaVersion := "2.10.4" (with your version/settings of SBT, at least) but your libraryDependencies still require 2.11.

How to declare play json dependency?

My build.sbt file (sbt version is 0.13.8):
lazy val commonSettings = Seq(
version := "1.0.0",
scalaVersion := "2.11.6"
)
resolvers += "Typesafe Repo" at "http://repo.typesafe.com/typesafe/releases/"
lazy val root = (project in file(".")).
settings(commonSettings: _*).
settings(
name := "myapp",
libraryDependencies ++= Seq(
"com.typesafe.play" % "play-json" % "2.3.4",
"org.scalatest" % "scalatest_2.11" % "2.2.4" % "test",
"junit" % "junit" % "4.12" % "test"
)
)
scalacOptions ++= Seq("-unchecked", "-feature", "-deprecation")
I get this error when trying to compile my project:
[trace] Stack trace suppressed: run last *:update for the full output.
[error] (*:update) sbt.ResolveException: unresolved dependency: com.typesafe.play#play-json_2.11;2.3.4: not found
[error] Total time: 0 s, completed Apr 17, 2015 5:59:28 PM
How can I get this play-json library for my scala 2.11.6?
You need to tell sbt which scala version should use.
You can either be explicit:
"com.typesafe.play" % "play-json_2.11" % "2.3.4",
Or use %% (sbt doc) as follows to tell sbt to use scalaVersion :
"com.typesafe.play" %% "play-json" % "2.3.4",
You can see all of com.typesafe.play's play-json versions here. They don't have a 2.3.4 version; try using 2.4.0-M3 instead.
"com.typesafe.play" %% "play-json" % "2.4.0-M3"
Mind the double %% so scalaVersion is used properly to resolve the dependency.

Errors while compiling project migrated to SBT - error while loading package and Assertions

I'm migrating a Scala application that compiles and runs fine by manually
including jars in the classpath to a SBT build configuration.
My build.sbt is as follows:
name := "hello"
version := "1.0"
scalaVersion := "2.9.2"
libraryDependencies += "org.slf4j" % "slf4j-simple" % "1.6.4"
libraryDependencies += "junit" % "junit" % "4.11"
libraryDependencies += "org.scalatest" % "scalatest_2.10" % "1.9.2"
libraryDependencies += "org.hamcrest" % "hamcrest-all" % "1.3"
libraryDependencies += "ch.qos.logback" % "logback-classic" % "1.0.13"
libraryDependencies += "com.github.scct" % "scct_2.10" % "0.2.1"
libraryDependencies += "org.scala-lang" % "scala-swing" % "2.9.2"
When I compile it I get the following errors:
Loading /usr/share/sbt/bin/sbt-launch-lib.bash
[info] Set current project to hello (in build file:/home/kevin/gitrepos/go-game-msc/)
> compile
[info] Updating {file:/home/kevin/gitrepos/go-game-msc/}go-game-msc...
[info] Resolving org.fusesource.jansi#jansi;1.4 ...
[info] Done updating.
[info] Compiling 25 Scala sources to /home/kevin/gitrepos/go-game-msc/target/scala-2.9.2/classes...
[error] error while loading package, class file needed by package is missing.
[error] reference value <init>$default$2 of object deprecated refers to nonexisting symbol.
[error] error while loading Assertions, class file needed by Assertions is missing.
[error] reference value <init>$default$2 of object deprecated refers to nonexisting symbol.
[error] two errors found
[error] (compile:compile) Compilation failed
[error] Total time: 21 s, completed 09-Mar-2014 12:07:14
I've tried matching up the dependencies with the jar files I am using:
hamcrest-all-1.3.jar
logback-classic-1.0.13.jar
scalaedit-assembly-0.3.7(1).jar
scalatest_2.9.0-1.9.1.jar
slf4j-simple-1.6.4.jar
hamcrest-core-1.3.jar
logback-core-1.0.13.jar
scalaedit-assembly-0.3.7.jar
scct_2.9.2-0.2-SNAPSHOT.jar
junit-4.11.jar
miglayout-4.0.jar
scalariform.jar
slf4j-api-1.7.5.jar
Please advise.
Never mix scala binary versions.
Use always %% (instead of % and _2.x.x):
libraryDependencies +="org.scalatest" %% "scalatest" % "1.9.2"
libraryDependencies +="com.github.scct" %% "scct" % "0.2-SNAPSHOT"