I've been trying to use the apache commons.
But, it fails and get the following errors.
I've no idea how to fix it. Maybe, need to add something into build.sbt?
$ sbt
> clean
> compile
[error] hw.scala:3: object compress is not a member of package org.apache.commons
[error] import org.apache.commons.compress.utils.IOUtils
[error] ^
[error] hw.scala:24: not found: value IOUtils
[error] val bytes = IOUtils.toByteArray(new FileInputStream(imgFile))
[error] ^
[error] two errors found
[error] (compile:compileIncremental) Compilation failed
hw.scala
import org.apache.commons.codec.binary.{ Base64 => ApacheBase64 }
import org.apache.commons.compress.utils.IOUtils
...
build.sbt
name := "hello"
version := "1.0"
scalaVersion := "2.11.8"
libraryDependencies ++= Seq(
"commons-codec" % "commons-codec" % "1.10",
"commons-io" % "commons-io" % "2.4"
)
Add this in your build.sbt:
// https://mvnrepository.com/artifact/org.apache.commons/commons-compress
libraryDependencies += "org.apache.commons" % "commons-compress" % "1.14"
To find this by yourself in the future, search on https://mvnrepository.com/
Related
I've scala grpc protobuf project with below build.sbt.
ThisBuild / version := "0.1.0-SNAPSHOT"
val openRtbCoreVersion = "1.5.5"
val googleCommonProtosVersion = "1.12.0"
val commonSettings: Seq[Def.Setting[_]] = Seq[Def.Setting[_]](
scalaVersion := "2.12.14",
organization := "com.td",
)
def scalaGrpcSettings: Seq[Def.Setting[_]] = Seq[Def.Setting[_]](
libraryDependencies += "com.thesamet.scalapb" %% "scalapb-runtime-grpc" % scalapb.compiler.Version.scalapbVersion,
libraryDependencies += "com.google.api.grpc" % "proto-google-common-protos" % googleCommonProtosVersion % "protobuf",
PB.targets in Compile := Seq(
PB.gens.java -> (sourceManaged in Compile).value,
scalapb.gen(javaConversions = true) -> (sourceManaged in Compile).value
),
PB.includePaths in Compile := Seq(
target.value / "protobuf_external",
),
PB.protoSources in Compile := Seq(
PB.externalIncludePath.value / "google" / "api",
target.value / "protobuf_external",
new File("definitions/common")
),
PB.additionalDependencies := Nil
)
lazy val root = (project in file("."))
.settings(
name := "proto-path-error"
).settings(scalaGrpcSettings)
project/plugins.sbt file is -
addSbtPlugin("com.thesamet" % "sbt-protoc" % "0.99.20")
libraryDependencies += "com.thesamet.scalapb" %% "compilerplugin" % "0.10.0"
project/build.properties is -
sbt.version = 1.3.13
The src/main/protobuf/definitions/common/common.proto -
syntax = "proto2";
option java_outer_classname = "CommonUtils";
package com.td.protobuf;
message CMap {
map<int32, Assets> cs = 1;
}
message Assets {
repeated int32 assets = 1;
}
The whole project is on github here
This project is working fine. I want to update the sbt-protoc plugin in project/plugins.sbt to 1.0.0.
After I upgrade the sbt-protoc version to 1.0.0 I get below error when I do sbt clean compile -
/home/rajkumar/Coding/scala/proto-path-error/definitions/common/common.proto: File does not reside within any path specified using --proto_path (or -I). You must specify a --proto_path which encompasses this file. Note that the proto_path must be an exact prefix of the .proto file names -- protoc is too dumb to figure out when two paths (e.g. absolute and relative) are equivalent (it's harder than you think).
[error] java.lang.RuntimeException: protoc returned exit code: 1
[error] at scala.sys.package$.error(package.scala:30)
[error] at sbtprotoc.ProtocPlugin$.compile(ProtocPlugin.scala:438)
[error] at sbtprotoc.ProtocPlugin$.compileProto$1(ProtocPlugin.scala:537)
[error] at sbtprotoc.ProtocPlugin$.$anonfun$sourceGeneratorTask$12(ProtocPlugin.scala:544)
[error] at sbt.util.FileFunction$.$anonfun$cached$1(FileFunction.scala:73)
[error] at sbt.util.FileFunction$.$anonfun$cached$4(FileFunction.scala:146)
[error] at sbt.util.Difference.apply(Tracked.scala:323)
[error] at sbt.util.Difference.apply(Tracked.scala:303)
[error] at sbt.util.FileFunction$.$anonfun$cached$3(FileFunction.scala:142)
[error] at sbt.util.Difference.apply(Tracked.scala:323)
[error] at sbt.util.Difference.apply(Tracked.scala:298)
[error] at sbt.util.FileFunction$.$anonfun$cached$2(FileFunction.scala:141)
[error] at sbtprotoc.ProtocPlugin$.$anonfun$sourceGeneratorTask$4(ProtocPlugin.scala:549)
[error] at scala.Function1.$anonfun$compose$1(Function1.scala:49)
[error] at sbt.internal.util.$tilde$greater.$anonfun$$u2219$1(TypeFunctions.scala:62)
[error] at sbt.std.Transform$$anon$4.work(Transform.scala:67)
[error] at sbt.Execute.$anonfun$submit$2(Execute.scala:281)
[error] at sbt.internal.util.ErrorHandling$.wideConvert(ErrorHandling.scala:19)
[error] at sbt.Execute.work(Execute.scala:290)
[error] at sbt.Execute.$anonfun$submit$1(Execute.scala:281)
[error] at sbt.ConcurrentRestrictions$$anon$4.$anonfun$submitValid$1(ConcurrentRestrictions.scala:178)
[error] at sbt.CompletionService$$anon$2.call(CompletionService.scala:37)
[error] at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
[error] at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
[error] at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
[error] at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
[error] at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
[error] at java.base/java.lang.Thread.run(Thread.java:829)
[error] (Compile / protocGenerate) protoc returned exit code: 1
[error] Total time: 2 s, completed Sep 9, 2022, 7:15:30 PM
what do I miss in the plugin upgrade. Any idea how can I resolve the error?
There are multiple things to fix in build.sbt to update for sbt-protoc 1.0.x. Perhaps it was relying on old behaviors that were fixed. Let's start first with the error that protoc is reporting:
File does not reside within any path specified using --proto_path (or -I).
You must specify a --proto_path which encompasses this file. Note that the
proto_path must be an exact prefix of the .proto file names -- protoc is too
dumb to figure out when two paths (e.g. absolute and relative) are
equivalent (it's harder than you think).
In other words, every file we generate the sources for, need to be under "proto_path". ScalaPB takes cares of setting proto_path for you automatically, unless you somehow circumvent it by manually resetting it with := like in here:
PB.includePaths in Compile := Seq(
target.value / "protobuf_external",
),
And since protobuf_external is in the proto_path by default (also in older version), those lines are unnecessary and should be removed.
It appears that you want to unpack the protos that are in proto-google-common-protos. There's a shortcut for that: use protobuf-src instead of protobuf as the scope:
libraryDependencies += "com.google.api.grpc" % "proto-google-common-protos" % googleCommonProtosVersion % "protobuf-src",
In summary, your config should be:
def scalaGrpcSettings: Seq[Def.Setting[_]] = Seq[Def.Setting[_]](
libraryDependencies += "com.thesamet.scalapb" %% "scalapb-runtime-grpc" % scalapb.compiler.Version.scalapbVersion,
libraryDependencies += "com.google.api.grpc" % "proto-google-common-protos" % googleCommonProtosVersion % "protobuf-src",
PB.targets in Compile := Seq(
PB.gens.java -> (sourceManaged in Compile).value,
scalapb.gen(javaConversions = true) -> (sourceManaged in Compile).value
),
PB.protoSources in Compile ++= Seq(
new File("definitions/common")
)
)
I have added following dependencies to built.sbt, after running the sbt run in terminal, I got the bellow error:
$ sbt run
[info] welcome to sbt 1.5.5 (Private Build Java 1.8.0_292)
[info] loading global plugins from /home/hayat/.sbt/1.0/plugins
[info] loading project definition from /home/hayat/myproject/project
[info] loading settings for project root from build.sbt ...
[info] set current project to scala3-simple (in build file:/home/hayat/myproject/)
[info] Updating
[info] Resolved dependencies
[warn]
[warn] Note: Unresolved dependencies path:
[error] sbt.librarymanagement.ResolveException: Error downloading org.apache.spark:spark-streaming:3.1.2
[error] Not found
[error] Not found
[error] not found: /home/hayat/.ivy2/localorg.apache.spark/spark-streaming/3.1.2/ivys/ivy.xml
[error] not found: https://repo1.maven.org/maven2/org/apache/spark/spark-streaming/3.1.2/spark-streaming-3.1.2.pom
[error] at lmcoursier.CoursierDependencyResolution.unresolvedWarningOrThrow(CoursierDependencyResolution.scala:258)
[error] at lmcoursier.CoursierDependencyResolution.$anonfun$update$38(CoursierDependencyResolution.scala:227)
[error] at scala.util.Either$LeftProjection.map(Either.scala:573)
[error] at lmcoursier.CoursierDependencyResolution.update(CoursierDependencyResolution.scala:227)
[error] at sbt.librarymanagement.DependencyResolution.update(DependencyResolution.scala:60)
[error] at sbt.internal.LibraryManagement$.resolve$1(LibraryManagement.scala:59)
[error] at sbt.internal.LibraryManagement$.$anonfun$cachedUpdate$12(LibraryManagement.scala:133)
[error] at sbt.util.Tracked$.$anonfun$lastOutput$1(Tracked.scala:73)
[error] at sbt.internal.LibraryManagement$.$anonfun$cachedUpdate$20(LibraryManagement.scala:146)
[error] at scala.util.control.Exception$Catch.apply(Exception.scala:228)
[error] at sbt.internal.LibraryManagement$.$anonfun$cachedUpdate$11(LibraryManagement.scala:146)
[error] at sbt.internal.LibraryManagement$.$anonfun$cachedUpdate$11$adapted(LibraryManagement.scala:127)
[error] at sbt.util.Tracked$.$anonfun$inputChangedW$1(Tracked.scala:219)
[error] at sbt.internal.LibraryManagement$.cachedUpdate(LibraryManagement.scala:160)
[error] at sbt.Classpaths$.$anonfun$updateTask0$1(Defaults.scala:3678)
[error] at scala.Function1.$anonfun$compose$1(Function1.scala:49)
[error] at sbt.internal.util.$tilde$greater.$anonfun$$u2219$1(TypeFunctions.scala:62)
[error] at sbt.std.Transform$$anon$4.work(Transform.scala:68)
[error] at sbt.Execute.$anonfun$submit$2(Execute.scala:282)
[error] at sbt.internal.util.ErrorHandling$.wideConvert(ErrorHandling.scala:23)
[error] at sbt.Execute.work(Execute.scala:291)
[error] at sbt.Execute.$anonfun$submit$1(Execute.scala:282)
[error] at sbt.ConcurrentRestrictions$$anon$4.$anonfun$submitValid$1(ConcurrentRestrictions.scala:265)
[error] at sbt.CompletionService$$anon$2.call(CompletionService.scala:64)
[error] at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[error] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[error] at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[error] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[error] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[error] at java.lang.Thread.run(Thread.java:748)
[error] (update) sbt.librarymanagement.ResolveException: Error downloading org.apache.spark:spark-streaming:3.1.2
[error] Not found
[error] Not found
[error] not found: /home/hayat/.ivy2/localorg.apache.spark/spark-streaming/3.1.2/ivys/ivy.xml
[error] not found: https://repo1.maven.org/maven2/org/apache/spark/spark-streaming/3.1.2/spark-streaming-3.1.2.pom
[error] Total time: 7 s, completed Sep 16, 2021 11:21:30 AM
Here is built.sbt:
val scala3Version = "3.0.2"
lazy val root = project
.in(file("."))
.settings(
name := "scala3-simple",
version := "0.1.0",
scalaVersion := scala3Version,
libraryDependencies += "com.novocode" % "junit-interface" % "0.11" % "test",
libraryDependencies += "org.apache.spark" % "spark-streaming" % "3.1.2",
libraryDependencies += "org.apache.spark" % "spark-core" % "3.1.2"
)
Scala version: 3.0.2
Sbt version: 1.5.5
Libraries spark-streaming and spark-core don't exist, it is spark-streaming_2.12 and spark-core_2.12, where 2.12 is the Scala version. Currently there are no spark-streaming_3.0 and spark-core_3.0 libraries.
So to solve your issue, you need to:
downgrade your version of scala from 3.0.2 to 2.12.x (latest current version, 2.12.15) as there is no version of Spark for Scala 3
use spark-streaming_2.12 library instead of spark-streaming
use spark-core_2.12 library instead of spark-core
To use _2.12 version of libraries, you can either add _2.12 to your library name:
libraryDependencies += "org.apache.spark" % "spark-streaming_2.12" % "3.1.2",
libraryDependencies += "org.apache.spark" % "spark-core_2.12" % "3.1.2"
or, better, use %% between group and library name to automatically add scala version to library name:
libraryDependencies += "org.apache.spark" %% "spark-streaming" % "3.1.2",
libraryDependencies += "org.apache.spark" %% "spark-core" % "3.1.2"
So your build.sbt should become:
val scala2Version = "2.12.15"
lazy val root = project
.in(file("."))
.settings(
name := "scala2-simple",
version := "0.1.0",
scalaVersion := scala2Version,
libraryDependencies += "com.novocode" % "junit-interface" % "0.11" % "test",
libraryDependencies += "org.apache.spark" %% "spark-streaming" % "3.1.2",
libraryDependencies += "org.apache.spark" %% "spark-core" % "3.1.2"
)
I'm a beginner, I'm trying to use sbt to import phoenix libraries for reading hbase tables in spark, but my build.sbt keeps giving me errors.
Error while importing sbt project:
[error] stack trace is suppressed; run 'last update' for the full output
[error] stack trace is suppressed; run 'last ssExtractDependencies' for the full output
[error] (update) sbt.librarymanagement.ResolveException: Error downloading org.apache.hbase:hbase-common:${cdh.hbase.version}
[error] Not found
[error] Not found
[error] not found: /Users/johnny/.ivy2/local/org.apache.hbase/hbase-common/${cdh.hbase.version}/ivys/ivy.xml
[error] not found: https://repo1.maven.org/maven2/org/apache/hbase/hbase-common/${cdh.hbase.version}/hbase-common-${cdh.hbase.version}.pom
[error] Error downloading org.apache.hbase:hbase-hadoop-compat:${cdh.hbase.version}
[error] Not found
[error] Not found
[error] not found: /Users/johnny/.ivy2/local/org.apache.hbase/hbase-hadoop-compat/${cdh.hbase.version}/ivys/ivy.xml
[error] not found: https://repo1.maven.org/maven2/org/apache/hbase/hbase-hadoop-compat/${cdh.hbase.version}/hbase-hadoop-compat-${cdh.hbase.version}.pom
[error] not found: https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-common/${cdh.hadoop.version}/hadoop-common-${cdh.hadoop.version}.pom
[error] Error downloading org.apache.hbase:hbase-hadoop2-compat:${cdh.hbase.version}
[error] Not found
[error] Not found
[error] not found: /Users/johnny/.ivy2/local/org.apache.hbase/hbase-hadoop2-compat/${cdh.hbase.version}/ivys/ivy.xml
[error] not found: https://repo1.maven.org/maven2/org/apache/hbase/hbase-hadoop2-compat/${cdh.hbase.version}/hbase-hadoop2-compat-${cdh.hbase.version}.pom
[error] Error downloading org.apache.hbase:hbase-annotations:${cdh.hbase.version}
[error] Not found
[error] Not found
[error] not found: /Users/johnny/.ivy2/local/org.apache.hbase/hbase-annotations/${cdh.hbase.version}/ivys/ivy.xml
[error] not found: https://repo1.maven.org/maven2/org/apache/hbase/hbase-annotations/${cdh.hbase.version}/hbase-annotations-${cdh.hbase.version}.pom
[error] Error downloading org.apache.hbase:hbase-protocol:${cdh.hbase.version}
[error] Not found
[error] Not found
[error] not found: /Users/johnny/.ivy2/local/org.apache.hbase/hbase-protocol/${cdh.hbase.version}/ivys/ivy.xml
[error] not found: https://repo1.maven.org/maven2/org/apache/hbase/hbase-protocol/${cdh.hbase.version}/hbase-protocol-${cdh.hbase.version}.pom
[error] Error downloading org.apache.hbase:hbase-client:${cdh.hbase.version}
[error] Not found
[error] Not found
[error] not found: /Users/johnny/.ivy2/local/org.apache.hbase/hbase-client/${cdh.hbase.version}/ivys/ivy.xml
[error] not found: https://repo1.maven.org/maven2/org/apache/hbase/hbase-client/${cdh.hbase.version}/hbase-client-${cdh.hbase.version}.pom
[error] Error downloading org.apache.hbase:hbase-server:${cdh.hbase.version}
[error] Not found
[error] Not found
[error] not found: /Users/johnny/.ivy2/local/org.apache.hbase/hbase-server/${cdh.hbase.version}/ivys/ivy.xml
[error] not found: https://repo1.maven.org/maven2/org/apache/hbase/hbase-server/${cdh.hbase.version}/hbase-server-${cdh.hbase.version}.pom
[error] Error downloading com.cloudera.cdh:cdh-root:5.11.2
[error] Not found
[error] Not found
[error] not found: /Users/johnny/.ivy2/local/com.cloudera.cdh/cdh-root/5.11.2/ivys/ivy.xml
[error] not found: https://repo1.maven.org/maven2/com/cloudera/cdh/cdh-root/5.11.2/cdh-root-5.11.2.pom
[error] Total time: 3 s, completed Sep 27, 2019, 4:54:09 PM
[info] shutting down sbt server)
My build.sbt is:
name := "SparkHbase"
version := "0.1"
scalaVersion := "2.11.12"
libraryDependencies ++= Seq(
"org.apache.spark" %% "spark-core" % "2.2.0" % "provided"
,"org.apache.spark" %% "spark-sql" % "2.2.0" % "provided"
,"org.apache.spark" %% "spark-hive" % "2.2.0" % "provided"
,"org.apache.phoenix" % "phoenix-spark" % "4.13.2-cdh5.11.2"
)
I even included this: resolvers += "ClouderaRepo" at "https://repository.cloudera.com/content/repositories/releases"
But still had the errors. Please, what I'm I doing wrong?
The problem is that you are trying to use a very old version of phoenix-spark. If you have HBase 1.3, you can use the version 4.14.3-HBase-1.3, see this build.sbt:
name := "SparkHbase"
version := "0.1"
scalaVersion := "2.11.12"
resolvers += "Cloudera" at "https://repository.cloudera.com/content/repositories/releases/"
resolvers += "Cloudera_Artifactory" at "https://repository.cloudera.com/artifactory/cloudera-repos/"
resolvers += Resolver.sonatypeRepo("releases")
libraryDependencies ++= Seq(
"org.apache.spark" %% "spark-core" % "2.2.0" % "provided"
,"org.apache.spark" %% "spark-sql" % "2.2.0" % "provided"
,"org.apache.spark" %% "spark-hive" % "2.2.0" % "provided"
,"org.apache.phoenix" % "phoenix-spark" % "4.14.3-HBase-1.3"
)
I am attempting to use sbt assembly on a spark project. sbt compile and package work but when I attempt sbt assembly I get the following error:
object spark is not a member of package org.apache
I have included the spark core and spark sql libraries and have sbt-assembly in my plugins file. Why is assembly producing these errors?
build.sbt:
name := "redis-record-loader"
scalaVersion := "2.11.8"
val sparkVersion = "2.3.1"
val scalatestVersion = "3.0.3"
val scalatest = "org.scalatest" %% "scalatest" % scalatestVersion
libraryDependencies ++=
Seq(
"com.amazonaws" % "aws-java-sdk-s3" % "1.11.347",
"com.typesafe" % "config" % "1.3.1",
"net.debasishg" %% "redisclient" % "3.0",
"org.slf4j" % "slf4j-log4j12" % "1.7.12",
"org.apache.commons" % "commons-lang3" % "3.0" % "test,it",
"org.apache.hadoop" % "hadoop-aws" % "2.8.1" % Provided,
"org.apache.spark" %% "spark-core" % sparkVersion % Provided,
"org.apache.spark" %% "spark-sql" % sparkVersion % Provided,
"org.mockito" % "mockito-core" % "2.21.0" % Test,
scalatest
)
val integrationTestsKey = "it"
val integrationTestLibs = scalatest % integrationTestsKey
lazy val IntegrationTestConfig = config(integrationTestsKey) extend Test
lazy val root = project.in(file("."))
.configs(IntegrationTestConfig)
.settings(inConfig(IntegrationTestConfig)(Defaults.testSettings): _*)
.settings(libraryDependencies ++= Seq(integrationTestLibs))
test in assembly := Seq(
(test in Test).value,
(test in IntegrationTestConfig).value
)
assemblyMergeStrategy in assembly := {
case PathList("META-INF", xs # _*) => MergeStrategy.discard
case x => MergeStrategy.first
}
plugins.sbt:
logLevel := Level.Warn
addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.14.6")
full error message:
/com/elsevier/bos/RedisRecordLoaderIntegrationSpec.scala:11: object spark is not a member of package org.apache
[error] import org.apache.spark.sql.{DataFrame, SaveMode, SparkSession}
[error] ^
[error] /Users/jones8/Work/redis-record-loader/src/it/scala/com/elsevier/bos/RedisRecordLoaderIntegrationSpec.scala:26: not found: type SparkSession
[error] implicit val spark: SparkSession = SparkSession.builder
[error] ^
[error] /Users/jones8/Work/redis-record-loader/src/it/scala/com/elsevier/bos/RedisRecordLoaderIntegrationSpec.scala:26: not found: value SparkSession
[error] implicit val spark: SparkSession = SparkSession.builder
[error] ^
[error] /Users/jones8/Work/redis-record-loader/src/it/scala/com/elsevier/bos/RedisRecordLoaderIntegrationSpec.scala:51: not found: type DataFrame
[error] val testDataframe0: DataFrame = testData0.toDF()
[error] ^
[error] /Users/jones8/Work/redis-record-loader/src/it/scala/com/elsevier/bos/RedisRecordLoaderIntegrationSpec.scala:51: value toDF is not a member of Seq[(String, String)]
[error] val testDataframe0: DataFrame = testData0.toDF()
[error] ^
[error] /Users/jones8/Work/redis-record-loader/src/it/scala/com/elsevier/bos/RedisRecordLoaderIntegrationSpec.scala:52: not found: type DataFrame
[error] val testDataframe1: DataFrame = testData1.toDF()
[error] ^
[error] /Users/jones8/Work/redis-record-loader/src/it/scala/com/elsevier/bos/RedisRecordLoaderIntegrationSpec.scala:52: value toDF is not a member of Seq[(String, String)]
[error] val testDataframe1: DataFrame = testData1.toDF()
[error] ^
[error] missing or invalid dependency detected while loading class file 'RedisRecordLoader.class'.
[error] Could not access term spark in package org.apache,
[error] because it (or its dependencies) are missing. Check your build definition for
[error] missing or conflicting dependencies. (Re-run with `-Ylog-classpath` to see the problematic classpath.)
[error] A full rebuild may help if 'RedisRecordLoader.class' was compiled against an incompatible version of org.apache.
[error] missing or invalid dependency detected while loading class file 'RedisRecordLoader.class'.
[error] Could not access type SparkSession in value org.apache.sql,
[error] because it (or its dependencies) are missing. Check your build definition for
[error] missing or conflicting dependencies. (Re-run with `-Ylog-classpath` to see the problematic classpath.)
[error] A full rebuild may help if 'RedisRecordLoader.class' was compiled against an incompatible version of org.apache.sql.
[error] 9 errors found
Cant comment on that I can say "I doubt the AWS SDK & hadoop-aws versions are going to work". You need the exact version of hadoop-aws to match the hadoop-common JAR on your CP, (it's all one project which releases in sync, after all), and the aws SDK Version built against was 1.10. The AWS SDK has a habit of (a) breaking APIs on every point release (b) aggressively pushing new versions of jackson down, even when they are incompatible and (c) causing regressions in the hadoop-aws code.
If you really want to work with S3A, best to go for hadoop-2.9, which pulls in a shaded 1.11.x version
this is my build.sbt file:
name := "words"
version := "1.0"
scalaVersion := "2.10.4"
libraryDependencies ++= Seq(
"org.apache.spark" %% "spark-core" % "1.3.0",
"org.apache.spark" %% "spark-sql" % "1.3.0"
)
sbt.version=0.13.8-RC1
When I compile the program, I have the following error:
[error] D:\projects\bd\words\src\main\scala\test.scala:8:
type SqlContext is not a member of package org.apache.spark.sql
[error] val sqlContext = new org.apache.spark.sql.SqlContext(sc)
[error] ^
[error] one error found
[error] (compile:compileIncremental) Compilation failed
It's SQLContext not SqlContext.