I have a text file in S3 that I would like to load into an RDD with spark-shell.
I have downloaded Spark 2.3.0 for Hadoop. Naively, I would expect that I just need to set the hadoop settings and I'd be set.
val inFile = "s3a://some/path"
val accessKey = "some-access-key"
val secretKey = "some-secret-key"
sc.hadoopConfiguration.set("fs.s3a.access.key", accessKey)
sc.hadoopConfiguration.set("fs.s3a.secret.key", secretKey)
sc.textFile(inFile).count()
println(run())
Invoking the final line returns:
Failure(java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.hadoop.fs.s3a.S3AFileSystem not found)
This seems to be asking that I provide the library which includes S3AFileSystem. No problem - I download the appropriate jar and add this line to the beginning of the script.
:require C:\{path-to-jar}\hadoop-aws-3.1.0.jar
Now, running the script fails at the final line with a variety of errors similar to this:
error: error while loading Partition, class file 'C:\spark\spark-2.3.0-bin-hadoop2.7\jars\spark-core_2.11-2.3.0.jar(org/apache/spark/Partition.class)' has location not matching its contents: contains class Partition
I'm lost at this point - clearly, it had no issue defining the run method before.
I can access the Partition class myself directly, but something is happening above that prevents the code from accessing Partition.
scala> new org.apache.spark.Partition {def index = 3}
res6: org.apache.spark.Partition = $anon$1#3
Curiously, running the final line of the script yields a different error in subsequent invocations.
scala> sc.textFile(inFile).count()
java.lang.NoClassDefFoundError: org/apache/hadoop/fs/StreamCapabilities
at java.lang.ClassLoader.defineClass1(Native Method)
...
The documentation claims this is part of hadoop 3.1.0, which I'm using, but when exploring hadoop-aws-3.1.0.jar I see no trace of StreamCapabilities.
Is there a different jar I should be using? Am I trying to solve this problem incorrectly? Or, have I fallen into the XY problem trap?
Answers I tried
The official docs assume I'm running the script on a cluster. But I'm running spark-shell locally.
This other StackOverflow question is for an older problem. I'm using s3a as a result, but am encountering a different problem.
I also tried using every jar of Hadoop from 2.6 to 3.1, to no avail.
org.apache.hadoop.fs.StreamCapabilities is in hadoop-common-3.1.jar
You are probably mixing version of Hadoop JARs, which, as coved in the s3a troubleshooting docs is doomed.
Spark shell works fine with the right JARs in. But ASF Spark releases don't work with Hadoop 3.x yet, due to some outstanding issues. Stick to Hadoop 2.8.x and you'll get good S3 performance without so much pain.
I found a path that fixed the issue, but I have no idea why.
Create an SBT IntelliJ project
Include the below dependencies and overrides
Run the script (sans require statement) from sbt console
scalaVersion := "2.11.12"
libraryDependencies += "org.apache.spark" %% "spark-core" % "2.3.0"
libraryDependencies += "org.apache.hadoop" % "hadoop-aws" % "3.1.0"
libraryDependencies += "org.apache.hadoop" % "hadoop-client" % "3.1.0"
dependencyOverrides += "com.fasterxml.jackson.core" % "jackson-core" % "2.8.7"
dependencyOverrides += "com.fasterxml.jackson.core" % "jackson-databind" % "2.8.7"
dependencyOverrides += "com.fasterxml.jackson.module" % "jackson-module-scala_2.11" % "2.8.7"
The key part, naturally, is overriding the jackson dependencies.
I have a Scala project that I build with sbt. It uses the sryza/spark-timeseries library.
I am trying to run the following simple code:
val tsAirPassengers = new DenseVector(Array(
112.0,118.0,132.0,129.0,121.0,135.0,148.0,148.0,136.0,119.0,104.0,118.0,115.0,126.0,
141.0,135.0,125.0,149.0,170.0,170.0,158.0,133.0,114.0,140.0,145.0,150.0,178.0,163.0,
172.0,178.0,199.0,199.0,184.0,162.0,146.0,166.0,171.0,180.0,193.0,181.0,183.0,218.0,
230.0,242.0,209.0,191.0,172.0,194.0,196.0,196.0,236.0,235.0,229.0,243.0,264.0,272.0,
237.0,211.0,180.0,201.0,204.0,188.0,235.0,227.0,234.0,264.0,302.0,293.0,259.0,229.0,
203.0,229.0,242.0,233.0,267.0,269.0,270.0,315.0,364.0,347.0,312.0,274.0,237.0,278.0,
284.0,277.0,317.0,313.0,318.0,374.0,413.0,405.0,355.0,306.0,271.0,306.0,315.0,301.0,
356.0,348.0,355.0,422.0,465.0,467.0,404.0,347.0,305.0,336.0,340.0,318.0,362.0,348.0,
363.0,435.0,491.0,505.0,404.0,359.0,310.0,337.0,360.0,342.0,406.0,396.0,420.0,472.0,
548.0,559.0,463.0,407.0,362.0,405.0,417.0,391.0,419.0,461.0,472.0,535.0,622.0,606.0,
508.0,461.0,390.0,432.0
))
val period = 12
val model = HoltWinters.fitModel(tsAirPassengers, period, "additive", "BOBYQA")
It builds fine, but when I try to run it, I get this error:
Exception in thread "main" java.lang.NoSuchMethodError: scala.runtime.IntRef.create(I)Lscala/runtime/IntRef;
at com.cloudera.sparkts.models.HoltWintersModel.convolve(HoltWinters.scala:252)
at com.cloudera.sparkts.models.HoltWintersModel.initHoltWinters(HoltWinters.scala:277)
at com.cloudera.sparkts.models.HoltWintersModel.getHoltWintersComponents(HoltWinters.scala:190)
.
.
.
The error occurs on this line:
val model = HoltWinters.fitModel(tsAirPassengers, period, "additive", "BOBYQA")
My build.sbt includes:
name := "acme-project"
version := "0.0.1"
scalaVersion := "2.10.5"
libraryDependencies ++= Seq(
"org.apache.spark" %% "spark-hive" % "1.6.0",
"net.liftweb" %% "lift-json" % "2.5+",
"com.github.seratch" %% "awscala" % "0.3.+",
"org.apache.spark" % "spark-mllib_2.10" % "1.6.2"
)
I have placed sparkts-0.4.0-SNAPSHOT.jar in the lib folder of my project. (I would have preferred to add a libraryDependency, but spark-ts does not appear to be on Maven Central.)
What is causing this run-time error?
The library requires Scala 2.11, not 2.10, and Spark 2.0, not 1.6.2, as you can see from
<scala.minor.version>2.11</scala.minor.version>
<scala.complete.version>${scala.minor.version}.8</scala.complete.version>
<spark.version>2.0.0</spark.version>
in pom.xml. You can try changing these and seeing if it still compiles, find which older version of sparkts is compatible with your versions, or update your project's Scala and Spark versions (don't miss spark-mllib_2.10 in this case).
Also, if you put the jar into lib folder, you also have to put its dependencies there (and their dependencies, etc.) or into libraryDependencies. Instead, publish sparkts into your local repository using mvn install (IIRC) and add it to libraryDependencies, which will allow SBT to resolve its dependencies.
i build tests on my Scala Spark App, but i get the exception below on Intellij while running the test. Other tests, without SparkContext are running fine. If i run the test on the terminal with "sbt test-only" the tests with sparkcontext works? Need i to specially configure intellij for tests with sparkcontext?
An exception or error caused a run to abort: org.apache.spark.rdd.ShuffledRDD.(Lorg/apache/spark/rdd/RDD;Lorg/apache/spark/Partitioner;)V
java.lang.NoSuchMethodError: org.apache.spark.rdd.ShuffledRDD.(Lorg/apache/spark/rdd/RDD;Lorg/apache/spark/Partitioner;)V
at org.apache.spark.graphx.impl.RoutingTableMessageRDDFunctions.copartitionWithVertices(RoutingTablePartition.scala:36)
at org.apache.spark.graphx.VertexRDD$.org$apache$spark$graphx$VertexRDD$$createRoutingTables(VertexRDD.scala:457)
at org.apache.spark.graphx.VertexRDD$.fromEdges(VertexRDD.scala:440)
at org.apache.spark.graphx.impl.GraphImpl$.fromEdgeRDD(GraphImpl.scala:336)
at org.apache.spark.graphx.impl.GraphImpl$.fromEdgePartitions(GraphImpl.scala:282)
at org.apache.spark.graphx.GraphLoader$.edgeListFile(GraphLoader.scala:91)
The most likely problem is that spark-core version doesn't match.
Check your sbt file to make sure using the corresponding spark core version you have.
libraryDependencies += "org.apache.spark" %% "spark-core" % "1.1.0"
libraryDependencies += "org.apache.spark" %% "spark-graphx" %"1.1.0"
I'm new to Scala & sbt, and I'm trying to compile a project that imports akka.stm._.
When I try running sbt compile the compilation fails with a message point to that point.
I tried using
https://github.com/paulp/sbt-extras
so I'll have the exact sbt version defined in the "build.sbt" file, but it did not help.
I downloaded the akka-2.0.3.tgz and opened the files, but I don't understand exactly how to install them by default or how to tell sbt to use them.
I also noticed that the build.sbt file contains:
resolvers += "repo.codahale.com" at "http://repo.codahale.com"
libraryDependencies ++= Seq(
// "com.codahale" % "simplespec_2.9.0-1" % "0.3.4"
"com.codahale" % "simplespec_2.9.0-1" % "0.4.1"
// "se.scalablesolutions.akka" %% "akka-sbt-plugin" % "2.0-SNAPSHOT"
I tried uncommenting out the "se.scalablesolutions.akka" (assuming the programmer used the akka-library of that version), but it only printed the message:
[error] Error parsing expression. Ensure that there are no blank lines within a setting.
(There are no blank lines, I just deleted the '//' and replaced the double '%' with a single one)
How do I tell the sbt to find the akka jar files in their place? Can I add another resolver to solve this problem?
I know this kind of question doesn't fit in stackoverflow, but can you at least refer me to some manuals I should read inorder to solve this?
OK, first of all I want to apologize for the newbie question.
(Stackoverflow should make a separate "newbie" section)
First, the elements in the Seq part should be separated by comma.
Second, the akka snapshots were moved to http://repo.akka.io/snapshots so I fixed the build.sbt to:
resolvers += "repo.codahale.com" at "http://repo.codahale.com"
resolvers += "akka" at "http://repo.akka.io/snapshots"
libraryDependencies ++= Seq(
// "com.codahale" % "simplespec_2.9.0-1" % "0.3.4"
"com.codahale" % "simplespec_2.9.0-1" % "0.4.1",
"com.typesafe.akka" % "akka-stm" % "2.0-SNAPSHOT"
)
And the compilation finished successfully.
I don't know if this is the exact configuration in which the original compilation was done, but this issue doesn't disturb me at the moment.