java.lang.ClassNotFoundException: org.apache.hadoop.hbase.HBaseConfiguration - scala

Question

I want to create my first scala program using the scala example HBaseTest2.scala, provided in Sparkd 1.4.1. The goal is to connect to HBase and do some basic stuff, such as counting rows or scan rows. However, when I tried to execute the program, I got an error. It seems that Spark couldn't find the class HBaseConfiguration. Assuming the we're located a the root path of my project HBaseTest2 /usr/local/Cellar/spark/programs/HBaseTest2. Here are some details for the exception :
./src/main/scala/com/orange/spark/examples/HBaseTest2.scala
package com.orange.spark.examples
import org.apache.hadoop.hbase.client.HBaseAdmin
import org.apache.hadoop.hbase.{HBaseConfiguration, HTableDescriptor, TableName}
import org.apache.hadoop.hbase.mapreduce.TableInputFormat
import org.apache.spark._
object HBaseTest2 {
def main(args: Array[String]) {
val sparkConf = new SparkConf().setAppName("HBaseTest2")
val sc = new SparkContext(sparkConf)
val tableName = "personal-cloud-test"
// please ensure HBASE_CONF_DIR is on classpath of spark driver
// e.g: set it through spark.driver.extraClassPath property
// in spark-defaults.conf or through --driver-class-path
// command line option of spark-submit
val conf = HBaseConfiguration.create()
// Other options for configuring scan behavior are available. More information available at
// http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/mapreduce/TableInputFormat.html
conf.set(TableInputFormat.INPUT_TABLE, tableName)
// Initialize hBase table if necessary
val admin = new HBaseAdmin(conf)
if (!admin.isTableAvailable(tableName)) {
val tableDesc = new HTableDescriptor(TableName.valueOf(tableName))
admin.createTable(tableDesc)
}
val hBaseRDD = sc.newAPIHadoopRDD(conf, classOf[TableInputFormat],
classOf[org.apache.hadoop.hbase.io.ImmutableBytesWritable],
classOf[org.apache.hadoop.hbase.client.Result])
println("hbaseRDD.count()")
println(hBaseRDD.count())
sc.stop()
admin.close()
}
}
./build.sbt
I've added dependencies in this file to ensure all classes called are included in the jar file.
name := "HBaseTest2"
version := "1.0"
scalaVersion := "2.11.7"
libraryDependencies += "org.apache.spark" %% "spark-core" % "1.4.1"
libraryDependencies ++= Seq(
"org.apache.hadoop" % "hadoop-core" % "1.2.1",
"org.apache.hbase" % "hbase" % "1.0.1.1",
"org.apache.hbase" % "hbase-client" % "1.0.1.1",
"org.apache.hbase" % "hbase-common" % "1.0.1.1",
"org.apache.hbase" % "hbase-server" % "1.0.1.1"
)
Run application
MacBook-Pro-de-Mincong:spark-1.4.1 minconghuang$ bin/spark-submit \
--class "com.orange.spark.examples.HBaseTest2" \
--master local[4] \
../programs/HBaseTest2/target/scala-2.11/hbasetest2_2.11-1.0.jar
Exception
15/08/18 12:06:17 INFO storage.BlockManagerMaster: Registered BlockManager
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/hbase/HBaseConfiguration
at com.orange.spark.examples.HBaseTest2$.main(HBaseTest2.scala:21)
at com.orange.spark.examples.HBaseTest2.main(HBaseTest2.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:665)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:170)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:193)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:112)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.hbase.HBaseConfiguration
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 11 more
15/08/18 12:06:17 INFO spark.SparkContext: Invoking stop() from shutdown hook
The problem might come from the HBase configuration as mentioned in HBaseTest2.scala line 16 :
// please ensure HBASE_CONF_DIR is on classpath of spark driver
// e.g: set it through spark.driver.extraClassPath property
// in spark-defaults.conf or through --driver-class-path
// command line option of spark-submit
But I don't know how to configure it... I've added the HBASE_CONF_DIR to CLASSPATH in my command line. The CLASSPATH is now /usr/local/Cellar/hadoop/hbase-1.0.1.1/conf. Nothing happened... T_T So what should I do to get this fixed ? I can add/delete details if needed. Thanks a lot !!

Answer 1

Have you tried
sparkConf.set("spark.driver.extraClassPath", "/usr/local/Cellar/hadoop/hbase-1.0.1.1/conf")

Answer 2

The problem came from class-path-setting as mentioned in HBaseTest2.scala line 33 :
// please ensure HBASE_CONF_DIR is on classpath of spark driver
// e.g: set it through spark.driver.extraClassPath property
// in spark-defaults.conf or through --driver-class-path
// command line option of spark-submit
As I'm using a MAC OS X, setting is different from Linux. When I tried echo $CLASSPATH, it returned empty. It seems that Mac doesn't use the CLASSPATH to do the driver job. So I need to add all jar files through spark.driver.extraClassPath in spark-defaults.conf file. My collegue did the same way in Linux. I think there's a better way to handle it elegantly, but we didn't find out. Please share if you know the answer. Thanks.
Mac / Linux
add all external jars in conf/spark-defaults.conf
spark.driver.extraClassPath /path/to/a.jar:/path/to/b.jar:/path/to/c.jar