hbase-spark load data raise NullPointerException Error (scala) - scala

Question

I want to load data from HBase by Spark SQL, I use the hbase-spark official example and it raise NullPointerException
My build.sbt file is:
name := "proj_1"
version := "0.1"
scalaVersion := "2.11.12"
libraryDependencies ++= Seq(
"org.apache.spark" % "spark-core_2.11" % "2.3.1",
"org.apache.spark" % "spark-sql_2.11" % "2.3.1",
"org.apache.spark" % "spark-mllib_2.11" % "2.3.1",
"org.apache.spark" % "spark-streaming_2.11" % "2.3.1",
"org.apache.spark" % "spark-hive_2.11" % "2.3.1",
"org.elasticsearch" % "elasticsearch-hadoop" % "6.4.0",
"org.apache.hadoop" % "hadoop-core" % "2.6.0-mr1-cdh5.15.1",
"org.apache.hbase" % "hbase" % "2.1.0",
"org.apache.hbase" % "hbase-server" % "2.1.0",
"org.apache.hbase" % "hbase-common" % "2.1.0",
"org.apache.hbase" % "hbase-client" % "2.1.0",
"org.apache.hbase" % "hbase-spark" % "2.1.0-cdh6.x-SNAPSHOT"
)
resolvers += "Cloudera Repository" at "https://repository.cloudera.com/artifactory/cloudera-repos/"
resolvers += "clojars" at "https://clojars.org/repo"
resolvers += "conjars" at "http://conjars.org/repo"
resolvers += "Apache HBase" at "https://repository.apache.org/content/repositories/releases"
The wrong code is:
def withCatalog(cat: String): DataFrame = {
sqlContext
.read
.options(Map(HBaseTableCatalog.tableCatalog->cat))
.format("org.apache.hadoop.hbase.spark")
.option("zkUrl", "127.0.0.1:2181/chen_test")
.load()
}
val df = withCatalog(catalog)
Exception info is:
Exception in thread "main" java.lang.NullPointerException
at org.apache.hadoop.hbase.spark.HBaseRelation.<init> (DefaultSource.scala:139)
at org.apache.hadoop.hbase.spark.DefaultSource.createRelation(DefaultSource.scala:70)
at org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:340)
at org.apache.spark.sql.DataFrameReader.loadV1Source(DataFrameReader.scala:239)
at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:227)
at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:164)
at hbase_test$.withCatalog$1(hbase_test.scala:57)
at hbase_test$.main(hbase_test.scala:59)
at hbase_test.main(hbase_test.scala)
How do I fix it? Can you help me?

Answer 1

Ran into this problem recently. Suggest you try this:
import org.apache.hadoop.fs.Path
val conf = HBaseConfiguration.create()
conf.addResource(new Path("/path/to/hbase-site.xml"))
new HBaseContext(sc, conf) // "sc" is the SparkContext you created earlier.
The last expression is introducing a stable value into the environment; found this quite accidentally by scanning Hbase's codebase.
Hope it helps.