How to unit test HBase in Spark streaming scala - scala

Question

I was trying to unit test doSomethingRdd which requires to read some reference data from HBase in rdd transformation.
def doSomethingRdd(in: DStream[String]): DStream[String] = {
in.map(i => {
val cell = HbaseUtil.getCell("myTable", "myRowKey", "myFamily", "myColumn")
i + cell.getOrElse("")
})
}
Object HBaseUtil {
def getCell(tableName: String, rowKey: String, columnFamily: String, column: String): Option[String] = {
val HBaseConn = ConnectionPool.getConnection()
//the rest of the code will use HBaseConn
//to get a HBase cell and convert to a string
}
}
I read this Cloudera article but I have some problem with their recommended methods.
This first thing I tried was using ScalaMock to mock HBaseUtil.getUtil method so I can bypass HBase connection. I also did some workaround in order to mock Object singleton suggested by this article. I updated my code a bit like below. However, doSomethingRdd failed because mocked hbaseUtil is not serialization which also explained by Paul Butcher in his reply
def doSomethingRdd(in: DStream[String], hbaseUtil: HBaseUtilBody:HBaseUtil): DStream[String] = {
in.map(i => {
val cell = HbaseUtil.getCell("myTable", "myRowKey", "myFamily", "myColumn")
i + cell.getOrElse("")
})
}
trait HBaseUtilBody {
def getCell(tableName: String, rowKey: String, columnFamily: String, column: String): Option[String] = {
val HBaseConn = ConnectionPool.getConnection()
//the rest of the code will use HBaseConn
//to get a HBase cell and convert to a string
}
}
object HBaseUtil extends HBaseUtilBody
I think getting data from HBase in RDD transformation would be a very common pattern. But I'm not sure how to unit test it without connecting to a real HBase instance.

Answer 1

In 2020 with HBase 2.x we use the hbase-testing-util. Simply add it to your SBT build file
// https://mvnrepository.com/artifact/org.apache.hbase/hbase-testing-util
libraryDependencies += "org.apache.hbase" % "hbase-testing-util" % "2.2.3" % Test
And then establish a connection like this
import org.apache.hadoop.hbase.HBaseTestingUtility
val utility = new HBaseTestingUtiliy
utility.startMiniCluster() // defaults to 1 master, 1 region server and 1 data node
val connection = utility.getConnection()
Starting the MiniCluster actually starts
MiniDFSCluster
MiniZKCluster, and
MiniHBaseCluster
In case you need to add some specific configuration (e.g. security settings) you can add hbase-site.xml to your resources.
For more information refer to section Integration Testing with an HBase Mini-Cluster in HBase Reference Guide.