Unit testing in Spark with SQLContext implicits - scala

Question

I'm trying to run multiple unit tests in Spark and have copied (and slightly adapted) the bit from the source code:
import org.apache.spark.sql.SQLContext
import org.apache.spark.{SparkConf, SparkContext}
import org.scalatest.{BeforeAndAfterAll, Suite}
trait SharedSparkContext extends BeforeAndAfterAll {
self: Suite =>
#transient private var _sc: SparkContext = _
#transient private var _sqlContext: SQLContext = _
def sc: SparkContext = _sc
def sqlContext: SQLContext = _sqlContext
private var conf = new SparkConf(false)
override def beforeAll() {
super.beforeAll()
_sc = new SparkContext("local[*]", "Test Suites", conf)
_sqlContext = new SQLContext(_sc)
}
override def afterAll() {
try {
LocalSparkContext.stop(_sc)
_sc = null
} finally {
super.afterAll()
}
}
}
The LocalSparkContext class with companion object are simply copied from the source.
I thought about using it as follows, which tells me that stable identifier required because the def sqlContext does not have the member implicits:
class MySuite extends FlatSpec with SharedSparkContext {
import sqlContext.implicits._
// ...
}
I have tried replacing it with the following, but that gives me null pointer exceptions:
class MySuite extends FlatSpec with SharedSparkContext {
val sqlCtxt = sqlContext
import sqlCtxt.implicits._
// ...
}
I am using Spark 1.4.1 and I have set parallelExecution in test := false.
How can get this to work (without using additional packages)?

Answer 1

Instead of using a trait, you can use a simple object that holds all your variables, here's what I do for my tests :
object TestConfiguration extends Serializable {
private val sparkConf = new SparkConf()
.setAppName("Tests")
.setMaster("local")
private lazy val sparkContext = new SparkContext(sparkConf)
private lazy val sqlContext = new SQLContext(sparkContext)
def getSqlContext() = {
sqlContext
}
}
Then, you'll be able to use the sqlContext in a test suite.
class MySuite extends FlatSpec with SharedSparkContext {
val sqlCtxt = TestConfiguration.getSqlContext
import sqlCtxt.implicits._
// ...
}