NotSerializableException with Neo4j Spark Streaming Scala - scala

Question

I am trying to run a query in Neo4j using Neo4j-Spark connector. I want to pass the values from the stream (produced by Kafka as a String) into my query. However, I get serialization exception:
Caused by: java.io.NotSerializableException: org.apache.spark.SparkContext
Serialization stack:
- object not serializable (class: org.apache.spark.SparkContext, value: org.apache.spark.SparkContext#54688d9f)
- field (class: consumer.SparkConsumer$$anonfun$processingLogic$2, name: sc$1, type: class org.apache.spark.SparkContext)
- object (class consumer.SparkConsumer$$anonfun$processingLogic$2, <function1>)
- field (class: consumer.SparkConsumer$$anonfun$processingLogic$2$$anonfun$apply$3, name: $outer, type: class consumer.SparkConsumer$$anonfun$processingLogic$2)
- object (class consumer.SparkConsumer$$anonfun$processingLogic$2$$anonfun$apply$3, <function1>)
Here is the code for the main function and querying logic:
object SparkConsumer {
def main(args: Array[String]) {
val config = "neo4j_local"
val sparkConf = new SparkConf().setMaster("local[*]").setAppName("KafkaSparkStreaming")
setNeo4jSparkConfig(config, sparkConf)
val sparkSession = SparkSession
.builder()
.config(sparkConf)
.getOrCreate()
val streamingContext = new StreamingContext(sparkSession.sparkContext, Seconds(3))
streamingContext.sparkContext.setLogLevel("ERROR")
val sqlContext = new SQLContext(streamingContext.sparkContext)
val numStreams = 2
val topics = Array("member_topic1")
def kafkaParams(i: Int) = Map[String, Object](
"bootstrap.servers" -> "localhost:9092",
"key.deserializer" -> classOf[StringDeserializer],
"value.deserializer" -> classOf[StringDeserializer],
"group.id" -> "group2",
"auto.offset.reset" -> "latest",
"enable.auto.commit" -> (false: java.lang.Boolean)
)
val lines = (1 to numStreams).map(i => KafkaUtils.createDirectStream[String, String](
streamingContext,
LocationStrategies.PreferConsistent,
ConsumerStrategies.Subscribe[String, String](topics, kafkaParams(i))
))
val messages = streamingContext.union(lines)
val wordsArrays = values.map(_.split(","))
wordsArrays.foreachRDD(rdd => rdd.foreach(
data => execNeo4jSearchQuery(data)(streamingContext.sparkContext)
))
streamingContext.start()
streamingContext.awaitTermination()
}
def execNeo4jSearchQuery(data: Array[String])(implicit sc: SparkContext) = {
val neo = Neo4j(sc)
val query = "my query"
val paramsMap = Map("lat" -> data(1).toDouble, "lon" -> data(2).toDouble, "id" -> data(0).toInt)
val df = neo.cypher(query, paramsMap).loadDataFrame("group_name" -> "string", "event_name" -> "string", "venue_name" -> "string", "distance" -> "double")
println("\ndf:")
df.show()
}
}

Answer 1

It is not allowed to access SparkContext, SparkSession or create distrbuted data structures from an executor. Therefore:
wordsArrays.foreachRDD(rdd => rdd.foreach(
data => execNeo4jSearchQuery(data)(streamingContext.sparkContext)
))
where execNeo4jSearchQuery calls:
neo.cypher(query, paramsMap).loadDataFrame
is not valid Spark code.
If you want to access Neo4j directly from RDD.foreach you have to use standard client (AnormCypher seems to provide very elegant API), without conversion to Spark distributed structures.
A bit unrelated note - you might consider using a single connection for the set of records with foreachPartition (also SPARK Cost of Initalizing Database Connection in map / mapPartitions context).