Reading and processing parallelism in Kafka Spark streaming - scala

Question

I am trying to parallelize reading of Kafka messages thereby processing them in parallel. My Kafka topic has 10 partitions. I'm trying to create 5 DStreams and applying Union method to operate on a single DStream. Here is the code I tried so far:
def main(args: scala.Array[String]): Unit = {
val properties = readProperties()
val streamConf = new SparkConf().setMaster("local[2]").setAppName("KafkaStream")
val ssc = new StreamingContext(streamConf, Seconds(1))
// println("defaultParallelism: "+ssc.sparkContext.defaultParallelism)
ssc.sparkContext.setLogLevel("WARN")
val numPartitionsOfInputTopic = 5
val group_id = Random.alphanumeric.take(4).mkString("consumer_group")
val kafkaStream = {
val kafkaParams = Map("zookeeper.connect" -> properties.getProperty("zookeeper_connection_str"),
"group.id" -> group_id,
"zookeeper.connection.timeout.ms" -> "3000")
val streams = (1 to numPartitionsOfInputTopic).map { _ =>
KafkaUtils.createStream[scala.Array[Byte], String, DefaultDecoder, StringDecoder](
ssc, kafkaParams, Map("kafka_topic" -> 1), StorageLevel.MEMORY_ONLY_SER).map(_._2)
}
val unifiedStream = ssc.union(streams)
val sparkProcessingParallelism = 5
unifiedStream.repartition(sparkProcessingParallelism)
}
kafkaStream.foreachRDD { x =>
x.foreach {
msg => println("Message: "+msg)
processMessage(msg)
}
}
ssc.start()
ssc.awaitTermination()
}
Upon execution, it's not even receiving a single message, let alone processing it further. Am I missing something here? Please suggest for changes if required. Thanks.

Answer 1

I highly recommend to switch to Direct Stream. Why?
Direct Stream by default sets the parallelism to the number of partition you have in Kafka. Nothing more must be done - just create Direct Stream and do your job :)
If you create 5 DStreams, you will by default read in 5 thread, one non-Direct-DStream = one thread