Spark streaming consumer streams null values - scala

Question

I have created a kafka producer using node js which is basically pushing the live data it is receiving from upstox into a kafka-topic. The kafka-producer snippet looks someting like this:
upstox.on("liveFeed", function(message) {
//message for live feed
var data = JSON.stringify(message);
var payload = [{
topic : 'live-feed',
message: data,
attributes: 1
}];
producer.send(payload, function(error, result) {
console.info('Sent payload to Kafka: ', payload);
if (error) {
console.error(error);
} else {
console.log('result: ', result)
}
});
It's giving me the live feed in the following format:
topic: live-feed,
message:{live-feed data},
attributes:1
Now I'm trying to code a spark streaming consumer which streams the data produced by this producer. I came up with something like this:
package com.senpuja.datastream
import kafka.serializer.StringDecoder
import org.apache.spark.streaming.kafka.KafkaUtils
import org.apache.spark.streaming.{Seconds, StreamingContext}
import org.apache.spark.SparkConf
import org.apache.spark.SparkContext
object LiveFeedStream {
def main(args: Array[String]): Unit = {
val brokers = util.Try(args(0)).getOrElse("localhost:9092")
val inTopic = util.Try(args(1)).getOrElse("live-feed")
val sparkConf = new SparkConf()
val spark = new SparkContext(sparkConf)
val streamCtx = new StreamingContext(spark, Seconds(10))
val inTopicSet = Set(inTopic)
val kafkaParams = Map[String, String](
"bootstrap.servers" -> brokers,
"key.deserializer" -> "org.apache.kafka.common.serialization.StringDeserializer",
"value.deserializer" -> "org.apache.kafka.common.serialization.StringDeserializer"
)
val msg = KafkaUtils.createDirectStream[String, String, StringDecoder, StringDecoder](
streamCtx,
kafkaParams,
inTopicSet
)
msg.print()
streamCtx.start()
streamCtx.awaitTermination()}
But when I submit the code, I get the following output which is just null:
{null}, {null}
{null}, {null}
{null}, {null}
{null}, {null}
{null}, {null}
I want to retrieve the message part from the producer topic. I think it has something to do with the key-value thing I guess, but I'm not able to figure out its solution. Any help would be really appreciated!

Answer 1

Add enable.auto.commit = false in Kafka parameter and try.

Answer 2

I found that the problem was this that I was directly passing the message while the spark streaming code was looking for a key-value pair. So I used KeyedMessage to produce a key-value pair.
upstox.on("liveFeed", function(message) {
//message for live feed var
var data = JSON.stringify(message);
var km = new KeyedMessage(Math.floor(Math.random() * 10000), data);
var payload = [{
topic : 'live-feed',
messages: km
}];
producer.send(payload, function(error, result) {
console.info('Sent payload to Kafka: ', payload);
if (error) {
console.error(error);
} else {
console.log('result: ', result)
}
)}
It solved my problem.