unable to fetch messages from Kafka topic using spark structured Streaming - spark-structured-streaming

Question

I have created kafka topic with below command,
kafka-topics --create --zookeeper 10.0.2.11:2181 --replication-factor 1 --partitions 1 --topic xmlStore
I am using spark structured streaming to subscribe and read from topic. below is the code
object Test {
val spark = SparkSession.builder
.appName("Spark-Kafka-Integration")
.master("local")
.getOrCreate()
def main(args: Array[String]): Unit = {
println("Inside Main ===>")
import spark.implicits._
val inputDf = spark
.readStream
.format("kafka")
.option("kafka.bootstrap.servers", "localhost:9092")
.option("subscribe", "xmlStore")
.option("startingOffsets" , "earliest")
.load()
inputDf.printSchema()
val dataSet: Dataset[(String, String)] =inputDf.selectExpr("CAST(key AS
STRING)", "CAST(value AS STRING)")
.as[(String, String)]
val query = dataSet
.writeStream
.outputMode("append")
.format("memory")
.queryName("op")
.start()
println("*********" +query.isActive)
spark.sql("select * from op").show(false)
query.explain()
query.awaitTermination()
}
}
After submitting above spark consumer job, I am producing values with below command
kafka-console-producer --broker-list 10.0.1.10:9092 --topic xmlStore
Hello how are you?
I am not seeing any values on console. i also tried with
dataSet
.writeStream
.outputMode("append")
.format("console")
.start()
Still no values are displayed. I dont know what went wrong. please help me out.thanks in advance
I am getting below output on console,
18/12/14 07:15:09 INFO metadata.Hive: Registering function
tempfromfartocelcius com.hive.temparature.udf.ConvertToCelcius
root
|-- key: binary (nullable = true)
|-- value: binary (nullable = true)
|-- topic: string (nullable = true)
|-- partition: integer (nullable = true)
|-- offset: long (nullable = true)
|-- timestamp: timestamp (nullable = true)
|-- timestampType: integer (nullable = true)
18/12/14 07:15:11 INFO streaming.MicroBatchExecution: Starting op [id =
dfceb317-4a27-4b8a-b2f8-3ff7aa4a0242, runId = f6fe77e1-cb64-4d41-938f-
db573380dc71]. Use hdfs://ip-10-0-1-20.ec2.internal:8020/tmp/temporary-
b6d69d12-26b4-4635-80bc-d923cf82dc23 to store the query checkpoint.
18/12/14 07:15:11 INFO consumer.ConsumerConfig: ConsumerConfig values:
metric.reporters = []
metadata.max.age.ms = 300000
partition.assignment.strategy =
[org.apache.kafka.clients.consumer.RangeAssignor]
reconnect.backoff.ms = 50
sasl.kerberos.ticket.renew.window.factor = 0.8
max.partition.fetch.bytes = 1048576
bootstrap.servers = [localhost:9092]
ssl.keystore.type = JKS
enable.auto.commit = false
sasl.mechanism = GSSAPI
interceptor.classes = null
exclude.internal.topics = true
ssl.truststore.password = null
client.id =
ssl.endpoint.identification.algorithm = null
max.poll.records = 1
check.crcs = true
request.timeout.ms = 40000
heartbeat.interval.ms = 3000
auto.commit.interval.ms = 5000
receive.buffer.bytes = 65536
ssl.truststore.type = JKS
ssl.truststore.location = null
ssl.keystore.password = null
fetch.min.bytes = 1
send.buffer.bytes = 131072
value.deserializer = class
org.apache.kafka.common.serialization.ByteArrayDeserializer
group.id = spark-kafka-source-4b2e17d1-f289-47be-ab4f-fce9fa1c1723-
-744805972-driver-0
retry.backoff.ms = 100
ssl.secure.random.implementation = null
sasl.kerberos.kinit.cmd = /usr/bin/kinit
sasl.kerberos.service.name = null
sasl.kerberos.ticket.renew.jitter = 0.05
ssl.trustmanager.algorithm = PKIX
ssl.key.password = null
fetch.max.wait.ms = 500
sasl.kerberos.min.time.before.relogin = 60000
connections.max.idle.ms = 540000
session.timeout.ms = 30000
metrics.num.samples = 2
key.deserializer = class
org.apache.kafka.common.serialization.ByteArrayDeserializer
ssl.protocol = TLS
ssl.provider = null
ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1]
ssl.keystore.location = null
ssl.cipher.suites = null
security.protocol = PLAINTEXT
ssl.keymanager.algorithm = SunX509
metrics.sample.window.ms = 30000
auto.offset.reset = earliest
18/12/14 07:15:11 INFO consumer.ConsumerConfig: ConsumerConfig values:
metric.reporters = []
metadata.max.age.ms = 300000
partition.assignment.strategy =
[org.apache.kafka.clients.consumer.RangeAssignor]
reconnect.backoff.ms = 50
sasl.kerberos.ticket.renew.window.factor = 0.8
max.partition.fetch.bytes = 1048576
bootstrap.servers = [localhost:9092]
ssl.keystore.type = JKS
enable.auto.commit = false
sasl.mechanism = GSSAPI
interceptor.classes = null
exclude.internal.topics = true
ssl.truststore.password = null
client.id = consumer-2
ssl.endpoint.identification.algorithm = null
max.poll.records = 1
check.crcs = true
request.timeout.ms = 40000
heartbeat.interval.ms = 3000
auto.commit.interval.ms = 5000
receive.buffer.bytes = 65536
ssl.truststore.type = JKS
ssl.truststore.location = null
ssl.keystore.password = null
fetch.min.bytes = 1
send.buffer.bytes = 131072
value.deserializer = class
org.apache.kafka.common.serialization.ByteArrayDeserializer
group.id = spark-kafka-source-4b2e17d1-f289-47be-ab4f-fce9fa1c1723-
-744805972-driver-0
retry.backoff.ms = 100
ssl.secure.random.implementation = null
sasl.kerberos.kinit.cmd = /usr/bin/kinit
sasl.kerberos.service.name = null
sasl.kerberos.ticket.renew.jitter = 0.05
ssl.trustmanager.algorithm = PKIX
ssl.key.password = null
fetch.max.wait.ms = 500
sasl.kerberos.min.time.before.relogin = 60000
connections.max.idle.ms = 540000
session.timeout.ms = 30000
metrics.num.samples = 2
key.deserializer = class
org.apache.kafka.common.serialization.ByteArrayDeserializer
ssl.protocol = TLS
ssl.provider = null
ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1]
ssl.keystore.location = null
ssl.cipher.suites = null
security.protocol = PLAINTEXT
ssl.keymanager.algorithm = SunX509
metrics.sample.window.ms = 30000
auto.offset.reset = earliest
INFO utils.AppInfoParser: Kafka version : 0.10.0-kafka-2.1.0
INFO utils.AppInfoParser: Kafka commitId : unknown
INFO streaming.MicroBatchExecution: Starting new streaming query.
+-----++-----+
|key ||value|
+-----++-----+
+-----++-----+

Answer 1

I was trying to to run on Cluster by submitting spark job. May be because of spark and kafka's versions mismatch messages were not consumed.
i set up kafka locally on windows and i am able to consume messages from topic.