Spark Kafka streamRead and streamWrite showing null records - scala

Question

Am writing a batch application to consume Kafka events and write it to GCS location. Tried deleting the checkpoint location and also verified kafka has 200 messages to consume
Spark - 2.4.8
Scala - 2.21
Submit Command : spark-shell --master yarn --packages org.apache.spark:spark-sql-kafka-0-10_2.12:2.4.8
import org.apache.spark.sql.streaming.Trigger
val readInputKafkaDataNew = spark.readStream
.format("kafka")
.option(
"kafka.bootstrap.servers",
"localhost:9092"
)
.option("subscribe", "changefeed")
.load()
.selectExpr("CAST(key AS STRING)", "CAST(value AS STRING)")
.as[(String, String)]
.writeStream
.outputMode("append")
.format("csv")
.option(
"path",
"gs://test-data-today/test_data/abc"
)
.option(
"checkpointLocation",
"gs://test-data-today/test_data/chkdir"
)
.trigger(Trigger.Once())
.start()
.awaitTermination();
The console log prints its committing the offsets but the dataframe is empty
22/06/24 22:25:35 INFO org.apache.spark.sql.execution.streaming.MicroBatchExecution: Streaming query made progress: {
"id" : "df24d47d-1dbb-4124-9d4f-0e4d9e6a0275",
"runId" : "e0f330e7-fdcf-47ba-85c6-417afbb3cff9",
"name" : null,
"timestamp" : "2022-06-24T22:25:23.257Z",
"batchId" : 0,
"numInputRows" : 0,
"processedRowsPerSecond" : 0.0,
"durationMs" : {
"addBatch" : 5675,
"getBatch" : 1,
"getEndOffset" : 0,
"queryPlanning" : 17,
"setOffsetRange" : 4870,
"triggerExecution" : 12018,
"walCommit" : 715
},
"stateOperators" : [ ],
"sources" : [ {
"description" : "KafkaV2[Subscribe[changefeed]]",
"startOffset" : null,
"endOffset" : {
"changefeed" : {
"2" : 34,
"5" : 28,
"4" : 26,
"1" : 39,
"3" : 49,
"0" : 30
}
},
"numInputRows" : 0,
"processedRowsPerSecond" : 0.0
} ],
"sink" : {
"description" : "FileSink[gs://gs://test-data-today/test_data/abc]"
}
}