Spark Streaming - Read Data using the metadata received from Kafka - apache-kafka

Question

I am pretty much new to the Spark world. I am trying to write an optimised solution for the below use case:
Need to read streaming data from Kafka which primarily is a S3 filepath of some compressed files.
Read the compressed file from the filepath received above and process it and store it back to some S3 bucket.
I am able to read the Kafka topic and get the filepath but not sure how do I read this file path now?
Something like spark.read.binaryFile(filePath).
Any help or guidance would be appreciated.

Answer 1

This has many examples
Read compressed file
rdd = sc.textFile("s3://bucket/lahs/blahblah.*.gz")
Without your code its hard, here is an outline on reading and writing it back
From this answer for the rest...
val spark = SparkSession.builder()
.appName("myKfconsumer")
.master("local[*]")
.getOrCreate()
//... create your schema
// you path
val filePath = "file:///tmp/spark/Blah/blah"
// create it as your batch data
// someBatchData
// now read it, you need your schema and write it back in the process section below
import spark.implicits._
val kafkaStream = spark.readStream
.format("kafka")
.option("kafka.xyz.servers", "localhost:0001")
.option("subscribe", "blah")
.option("startingOffsets", "latest")
.option("failOnDataLoss", "true") // stop and debug it
.load()
.as[String]
kafkaStream.writeStream.foreachBatch((someBatchData:Dataset[String], batchId:Long) => {
val records = someBatchData.collect()
// go through all the records
records.foreach((path: String) => {
val yourData = spark.read.schema(.. youfileSchema).json(..youPath)
// write it back as you wanted..
//
})
}).start()
spark.streams.awaitAnyTermination()