Returning values from kafka consumer - scala

Question

I am working on a scala application. I am using kafka in it. I am to consume message from kafka topic. Since I am writing a test case, I need to get record do some assertion to get pass my testcase. I am using following code to consume kafka message:
import java.util.{Collections, Properties}
import java.util.regex.Pattern
import org.apache.kafka.clients.consumer.KafkaConsumer
import scala.collection.JavaConverters._
object KafkaConsumerSubscribeApp extends App {
val props:Properties = new Properties()
props.put("group.id", "test")
props.put("bootstrap.servers","localhost:9092")
props.put("key.deserializer",
"org.apache.kafka.common.serialization.StringDeserializer")
Props.put("value.deserializer",
"org.apache.kafka.common.serialization.StringDeserializer")
props.put("enable.auto.commit", "true")
props.put("auto.commit.interval.ms", "1000")
val consumer = new KafkaConsumer(props)
val topics = List("topic_text")
try {
consumer.subscribe(topics.asJava)
while (true) {
val records = consumer.poll(10)
for (record <- records.asScala) {
println("Topic: " + record.topic() +
",Key: " + record.key() +
",Value: " + record.value() +
", Offset: " + record.offset() +
", Partition: " + record.partition())
}
}
}catch{
case e:Exception => e.printStackTrace()
}finally {
consumer.close()
}
}
There are two problems I am facing with this code. In intellij it is giving warning that poll method is deprecated. How can I modify this code for deprecation? Second problem is I want to this method to return the message which it gets from kafka topic. That message is in record.value(). How can I return it? With this code since while(true) is used, so it will be a endless loop and it will keep listening to messages from topic. How can I return record.value() from this method so that I can use data which I got from topic in other methods.

Answer 1

If you want to return you have to modify the codebase as follows. To remove the deprecation of the poll, you need to just pass the durations. For more info refer
def readFromKafka(//arguments) = {
// other code
consumer.subscribe(util.Collections.singletonList(topic))
consumer.poll(Duration.ofMillis(5000)).asScala.toList.map(_.value())
}
To keep on reading from Kafka, you have to keep on calling the function. For that, you may use the scheduler.
I hope it will help.