Is there any example of Spring Schedule that reads Kafka topic? - apache-kafka

Question

We're trying to read the data from Kafka at specified window time (so we have Kafka consumer), that means avoiding the data read at other times. However we're not sure how to shut down the consumer after the time period is expired. I wonder if there are any example of how to do that? Many thanks in advance for helping us.

Answer 1

You can disable the autoStartup and then manually start the kafka containers using KafkaListenerEndpointRegistry start and stop methods
#KafkaListener Lifecycle Management
public class KafkaConsumer {
#Autowired
private KafkaListenerEndpointRegistry registry;
#KafkaListener(id = "myContainer", topics = "myTopic", autoStartup = "false")
public void listen(...) { ... }
#Schedule(cron = "")
public void scheduledMethod() {
registry.start();
registry.stop()
}
But in the above approach there is no guarantee that all messages from kafka will be consumed in that time frame (It depends on load and processing speed)

Answer 2

I had the same use case but I wrote the scheduler specifying the max poll record for one batch and kept a counter if the counter matched the max polled record then I consider that the processing for the batch is finished as it has processed the record which it got during the one poll.
And then I am unsubscribing to the topic and closing the consumer. The next time when the scheduler will run it will again process the max poll record specified limit.
fixedDelayString fulfil the purpose that the scheduler starts after the specified time limit once the previous is finished.
#EnableScheduling
public class MessageScheduler{
#Scheduled(initialDelayString = "${fixedInitialDelay.in.milliseconds}", fixedDelayString = "${fixedDelay.in.milliseconds}")
public void run(){
/*write your kafka consumer here with manual commit*/
/*once your batch is finished processing unsubcribe and close the consumer*/
kafkaConsumer.unsubscribe();
kafkaConsumer.close();
}
}