How to get a Scala Kafka Consumer in a Play application to continously listen to the broker through the life of the application? - scala

Question

I am trying to create a Play-Scala application that uses a Scala Kafka Consumer to listen to a Kafka broker. I am using the Cake Solutions Scala Kafka Client library, and following their example here.
I have created a containing class to act as a Kafka consumer provider, and I have bound this as an eager singleton so that it is created when the application starts up.
The problem is that the consumer will listen to the broker when the application starts up, but not after that.
Here is my code for the ConsumerProvider:
trait KafkaConsumerProvider {
def consumer: ActorRef
}
#Singleton
class KafkaConsumerProviderImpl #Inject() (actorSystem: ActorSystem, configuration: Configuration)
extends KafkaConsumerProvider {
private val consumerConf: KafkaConsumer.Conf[String, String] = KafkaConsumer.Conf(
keyDeserializer = new StringDeserializer,
valueDeserializer = new StringDeserializer,
bootstrapServers = configuration.get[String]("messageBroker.bootstrapServers"),
groupId = configuration.get[String]("messageBroker.consumer.groupId"),
enableAutoCommit = false,
autoCommitInterval= 1000,
sessionTimeoutMs = 10000,
maxPartitionFetchBytes = ConsumerConfig.DEFAULT_MAX_PARTITION_FETCH_BYTES,
maxPollRecords = 500,
maxPollInterval = 300000,
maxMetaDataAge = 300000,
autoOffsetReset = OffsetResetStrategy.LATEST,
isolationLevel = IsolationLevel.READ_UNCOMMITTED,
)
private val actorConf: KafkaConsumerActor.Conf = KafkaConsumerActor.Conf(
scheduleInterval = 1.seconds, // scheduling interval for Kafka polling when consumer is inactive
unconfirmedTimeout = 3.seconds, // duration for how long to wait for a confirmation before redelivery
maxRedeliveries = 3 // maximum number of times a unconfirmed message will be redelivered
)
override val consumer: ActorRef = {
val receiverActor = actorSystem.actorOf(ReceiverActor.props)
val topics = configuration.get[String]("messageBroker.consumer.topics").split(",").toSeq
val _consumer = actorSystem.actorOf(KafkaConsumerActor.props(consumerConf, actorConf, receiverActor))
_consumer ! Subscribe.AutoPartition(topics)
_consumer
}
}
and here is how I am binding the dependency as an eager singleton in Module.scala:
class Module extends AbstractModule with ScalaModule {
override def configure(): Unit = {
bind[KafkaMessageBrokerWriter].to[KafkaMessageBrokerWriterImpl].asEagerSingleton()
bind[KafkaConsumerProvider].to[KafkaConsumerProviderImpl].asEagerSingleton()
}
}
How do I get the consumer to keep listening?

Answer 1

The problem was that, in the ReceiverActor, I forgot to confirm the offsets:
sender() ! Confirm(records.offsets)