Using Kafka with Micronaut - apache-kafka

Question

Are there any example projects showing how to use Kafka with Micronaut? I am having problems with getting it to work.
I have the following producer:
#KafkaClient
interface AppClient {
#Topic("topic-name")
void sendMessage(#KafkaKey String id, Event event)
}
and listener:
#KafkaListener(
groupId="group-id",
offsetReset = OffsetReset.EARLIEST
)
class AppListener {
#Topic("topic-name")
void onMessage(Event event) {
// do stuff
}
}
My application.yml contains:
kafka:
bootstrap:
servers: localhost:2181
and application-test.yml (is this right and should it be in the same directory as application.yml?. Also unsure how the embedded server should be used):
kafka:
# embedded:
# enabled: true
# topics: promo-api-promotions
bootstrap:
servers: localhost:9092
My test looks like:
#MicronautTest
class AppSpec extends Specification {
#Shared
#AutoCleanup
EmbeddedServer server = ApplicationContext.run(EmbeddedServer)
#Shared
private AppClient appClient =
server.applicationContext.getBean(AppClient)
def 'The upload endpoint is called'() {
// test here
appClient.sendMessage(id, event)
// other test stuff
}
The main problems I am having are:
My consumer is not consuming from my topic. I can see the producer creates the topic in Kafka and the client group is created, but the offset stays at 0.
I am having problems when the test is started up where it looks as if two instances of the client are created and therefore the MBean registration fails (also, if I try to use the embedded Kafka, I get a different message about port 9092 already being in use because it tries to start the server up twice):
javax.management.InstanceAlreadyExistsException:
kafka.consumer:type=app-info,id=app-kafka-client-app-listener
at com.sun.jmx.mbeanserver.Repository.addMBean(Repository.java:437)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerWithRepository(DefaultMBeanServerInterceptor.java:1898)

Answer 1

Managed to fix the second problem - the object passed into the listener did not have a #JsonCreator. I found this out by trying to use the Jackson object mapper to construct the object from it's JSON while playing around.
If anyone else has the same problem - make sure that the object model works with Jackson before going any further!

Answer 2

You should add the embedded configuration kafka.embedded.enabled to a map with configuration and pass it to the ApplicationContext.run method.
Map<String, Object> config = Collections.
unmodifiableMap(new HashMap<String, Object>() {
{
put(AbstractKafkaConfiguration.EMBEDDED, true);
put(AbstractKafkaConfiguration.EMBEDDED_TOPICS, "test_topic");
}
});
try (ApplicationContext ctx = ApplicationContext.run(config)) {
The consumer consumes from Kafka in another thread and you have to wait for a while until your AppListener catches up.
You can see a short example in KafkaProducerListenerTest
Remember the Kafka dependencies described in the Micronaut doc: Embedding Kafka