I am consuming Kafka events using #KafkaHandler on the method level (#KafkaListener on class level).
I have seen a lot of examples where an "Acknowledgement" argument is available, on which the "acknowledge()" method can be called to commit consumption of the event, however, I am not able to get the acknowledgement object populated when including it as an argument to my method. How do I manual commit when using a KafkaHandler? Is it possible at all?
Code example:
#Service
#KafkaListener(topics = "mytopic", groupId = "mygroup")
public class TestListener {
#KafkaHandler
public void consumeEvent(MyEvent event, Acknowledgement ack) throws Exception {
//... processing
ack.acknowledge(); // ack is not available
}
Using SpringBoot and Spring-kafka.
You must configure the listener container with AckMode.MANUAL or AckMode.MANUAL_IMMEDIATE to get this functionality.
However, it's generally better to let the container take care of committing the offset with AckMode.RECORD or AckMode.BATCH (default).
https://docs.spring.io/spring-kafka/docs/current/reference/html/#committing-offsets
EDIT
spring.kafka.consumer.auto-offset-reset=earliest
spring.kafka.listener.ack-mode=MANUAL
#SpringBootApplication
public class So68844554Application {
public static void main(String[] args) {
SpringApplication.run(So68844554Application.class, args);
}
#Bean
public NewTopic topic() {
return TopicBuilder.name("so68844554").partitions(1).replicas(1).build();
}
}
#Component
#KafkaListener(id = "so68844554", topics = "so68844554")
class Foo {
#KafkaHandler
void listen(String in, Acknowledgment ack) {
System.out.println(in);
ack.acknowledge();
}
}
% kafka-consumer-groups --bootstrap-server localhost:9092 --describe -group so68844554
Consumer group 'so68844554' has no active members.
GROUP TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID HOST CLIENT-ID
so68844554 so68844554 0 2 2 0 - - -
Related
The Flink consumer application I am developing reads from multiple Kafka topics. The messages published in the different topics adhere to the same schema (formatted as Avro). For schema management, I am using the Confluent Schema Registry.
I have been using the following snippet for the KafkaSource and it works just fine.
KafkaSource<MyObject> source = KafkaSource.<MyObject>builder()
.setBootstrapServers(BOOTSTRAP_SERVERS)
.setTopics(TOPIC-1, TOPIC-2)
.setGroupId(GROUP_ID)
.setStartingOffsets(OffsetsInitializer.earliest())
.setValueOnlyDeserializer(ConfluentRegistryAvroDeserializationSchema.forSpecific(MyObject.class, SCHEMA_REGISTRY_URL))
.build();
Now, I want to determine the topic-name for each message that I process. Since the current deserializer is ValueOnly, I started looking into the setDeserializer() method which I felt would give me access to the whole ConsumerRecord object and I can fetch the topic-name from that.
However, I am unable to figure out how to use that implementation. Should I implement my own deserializer? If so, how does the Schema registry fit into that implementation?
You can use the setDeserializer method with a KafkaRecordDeserializationSchema that might look something like this:
public class KafkaUsageRecordDeserializationSchema
implements KafkaRecordDeserializationSchema<UsageRecord> {
private static final long serialVersionUID = 1L;
private transient ObjectMapper objectMapper;
#Override
public void open(DeserializationSchema.InitializationContext context) throws Exception {
KafkaRecordDeserializationSchema.super.open(context);
objectMapper = JsonMapper.builder().build();
}
#Override
public void deserialize(
ConsumerRecord<byte[], byte[]> consumerRecord,
Collector<UsageRecord> collector) throws IOException {
collector.collect(objectMapper.readValue(consumerRecord.value(), UsageRecord.class));
}
#Override
public TypeInformation<UsageRecord> getProducedType() {
return TypeInformation.of(UsageRecord.class);
}
}
Then you can use the ConsumerRecord to access the topic and other metadata.
I took inspiration from the above answer (by David) and added the following custom deserializer -
KafkaSource<MyObject> source = KafkaSource.<MyObject>builder()
.setBootstrapServers(BOOTSTRAP_SERVERS)
.setTopics(TOPIC-1, TOPIC-2)
.setGroupId(GROUP_ID)
.setStartingOffsets(OffsetsInitializer.earliest())
.setDeserializer(KafkaRecordDeserializationSchema.of(new KafkaDeserializationSchema<Event>{
DeserializationSchema deserialzationSchema = ConfluentRegistryAvroDeserializationSchema.forSpecific(MyObject.class, SCHEMA_REGISTRY_URL);
#Override
public boolean isEndOfStream(Event nextElement) {
return false;
}
#Override
public String deserialize(ConsumerRecord<byte[], byte[]> consumerRecord) throws Exception {
Event event = new Event();
event.setTopicName(record.topic());
event.setMyObject((MyObject) deserializationSchema.deserialize(record.value()));
return event;
}
#Override
public TypeInformation<String> getProducedType() {
return TypeInformation.of(Event.class);
}
})).build();
The Event class is a wrapper over the MyObject class with additional field for storing the topic name.
i have two kafka listeners like below:
#KafkaListener(topics = "foo1, foo2", groupId = foo.id, id = "foo")
public void fooTopics(#Header(KafkaHeaders.RECEIVED_TOPIC) String topic, String message, Acknowledgment acknowledgment) {
//processing
}
#KafkaListener(topics = "Bar1, Bar2", groupId = bar.id, id = "bar")
public void barTopics(#Header(KafkaHeaders.RECEIVED_TOPIC) String topic, String message, Acknowledgment acknowledgment) {
//processing
same application is running on two instances like inc1 and inc2. is there a way if i can assign foo listener to inc1 and bar listener to inc2. and if one instance is going down both the listener(foo and bar) assign to the running instance.
You can use the #KafkaListener property autoStartup, introduced since 2.2.
When an instance die, you can automatically start it up in the other instance like so:
#Autowired
private KafkaListenerEndpointRegistry registry;
...
#KafkaListener(topics = "foo1, foo2", groupId = foo.id, id = "foo", autoStartup = "false")
public void fooTopics(#Header(KafkaHeaders.RECEIVED_TOPIC) String topic, String message, Acknowledgment acknowledgment) {
//processing
}
//Start up condition
registry.getListenerContainer("foo").start();
I have two kafka consumers defined in two micro services from one producer with different group id defined.One of the micro service is not consuming the message from the producer
#Component
#ConditionalOnProperty(value = "XXX", havingValue = "true")
public class EventListener {
#KafkaListener(id = "XX", topics = "#{'${topic'}")
public void consumeMessageEvent(Event messageEvent, ConsumerRecord<String, ?> record) {
}
}
I have one publisher which is publishing messages on topic, and I have 2 subscribers S1 & S2 which are receiving the messages. When my publisher sends a message and both subscribers are up then they both receive the message. However, when my subscribers are not up and my publisher sends a message then when the subscribers come up they do not receive the message. How can my subscribers receive messages sent when they are not up?
Note: I am using Spring Boot.
MessageProducer.java
#RestController
#RequestMapping("/rest/produce")
public class MessageProducer {
private static final Logger LOG = LoggerFactory.getLogger(MessageProducer.class);
#Autowired
public JmsTemplate jmsTemplate;
#GetMapping("/{message}")
public void run(#PathVariable("message") final String message) throws Exception {
final String messageText = "Hello Blockchain World";
LOG.info("============= Sending " + message);
sendMessage(message);
}
public void sendMessage(String payload) {
this.jmsTemplate.convertAndSend("example", payload);
}
}
application.properties - (MessageProducer)
spring.qpidjms.remoteURL=amqp://127.0.0.1:5672
spring.qpidjms.username=admin
spring.qpidjms.password=admin
activemq.broker-url=tcp://localhost:61616
server.port=8888
spring.jms.pub-sub-domain=true
MessageConsumer.java
#Component
public class MessageConsumer {
private static final Logger LOG = LoggerFactory.getLogger(MessageConsumer.class);
#JmsListener( destination = "example")
public void processMsg(String message) {
LOG.info("============= Received: " + message);
}
}
MessageConsumer main Initiator class (ignore class name)
#SpringBootApplication
#EnableJms
public class QpidJMSSpringBootHelloWorld {
public static void main(String[] args) {
SpringApplication.run(QpidJMSSpringBootHelloWorld.class, args);
}
}
Second consumer is same as first one just port no has been changed in application.properties
application.properties (MessageConsumer-1, S1)
spring.qpidjms.remoteURL=amqp://127.0.0.1:5672
spring.qpidjms.username=admin
spring.qpidjms.password=admin
activemq.broker-url=tcp://localhost:61616
server.port=9999
spring.jms.pub-sub-domain=true
application.properties (S2)
spring.qpidjms.remoteURL=amqp://127.0.0.1:5672
spring.qpidjms.username=admin
spring.qpidjms.password=admin
activemq.broker-url=tcp://localhost:61616
server.port=9990
spring.jms.pub-sub-domain=true
Messages sent to a multicast address (i.e. a JMS topic) are routed to all existing multicast queues (i.e. JMS subscriptions). If no subscriptions exist then the messages are discarded. This is the fundamental semantics of multicast routing (i.e. JMS publish-subscribe).
If you want messages for a subscriber to be stored when the subscriber is not connected then the subscriber must create a durable subscription before any messages which it wants are sent. Once the durable subscription is created the messages sent to the topic will be stored in that subscription even if the subscriber is not connected.
I know I can find out from which partition record comes in, but I wonder is any way to dynamically get which partitions are assigned for consumers at specific moment? Maybe I need to implement some listener to detect and follow up partitions assignation info?
I am using spring-kafka 1.3.2 with ConcurrentKafkaListenerContainerFactory and #KafkaListener.
Yes, you can do:
#Bean
public ConsumerAwareRebalanceListener rebalanceListener() {
return new ConsumerAwareRebalanceListener() {
#Override
public void onPartitionsAssigned(Consumer<?, ?> consumer, Collection<TopicPartition> partitions) {
// here partitions
}
};
}
And then add it, for example, to ConcurrentKafkaListenerContainerFactory
#Bean
public ConcurrentKafkaListenerContainerFactory<Object, Object> kafkaListenerContainerFactory() {
ConcurrentKafkaListenerContainerFactory<Object, Object> factory = new ConcurrentKafkaListenerContainerFactory<>();
ContainerProperties props = factory.getContainerProperties();
props.setConsumerRebalanceListener(rebalanceListener());
return factory;
}
I did it in different way by using KafkaListenerEndpointRegistry
for (MessageListenerContainer messageListenerContainer : kafkaListenerEndpointRegistry.getListenerContainers()) {
List<KafkaMessageListenerContainer> containers = ((ConcurrentMessageListenerContainer) messageListenerContainer).getContainers();
List<TopicPartition> topicPartitions = (List<TopicPartition>) containers.stream().flatMap(kafkaMessageListenerContainer ->
kafkaMessageListenerContainer.getAssignedPartitions().stream()).collect(Collectors.toList());
partitions.addAll(topicPartitions.stream().map(TopicPartition::partition).collect(Collectors.toList()));
}