I have a need to turn Kafka consumer on/off on the basis of some Database driven property. How can it be achieved.
one way that i have thought of is : throwing exception from consumer when consumer flag is turned off. and container factory config is defined as
factory.setErrorHandler(new SeekToCurrentErrorHandler());
But it actively seeks the same message.
is there any way to turn heart-beat off and then on back again on demand.
You can stop() and start() the listener container.
It appears you are using #KafkaListener since you are using a container factory.
In that case
#KafkaListener(id = "foo" ...)
and then use the KafkaListenerEndpointRegistry bean ...
registry.getListenerContainer("foo").stop();
The Consumer has a few APIs to control its state:
pause()/resume(): Allows to stop/resume consusuming from a set of partitions. The Consumer stays subscribed (so no rebalance) but just does not fetch any new records until resumed
unsubscribe(): Allows to change consumer subscription, if not subscribed to anything, it will just stay connected to the cluster.
If you are "done" with the Consumer, you can also call close() and start a new one when needed
Related
Currently, I have a Kafka Listener configured with a ConcurrentKafkaListenerContainerFactory and a SeekToCurrentErrorHandler (with a DeadLetterPublishingRecoverer configured with 1 retry).
My Listener method is annotated with #Transactional (and also all the methods in my Services that interact with the DB).
My Listener method does the following:
Receive message from Kafka
Interact with several services that save different parts of the received data to the DB
Ack message in Kafka (i.e., commit offset)
If it fails somewhere in the middle, it should rollback and retry until max retries.
Then send message to DLT.
I'm trying to make this method fully transactional, i.e., if something fails all previous changes are rolled back.
However, the #Transactional annotation in the Listener method is not enough.
How can I achieve this?
What configurations should I employ to make the Listener method fully transactional?
If you are not also publishing to Kafka from the listener, there is no need (or benefit) to using Kafka transactions; just overhead. The STCEH + DLPR is enough.
If you are also publishing to Kafka (and want those to be rolled back too), then see the documentation - configure a KafkaTransactionManager in the listener container.
I am studying cqrs and would like to implement the saga pattern to handle distributed transaction using Kafka.
A saga would subscribe the domain event from other aggregate to send a command. My problem is the domain event would be handled by the aggregate event handler as well.
If the aggregate event handler handled the event successfully, the offset would be committed so that the job in the broker gone.
Let's say if the aggregate event handler handle that event successfully, but the saga is not triggered because of some unexpected issue. Since the job is gone, the event would never be picked up by the saga if they both live in same consumer group...
So does it make sense to have a consumer group for the aggregate and create another consumer group for the saga?
so that the job in the broker gone.
I think you're misunderstanding that consuming messages from a Kafka topic does not remove them.
Part of your saga flow should be an action for committing the full offset trace once each service does its complete action
does it make sense to have a consumer group for the aggregate and create another consumer group for the saga?
It makes sense to have intermediate topics, sure. You could use Kafka Streams processors to easily implement that
I have to log the time taken by a consumer in spring kafka. As the kafkaListener method execute for each messages, putting a logger over there does not work. Also sometimes a few messages are getting lost and not consumed by consumer. Where should I put logger to find out the time elasped since the consumer starts. The consumer does not exit or shutdown, its polling indefinitely
You can set the idleEventInterval container property and use an ApplicationListener<ListenerContainerIdleEvent> bean (or an #EventListener method) to receive these events.
The events are published every idleEventInterval and contain an idleTime property which is the time since we last received a message.
For the start time, you can see the partitions assigned log message or just use the time the container was start()ed.
My app has 5+ consumers consuming off of five partitions on a kafka topic.(using kafka version 11) My consumer's each produce a message to another topic then save some state to the database, then do a manual_ immediate acknowledgement and move onto the next message.
I'm trying to solve the scenario when they emit successful to the outbound topic. then we have a failure/lose the consumer. When another consumer takes over the partition it will emit ANOTHER message to the outbound topic. This is bad :(
I discovered that kafka now has idempotent producers but from what I read it only guarantees for a producers session.
"When producer restarts, new PID gets assigned. So the idempotency is promised only for a single producer session" - (blog) - https://hevodata.com/blog/kafka-exactly-once
This seems largely useless to me. In my use-case the whole point is when I replay a message on another consumer it does not duplicate the outbound message.
Is there something i'm missing?
When using transactions, you shouldn't use ANY consumer-based mechanism, manual or otherwise, to commit the offsets.
Instead, you use the producer to send the offsets to the transaction so the offset commit is part of the transaction.
If configured with a KafkaTransactionManager, or ChainedKafkaTransactionManager the Spring listener container will send the offsets to the transaction when the listener exits normally.
If you don't use a Kafka transaction manager, you need to use the KafkaTemplate (or Producer if you are using the native APIs) to send the offsets to the transaction.
Using the consumer to commit the offset is not part of the transaction, so things will not work as expected.
When using a transaction manager, the listener container binds the Producer to the thread so any downstream KafkaTemplate operations participate in the transaction that the consumer starts. See the documentation.
We are going to implement a Kafka Publish Subscribe system.
Now, in the worst of the worst cases - if all the kafka brokers for a given topic go down -- what happens?
I tried this out...the publisher detects it after the default timeout for metadata fetch & throws exception if not successful.
In this case, we can monitor the exception and restart Publisher after fixing Kafka.
But, what about the consumers -- they don't seem to get any exceptions once Kafka goes down. We simply can't ask "all" the consumers to restart their systems. Any better way to solve this problem?
But, what about the consumers -- they don't seem to get any exceptions
once Kafka goes down. We simply can't ask "all" the consumers to
restart their systems. Any better way to solve this problem?
Yes, consumer won't get any exceptions and the behavior is work as designed. However, you don't need to restart all the consumers, just make sure in your logic that consumer is calling the poll()method call regularly. Consumer is designed in a way that it does not get effected, even if there is no cluster alive. Consider the following steps to understand what will happen actually:
1: All clusters are down, there is no active cluster.
2: consumer.poll(timeout) // This will be called form you portion of code
3: Inside poll() method call in KafkaConsumer.java, following sequence of calls will take place.
poll() --> pollOnce() --> ensureCoordinatorKnown() --> awaitMetaDataUpdate()
I have highlighted the main method calls that will be called after performing logical checks internally. Now, at this point your consumer will wait until the cluster is up again.
4: Cluster up again or restarted
5: Consumer will be notified and it will start working again as normally it was before the cluster goes down.
Note:- Consumer will start receiving messages from the last offset commit, message received successfully won't be duplicated.
The described behavior is valid for (0.9.x version)
If the consumer (0.9.x version) is polling and the cluster goes down it should get the following exception
java.net.ConnectException: Connection refused
You can keep polling until the cluster is back again there is no need to restart the consumer, it will re-establish the connection.