Our application uses akka actors (v. 1.1.2) that throw exception from time to time. We would like to have those exceptions logged and are using event-handlers = ["akka.event.slf4j.Slf4jEventHandler"] to turn on the logging using slf4j/logback.
However, error log messages are still also written to stdout.
Is there any setting in akka how to disable EventHandler$DefaultListener which seems to cause that behavior?
Thanks in advance!
Related
Is there any event callback for producer-broker connection failure like we have for consumer-broker failure in APache Kafka like for Consumer we have KafkaRebalanceCallback? Does there exist something similar for the producer since my producer does reconnect if my broker goes down and comes up again or show some kinda log when this happens.
Kafka consumer application is throwing ObjectDisposedException when we initiate the process to stop the consumer service.I would like to understand what process destroyed the kafka handle
System.ObjectDisposedException: handle is destroyed
at Confluent.Kafka.Impl.SafeKafkaHandle.ThrowIfHandleClosed()
at Confluent.Kafka.Consumer`2.Consume(Int32 millisecondsTimeout)
at Confluent.Kafka.Consumer`2.Consume(CancellationToken cancellationToken)
in my case this exception was thrown in .Net Library Confluent Kafka.
If you are also using this library, please make sure what you call Consumer.Close() BEFORE Consumer.Dispose().
I have a Kafka Streams application that launches and runs successfully. We have 4 instances of the application running. Occasionally one of our instance of the application is legitimately killed which causes several rounds of rebalancing until the old node is replaced.
Sometimes during the rebalance, one ore more previously healthy nodes fail. The logs are indicating that the Streams application transitions into a PENDING_SHUTDOWN state directly after receiving the following exception:
java.lang.IllegalStateException: No current assignment for partition public.chat.message-28
at org.apache.kafka.clients.consumer.internals.SubscriptionState.assignedState(SubscriptionState.java:256)
at org.apache.kafka.clients.consumer.internals.SubscriptionState.resetFailed(SubscriptionState.java:418)
at org.apache.kafka.clients.consumer.internals.Fetcher$2.onFailure(Fetcher.java:621)
at org.apache.kafka.clients.consumer.internals.RequestFuture.fireFailure(RequestFuture.java:177)
at org.apache.kafka.clients.consumer.internals.RequestFuture.raise(RequestFuture.java:147)
at org.apache.kafka.clients.consumer.internals.RequestFutureAdapter.onFailure(RequestFutureAdapter.java:30)
at org.apache.kafka.clients.consumer.internals.RequestFuture$1.onFailure(RequestFuture.java:209)
at org.apache.kafka.clients.consumer.internals.RequestFuture.fireFailure(RequestFuture.java:177)
at org.apache.kafka.clients.consumer.internals.RequestFuture.raise(RequestFuture.java:147)
at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient$RequestFutureCompletionHandler.fireCompletion(ConsumerNetworkClient.java:571)
at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.firePendingCompletedRequests(ConsumerNetworkClient.java:389)
at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:297)
at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:236)
at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:215)
at org.apache.kafka.clients.consumer.internals.Fetcher.getTopicMetadata(Fetcher.java:292)
at org.apache.kafka.clients.consumer.internals.Fetcher.getAllTopicMetadata(Fetcher.java:275)
at org.apache.kafka.clients.consumer.KafkaConsumer.listTopics(KafkaConsumer.java:1849)
at org.apache.kafka.clients.consumer.KafkaConsumer.listTopics(KafkaConsumer.java:1827)
at org.apache.kafka.streams.processor.internals.StoreChangelogReader.refreshChangelogInfo(StoreChangelogReader.java:259)
at org.apache.kafka.streams.processor.internals.StoreChangelogReader.initialize(StoreChangelogReader.java:133)
at org.apache.kafka.streams.processor.internals.StoreChangelogReader.restore(StoreChangelogReader.java:79)
at org.apache.kafka.streams.processor.internals.TaskManager.updateNewAndRestoringTasks(TaskManager.java:328)
at org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:866)
at org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:804)
at org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:773)
Prior to this error we often seem to also recieve some informational logs reporting a disconnect exception:
Error sending fetch request (sessionId=568252460, epoch=7) to node 4: org.apache.kafka.common.errors.DisconnectException
I have a feeling the two are related but I'm unable to reason why at present.
Is anyone able to give me some hints as to what may be causing this issue and any possible solutions?
Additional Info:
Kafka 2.2.1
32 partitions spread evenly across the 4 worker nodes
StreamsConfig settings:
kafkaStreamProps.put(StreamsConfig.REPLICATION_FACTOR_CONFIG, 2);
kafkaStreamProps.put(StreamsConfig.NUM_STANDBY_REPLICAS_CONFIG, 1);
kafkaStreamProps.put(StreamsConfig.NUM_STREAM_THREADS_CONFIG, 4);
kafkaStreamProps.put(StreamsConfig.COMMIT_INTERVAL_MS_CONFIG, 120000);
kafkaStreamProps.put(StreamsConfig.TOPOLOGY_OPTIMIZATION, StreamsConfig.OPTIMIZE);
This looks like it could be related to https://issues.apache.org/jira/browse/KAFKA-9073, which has been fixed in Kafka Streams 2.3.2.
If you can't wait for that release, you could try creating a private build using the changeset from this pull request: https://github.com/apache/kafka/pull/7630/files
I am using lagom for building a, small sample application. I am following the structure of "hello-world" lagom sample application. But whenever I am trying to run the application, I am facing, bellow error:
[error] a.c.s.PersistentShardCoordinator - Persistence failure when replaying events for persistenceId [/sharding/TwitterEntityCoordinator]. Last known sequence number [0]
akka.pattern.CircuitBreaker$$anon$1: Circuit Breaker Timed out.
[error] a.c.s.PersistentShardCoordinator - Persistence failure when replaying events for persistenceId [/sharding/kafkaProducer-tweetsCoordinator]. Last known sequence number [0]
Still, I am not getting what is the meaning of this error and to resolve them?? Because, I am using Apache Kafka also, and my messages are successfully stored into topic but in logs, this error show every time.
my actor runs on default akka dispatcher, which then calls a method which returns a future. I have configured different executioncontexts for all futures to run (since they are blocking(due to db calls) and to keep actors dispatcher dedicated to non blocking actors only. Wondering if this code can be tested (continue using two execution contexts etc) using Akka Testkit? If so what would be the way to configure a test so Actor runs on default dispatcher and futures can find "custom-dispatcher" as well for them to run? Obviously currently test throws following.
Caused by: akka.ConfigurationException: Dispatcher [custom-dispatcher] not configured
When you create an Akka Testkit's TestActorRef for an Actor it will use PinnedDispatcher except you've specified a different one in Actor's Props and passed that Props when creating the TestActorRef.
The exception "Dispatcher [custom-dispatcher] not configured" may mean that you are using different Akka config for your tests in which no dispatcher with name [custom-dispatcher] configured.
create a file application.conf in your test/resources directory
my-custom-dispatcher {
executor = "thread-pool-executor"
type = PinnedDispatcher
}
then in your test when you create the actor
val boothWorker = system.actorOf(Props(classOf[WorkerTest]))
.withDispatcher("my-custom-dispatcher"))