few kafka stream tasks stopped running in a stream application - apache-kafka

Question

We have a Stream Topology with 2 source topics, 3 processors, 5 state stores and 4 sink topics.
Those two topics have 12 partitions.
Our application is running in kubernetes. It has three pods and so we have 3 instances of this streams running in one pod each.
When we install the application the rebalancing happens and the stream instances in the three pod goes to running state with 4 tasks each.
These streams work fine consuming events, updating the state store and producing messages through sinks.
But some times message consumption from some partitions is stopped in one or two stream instances.Message consumption of other partitions in the same instance goes fine. So I think that the corresponding tasks suddenly stopped running in those pods. There is no warning or error messages in streams client's log files.
Edited: Added Section:
As there is no consumer join request or rebalance triggered I think the tasks are running but some how no records are consumed.
There are two log messages after this.
17:43:44.105 [kafka-admin-client-thread | XXXXX-admin] INFO o.apache.kafka.clients.NetworkClient - [AdminClient clientId=XXXXX-admin] Node -1 disconnected.
17:42:45.050 [XXXXX-StreamThread-1] INFO o.a.k.s.p.internals.StreamThread - stream-thread [XXXXX-StreamThread-1] Processed 11036 total records, ran 0 punctuators, and committed 45 total tasks since the last update
what could be the reason for that? I could not find the root cause and am out of ideas. Could some one please help.
we are using kafka-streams-3.1.1.
we have the config: session.timeout.ms: 10000
StreamsConfig values:
acceptable.recovery.lag = 10000
application.id = XXXXX
application.server =
bootstrap.servers = XXXXX
buffered.records.per.partition = 1000
built.in.metrics.version = latest
cache.max.bytes.buffering = 10485760
client.id =
commit.interval.ms = 1000
connections.max.idle.ms = 540000
default.deserialization.exception.handler = class org.apache.kafka.streams.errors.LogAndContinueExceptionHandler
default.key.serde = class org.apache.kafka.common.serialization.Serdes$StringSerde
default.list.key.serde.inner = null
default.list.key.serde.type = null
default.list.value.serde.inner = null
default.list.value.serde.type = null
default.production.exception.handler = class org.apache.kafka.streams.errors.DefaultProductionExceptionHandler
default.timestamp.extractor = class org.apache.kafka.streams.processor.FailOnInvalidTimestamp
default.value.serde = null
max.task.idle.ms = 0
max.warmup.replicas = 2
metadata.max.age.ms = 300000
metric.reporters = []
metrics.num.samples = 2
metrics.recording.level = INFO
metrics.sample.window.ms = 30000
num.standby.replicas = 0
num.stream.threads = 1
poll.ms = 50
probing.rebalance.interval.ms = 600000
processing.guarantee = at_least_once
receive.buffer.bytes = 32768
reconnect.backoff.max.ms = 1000
reconnect.backoff.ms = 50
replication.factor = 3
request.timeout.ms = 40000
retries = 0
retry.backoff.ms = 100
rocksdb.config.setter = null
security.protocol = SASL_SSL
send.buffer.bytes = 131072
state.cleanup.delay.ms = 600000
state.dir = /tmp/kafka-streams
task.timeout.ms = 300000
topology.optimization = none
upgrade.from = null
window.size.ms = null
windowed.inner.class.serde = null
windowstore.changelog.additional.retention.ms = 86400000