Kafka stop starting correctly - apache-kafka

Yesterday I sent a lot of messages to Kafka and trying to consume them and everything worked fine but today when I'm starting server it gets shutdown. I have no idea why it's happening because I still learning about Kafka. This is what appears in the server console (I notice repeating lines for offset from 0 to 49):
INFO [Log partition=Ranking-0, dir=C:\kafka-logs\kafka-logs-0] Completed load of log with 1 segments, log start offset 0 and log end offset 5 in 120 ms (kafka.log.Log)
[2018-06-17 12:18:08,379] WARN [Log partition=__consumer_offsets-0, dir=C:\kafka-logs\kafka-logs-0] Found a corrupted index file corresponding to log file C:\kafka-logs\kafka-logs-0\__consumer_offsets-0\00000000000000000000.log due to Corrupt index found, index file (C:\kafka-logs\kafka-logs-0\__consumer_offsets-0\00000000000000000000.index) has non-zero size but the last offset is 0 which is no greater than the base offset 0.}, recovering segment and rebuilding index files... (kafka.log.Log)
[2018-06-17 12:18:08,383] INFO [Log partition=__consumer_offsets-0, dir=C:\kafka-logs\kafka-logs-0] Recovering unflushed segment 0 (kafka.log.Log)
[2018-06-17 12:18:08,394] INFO [Log partition=__consumer_offsets-0, dir=C:\kafka-logs\kafka-logs-0] Loading producer state from offset 0 with message format version 2 (kafka.log.Log)
[2018-06-17 12:18:08,396] INFO [Log partition=__consumer_offsets-0, dir=C:\kafka-logs\kafka-logs-0] Completed load of log with 1 segments, log start offset 0 and log end offset 0 in 23 ms (kafka.log.Log)
And some lines like:
[2018-06-17 12:18:09,170] INFO [ThrottledRequestReaper-Fetch]: Shutdown completed (kafka.server.ClientQuotaManager$ThrottledRequestReaper)
[2018-06-17 12:18:09,170] INFO [ThrottledRequestReaper-Fetch]: Stopped (kafka.server.ClientQuotaManager$ThrottledRequestReaper)
[2018-06-17 12:18:09,170] INFO [ThrottledRequestReaper-Produce]: Shutting down (kafka.server.ClientQuotaManager$ThrottledRequestReaper)
And ending with:
[2018-06-17 12:18:10,173] INFO [ThrottledRequestReaper-Produce]: Stopped (kafka.server.ClientQuotaManager$ThrottledRequestReaper)
[2018-06-17 12:18:10,173] INFO [ThrottledRequestReaper-Produce]: Shutdown completed (kafka.server.ClientQuotaManager$ThrottledRequestReaper)
[2018-06-17 12:18:10,173] INFO [ThrottledRequestReaper-Request]: Shutting down (kafka.server.ClientQuotaManager$ThrottledRequestReaper)
[2018-06-17 12:18:11,175] INFO [ThrottledRequestReaper-Request]: Stopped (kafka.server.ClientQuotaManager$ThrottledRequestReaper)
[2018-06-17 12:18:11,175] INFO [ThrottledRequestReaper-Request]: Shutdown completed (kafka.server.ClientQuotaManager$ThrottledRequestReaper)
[2018-06-17 12:18:11,181] INFO [KafkaServer id=0] shut down completed (kafka.server.KafkaServer)
[2018-06-17 12:18:11,181] ERROR Exiting Kafka. (kafka.server.KafkaServerStartable)
[2018-06-17 12:18:11,183] INFO [KafkaServer id=0] shutting down (kafka.server.KafkaServer)
Below my server properties:
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# see kafka.server.KafkaConfig for additional details and defaults
############################# Server Basics #############################
# The id of the broker. This must be set to a unique integer for each broker.
broker.id=0
############################# Socket Server Settings #############################
# The address the socket server listens on. It will get the value returned from
# java.net.InetAddress.getCanonicalHostName() if not configured.
# FORMAT:
# listeners = listener_name://host_name:port
# EXAMPLE:
# listeners = PLAINTEXT://your.host.name:9092
#listeners=PLAINTEXT://:9092
# Hostname and port the broker will advertise to producers and consumers. If not set,
# it uses the value for "listeners" if configured. Otherwise, it will use the value
# returned from java.net.InetAddress.getCanonicalHostName().
#advertised.listeners=PLAINTEXT://your.host.name:9092
host=9092
# Maps listener names to security protocols, the default is for them to be the same. See the config documentation for more details
#listener.security.protocol.map=PLAINTEXT:PLAINTEXT,SSL:SSL,SASL_PLAINTEXT:SASL_PLAINTEXT,SASL_SSL:SASL_SSL
# The number of threads that the server uses for receiving requests from the network and sending responses to the network
num.network.threads=3
# The number of threads that the server uses for processing requests, which may include disk I/O
num.io.threads=8
# The send buffer (SO_SNDBUF) used by the socket server
socket.send.buffer.bytes=102400
# The receive buffer (SO_RCVBUF) used by the socket server
socket.receive.buffer.bytes=102400
# The maximum size of a request that the socket server will accept (protection against OOM)
socket.request.max.bytes=104857600
############################# Log Basics #############################
# A comma separated list of directories under which to store log files
log.dirs=C:/kafka-logs/kafka-logs-0
# The default number of log partitions per topic. More partitions allow greater
# parallelism for consumption, but this will also result in more files across
# the brokers.
num.partitions=1
# The number of threads per data directory to be used for log recovery at startup and flushing at shutdown.
# This value is recommended to be increased for installations with data dirs located in RAID array.
num.recovery.threads.per.data.dir=1
############################# Internal Topic Settings #############################
# The replication factor for the group metadata internal topics "__consumer_offsets" and "__transaction_state"
# For anything other than development testing, a value greater than 1 is recommended for to ensure availability such as 3.
offsets.topic.replication.factor=1
transaction.state.log.replication.factor=1
transaction.state.log.min.isr=1
############################# Log Flush Policy #############################
# Messages are immediately written to the filesystem but by default we only fsync() to sync
# the OS cache lazily. The following configurations control the flush of data to disk.
# There are a few important trade-offs here:
# 1. Durability: Unflushed data may be lost if you are not using replication.
# 2. Latency: Very large flush intervals may lead to latency spikes when the flush does occur as there will be a lot of data to flush.
# 3. Throughput: The flush is generally the most expensive operation, and a small flush interval may lead to excessive seeks.
# The settings below allow one to configure the flush policy to flush data after a period of time or
# every N messages (or both). This can be done globally and overridden on a per-topic basis.
# The number of messages to accept before forcing a flush of data to disk
#log.flush.interval.messages=10000
# The maximum amount of time a message can sit in a log before we force a flush
#log.flush.interval.ms=1000
############################# Log Retention Policy #############################
# The following configurations control the disposal of log segments. The policy can
# be set to delete segments after a period of time, or after a given size has accumulated.
# A segment will be deleted whenever *either* of these criteria are met. Deletion always happens
# from the end of the log.
# The minimum age of a log file to be eligible for deletion due to age
log.retention.hours=168
# A size-based retention policy for logs. Segments are pruned from the log unless the remaining
# segments drop below log.retention.bytes. Functions independently of log.retention.hours.
#log.retention.bytes=1073741824
# The maximum size of a log segment file. When this size is reached a new log segment will be created.
log.segment.bytes=1073741824
# The interval at which log segments are checked to see if they can be deleted according
# to the retention policies
log.retention.check.interval.ms=300000
############################# Zookeeper #############################
# Zookeeper connection string (see zookeeper docs for details).
# This is a comma separated host:port pairs, each corresponding to a zk
# server. e.g. "127.0.0.1:3000,127.0.0.1:3001,127.0.0.1:3002".
# You can also append an optional chroot string to the urls to specify the
# root directory for all kafka znodes.
zookeeper.connect=localhost:2181
# Timeout in ms for connecting to zookeeper
zookeeper.connection.timeout.ms=6000
############################# Group Coordinator Settings #############################
# The following configuration specifies the time, in milliseconds, that the GroupCoordinator will delay the initial consumer rebalance.
# The rebalance will be further delayed by the value of group.initial.rebalance.delay.ms as new members join the group, up to a maximum of max.poll.interval.ms.
# The default value for this is 3 seconds.
# We override this to 0 here as it makes for a better out-of-the-box experience for development and testing.
# However, in production environments the default value of 3 seconds is more suitable as this will help to avoid unnecessary, and potentially expensive, rebalances during application startup.
group.initial.rebalance.delay.ms=0
delete.topic.enable=true

Related

Apache Kafka Server Stops Automatically after some time

I am using apache Kafka for one of the projects. Kafka server stops automatically after I kept it running for 1 day. Can anyone tell me which config property I need to modify?
Its works after deleting log files for some time and the same issue repeats continuously.
can anyone please provide a solution for this?
Kafka server.properties
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# see kafka.server.KafkaConfig for additional details and defaults
############################# Server Basics #############################
# The id of the broker. This must be set to a unique integer for each broker.
broker.id=0
############################# Socket Server Settings #############################
# The address the socket server listens on. It will get the value returned from
# java.net.InetAddress.getCanonicalHostName() if not configured.
# FORMAT:
# listeners = listener_name://host_name:port
# EXAMPLE:
# listeners = PLAINTEXT://your.host.name:9092
listeners=PLAINTEXT://:9092
# Hostname and port the broker will advertise to producers and consumers. If not set,
# it uses the value for "listeners" if configured. Otherwise, it will use the value
# returned from java.net.InetAddress.getCanonicalHostName().
#advertised.listeners=PLAINTEXT://your.host.name:9092
advertised.listeners=PLAINTEXT://*.*.*.*:9092
# Maps listener names to security protocols, the default is for them to be the same. See the config documentation for more details
#listener.security.protocol.map=PLAINTEXT:PLAINTEXT,SSL:SSL,SASL_PLAINTEXT:SASL_PLAINTEXT,SASL_SSL:SASL_SSL
# The number of threads that the server uses for receiving requests from the network and sending responses to the network
num.network.threads=3
# The number of threads that the server uses for processing requests, which may include disk I/O
num.io.threads=8
# The send buffer (SO_SNDBUF) used by the socket server
socket.send.buffer.bytes=102400
# The receive buffer (SO_RCVBUF) used by the socket server
socket.receive.buffer.bytes=102400
# The maximum size of a request that the socket server will accept (protection against OOM)
socket.request.max.bytes=104857600
############################# Log Basics #############################
# A comma separated list of directories under which to store log files
log.dirs=/centos/home/kafka/tmp/kafka-logs
# The default number of log partitions per topic. More partitions allow greater
# parallelism for consumption, but this will also result in more files across
# the brokers.
num.partitions=1
# The number of threads per data directory to be used for log recovery at startup and flushing at shutdown.
# This value is recommended to be increased for installations with data dirs located in RAID array.
num.recovery.threads.per.data.dir=1
############################# Internal Topic Settings #############################
# The replication factor for the group metadata internal topics "__consumer_offsets" and "__transaction_state"
# For anything other than development testing, a value greater than 1 is recommended to ensure availability such as 3.
offsets.topic.replication.factor=1
transaction.state.log.replication.factor=1
transaction.state.log.min.isr=1
############################# Log Flush Policy #############################
# Messages are immediately written to the filesystem but by default we only fsync() to sync
# the OS cache lazily. The following configurations control the flush of data to disk.
# There are a few important trade-offs here:
# 1. Durability: Unflushed data may be lost if you are not using replication.
# 2. Latency: Very large flush intervals may lead to latency spikes when the flush does occur as there will be a lot of data to flush.
# 3. Throughput: The flush is generally the most expensive operation, and a small flush interval may lead to excessive seeks.
# The settings below allow one to configure the flush policy to flush data after a period of time or
# every N messages (or both). This can be done globally and overridden on a per-topic basis.
# The number of messages to accept before forcing a flush of data to disk
#log.flush.interval.messages=10000
# The maximum amount of time a message can sit in a log before we force a flush
#log.flush.interval.ms=1000
############################# Log Retention Policy #############################
# The following configurations control the disposal of log segments. The policy can
# be set to delete segments after a period of time, or after a given size has accumulated.
# A segment will be deleted whenever *either* of these criteria are met. Deletion always happens
# from the end of the log.
# The minimum age of a log file to be eligible for deletion due to age
log.retention.hours=168
# A size-based retention policy for logs. Segments are pruned from the log unless the remaining
# segments drop below log.retention.bytes. Functions independently of log.retention.hours.
#log.retention.bytes=1073741824
# The maximum size of a log segment file. When this size is reached a new log segment will be created.
log.segment.bytes=1073741824
# The interval at which log segments are checked to see if they can be deleted according
# to the retention policies
log.retention.check.interval.ms=300000
############################# Zookeeper #############################
# Zookeeper connection string (see zookeeper docs for details).
# This is a comma separated host:port pairs, each corresponding to a zk
# server. e.g. "127.0.0.1:3000,127.0.0.1:3001,127.0.0.1:3002".
# You can also append an optional chroot string to the urls to specify the
# root directory for all kafka znodes.
zookeeper.connect=localhost:2181
# Timeout in ms for connecting to zookeeper
zookeeper.connection.timeout.ms=18000
############################# Group Coordinator Settings #############################
# The following configuration specifies the time, in milliseconds, that the GroupCoordinator will delay the initial consumer rebalance.
# The rebalance will be further delayed by the value of group.initial.rebalance.delay.ms as new members join the group, up to a maximum of max.poll.interval.ms.
# The default value for this is 3 seconds.
# We override this to 0 here as it makes for a better out-of-the-box experience for development and testing.
# However, in production environments the default value of 3 seconds is more suitable as this will help to avoid unnecessary, and potentially expensive, rebalances during application startup.
group.initial.rebalance.delay.ms=0
and zookeeper.properties
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# the directory where the snapshot is stored.
dataDir=/tmp/zookeeper
# the port at which the clients will connect
clientPort=2181
# disable the per-ip limit on the number of connections since this is a non-production config
maxClientCnxns=0
# Disable the adminserver by default to avoid port conflicts.
# Set the port to something non-conflicting if choosing to enable this
admin.enableServer=false
# admin.serverPort=8080
And Serverlog file has below message
[2021-11-16 06:25:07,521] WARN Unexpected exception (org.apache.zookeeper.server.NIOServerCnxn)
EndOfStreamException: Unable to read additional data from client, it probably closed the socket: address = /0:0:0:0:0:0:0:1:52984, session = 0x1005ceb08740001
at org.apache.zookeeper.server.NIOServerCnxn.handleFailedRead(NIOServerCnxn.java:163)
at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:326)
at org.apache.zookeeper.server.NIOServerCnxnFactory$IOWorkRequest.doWork(NIOServerCnxnFactory.java:522)
at org.apache.zookeeper.server.WorkerService$ScheduledWorkRequest.run(WorkerService.java:154)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
[2021-11-16 06:25:07,537] INFO Expiring session 0x1005ceb08740001, timeout of 18000ms exceeded (org.apache.zookeeper.server.ZooKeeperServer)
[2021-11-16 06:25:58,618] INFO Registered kafka:type=kafka.Log4jController MBean (kafka.utils.Log4jControllerRegistration$)
[2021-11-16 06:25:59,324] INFO Setting -D jdk.tls.rejectClientInitiatedRenegotiation=true to disable client-initiated TLS renegotiation (org.apache.zookeeper.common.X509Util)
[2021-11-16 06:25:59,488] INFO Registered signal handlers for TERM, INT, HUP (org.apache.kafka.common.utils.LoggingSignalHandler)
[2021-11-16 06:25:59,492] INFO starting (kafka.server.KafkaServer)
[2021-11-16 06:25:59,499] INFO Connecting to zookeeper on localhost:2181 (kafka.server.KafkaServer)
[2021-11-16 06:25:59,520] INFO [ZooKeeperClient Kafka server] Initializing a new session to localhost:2181. (kafka.zookeeper.ZooKeeperClient)
[2021-11-16 06:25:59,533] INFO Client environment:zookeeper.version=3.6.3--6401e4ad2087061bc6b9f80dec2d69f2e3c8660a, built on 04/08/2021 16:35 GMT (org.apache.zookeeper.ZooKeeper)
[2021-11-16 06:25:59,533] INFO Client environment:host.name=ip-172-31-32-15.ap-south-1.compute.internal (org.apache.zookeeper.ZooKeeper)
[2021-11-16 06:25:59,533] INFO Client environment:java.version=1.8.0_302 (org.apache.zookeeper.ZooKeeper)
[2021-11-16 06:25:59,533] INFO Client environment:java.vendor=Red Hat, Inc. (org.apache.zookeeper.ZooKeeper)
[2021-11-16 06:25:59,533] INFO Client environment:java.home=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.302.b08-0.el8_4.x86_64/jre (org.apache.zookeeper.ZooKeeper)

kafka is not retaining 7 days of logs

I am using kafka2.11 i am pasting my server.properties.I have 3 nodes of kafka cluster
############################# Server Basics #############################
# The id of the broker. This must be set to a unique integer for each broker.
broker.id=1
#broker.rack=1
############################# Socket Server Settings #############################
# Hostname and port the broker will advertise to producers and consumers. If not set,
# it uses the value for "listeners" if configured. Otherwise, it will use the value
# returned from java.net.InetAddress.getCanonicalHostName().
advertised.listeners=PLAINTEXT://xx.xx.xx.xx:9092
# The number of threads that the server uses for processing requests, which may include disk I/O
num.io.threads=8
# The send buffer (SO_SNDBUF) used by the socket server
socket.send.buffer.bytes=102400
# The receive buffer (SO_RCVBUF) used by the socket server
socket.receive.buffer.bytes=102400
# The maximum size of a request that the socket server will accept (protection against OOM)
socket.request.max.bytes=104857600
############################# Log Basics #############################
# A comma separated list of directories under which to store log files
log.dirs=/home/kafka_data/
# The number of threads per data directory to be used for log recovery at startup and flushing at shutdown.
# This value is recommended to be increased for installations with data dirs located in RAID array.
num.recovery.threads.per.data.dir=1
num.replica.fetchers=1
############################# Internal Topic Settings #############################
# The replication factor for the group metadata internal topics "__consumer_offsets" and "__transaction_state"
# For anything other than development testing, a value greater than 1 is recommended to ensure availability such as 3.
offsets.topic.replication.factor=3
transaction.state.log.replication.factor=3
transaction.state.log.min.isr=3
############################# Log Flush Policy #############################
#log.flush.interval.messages=10000
# The maximum amount of time a message can sit in a log before we force a flush
#log.flush.interval.ms=1000
############################# Log Retention Policy #############################
# The minimum age of a log file to be eligible for deletion due to age
log.retention.hours=168
# A size-based retention policy for logs. Segments are pruned from the log unless the remaining
# segments drop below log.retention.bytes. Functions independently of log.retention.hours.
log.retention.bytes=1000000
# The maximum size of a log segment file. When this size is reached a new log segment will be created.
log.segment.bytes=1073741824
# The interval at which log segments are checked to see if they can be deleted according
# to the retention policies
log.retention.check.interval.ms=300000
############################# Zookeeper #############################
# Zookeeper connection string (see zookeeper docs for details).
# This is a comma separated host:port pairs, each corresponding to a zk
# server. e.g. "127.0.0.1:3000,127.0.0.1:3001,127.0.0.1:3002".
# You can also append an optional chroot string to the urls to specify the
# root directory for all kafka znodes.
zookeeper.connect=xx.xx.xx.xx:2181,xx.xx.xx.xx:2181,xx.xx.xx.xx:2181/kafka
# Timeout in ms for connecting to zookeeper
zookeeper.connection.timeout.ms=18000
############################# Group Coordinator Settings #############################
# The following configuration specifies the time, in milliseconds, that the GroupCoordinator will delay the initial consumer rebalance.
# The rebalance will be further delayed by the value of group.initial.rebalance.delay.ms as new members join the group, up to a maximum of max.poll.interval.ms.
# The default value for this is 3 seconds.
# We override this to 0 here as it makes for a better out-of-the-box experience for development and testing.
# However, in production environments the default value of 3 seconds is more suitable as this will help to avoid unnecessary, and potentially expensive, rebalances during application startup.
group.initial.rebalance.delay.ms=0
###################### Topic configuration ##############################
delete.topic.enable=true
num.partitions=3
default.replication.factor=3
auto.create.topic.enable=false
# Enables automatic leader balancing.
auto.leader.rebalance.enable=true
# The frequency with which the partition rebalance check is triggered by the controller
leader.imbalance.check.interval.seconds=60
#offsets.topic.replication.factor=1
############################## REPLICA SETTING ##########################i#######
# max wait time for each fetcher request issued by follower replicas
replica.fetch.wait.max.ms=300
#replica.fetch.min.bytes=1
# If a follower hasn't sent any fetch requests or hasn't consumed up to the leaders log end offset for at least this time, the leader will remove the follower from isr
replica.lag.time.max.ms=30000
# The socket receive buffer for network requests
replica.socket.receive.buffer.bytes=1000000
# The socket timeout for network requests.
replica.socket.timeout.ms=30000
# The configuration controls the maximum amount of time the client will wait for the response of a request. If the response is not received
# before the timeout elapses the client will resend the request if necessary or fail the request if retries are exhausted.
request.timeout.ms=30000
metadata.max.age.ms=30000
####################### PRODUCER CONFIG ##################################
acks=all
#unclean.leader.election.enable=false
message.max.bytes=1000000000
fetch.message.max.bytes=1000000000
replica.fetch.max.bytes=1000000000
unclean.leader.election.enable=true
#enable.idempotence = true
#message.send.max.retries= 10000000
#max.in.flight.requests.per.connection=1
I have set log.retention.hours=168 that is 7 days retention
i consumed the data by creating new group and add --from-beginning attribute but i've got 4 days of logs.
bin/kafka-consumer-groups.sh --describe --group consumer2 --bootstrap-server lc-helk.nic.in:9092
TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG
ise 2 1114176 1114176 0
ise 0 1113757 1113757 0
ise 1 1113627 1113627 0
In log entry i am seeing 17th July,21 logs it should show 13th july,21 logs
If i want to change this kafka topic configuration how can i change this in existing kafka topic.
Topic retention is configured by 3 properties(besides some others):
log.retention.bytes
log.retention.hours
log.cleanup.policy
I can see that you set log.retention.bytes=1000000 have you tried setting it to -1 ?

Kafka server is not connecting with zookeeper server remotly

I am trying to connect kafka server (on windows system) to Zookeeper server. I am facing
Opening socket connection to server 10.160.10.25/10.160.10.25:2181. Will not attempt to authenticate using SASL (unknown error)
I am not able to understand how can i resolve.I have tried to more relevant question regarding this but didn't get answer. I have attached log and command as well to connect.
Also port is accessible from client
**bin\windows\kafka-server-start.bat config\server.properties**
Log attached below
[2020-05-01 16:25:37,082] INFO Client environment:java.io.tmpdir=C:\Users\Admin\AppData\Local\Temp\ (org.apache.zookeeper.ZooKeeper)
[2020-05-01 16:25:37,083] INFO Client environment:java.compiler=<NA> (org.apache.zookeeper.ZooKeeper)
[2020-05-01 16:25:37,083] INFO Client environment:os.name=Windows 7 (org.apache.zookeeper.ZooKeeper)
[2020-05-01 16:25:37,084] INFO Client environment:os.arch=amd64 (org.apache.zookeeper.ZooKeeper)
[2020-05-01 16:25:37,084] INFO Client environment:os.version=6.1 (org.apache.zookeeper.ZooKeeper)
[2020-05-01 16:25:37,085] INFO Client environment:user.name=Admin (org.apache.zookeeper.ZooKeeper)
[2020-05-01 16:25:37,085] INFO Client environment:user.home=C:\Users\Admin (org.apache.zookeeper.ZooKeeper)
[2020-05-01 16:25:37,086] INFO Client environment:user.dir=D:\Software\kafka_2.12-2.4.1 (org.apache.zookeeper.ZooKeeper)
[2020-05-01 16:25:37,086] INFO Client environment:os.memory.free=976MB (org.apache.zookeeper.ZooKeeper)
[2020-05-01 16:25:37,087] INFO Client environment:os.memory.max=1024MB (org.apache.zookeeper.ZooKeeper)
[2020-05-01 16:25:37,087] INFO Client environment:os.memory.total=1024MB (org.apache.zookeeper.ZooKeeper)
[2020-05-01 16:25:37,099] INFO Initiating client connection, connectString=X.X.X.X:2181 sessionTimeout=6000 watcher=kafka.zookeeper.ZooKeeperClient$ZooKeep
erClientWatcher$#31c88ec8 (org.apache.zookeeper.ZooKeeper)
[2020-05-01 16:25:37,192] INFO Setting -D jdk.tls.rejectClientInitiatedRenegotiation=true to disable client-initiated TLS renegotiation (org.apache.zookeeper.co
mmon.X509Util)
[2020-05-01 16:25:37,357] INFO jute.maxbuffer value is 4194304 Bytes (org.apache.zookeeper.ClientCnxnSocket)
[2020-05-01 16:25:37,393] INFO zookeeper.request.timeout value is 0. feature enabled= (org.apache.zookeeper.ClientCnxn)
[2020-05-01 16:25:37,402] INFO [ZooKeeperClient Kafka server] Waiting until connected. (kafka.zookeeper.ZooKeeperClient)
[2020-05-01 16:25:43,416] INFO [ZooKeeperClient Kafka server] Closing. (kafka.zookeeper.ZooKeeperClient)
[2020-05-01 16:25:46,564] INFO Opening socket connection to server XX.XX.XX.XX/XX.XX.XX.XX:2181. Will not attempt to authenticate using SASL (unknown error) (
org.apache.zookeeper.ClientCnxn)
[2020-05-01 16:25:46,569] WARN Client session timed out, have not heard from server in 9172ms for sessionid 0x0 (org.apache.zookeeper.ClientCnxn)
[2020-05-01 16:25:46,699] INFO Session: 0x0 closed (org.apache.zookeeper.ZooKeeper)
[2020-05-01 16:25:46,706] INFO [ZooKeeperClient Kafka server] Closed. (kafka.zookeeper.ZooKeeperClient)
[2020-05-01 16:25:46,711] INFO EventThread shut down for session: 0x0 (org.apache.zookeeper.ClientCnxn)
[2020-05-01 16:25:46,724] ERROR Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)
kafka.zookeeper.ZooKeeperClientTimeoutException: Timed out waiting for connection while in state: CONNECTING
at kafka.zookeeper.ZooKeeperClient.$anonfun$waitUntilConnected$3(ZooKeeperClient.scala:259)
at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:253)
at kafka.zookeeper.ZooKeeperClient.waitUntilConnected(ZooKeeperClient.scala:255)
at kafka.zookeeper.ZooKeeperClient.<init>(ZooKeeperClient.scala:113)
at kafka.zk.KafkaZkClient$.apply(KafkaZkClient.scala:1858)
at kafka.server.KafkaServer.createZkClient$1(KafkaServer.scala:375)
at kafka.server.KafkaServer.initZkClient(KafkaServer.scala:399)
at kafka.server.KafkaServer.startup(KafkaServer.scala:207)
at kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:44)
at kafka.Kafka$.main(Kafka.scala:84)
at kafka.Kafka.main(Kafka.scala)
[2020-05-01 16:25:46,749] INFO shutting down (kafka.server.KafkaServer)
[2020-05-01 16:25:46,844] INFO shut down completed (kafka.server.KafkaServer)
[2020-05-01 16:25:46,886] ERROR Exiting Kafka. (kafka.server.KafkaServerStartable)
[2020-05-01 16:25:46,993] INFO shutting down (kafka.server.KafkaServer)
zookeeper.properties
icensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# the directory where the snapshot is stored.
dataDir=/home/weaver/app_server/zookeeper
# the port at which the clients will connect
clientPort=2181
# disable the per-ip limit on the number of connections since this is a non-production config
maxClientCnxns=0
# Disable the adminserver by default to avoid port conflicts.
# Set the port to something non-conflicting if choosing to enable this
admin.enableServer=false
# admin.serverPort=8080
server.properties
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# see kafka.server.KafkaConfig for additional details and defaults
############################# Server Basics #############################
# The id of the broker. This must be set to a unique integer for each broker.
broker.id=0
############################# Socket Server Settings #############################
# The address the socket server listens on. It will get the value returned from
# java.net.InetAddress.getCanonicalHostName() if not configured.
# FORMAT:
# listeners = listener_name://host_name:port
# EXAMPLE:
# listeners = PLAINTEXT://your.host.name:9092
#listeners=PLAINTEXT://:9092
# Hostname and port the broker will advertise to producers and consumers. If not set,
# it uses the value for "listeners" if configured. Otherwise, it will use the value
# returned from java.net.InetAddress.getCanonicalHostName().
#advertised.listeners=PLAINTEXT://your.host.name:9092
# Maps listener names to security protocols, the default is for them to be the same. See the config documentation for more details
#listener.security.protocol.map=PLAINTEXT:PLAINTEXT,SSL:SSL,SASL_PLAINTEXT:SASL_PLAINTEXT,SASL_SSL:SASL_SSL
# The number of threads that the server uses for receiving requests from the network and sending responses to the network
num.network.threads=3
# The number of threads that the server uses for processing requests, which may include disk I/O
num.io.threads=8
# The send buffer (SO_SNDBUF) used by the socket server
socket.send.buffer.bytes=102400
# The receive buffer (SO_RCVBUF) used by the socket server
socket.receive.buffer.bytes=102400
# The maximum size of a request that the socket server will accept (protection against OOM)
socket.request.max.bytes=104857600
############################# Log Basics #############################
# A comma separated list of directories under which to store log files
log.dirs=/tmp/kafka-logs
# The default number of log partitions per topic. More partitions allow greater
# parallelism for consumption, but this will also result in more files across
# the brokers.
num.partitions=1
# The number of threads per data directory to be used for log recovery at startup and flushing at shutdown.
# This value is recommended to be increased for installations with data dirs located in RAID array.
num.recovery.threads.per.data.dir=1
############################# Internal Topic Settings #############################
# The replication factor for the group metadata internal topics "__consumer_offsets" and "__transaction_state"
# For anything other than development testing, a value greater than 1 is recommended to ensure availability such as 3.
offsets.topic.replication.factor=1
transaction.state.log.replication.factor=1
transaction.state.log.min.isr=1
############################# Log Flush Policy #############################
# Messages are immediately written to the filesystem but by default we only fsync() to sync
# the OS cache lazily. The following configurations control the flush of data to disk.
# There are a few important trade-offs here:
# 1. Durability: Unflushed data may be lost if you are not using replication.
# 2. Latency: Very large flush intervals may lead to latency spikes when the flush does occur as there will be a lot of data to flush.
# 3. Throughput: The flush is generally the most expensive operation, and a small flush interval may lead to excessive seeks.
# The settings below allow one to configure the flush policy to flush data after a period of time or
# every N messages (or both). This can be done globally and overridden on a per-topic basis.
# The number of messages to accept before forcing a flush of data to disk
#log.flush.interval.messages=10000
# The maximum amount of time a message can sit in a log before we force a flush
#log.flush.interval.ms=1000
############################# Log Retention Policy #############################
# The following configurations control the disposal of log segments. The policy can
# be set to delete segments after a period of time, or after a given size has accumulated.
# A segment will be deleted whenever *either* of these criteria are met. Deletion always happens
# from the end of the log.
# The minimum age of a log file to be eligible for deletion due to age
log.retention.hours=168
# A size-based retention policy for logs. Segments are pruned from the log unless the remaining
# segments drop below log.retention.bytes. Functions independently of log.retention.hours.
#log.retention.bytes=1073741824
# The maximum size of a log segment file. When this size is reached a new log segment will be created.
log.segment.bytes=1073741824
# The interval at which log segments are checked to see if they can be deleted according
# to the retention policies
log.retention.check.interval.ms=300000
############################# Zookeeper #############################
# Zookeeper connection string (see zookeeper docs for details).
# This is a comma separated host:port pairs, each corresponding to a zk
# server. e.g. "127.0.0.1:3000,127.0.0.1:3001,127.0.0.1:3002".
# You can also append an optional chroot string to the urls to specify the
# root directory for all kafka znodes.
zookeeper.connect=10.160.10.25:2181
# Timeout in ms for connecting to zookeeper
zookeeper.connection.timeout.ms=6000
############################# Group Coordinator Settings #############################
# The following configuration specifies the time, in milliseconds, that the GroupCoordinator will delay the initial consumer rebalance.
# The rebalance will be further delayed by the value of group.initial.rebalance.delay.ms as new members join the group, up to a maximum of max.poll.interval.ms.
# The default value for this is 3 seconds.
# We override this to 0 here as it makes for a better out-of-the-box experience for development and testing.
# However, in production environments the default value of 3 seconds is more suitable as this will help to avoid unnecessary, and potentially expensive, rebalances during application startup.
group.initial.rebalance.delay.ms=0

How to run confluent-5.3.2-2.12 platform?

Environment:
CentOS7
openjdk version "1.8.0_181"
I downloaded confluent-5.3.2-2.12.tar.gz and extracted to /opt/confluent.
I am following "Installing and Running KSQL | Level Up your KSQL by Confluent" (https://youtu.be/icwHpPm-TCA).
Executed the following commands:
[root#srvr0 ~]# cd /opt/confluent/confluent-5.3.2/bin/
[root#srvr0 bin]# confluent start
bash: confluent: command not found...
Update1:
With reference to, https://docs.confluent.io/current/quickstart/ce-quickstart.html, executed the following commands:
curl -L https://cnfl.io/cli | sh -s -- -b /opt/confluent/confluent-5.3.2/bin
/opt/confluent/confluent-5.3.2/bin/confluent-hub install --no-prompt confluentinc/kafka-connect-datagen:latest
/opt/confluent/confluent-5.3.2/bin/confluent local start
Logs:
[root#srvr0 ~]# curl -L https://cnfl.io/cli | sh -s -- -b /opt/confluent/confluent-5.3.2/bin
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 162 0 0 0 0 0 0 --:--:-- 0:00:01 --:--:-- 0
100 10288 100 10288 0 0 3567 0 0:00:02 0:00:02 --:--:-- 16176
confluentinc/cli info checking S3 for latest tag
confluentinc/cli info found version: latest for latest/linux/amd64
confluentinc/cli info NOTICE: see licenses located in /tmp/tmp.h8m7jASeAh/confluent
confluentinc/cli info installed /opt/confluent/confluent-5.3.2/bin/confluent
confluentinc/cli info please ensure /opt/confluent/confluent-5.3.2/bin is in your PATH
[root#srvr0 ~]# cp /tmp/tmp.h8m7jASeAh/confluent
cp: missing destination file operand after ‘/tmp/tmp.h8m7jASeAh/confluent’
Try 'cp --help' for more information.
[root#srvr0 ~]# cp -a /tmp/tmp.h8m7jASeAh/confluent /opt/confluent
[root#srvr0 ~]# /opt/confluent/confluent-5.3.2/bin/confluent-hub install --no-prompt confluentinc/kafka-connect-datagen:latest
Running in a "--no-prompt" mode
Implicit acceptance of the license below:
Apache License 2.0
https://www.apache.org/licenses/LICENSE-2.0
Downloading component Kafka Connect Datagen 0.2.0, provided by Confluent, Inc. from Confluent Hub and installing into /opt/confluent/confluent-5.3.2/share/confluent-hub-components
Adding installation directory to plugin path in the following files:
/opt/confluent/confluent-5.3.2/etc/kafka/connect-distributed.properties
/opt/confluent/confluent-5.3.2/etc/kafka/connect-standalone.properties
/opt/confluent/confluent-5.3.2/etc/schema-registry/connect-avro-distributed.properties
/opt/confluent/confluent-5.3.2/etc/schema-registry/connect-avro-standalone.properties
Completed
[root#srvr0 ~]# /opt/confluent/confluent-5.3.2/bin/confluent local start
The local commands are intended for a single-node development environment
only, NOT for production usage. https://docs.confluent.io/current/cli/index.html
Using CONFLUENT_CURRENT: /tmp/confluent.R3YJZ2UC
Starting zookeeper
zookeeper is [UP]
Starting kafka
kafka is [UP]
Starting schema-registry
schema-registry is [UP]
Starting kafka-rest
kafka-rest is [UP]
Starting connect
connect is [UP]
Starting ksql-server
ksql-server is [UP]
Starting control-center
|control-center failed to start
control-center is [DOWN]
Update2:
Logs:
[root#srvr0 confluent-5.3.2]# cat ./logs/controller.log
[2020-01-16 12:20:40,220] DEBUG preRegister called. Server=com.sun.jmx.mbeanserver.JmxMBeanServer#66d3c617, name=log4j:logger=kafka.controller (kafka.controller)
[2020-01-16 12:21:40,097] INFO [ControllerEventThread controllerId=0] Starting (kafka.controller.ControllerEventManager$ControllerEventThread)
[2020-01-16 12:21:40,174] INFO [Controller id=0] 0 successfully elected as the controller. Epoch incremented to 1 and epoch zk version is now 1 (kafka.controller.KafkaController)
[2020-01-16 12:21:40,176] INFO [Controller id=0] Registering handlers (kafka.controller.KafkaController)
[2020-01-16 12:21:40,182] INFO [Controller id=0] Deleting log dir event notifications (kafka.controller.KafkaController)
[2020-01-16 12:21:40,193] INFO [Controller id=0] Deleting isr change notifications (kafka.controller.KafkaController)
[2020-01-16 12:21:40,197] INFO [Controller id=0] Initializing controller context (kafka.controller.KafkaController)
[2020-01-16 12:21:40,361] INFO [Controller id=0] Initialized broker epochs cache: Map(0 -> 24) (kafka.controller.KafkaController)
[2020-01-16 12:21:40,370] DEBUG [Controller id=0] Register BrokerModifications handler for Set(0) (kafka.controller.KafkaController)
[2020-01-16 12:21:40,384] DEBUG [Channel manager on controller 0]: Controller 0 trying to connect to broker 0 (kafka.controller.ControllerChannelManager)
[2020-01-16 12:21:40,444] INFO [RequestSendThread controllerId=0] Starting (kafka.controller.RequestSendThread)
[2020-01-16 12:21:40,445] INFO [Controller id=0] Partitions being reassigned: Map() (kafka.controller.KafkaController)
[2020-01-16 12:21:40,447] INFO [Controller id=0] Currently active brokers in the cluster: Set(0) (kafka.controller.KafkaController)
[2020-01-16 12:21:40,448] INFO [Controller id=0] Currently shutting brokers in the cluster: Set() (kafka.controller.KafkaController)
[2020-01-16 12:21:40,448] INFO [Controller id=0] Current list of topics in the cluster: Set() (kafka.controller.KafkaController)
[2020-01-16 12:21:40,449] INFO [Controller id=0] Fetching topic deletions in progress (kafka.controller.KafkaController)
[2020-01-16 12:21:40,456] INFO [Controller id=0] List of topics to be deleted: (kafka.controller.KafkaController)
[2020-01-16 12:21:40,456] INFO [Controller id=0] List of topics ineligible for deletion: (kafka.controller.KafkaController)
[2020-01-16 12:21:40,457] INFO [Controller id=0] Initializing topic deletion manager (kafka.controller.KafkaController)
[2020-01-16 12:21:40,458] INFO [Topic Deletion Manager 0] Initializing manager with initial deletions: Set(), initial ineligible deletions: Set() (kafka.controller.TopicDeletionManager)
[2020-01-16 12:21:40,459] INFO [Controller id=0] Sending update metadata request (kafka.controller.KafkaController)
[2020-01-16 12:21:40,485] INFO [ReplicaStateMachine controllerId=0] Initializing replica state (kafka.controller.ZkReplicaStateMachine)
[2020-01-16 12:21:40,487] INFO [ReplicaStateMachine controllerId=0] Triggering online replica state changes (kafka.controller.ZkReplicaStateMachine)
[2020-01-16 12:21:40,518] INFO [ReplicaStateMachine controllerId=0] Triggering offline replica state changes (kafka.controller.ZkReplicaStateMachine)
[2020-01-16 12:21:40,519] DEBUG [ReplicaStateMachine controllerId=0] Started replica state machine with initial state -> Map() (kafka.controller.ZkReplicaStateMachine)
[2020-01-16 12:21:40,523] INFO [PartitionStateMachine controllerId=0] Initializing partition state (kafka.controller.ZkPartitionStateMachine)
[2020-01-16 12:21:40,525] INFO [PartitionStateMachine controllerId=0] Triggering online partition state changes (kafka.controller.ZkPartitionStateMachine)
[2020-01-16 12:21:40,535] WARN [RequestSendThread controllerId=0] Controller 0's connection to broker srvr0:9092 (id: 0 rack: null) was unsuccessful (kafka.controller.RequestSendThread)
java.io.IOException: Connection to srvr0:9092 (id: 0 rack: null) failed.
at org.apache.kafka.clients.NetworkClientUtils.awaitReady(NetworkClientUtils.java:71)
at kafka.controller.RequestSendThread.brokerReady(ControllerChannelManager.scala:295)
at kafka.controller.RequestSendThread.doWork(ControllerChannelManager.scala:249)
at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:89)
[2020-01-16 12:21:40,539] DEBUG [PartitionStateMachine controllerId=0] Started partition state machine with initial state -> Map() (kafka.controller.ZkPartitionStateMachine)
[2020-01-16 12:21:40,540] INFO [Controller id=0] Ready to serve as the new controller with epoch 1 (kafka.controller.KafkaController)
[2020-01-16 12:21:40,542] INFO [Controller id=0] Removing partitions Set() from the list of reassigned partitions in zookeeper (kafka.controller.KafkaController)
[2020-01-16 12:21:40,543] INFO [Controller id=0] No more partitions need to be reassigned. Deleting zk path /admin/reassign_partitions (kafka.controller.KafkaController)
[2020-01-16 12:21:40,550] INFO [Controller id=0] Partitions undergoing preferred replica election: (kafka.controller.KafkaController)
[2020-01-16 12:21:40,551] INFO [Controller id=0] Partitions that completed preferred replica election: (kafka.controller.KafkaController)
[2020-01-16 12:21:40,553] INFO [Controller id=0] Skipping preferred replica election for partitions due to topic deletion: (kafka.controller.KafkaController)
[2020-01-16 12:21:40,554] INFO [Controller id=0] Resuming preferred replica election for partitions: (kafka.controller.KafkaController)
[2020-01-16 12:21:40,555] INFO [Controller id=0] Starting preferred replica leader election for partitions (kafka.controller.KafkaController)
[2020-01-16 12:21:40,593] INFO [Controller id=0] Starting the controller scheduler (kafka.controller.KafkaController)
[2020-01-16 12:21:40,637] WARN [RequestSendThread controllerId=0] Controller 0's connection to broker srvr0:9092 (id: 0 rack: null) was unsuccessful (kafka.controller.RequestSendThread)
java.io.IOException: Connection to srvr0:9092 (id: 0 rack: null) failed.
at org.apache.kafka.clients.NetworkClientUtils.awaitReady(NetworkClientUtils.java:71)
at kafka.controller.RequestSendThread.brokerReady(ControllerChannelManager.scala:295)
at kafka.controller.RequestSendThread.doWork(ControllerChannelManager.scala:249)
at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:89)
[2020-01-16 12:21:40,738] WARN [RequestSendThread controllerId=0] Controller 0's connection to broker srvr0:9092 (id: 0 rack: null) was unsuccessful (kafka.controller.RequestSendThread)
java.io.IOException: Connection to srvr0:9092 (id: 0 rack: null) failed.
...
[2020-01-16 12:21:41,559] INFO [Controller id=0] New topics: [Set(__confluent.support.metrics)], deleted topics: [Set()], new partition replica assignment [Map(__confluent.support.metrics-0 -> Vector(0))] (kafka.controller.KafkaController)
[2020-01-16 12:21:41,559] INFO [Controller id=0] New partition creation callback for __confluent.support.metrics-0 (kafka.controller.KafkaController)
[2020-01-16 12:21:41,653] WARN [RequestSendThread controllerId=0] Controller 0's connection to broker srvr0:9092 (id: 0 rack: null) was unsuccessful (kafka.controller.RequestSendThread)
java.io.IOException: Connection to srvr0:9092 (id: 0 rack: null) failed.
at org.apache.kafka.clients.NetworkClientUtils.awaitReady(NetworkClientUtils.java:71)
at kafka.controller.RequestSendThread.brokerReady(ControllerChannelManager.scala:295)
at kafka.controller.RequestSendThread.doWork(ControllerChannelManager.scala:249)
at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:89)
[2020-01-16 12:21:41,754] WARN [RequestSendThread controllerId=0] Controller 0's connection to broker srvr0:9092 (id: 0 rack: null) was unsuccessful (kafka.controller.RequestSendThread)
java.io.IOException: Connection to srvr0:9092 (id: 0 rack: null) failed.
...
[2020-01-16 12:21:45,596] INFO [Controller id=0] Processing automatic preferred replica leader election (kafka.controller.KafkaController)
[2020-01-16 12:21:45,597] TRACE [Controller id=0] Checking need to trigger auto leader balancing (kafka.controller.KafkaController)
[2020-01-16 12:21:45,601] DEBUG [Controller id=0] Preferred replicas by broker Map(0 -> Map(__confluent.support.metrics-0 -> Vector(0))) (kafka.controller.KafkaController)
[2020-01-16 12:21:45,605] DEBUG [Controller id=0] Topics not in preferred replica for broker 0 Map() (kafka.controller.KafkaController)
[2020-01-16 12:21:45,609] TRACE [Controller id=0] Leader imbalance ratio for broker 0 is 0.0 (kafka.controller.KafkaController)
[2020-01-16 12:21:45,616] WARN [RequestSendThread controllerId=0] Controller 0's connection to broker srvr0:9092 (id: 0 rack: null) was unsuccessful (kafka.controller.RequestSendThread)
java.io.IOException: Connection to srvr0:9092 (id: 0 rack: null) failed.
at org.apache.kafka.clients.NetworkClientUtils.awaitReady(NetworkClientUtils.java:71)
at kafka.controller.RequestSendThread.brokerReady(ControllerChannelManager.scala:295)
at kafka.controller.RequestSendThread.doWork(ControllerChannelManager.scala:249)
at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:89)
[2020-01-16 12:21:45,717] WARN [RequestSendThread controllerId=0] Controller 0's connection to broker srvr0:9092 (id: 0 rack: null) was unsuccessful (kafka.controller.RequestSendThread)
java.io.IOException: Connection to srvr0:9092 (id: 0 rack: null) failed.
...
[2020-01-16 12:22:10,944] INFO [ControllerEventThread controllerId=0] Shutting down (kafka.controller.ControllerEventManager$ControllerEventThread)
[2020-01-16 12:22:10,946] INFO [ControllerEventThread controllerId=0] Stopped (kafka.controller.ControllerEventManager$ControllerEventThread)
[2020-01-16 12:22:10,946] INFO [ControllerEventThread controllerId=0] Shutdown completed (kafka.controller.ControllerEventManager$ControllerEventThread)
[2020-01-16 12:22:10,947] DEBUG [Controller id=0] Resigning (kafka.controller.KafkaController)
[2020-01-16 12:22:10,948] DEBUG [Controller id=0] Unregister BrokerModifications handler for Set(0) (kafka.controller.KafkaController)
[2020-01-16 12:22:10,951] INFO [PartitionStateMachine controllerId=0] Stopped partition state machine (kafka.controller.ZkPartitionStateMachine)
[2020-01-16 12:22:10,953] INFO [ReplicaStateMachine controllerId=0] Stopped replica state machine (kafka.controller.ZkReplicaStateMachine)
[2020-01-16 12:22:10,955] INFO [RequestSendThread controllerId=0] Shutting down (kafka.controller.RequestSendThread)
[2020-01-16 12:22:10,956] TRACE [RequestSendThread controllerId=0] shutdownInitiated latch count reached zero. Shutdown called. (kafka.controller.RequestSendThread)
[2020-01-16 12:22:10,956] INFO [RequestSendThread controllerId=0] Stopped (kafka.controller.RequestSendThread)
[2020-01-16 12:22:10,956] INFO [RequestSendThread controllerId=0] Shutdown completed (kafka.controller.RequestSendThread)
[2020-01-16 12:22:10,960] INFO [Controller id=0] Resigned (kafka.controller.KafkaController)
Update 3:
Now, even worst... only zookeeper is starting. other services are failing to start...
Logs:
[root#srvr0 ~]# /opt/confluent/confluent-5.3.2/bin/confluent local start
The local commands are intended for a single-node development environment
only, NOT for production usage. https://docs.confluent.io/current/cli/index.html
Using CONFLUENT_CURRENT: /tmp/confluent.R3YJZ2UC
Starting zookeeper
zookeeper is [UP]
Starting kafka
-Kafka failed to start
kafka is [DOWN]
Cannot start Schema Registry, Kafka Server is not running. Check your deployment
Error: exit status 127
[root#srvr0 ~]#
Update 4:
confluent local start, zookeeper-server-start and kafka-server-start logs:
[root#srvr0 ~]# /opt/confluent/confluent-5.3.2/bin/confluent local start
Updates are available for confluent. To install them, please run:
$ confluent update
The local commands are intended for a single-node development environment
only, NOT for production usage. https://docs.confluent.io/current/cli/index.html
Using CONFLUENT_CURRENT: /tmp/confluent.xgVLokw7
Starting zookeeper
zookeeper is [UP]
Starting kafka
|Kafka failed to start
kafka is [DOWN]
Cannot start Schema Registry, Kafka Server is not running. Check your deployment
Error: exit status 127
[root#srvr0 ~]# /opt/confluent/confluent-5.3.2/bin/zookeeper-server-start
USAGE: /opt/confluent/confluent-5.3.2/bin/zookeeper-server-start [-daemon] zookeeper.properties
[root#srvr0 ~]# /opt/confluent/confluent-5.3.2/bin/kafka-server-start
USAGE: /opt/confluent/confluent-5.3.2/bin/kafka-server-start [-daemon] server.properties [--override property=value]*
server.properties hasn't been edited and its contents as follows:
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# see kafka.server.KafkaConfig for additional details and defaults
############################# Server Basics #############################
# The id of the broker. This must be set to a unique integer for each broker.
broker.id=0
############################# Socket Server Settings #############################
# The address the socket server listens on. It will get the value returned from
# java.net.InetAddress.getCanonicalHostName() if not configured.
# FORMAT:
# listeners = listener_name://host_name:port
# EXAMPLE:
# listeners = PLAINTEXT://your.host.name:9092
#listeners=PLAINTEXT://:9092
# Hostname and port the broker will advertise to producers and consumers. If not set,
# it uses the value for "listeners" if configured. Otherwise, it will use the value
# returned from java.net.InetAddress.getCanonicalHostName().
#advertised.listeners=PLAINTEXT://your.host.name:9092
# Maps listener names to security protocols, the default is for them to be the same. See the config documentation for more details
#listener.security.protocol.map=PLAINTEXT:PLAINTEXT,SSL:SSL,SASL_PLAINTEXT:SASL_PLAINTEXT,SASL_SSL:SASL_SSL
# The number of threads that the server uses for receiving requests from the network and sending responses to the network
num.network.threads=3
# The number of threads that the server uses for processing requests, which may include disk I/O
num.io.threads=8
# The send buffer (SO_SNDBUF) used by the socket server
socket.send.buffer.bytes=102400
# The receive buffer (SO_RCVBUF) used by the socket server
socket.receive.buffer.bytes=102400
# The maximum size of a request that the socket server will accept (protection against OOM)
socket.request.max.bytes=104857600
############################# Log Basics #############################
# A comma separated list of directories under which to store log files
log.dirs=/tmp/kafka-logs
# The default number of log partitions per topic. More partitions allow greater
# parallelism for consumption, but this will also result in more files across
# the brokers.
num.partitions=1
# The number of threads per data directory to be used for log recovery at startup and flushing at shutdown.
# This value is recommended to be increased for installations with data dirs located in RAID array.
num.recovery.threads.per.data.dir=1
############################# Internal Topic Settings #############################
# The replication factor for the group metadata internal topics "__consumer_offsets" and "__transaction_state"
# For anything other than development testing, a value greater than 1 is recommended for to ensure availability such as 3.
offsets.topic.replication.factor=1
transaction.state.log.replication.factor=1
transaction.state.log.min.isr=1
############################# Log Flush Policy #############################
# Messages are immediately written to the filesystem but by default we only fsync() to sync
# the OS cache lazily. The following configurations control the flush of data to disk.
# There are a few important trade-offs here:
# 1. Durability: Unflushed data may be lost if you are not using replication.
# 2. Latency: Very large flush intervals may lead to latency spikes when the flush does occur as there will be a lot of data to flush.
# 3. Throughput: The flush is generally the most expensive operation, and a small flush interval may lead to excessive seeks.
# The settings below allow one to configure the flush policy to flush data after a period of time or
# every N messages (or both). This can be done globally and overridden on a per-topic basis.
# The number of messages to accept before forcing a flush of data to disk
#log.flush.interval.messages=10000
# The maximum amount of time a message can sit in a log before we force a flush
#log.flush.interval.ms=1000
############################# Log Retention Policy #############################
# The following configurations control the disposal of log segments. The policy can
# be set to delete segments after a period of time, or after a given size has accumulated.
# A segment will be deleted whenever *either* of these criteria are met. Deletion always happens
# from the end of the log.
# The minimum age of a log file to be eligible for deletion due to age
log.retention.hours=168
# A size-based retention policy for logs. Segments are pruned from the log unless the remaining
# segments drop below log.retention.bytes. Functions independently of log.retention.hours.
#log.retention.bytes=1073741824
# The maximum size of a log segment file. When this size is reached a new log segment will be created.
log.segment.bytes=1073741824
# The interval at which log segments are checked to see if they can be deleted according
# to the retention policies
log.retention.check.interval.ms=300000
############################# Zookeeper #############################
# Zookeeper connection string (see zookeeper docs for details).
# This is a comma separated host:port pairs, each corresponding to a zk
# server. e.g. "127.0.0.1:3000,127.0.0.1:3001,127.0.0.1:3002".
# You can also append an optional chroot string to the urls to specify the
# root directory for all kafka znodes.
zookeeper.connect=localhost:2181
# Timeout in ms for connecting to zookeeper
zookeeper.connection.timeout.ms=6000
##################### Confluent Metrics Reporter #######################
# Confluent Control Center and Confluent Auto Data Balancer integration
#
# Uncomment the following lines to publish monitoring data for
# Confluent Control Center and Confluent Auto Data Balancer
# If you are using a dedicated metrics cluster, also adjust the settings
# to point to your metrics kakfa cluster.
#metric.reporters=io.confluent.metrics.reporter.ConfluentMetricsReporter
#confluent.metrics.reporter.bootstrap.servers=localhost:9092
#
# Uncomment the following line if the metrics cluster has a single broker
#confluent.metrics.reporter.topic.replicas=1
##################### Confluent Proactive Support ######################
# If set to true, and confluent-support-metrics package is installed
# then the feature to collect and report support metrics
# ("Metrics") is enabled. If set to false, the feature is disabled.
#
confluent.support.metrics.enable=true
# The customer ID under which support metrics will be collected and
# reported.
#
# When the customer ID is set to "anonymous" (the default), then only a
# reduced set of metrics is being collected and reported.
#
# Confluent customers
# -------------------
# If you are a Confluent customer, then you should replace the default
# value with your actual Confluent customer ID. Doing so will ensure
# that additional support metrics will be collected and reported.
#
confluent.support.customer.id=anonymous
############################# Group Coordinator Settings #############################
# The following configuration specifies the time, in milliseconds, that the GroupCoordinator will delay the initial consumer rebalance.
# The rebalance will be further delayed by the value of group.initial.rebalance.delay.ms as new members join the group, up to a maximum of max.poll.interval.ms.
# The default value for this is 3 seconds.
# We override this to 0 here as it makes for a better out-of-the-box experience for development and testing.
# However, in production environments the default value of 3 seconds is more suitable as this will help to avoid unnecessary, and potentially expensive, rebalances during application startup.
group.initial.rebalance.delay.ms=0
############################# Confluent Authorizer Settings #############################
# Uncomment to enable Confluent Authorizer with support for ACLs, LDAP groups and RBAC
#authorizer.class.name=io.confluent.kafka.security.authorizer.ConfluentServerAuthorizer
# Semi-colon separated list of super users in the format <principalType>:<principalName>
#super.users=
# Specify a valid Confluent license. By default free-tier license will be used
#confluent.license=
# Replication factor for the topic used for licensing. Default is 3.
confluent.license.topic.replication.factor=1
# Uncomment the following lines and specify values where required to enable RBAC
# Enable RBAC provider
#confluent.authorizer.access.rule.providers=ACL,RBAC
# Bootstrap servers for RBAC metadata. Must be provided if this broker is not in the metadata cluster
#confluent.metadata.bootstrap.servers=PLAINTEXT://127.0.0.1:9092
# Replication factor for the metadata topic used for authorization. Default is 3.
confluent.metadata.topic.replication.factor=1
# Listeners for metadata server
#confluent.metadata.server.listeners=http://0.0.0.0:8090
# Advertised listeners for metadata server
#confluent.metadata.server.advertised.listeners=http://127.0.0.1:8090
Please help me in resolving the issue!
It's not clear where srvr0:9092 is defined; I suggest reviewing your server.properties file to fix the connection strings.
You don't need to run confluent at all. You can follow the base Apache Kafka guides for running both Zookeeper and Kafka
zookeeper-server-start + kafka-server-start
Or you can use Confluent's APT/YUM repos rather than just extracting tarballs, then use systemctl to control services.
Or, using Docker is another way to get started quickly.

When does kafka change leader?

I was running my services that work with kafka already for a year and no spontaneous changes of leader happens.
But for the last 2 weeks that started happens quite often.
Kafka log on that:
[2015-09-27 15:35:14,826] INFO [ReplicaFetcherManager on broker 2]
Removed fetcher for partitions [myTopic] (kafka.server.ReplicaFetcherManager)
[2015-09-27 15:35:14,830] INFO Truncating log myTopic-0 to offset 11520979. (kafka.log.Log)
[2015-09-27 15:35:14,845] WARN [Replica Manager on Broker 2]: Fetch request with correlation id 713276 from client ReplicaFetcherThread-0-2 on partition [myTopic,0] failed due to Leader not local for partition [myTopic,0] on broker 2 (kafka.server.ReplicaManager)
[2015-09-27 15:35:14,857] WARN [Replica Manager on Broker 2]: Fetch request with correlation id 256685 from client mirrormaker-1 on partition [myTopic,0] failed due to Leader not local for partition [myTopic,0] on broker 2 (kafka.server.ReplicaManager)
[2015-09-27 15:35:20,171] INFO [ReplicaFetcherManager on broker 2] Removed fetcher for partitions [myTopic,0] (kafka.server.ReplicaFetcherManager)
What can cause switching leader? If there is info in some kafka documentation - please - just point the link. I've failed to find.
System configuration
kafka version: kafka_2.10-0.8.2.1
os: Red Hat Enterprise Linux Server release 6.5 (Santiago)
server.properties (differs from default):
broker.id=001
socket.send.buffer.bytes=1048576
socket.receive.buffer.bytes=1048576
socket.request.max.bytes=104857600
log.flush.interval.messages=10000
log.flush.interval.ms=1000
log.retention.bytes=-1
controlled.shutdown.enable=true
auto.create.topics.enable=false
It appears like lead broker is down for that partition. It might be that data directroy(log.dirs) configured in server.properties is out of space and broker is not able to accommodate.
Also, what is replication factor of topic and cluster size of brokers?
I am assuming you have one topic and one partition with a replication factor of 2. Which is not a good configuration for optimal Kafka performance and consumers.
Your Logs are not clear enough for leader switch. Major issue in your topic may be having the only one leader due to the only partition. Now the single file in your logs is getting bigger in size day by day. Kafka internally does rebalancing at some level(details are not confirmed). That can be the reason for your leader switch. But i am not sure.
Also in your 2nd log line its says some of the logs are truncated. Can you please go though the logs in details and check is this happening only after truncation?
As you already mentioned you already checked your Kafka log directory files and their size. Please run the describe when you got this issue. The leader switch will reflect here as well. Or if you can setup some dashboard that will display the leader for past time. Then it will be easy for you to find the root cause.
bin/kafka-topics.sh --describe --zookeeper Zookeeperhost:Port --topic TopicName
Suggestion: i will suggest you to create a new topic with more partitions(read Kafka documentation to get a good idea about optimum number of partitions) and start writing to it. Or you can check, how to change partitions for current topic.
Last Thing: Is leader switch causing some issues in your Clients or you are worried only about warnings?