Brokers are not available - apache-kafka

To /etc/kafka/server.properties the broker is configured using:
############################# Server Basics #############################
# The id of the broker. This must be set to a unique integer for each broker.
broker.id=0
############################# Socket Server Settings #############################
# The address the socket server listens on. It will get the value returned from
# java.net.InetAddress.getCanonicalHostName() if not configured.
# FORMAT:
# listeners = listener_name://host_name:port
# EXAMPLE:
# listeners = PLAINTEXT://your.host.name:9092
listeners=PLAINTEXT://<MY_HOST_NAME>:9092,SASL_SSL://<MY_HOST_NAME>:9093
To check the brokers I run:
/opt/confluent-kafka/v5.5.0/bin/zookeeper-shell localhost:2181 ls /brokers/ids
Which returns:
Connecting to localhost:2181
WATCHER::
WatchedEvent state:SyncConnected type:None path:null
[]
So, it seems that the brokers are not running. As kafka server.properties(above) contains:
listeners=PLAINTEXT://<MY_HOST_NAME>:9092,SASL_SSL://<MY_HOST_NAME>:9093
shouldn't the broker be available ?
My understanding is listeners=PLAINTEXT://<MY_HOST_NAME>:9092,SASL_SSL://<MY_HOST_NAME>:9093 is the config that allows the brokers to be accessible ?

I assume that the connection is within the same host (no advertised listeners nor external clients involved here).
Change the listeners property to make the broker listen to all interfaces:
listeners=PLAINTEXT://0.0.0.0:9092,SASL_SSL://0.0.0.0:9093
If you set your <hostname> as the listener, localhost won't be recognized as a valid binding point: there's no registered "localhost" listener, just the one specified on your config, <hostname>.
By setting 0.0.0.0 as listener, the broker will bind to all interfaces, hence accepting localhost as a valid endpoint. You could find more info regarding this here.

Related

The Cluster ID XXXXXXXXXXXX doesn't match stored clusterId Some

On two nodes Kafka cluster after restart one node I have get error:
ERROR Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)
kafka.common.InconsistentClusterIdException: The Cluster ID hJGLXQwERc2rTXI_dWLwcg doesn't match stored clusterId Some(9VcFGDwSfke5JWuzA6f9mw) in meta.properties. The broker is trying to join the wrong cluster. Configured zookeeper.connect may be wrong.
at kafka.server.KafkaServer.startup(KafkaServer.scala:223)
at kafka.Kafka$.main(Kafka.scala:109)
at kafka.Kafka.main(Kafka.scala)
Where hJGLXQwERc2rTXI_dWLwcg - ID of worked cluster from meta.properties
9VcFGDwSfke5JWuzA6f9mw - new generated ID.
Kafka version 3.0.0
On Node1 :
server.properties:
broker.id=1
listeners=SASL_PLAINTEXT://:9092
advertised.listeners=SASL_PLAINTEXT://loc-kafka1:9092
num.network.threads=3
num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
log.dirs=/opt/kafka/logs
num.partitions=1
num.recovery.threads.per.data.dir=1
offsets.topic.replication.factor=1
transaction.state.log.replication.factor=1
transaction.state.log.min.isr=1
log.retention.hours=168
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000
zookeeper.connect=loc-kafka1:2181, loc-kafka2:2181
zookeeper.connection.timeout.ms=18000
group.initial.rebalance.delay.ms=0
compression.type=gzip
security.inter.broker.protocol=SASL_PLAINTEXT
sasl.mechanism.inter.broker.protocol=PLAIN
sasl.enabled.mechanisms=PLAIN
authorizer.class.name=kafka.security.authorizer.AclAuthorizer
ssl.endpoint.identification.algorithm=
super.users=User:admin
sasl.jaas.config=/opt/kafka/config/kafka_server_jaas.conf
delete.topic.enable=true
zookeeper.properties
dataDir=/opt/kafka/zookeeper/data
clientPort=2181
maxClientCnxns=10
admin.enableServer=false
server1=loc-kafka1:2888:3888
server2=loc-kafka2:2888:3888
On Node2 the same config except broker.id=2 and myid file in dataDir.
All information what I can find sad : you should place log.dirs and dataDir not in /tmp folder. And all will done. But it's not like that.
One solution is clear log.dirs and dataDir folders allow this node connect to cluster. But this doesn't seem good solution.
If I just delete meta.properties, new cluster.id is plased in it and node don't connect to cluster. At the same time, in server.logs, the following lines show that the node is connecting to another cluster node:
INFO Opening socket connection to server loc-kafka1/10.1.1.3:2181. (org.apache.zookeeper.ClientCnxn)
INFO SASL config status: Will attempt to SASL-authenticate using Login Context section 'Client' (org.apache.zookeeper.ClientCnxn)
INFO Socket connection established, initiating session, client: loc-kafka2/10.1.1.4:38672, server: loc-kafka1/10.1.1.3:2181 (org.apache.zookeeper.ClientCnxn)
INFO Session establishment complete on server loc-kafka1/10.0.0.3:3001, session id = 0x100000043bb0006, negotiated timeout = 18000 (org.apache.zookeeper.ClientCnxn)
What I can do to fix it?

Unable to connect kafka server if i set broker.id other than 0

I am trying to connect kafka server with one node. If i set broker id = 0 then it runs fine, but if I change this from 0 to any number (like 1,2,3,etc) then not able to connect server
my configuration of zookeeper.properties are
tickTime=2000
initLimit=10
syncLimit=5
dataDir=/tmp/zookeeper
clientPort=2181
maxClientCnxns=60
and server.properties file
broker.id=1
zookeeper.connect=localhost:2181
listeners=PLAINTEXT://localhost:9092

Failed to connect to and describe Kafka cluster In Standalone mode

I have installed zookeeper and kafka,zookeeper is ruuning fine but as per my requirement I want to run Kafka-connect standalone to connect kafka and cassandsra. I am using following command to start kafka server kafka/bin/connect-standalone.sh kafka/config/connect-standalone.properties kafka-connect-cassandra-sink-1.4.0/conf/cassandra-sink-standalone.properties . The connect-standalone.properties configuration file is given below:
bootstrap.servers=localhost:9092
key.converter=org.apache.kafka.connect.storage.StringConverter
value.converter=org.apache.kafka.connect.json.JsonConverter
value.converter.schemas.enable=true
offset.storage.file.filename=/tmp/connect.offsets
offset.flush.interval.ms=10000
plugin.path=/home/shamama/kafka-connect-cassandra-sink-1.4.0/kafka-connect-cassandra-sink-1.4.0.jar
############################# Zookeeper #############################
# Zookeeper connection string (see zookeeper docs for details).
# This is a comma separated host:port pairs, each corresponding to a zk
# server. e.g. "127.0.0.1:3000,127.0.0.1:3001,127.0.0.1:3002".
# You can also append an optional chroot string to the urls to specify the
# root directory for all kafka znodes.
zookeeper.connect=localhost:2181
# Timeout in ms for connecting to zookeeper
zookeeper.connection.timeout.ms=18000
listeners=PLAINTEXT://localhost:9092
#advertised.listeners=PLAINTEXT://localhost:9092
The following error is occurring after starting kafka standalone server:
org.apache.kafka.connect.errors.ConnectException: Failed to connect to and describe Kafka cluster. Check worker's broker connection and security properties.
at org.apache.kafka.connect.util.ConnectUtils.lookupKafkaClusterId(ConnectUtils.java:70)
at org.apache.kafka.connect.util.ConnectUtils.lookupKafkaClusterId(ConnectUtils.java:51)
at org.apache.kafka.connect.cli.ConnectStandalone.main(ConnectStandalone.java:82)
Caused by: java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.TimeoutException: Call(callName=listNodes, deadlineMs=1628583433705, tries=1, nextAllowedTryMs=1628583433806) timed out at 1628583433706 after 1 attempt(s)
at org.apache.kafka.common.internals.KafkaFutureImpl.wrapAndThrow(KafkaFutureImpl.java:45)
at org.apache.kafka.common.internals.KafkaFutureImpl.access$000(KafkaFutureImpl.java:32)
at org.apache.kafka.common.internals.KafkaFutureImpl$SingleWaiter.await(KafkaFutureImpl.java:89)
at org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:260)
at org.apache.kafka.connect.util.ConnectUtils.lookupKafkaClusterId(ConnectUtils.java:64)
I cannot figure out my mistake.Please help.
The error indicates Kafka isn't running.
Based on your previous question, you're running kafka-server-start with the wrong property file
Specifically, zookeeper.connect, listeners, and advertised.listeners should not be part of connect-standalone.properties, and you need broker.id, at a minimum, to start a broker (already part of server.properties)

kafka is not retaining 7 days of logs

I am using kafka2.11 i am pasting my server.properties.I have 3 nodes of kafka cluster
############################# Server Basics #############################
# The id of the broker. This must be set to a unique integer for each broker.
broker.id=1
#broker.rack=1
############################# Socket Server Settings #############################
# Hostname and port the broker will advertise to producers and consumers. If not set,
# it uses the value for "listeners" if configured. Otherwise, it will use the value
# returned from java.net.InetAddress.getCanonicalHostName().
advertised.listeners=PLAINTEXT://xx.xx.xx.xx:9092
# The number of threads that the server uses for processing requests, which may include disk I/O
num.io.threads=8
# The send buffer (SO_SNDBUF) used by the socket server
socket.send.buffer.bytes=102400
# The receive buffer (SO_RCVBUF) used by the socket server
socket.receive.buffer.bytes=102400
# The maximum size of a request that the socket server will accept (protection against OOM)
socket.request.max.bytes=104857600
############################# Log Basics #############################
# A comma separated list of directories under which to store log files
log.dirs=/home/kafka_data/
# The number of threads per data directory to be used for log recovery at startup and flushing at shutdown.
# This value is recommended to be increased for installations with data dirs located in RAID array.
num.recovery.threads.per.data.dir=1
num.replica.fetchers=1
############################# Internal Topic Settings #############################
# The replication factor for the group metadata internal topics "__consumer_offsets" and "__transaction_state"
# For anything other than development testing, a value greater than 1 is recommended to ensure availability such as 3.
offsets.topic.replication.factor=3
transaction.state.log.replication.factor=3
transaction.state.log.min.isr=3
############################# Log Flush Policy #############################
#log.flush.interval.messages=10000
# The maximum amount of time a message can sit in a log before we force a flush
#log.flush.interval.ms=1000
############################# Log Retention Policy #############################
# The minimum age of a log file to be eligible for deletion due to age
log.retention.hours=168
# A size-based retention policy for logs. Segments are pruned from the log unless the remaining
# segments drop below log.retention.bytes. Functions independently of log.retention.hours.
log.retention.bytes=1000000
# The maximum size of a log segment file. When this size is reached a new log segment will be created.
log.segment.bytes=1073741824
# The interval at which log segments are checked to see if they can be deleted according
# to the retention policies
log.retention.check.interval.ms=300000
############################# Zookeeper #############################
# Zookeeper connection string (see zookeeper docs for details).
# This is a comma separated host:port pairs, each corresponding to a zk
# server. e.g. "127.0.0.1:3000,127.0.0.1:3001,127.0.0.1:3002".
# You can also append an optional chroot string to the urls to specify the
# root directory for all kafka znodes.
zookeeper.connect=xx.xx.xx.xx:2181,xx.xx.xx.xx:2181,xx.xx.xx.xx:2181/kafka
# Timeout in ms for connecting to zookeeper
zookeeper.connection.timeout.ms=18000
############################# Group Coordinator Settings #############################
# The following configuration specifies the time, in milliseconds, that the GroupCoordinator will delay the initial consumer rebalance.
# The rebalance will be further delayed by the value of group.initial.rebalance.delay.ms as new members join the group, up to a maximum of max.poll.interval.ms.
# The default value for this is 3 seconds.
# We override this to 0 here as it makes for a better out-of-the-box experience for development and testing.
# However, in production environments the default value of 3 seconds is more suitable as this will help to avoid unnecessary, and potentially expensive, rebalances during application startup.
group.initial.rebalance.delay.ms=0
###################### Topic configuration ##############################
delete.topic.enable=true
num.partitions=3
default.replication.factor=3
auto.create.topic.enable=false
# Enables automatic leader balancing.
auto.leader.rebalance.enable=true
# The frequency with which the partition rebalance check is triggered by the controller
leader.imbalance.check.interval.seconds=60
#offsets.topic.replication.factor=1
############################## REPLICA SETTING ##########################i#######
# max wait time for each fetcher request issued by follower replicas
replica.fetch.wait.max.ms=300
#replica.fetch.min.bytes=1
# If a follower hasn't sent any fetch requests or hasn't consumed up to the leaders log end offset for at least this time, the leader will remove the follower from isr
replica.lag.time.max.ms=30000
# The socket receive buffer for network requests
replica.socket.receive.buffer.bytes=1000000
# The socket timeout for network requests.
replica.socket.timeout.ms=30000
# The configuration controls the maximum amount of time the client will wait for the response of a request. If the response is not received
# before the timeout elapses the client will resend the request if necessary or fail the request if retries are exhausted.
request.timeout.ms=30000
metadata.max.age.ms=30000
####################### PRODUCER CONFIG ##################################
acks=all
#unclean.leader.election.enable=false
message.max.bytes=1000000000
fetch.message.max.bytes=1000000000
replica.fetch.max.bytes=1000000000
unclean.leader.election.enable=true
#enable.idempotence = true
#message.send.max.retries= 10000000
#max.in.flight.requests.per.connection=1
I have set log.retention.hours=168 that is 7 days retention
i consumed the data by creating new group and add --from-beginning attribute but i've got 4 days of logs.
bin/kafka-consumer-groups.sh --describe --group consumer2 --bootstrap-server lc-helk.nic.in:9092
TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG
ise 2 1114176 1114176 0
ise 0 1113757 1113757 0
ise 1 1113627 1113627 0
In log entry i am seeing 17th July,21 logs it should show 13th july,21 logs
If i want to change this kafka topic configuration how can i change this in existing kafka topic.
Topic retention is configured by 3 properties(besides some others):
log.retention.bytes
log.retention.hours
log.cleanup.policy
I can see that you set log.retention.bytes=1000000 have you tried setting it to -1 ?

Kafka client can't receive messages

I have kafka and zookeeper set up on a remote machine. On that machine I'm able to see below working using the test method on official website.
> bin/kafka-console-producer.sh --broker-list localhost:9092 --topic listings-incoming
This is a message
This is another message
> bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --listings-incoming --from-beginning
This is a message
This is another message
but when I use my local consumer script it is not working:
bin/kafka-console-consumer.sh —bootstrap-server X.X.X.X:9092 —listings-incoming —from-beginning —consumer-property group.id=group2
Haven't seen messages showing up but what is showing is:
[2017-08-11 14:39:56,425] WARN Auto-commit of offsets {listings-incoming-4=OffsetAndMetadata{offset=0, metadata=''}, listings-incoming-2=OffsetAndMetadata{offset=0, metadata=''}, listings-incoming-3=OffsetAndMetadata{offset=0, metadata=''}, listings-incoming-0=OffsetAndMetadata{offset=0, metadata=''}, listings-incoming-1=OffsetAndMetadata{offset=0, metadata=''}} failed for group group1: Commit cannot be completed since the group has already rebalanced and assigned the partitions to another member. This means that the time between subsequent calls to poll() was longer than the configured max.poll.interval.ms, which typically implies that the poll loop is spending too much time message processing. You can address this either by increasing the session timeout or by reducing the maximum size of batches returned in poll() with max.poll.records. (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator)
*****************update**********************
My zookeeper and kafka is running on the same machine, right now my configuration on advertised.listeners is this:
advertised.listeners=PLAINTEXT://the.machine.ip.address:9092
I tried to change it to:
advertised.listeners=PLAINTEXT://my.client.ip.address:9092
and then run the client side consumer script, it gives error:
[2017-08-11 15:49:01,591] WARN Error while fetching metadata with
correlation id 3 : {listings-incoming=LEADER_NOT_AVAILABLE}
(org.apache.kafka.clients.NetworkClient) [2017-08-11 15:49:22,106]
WARN Bootstrap broker 10.161.128.238:9092 disconnected
(org.apache.kafka.clients.NetworkClient) [2017-08-11 15:49:22,232]
WARN Error while fetching metadata with correlation id 7 :
{listings-incoming=LEADER_NOT_AVAILABLE}
(org.apache.kafka.clients.NetworkClient) [2017-08-11 15:49:22,340]
WARN Error while fetching metadata with correlation id 8 :
{listings-incoming=LEADER_NOT_AVAILABLE}
(org.apache.kafka.clients.NetworkClient) [2017-08-11 15:49:40,453]
WARN Bootstrap broker 10.161.128.238:9092 disconnected
(org.apache.kafka.clients.NetworkClient) [2017-08-11 15:49:40,531]
WARN Error while fetching metadata with correlation id 12 :
{listings-incoming=LEADER_NOT_AVAILABLE}
(org.apache.kafka.clients.NetworkClient)
You probably have not configured your advertised.listeners properly in the brokers server.properties file.
From https://kafka.apache.org/documentation/
advertised.listeners Listeners to publish to ZooKeeper for clients to
use, if different than the listeners above. In IaaS environments, this
may need to be different from the interface to which the broker binds.
If this is not set, the value for listeners will be used.
and in the same documentation
listeners Listener List - Comma-separated list of URIs we will listen
on and the listener names. If the listener name is not a security
protocol, listener.security.protocol.map must also be set. Specify
hostname as 0.0.0.0 to bind to all interfaces. Leave hostname empty to
bind to default interface. Examples of legal listener lists:
PLAINTEXT://myhost:9092,SSL://:9091
CLIENT://0.0.0.0:9092,REPLICATION://localhost:9093
So if advertised.listeners is not set and listeners is just listening to localhost:9092 or 127.0.0.1:9092 or 0.0.0.0:9092 then the clients will be told to connect to localhost when they make a meta-data request to the bootstrap server. That works when the client is actually running in the same machine as the broker but it will fail when you connect remotely.
You should set advertised.listeners to be a fully qualified domain name or public IP address for the host that the broker is running on.
For example
advertised.listeners=PLAINTEXT://kafkabrokerhostname.confluent.io:9092
or
advertised.listeners=PLAINTEXT://192.168.1.101:9092