I wish to monitor Kafka with Prometheus and Grafana.
I have downloaded kafka_2.11-0.10.0.0
cd kafka_2.11-0.10.0.0
and downloaded :
wget https://repo1.maven.org/maven2/io/prometheus/jmx/jmx_prometheus_javaagent/0.6/jmx_prometheus_javaagent-0.6.jar
wget https://raw.githubusercontent.com/prometheus/jmx_exporter/master/example_configs/kafka-0-8-2.yml
Started Zookeeper using :
nohup bin/zookeeper-server-start.sh config/zookeeper.properties >> zookeeper.log &
KAFKA_OPTS="$KAFKA_OPTS -javaagent:$PWD/jmx_prometheus_javaagent-0.6.jar=7071:$PWD/kafka-0-8-2.yml"
Started kafka using :
nohup bin/kafka-server-start.sh config/server.properties >> kafka.log &
logs of zookeeper :
INFO Got user-level KeeperException when processing sessionid:0x15b18c79a630075 type:create cxid:0x81f216 zxid:0x2b87c2 txntype:-1 reqpath:n/a Error
Path:/consumers/logstash/ids/logstash_wavescore-staging-1490769576466-92cd1041 Error:KeeperErrorCode = NodeExists for /consumers/logstash/ids/logstash_wavescore-staging-149076
9576466-92cd1041 (org.apache.zookeeper.server.PrepRequestProcessor)
INFO Got user-level KeeperException when processing sessionid:0x15b18c79a630075 type:create cxid:0x81f219 zxid:0x2b87c3 txntype:-1 reqpath:n/a Error
Path:/consumers/logstash/ids/logstash_wavescore-staging-1490769576466-92cd1041 Error:KeeperErrorCode = NodeExists for /consumers/logstash/ids/logstash_wavescore-staging-149076
9576466-92cd1041 (org.apache.zookeeper.server.PrepRequestProcessor)
INFO Got user-level KeeperException when processing sessionid:0x15b18c79a630075 type:create cxid:0x81f21c zxid:0x2b87c4 txntype:-1 reqpath:n/a Error
Path:/consumers/logstash/ids/logstash_wavescore-staging-1490769576466-92cd1041 Error:KeeperErrorCode = NodeExists for /consumers/logstash/ids/logstash_wavescore-staging-149076
9576466-92cd1041 (org.apache.zookeeper.server.PrepRequestProcessor)
logs of kafka :
INFO [Group Metadata Manager on Broker 0]: Finished loading offsets from [__consumer_offsets,36] in 1 milliseconds. (kafka.coordinator.GroupMetadataM
anager)
INFO [Group Metadata Manager on Broker 0]: Loading offsets and group metadata from [__consumer_offsets,39] (kafka.coordinator.GroupMetadataManager)
INFO [Group Metadata Manager on Broker 0]: Finished loading offsets from [__consumer_offsets,39] in 0 milliseconds. (kafka.coordinator.GroupMetadataM
anager)
INFO [Group Metadata Manager on Broker 0]: Loading offsets and group metadata from [__consumer_offsets,42] (kafka.coordinator.GroupMetadataManager)
INFO [Group Metadata Manager on Broker 0]: Finished loading offsets from [__consumer_offsets,42] in 1 milliseconds. (kafka.coordinator.GroupMetadataM
anager)
INFO [Group Metadata Manager on Broker 0]: Loading offsets and group metadata from [__consumer_offsets,45] (kafka.coordinator.GroupMetadataManager)
INFO [Group Metadata Manager on Broker 0]: Finished loading offsets from [__consumer_offsets,45] in 0 milliseconds. (kafka.coordinator.GroupMetadataM
anager)
INFO [Group Metadata Manager on Broker 0]: Loading offsets and group metadata from [__consumer_offsets,48] (kafka.coordinator.GroupMetadataManager)
INFO [Group Metadata Manager on Broker 0]: Finished loading offsets from [__consumer_offsets,48] in 1 milliseconds. (kafka.coordinator.GroupMetadataM
anager)
INFO [Group Metadata Manager on Broker 0]: Removed 0 expired offsets in 1 milliseconds. (kafka.coordinator.GroupMetadataManager
When i check the netstat in the server , there is no port 7071 was opened and curl localhost:7071 was resulted curl: (7) couldn't connect to host .
Reference Link :
https://www.robustperception.io/monitoring-kafka-with-prometheus/
in this link they were using kafka version : kafka_2.11-0.10.1.0.tgz
I have downloaded : kafka_2.11-0.10.0.0
It might be the wrong variable. I have managed to monitor my kafka broker with jolokia agent and metricbeat but it should be the same. Here is the script i am using to start the broker:
export KAFKA_JMX_OPTS=-javaagent:/opt/kafka/jolokia-jvm-1.3.7-agent.jar=port=8778,host=localhost
./bin/kafka-server-start.sh -daemon config/server_cluster.properties
Hope it can help.
NB: Make sure the owner of the agent jar is the same than the user you use to launch the broker.
Related
Running Kafka Connect via this repo https://github.com/entechlog/kafka-examples/tree/master/kafka-connect-standalone except I have added extra configs for AWS MSK IAM authentication. I've also updated the .env file to use different variables, like the AWS MSK IAM jar file, AWS key/secret key credentials, and a few other minor things. Note that this repo runs in standalone mode, but I have updated the launch shell script to run in distributed mode: exec connect-distributed /etc/"${COMPONENT}"/"${COMPONENT}".properties. But I have NOT created a file called kafka-connect.properties.template because when I do, I get a whole host of errors like Missing required configuration "group.id" which has no default value. which makes no sense to me as I can see it in the docker-compose.yml file.
My goal is to get data from a third party Kafka cluster into BigQuery.
When I run my docker-compose file, I see no errors, a few warnings, but nothing that stands out to me. I get a lot of warnings like this: [2021-12-14 01:57:37,917] WARN The configuration 'camelcase.default.dataset' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig:380), and this dataset is required to send data to BigQuery. Not making sense why things like dataset are not being used.
Here are the latest logs:
[2021-12-14 01:57:37,921] INFO Kafka version: 6.2.1-ce (org.apache.kafka.common.utils.AppInfoParser:119)
[2021-12-14 01:57:37,922] INFO Kafka commitId: 14770bfc4e973178 (org.apache.kafka.common.utils.AppInfoParser:120)
[2021-12-14 01:57:37,922] INFO Kafka startTimeMs: 1639447057921 (org.apache.kafka.common.utils.AppInfoParser:121)
[2021-12-14 01:57:39,332] INFO [Producer clientId=producer-3] Cluster ID: k2eIXxm_RkmWu2-R2d0N1Q (org.apache.kafka.clients.Metadata:279)
[2021-12-14 01:57:39,413] INFO [Consumer clientId=consumer-connect-kafka-connect-group-3, groupId=connect-kafka-connect-group] Cluster ID: k2eIXxm_RkmWu2-R2d0N1Q (org.apache.kafka.clients.Metadata:279)
[2021-12-14 01:57:39,428] INFO [Consumer clientId=consumer-connect-kafka-connect-group-3, groupId=connect-kafka-connect-group] Subscribed to partition(s): connect-configs-0 (org.apache.kafka.clients.consumer.KafkaConsumer:1123)
[2021-12-14 01:57:39,428] INFO [Consumer clientId=consumer-connect-kafka-connect-group-3, groupId=connect-kafka-connect-group] Seeking to EARLIEST offset of partition connect-configs-0 (org.apache.kafka.clients.consumer.internals.SubscriptionState:619)
[2021-12-14 01:57:40,727] INFO Finished reading KafkaBasedLog for topic connect-configs (org.apache.kafka.connect.util.KafkaBasedLog:228)
[2021-12-14 01:57:40,728] INFO Started KafkaBasedLog for topic connect-configs (org.apache.kafka.connect.util.KafkaBasedLog:230)
[2021-12-14 01:57:40,729] INFO Started KafkaConfigBackingStore (org.apache.kafka.connect.storage.KafkaConfigBackingStore:290)
[2021-12-14 01:57:40,729] INFO [Worker clientId=connect-1, groupId=connect-kafka-connect-group] Herder started (org.apache.kafka.connect.runtime.distributed.DistributedHerder:312)
[2021-12-14 01:57:45,059] INFO [Worker clientId=connect-1, groupId=connect-kafka-connect-group] Cluster ID: k2eIXxm_RkmWu2-R2d0N1Q (org.apache.kafka.clients.Metadata:279)
[2021-12-14 01:57:45,089] INFO [Worker clientId=connect-1, groupId=connect-kafka-connect-group] Discovered group coordinator <bootstrap server and port here> (id: 2147483643 rack: null) (org.apache.kafka.clients.consumer.internals.AbstractCoordinator:848)
[2021-12-14 01:57:45,095] INFO [Worker clientId=connect-1, groupId=connect-kafka-connect-group] Rebalance started (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator:221)
[2021-12-14 01:57:45,095] INFO [Worker clientId=connect-1, groupId=connect-kafka-connect-group] (Re-)joining group (org.apache.kafka.clients.consumer.internals.AbstractCoordinator:538)
[2021-12-14 01:57:45,957] INFO [Worker clientId=connect-1, groupId=connect-kafka-connect-group] (Re-)joining group (org.apache.kafka.clients.consumer.internals.AbstractCoordinator:538)
[2021-12-14 01:57:49,038] INFO [Worker clientId=connect-1, groupId=connect-kafka-connect-group] Successfully joined group with generation Generation{generationId=1, memberId='connect-1-a4cc5355-60da-46c3-8228-bff2de664f2c', protocol='sessioned'} (org.apache.kafka.clients.consumer.internals.AbstractCoordinator:594)
[2021-12-14 01:57:49,160] INFO [Worker clientId=connect-1, groupId=connect-kafka-connect-group] Successfully synced group in generation Generation{generationId=1, memberId='connect-1-a4cc5355-60da-46c3-8228-bff2de664f2c', protocol='sessioned'} (org.apache.kafka.clients.consumer.internals.AbstractCoordinator:758)
[2021-12-14 01:57:49,161] INFO [Worker clientId=connect-1, groupId=connect-kafka-connect-group] Joined group at generation 1 with protocol version 2 and got assignment: Assignment{error=0, leader='connect-1-a4cc5355-60da-46c3-8228-bff2de664f2c', leaderUrl='http://kafka-connect:8083/', offset=10, connectorIds=[], taskIds=[], revokedConnectorIds=[], revokedTaskIds=[], delay=0} with rebalance delay: 0 (org.apache.kafka.connect.runtime.distributed.DistributedHerder:1694)
[2021-12-14 01:57:49,162] WARN [Worker clientId=connect-1, groupId=connect-kafka-connect-group] Catching up to assignment's config offset. (org.apache.kafka.connect.runtime.distributed.DistributedHerder:1119)
[2021-12-14 01:57:49,162] INFO [Worker clientId=connect-1, groupId=connect-kafka-connect-group] Current config state offset -1 is behind group assignment 10, reading to end of config log (org.apache.kafka.connect.runtime.distributed.DistributedHerder:1183)
[2021-12-14 01:57:49,359] INFO [Worker clientId=connect-1, groupId=connect-kafka-connect-group] Finished reading to end of log and updated config snapshot, new config log offset: 10 (org.apache.kafka.connect.runtime.distributed.DistributedHerder:1190)
[2021-12-14 01:57:49,359] INFO [Worker clientId=connect-1, groupId=connect-kafka-connect-group] Starting connectors and tasks using config offset 10 (org.apache.kafka.connect.runtime.distributed.DistributedHerder:1244)
[2021-12-14 01:57:49,359] INFO [Worker clientId=connect-1, groupId=connect-kafka-connect-group] Finished starting connectors and tasks (org.apache.kafka.connect.runtime.distributed.DistributedHerder:1272)
[2021-12-14 01:57:50,791] INFO [Worker clientId=connect-1, groupId=connect-kafka-connect-group] Session key updated (org.apache.kafka.connect.runtime.distributed.DistributedHerder:1582)
And also this:
WARNING: A provider org.apache.kafka.connect.runtime.rest.resources.ConnectorsResource registered in SERVER runtime does not implement any provider interfaces applicable in the SERVER runtime. Due to constraint configuration problems the provider org.apache.kafka.connect.runtime.rest.resources.ConnectorsResource will be ignored.
WARNING: A provider org.apache.kafka.connect.runtime.rest.resources.ConnectorPluginsResource registered in SERVER runtime does not implement any provider interfaces applicable in the SERVER runtime. Due to constraint configuration problems the provider org.apache.kafka.connect.runtime.rest.resources.ConnectorPluginsResource will be ignored.
WARNING: The (sub)resource method createConnector in org.apache.kafka.connect.runtime.rest.resources.ConnectorsResource contains empty path annotation.
WARNING: The (sub)resource method listConnectors in org.apache.kafka.connect.runtime.rest.resources.ConnectorsResource contains empty path annotation.
WARNING: The (sub)resource method listConnectorPlugins in org.apache.kafka.connect.runtime.rest.resources.ConnectorPluginsResource contains empty path annotation.
And when I navigate to http://localhost:8083/connectors in my browser, I get an empty list [].
Environment:
CentOS7
openjdk version "1.8.0_181"
I downloaded confluent-5.3.2-2.12.tar.gz and extracted to /opt/confluent.
I am following "Installing and Running KSQL | Level Up your KSQL by Confluent" (https://youtu.be/icwHpPm-TCA).
Executed the following commands:
[root#srvr0 ~]# cd /opt/confluent/confluent-5.3.2/bin/
[root#srvr0 bin]# confluent start
bash: confluent: command not found...
Update1:
With reference to, https://docs.confluent.io/current/quickstart/ce-quickstart.html, executed the following commands:
curl -L https://cnfl.io/cli | sh -s -- -b /opt/confluent/confluent-5.3.2/bin
/opt/confluent/confluent-5.3.2/bin/confluent-hub install --no-prompt confluentinc/kafka-connect-datagen:latest
/opt/confluent/confluent-5.3.2/bin/confluent local start
Logs:
[root#srvr0 ~]# curl -L https://cnfl.io/cli | sh -s -- -b /opt/confluent/confluent-5.3.2/bin
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 162 0 0 0 0 0 0 --:--:-- 0:00:01 --:--:-- 0
100 10288 100 10288 0 0 3567 0 0:00:02 0:00:02 --:--:-- 16176
confluentinc/cli info checking S3 for latest tag
confluentinc/cli info found version: latest for latest/linux/amd64
confluentinc/cli info NOTICE: see licenses located in /tmp/tmp.h8m7jASeAh/confluent
confluentinc/cli info installed /opt/confluent/confluent-5.3.2/bin/confluent
confluentinc/cli info please ensure /opt/confluent/confluent-5.3.2/bin is in your PATH
[root#srvr0 ~]# cp /tmp/tmp.h8m7jASeAh/confluent
cp: missing destination file operand after ‘/tmp/tmp.h8m7jASeAh/confluent’
Try 'cp --help' for more information.
[root#srvr0 ~]# cp -a /tmp/tmp.h8m7jASeAh/confluent /opt/confluent
[root#srvr0 ~]# /opt/confluent/confluent-5.3.2/bin/confluent-hub install --no-prompt confluentinc/kafka-connect-datagen:latest
Running in a "--no-prompt" mode
Implicit acceptance of the license below:
Apache License 2.0
https://www.apache.org/licenses/LICENSE-2.0
Downloading component Kafka Connect Datagen 0.2.0, provided by Confluent, Inc. from Confluent Hub and installing into /opt/confluent/confluent-5.3.2/share/confluent-hub-components
Adding installation directory to plugin path in the following files:
/opt/confluent/confluent-5.3.2/etc/kafka/connect-distributed.properties
/opt/confluent/confluent-5.3.2/etc/kafka/connect-standalone.properties
/opt/confluent/confluent-5.3.2/etc/schema-registry/connect-avro-distributed.properties
/opt/confluent/confluent-5.3.2/etc/schema-registry/connect-avro-standalone.properties
Completed
[root#srvr0 ~]# /opt/confluent/confluent-5.3.2/bin/confluent local start
The local commands are intended for a single-node development environment
only, NOT for production usage. https://docs.confluent.io/current/cli/index.html
Using CONFLUENT_CURRENT: /tmp/confluent.R3YJZ2UC
Starting zookeeper
zookeeper is [UP]
Starting kafka
kafka is [UP]
Starting schema-registry
schema-registry is [UP]
Starting kafka-rest
kafka-rest is [UP]
Starting connect
connect is [UP]
Starting ksql-server
ksql-server is [UP]
Starting control-center
|control-center failed to start
control-center is [DOWN]
Update2:
Logs:
[root#srvr0 confluent-5.3.2]# cat ./logs/controller.log
[2020-01-16 12:20:40,220] DEBUG preRegister called. Server=com.sun.jmx.mbeanserver.JmxMBeanServer#66d3c617, name=log4j:logger=kafka.controller (kafka.controller)
[2020-01-16 12:21:40,097] INFO [ControllerEventThread controllerId=0] Starting (kafka.controller.ControllerEventManager$ControllerEventThread)
[2020-01-16 12:21:40,174] INFO [Controller id=0] 0 successfully elected as the controller. Epoch incremented to 1 and epoch zk version is now 1 (kafka.controller.KafkaController)
[2020-01-16 12:21:40,176] INFO [Controller id=0] Registering handlers (kafka.controller.KafkaController)
[2020-01-16 12:21:40,182] INFO [Controller id=0] Deleting log dir event notifications (kafka.controller.KafkaController)
[2020-01-16 12:21:40,193] INFO [Controller id=0] Deleting isr change notifications (kafka.controller.KafkaController)
[2020-01-16 12:21:40,197] INFO [Controller id=0] Initializing controller context (kafka.controller.KafkaController)
[2020-01-16 12:21:40,361] INFO [Controller id=0] Initialized broker epochs cache: Map(0 -> 24) (kafka.controller.KafkaController)
[2020-01-16 12:21:40,370] DEBUG [Controller id=0] Register BrokerModifications handler for Set(0) (kafka.controller.KafkaController)
[2020-01-16 12:21:40,384] DEBUG [Channel manager on controller 0]: Controller 0 trying to connect to broker 0 (kafka.controller.ControllerChannelManager)
[2020-01-16 12:21:40,444] INFO [RequestSendThread controllerId=0] Starting (kafka.controller.RequestSendThread)
[2020-01-16 12:21:40,445] INFO [Controller id=0] Partitions being reassigned: Map() (kafka.controller.KafkaController)
[2020-01-16 12:21:40,447] INFO [Controller id=0] Currently active brokers in the cluster: Set(0) (kafka.controller.KafkaController)
[2020-01-16 12:21:40,448] INFO [Controller id=0] Currently shutting brokers in the cluster: Set() (kafka.controller.KafkaController)
[2020-01-16 12:21:40,448] INFO [Controller id=0] Current list of topics in the cluster: Set() (kafka.controller.KafkaController)
[2020-01-16 12:21:40,449] INFO [Controller id=0] Fetching topic deletions in progress (kafka.controller.KafkaController)
[2020-01-16 12:21:40,456] INFO [Controller id=0] List of topics to be deleted: (kafka.controller.KafkaController)
[2020-01-16 12:21:40,456] INFO [Controller id=0] List of topics ineligible for deletion: (kafka.controller.KafkaController)
[2020-01-16 12:21:40,457] INFO [Controller id=0] Initializing topic deletion manager (kafka.controller.KafkaController)
[2020-01-16 12:21:40,458] INFO [Topic Deletion Manager 0] Initializing manager with initial deletions: Set(), initial ineligible deletions: Set() (kafka.controller.TopicDeletionManager)
[2020-01-16 12:21:40,459] INFO [Controller id=0] Sending update metadata request (kafka.controller.KafkaController)
[2020-01-16 12:21:40,485] INFO [ReplicaStateMachine controllerId=0] Initializing replica state (kafka.controller.ZkReplicaStateMachine)
[2020-01-16 12:21:40,487] INFO [ReplicaStateMachine controllerId=0] Triggering online replica state changes (kafka.controller.ZkReplicaStateMachine)
[2020-01-16 12:21:40,518] INFO [ReplicaStateMachine controllerId=0] Triggering offline replica state changes (kafka.controller.ZkReplicaStateMachine)
[2020-01-16 12:21:40,519] DEBUG [ReplicaStateMachine controllerId=0] Started replica state machine with initial state -> Map() (kafka.controller.ZkReplicaStateMachine)
[2020-01-16 12:21:40,523] INFO [PartitionStateMachine controllerId=0] Initializing partition state (kafka.controller.ZkPartitionStateMachine)
[2020-01-16 12:21:40,525] INFO [PartitionStateMachine controllerId=0] Triggering online partition state changes (kafka.controller.ZkPartitionStateMachine)
[2020-01-16 12:21:40,535] WARN [RequestSendThread controllerId=0] Controller 0's connection to broker srvr0:9092 (id: 0 rack: null) was unsuccessful (kafka.controller.RequestSendThread)
java.io.IOException: Connection to srvr0:9092 (id: 0 rack: null) failed.
at org.apache.kafka.clients.NetworkClientUtils.awaitReady(NetworkClientUtils.java:71)
at kafka.controller.RequestSendThread.brokerReady(ControllerChannelManager.scala:295)
at kafka.controller.RequestSendThread.doWork(ControllerChannelManager.scala:249)
at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:89)
[2020-01-16 12:21:40,539] DEBUG [PartitionStateMachine controllerId=0] Started partition state machine with initial state -> Map() (kafka.controller.ZkPartitionStateMachine)
[2020-01-16 12:21:40,540] INFO [Controller id=0] Ready to serve as the new controller with epoch 1 (kafka.controller.KafkaController)
[2020-01-16 12:21:40,542] INFO [Controller id=0] Removing partitions Set() from the list of reassigned partitions in zookeeper (kafka.controller.KafkaController)
[2020-01-16 12:21:40,543] INFO [Controller id=0] No more partitions need to be reassigned. Deleting zk path /admin/reassign_partitions (kafka.controller.KafkaController)
[2020-01-16 12:21:40,550] INFO [Controller id=0] Partitions undergoing preferred replica election: (kafka.controller.KafkaController)
[2020-01-16 12:21:40,551] INFO [Controller id=0] Partitions that completed preferred replica election: (kafka.controller.KafkaController)
[2020-01-16 12:21:40,553] INFO [Controller id=0] Skipping preferred replica election for partitions due to topic deletion: (kafka.controller.KafkaController)
[2020-01-16 12:21:40,554] INFO [Controller id=0] Resuming preferred replica election for partitions: (kafka.controller.KafkaController)
[2020-01-16 12:21:40,555] INFO [Controller id=0] Starting preferred replica leader election for partitions (kafka.controller.KafkaController)
[2020-01-16 12:21:40,593] INFO [Controller id=0] Starting the controller scheduler (kafka.controller.KafkaController)
[2020-01-16 12:21:40,637] WARN [RequestSendThread controllerId=0] Controller 0's connection to broker srvr0:9092 (id: 0 rack: null) was unsuccessful (kafka.controller.RequestSendThread)
java.io.IOException: Connection to srvr0:9092 (id: 0 rack: null) failed.
at org.apache.kafka.clients.NetworkClientUtils.awaitReady(NetworkClientUtils.java:71)
at kafka.controller.RequestSendThread.brokerReady(ControllerChannelManager.scala:295)
at kafka.controller.RequestSendThread.doWork(ControllerChannelManager.scala:249)
at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:89)
[2020-01-16 12:21:40,738] WARN [RequestSendThread controllerId=0] Controller 0's connection to broker srvr0:9092 (id: 0 rack: null) was unsuccessful (kafka.controller.RequestSendThread)
java.io.IOException: Connection to srvr0:9092 (id: 0 rack: null) failed.
...
[2020-01-16 12:21:41,559] INFO [Controller id=0] New topics: [Set(__confluent.support.metrics)], deleted topics: [Set()], new partition replica assignment [Map(__confluent.support.metrics-0 -> Vector(0))] (kafka.controller.KafkaController)
[2020-01-16 12:21:41,559] INFO [Controller id=0] New partition creation callback for __confluent.support.metrics-0 (kafka.controller.KafkaController)
[2020-01-16 12:21:41,653] WARN [RequestSendThread controllerId=0] Controller 0's connection to broker srvr0:9092 (id: 0 rack: null) was unsuccessful (kafka.controller.RequestSendThread)
java.io.IOException: Connection to srvr0:9092 (id: 0 rack: null) failed.
at org.apache.kafka.clients.NetworkClientUtils.awaitReady(NetworkClientUtils.java:71)
at kafka.controller.RequestSendThread.brokerReady(ControllerChannelManager.scala:295)
at kafka.controller.RequestSendThread.doWork(ControllerChannelManager.scala:249)
at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:89)
[2020-01-16 12:21:41,754] WARN [RequestSendThread controllerId=0] Controller 0's connection to broker srvr0:9092 (id: 0 rack: null) was unsuccessful (kafka.controller.RequestSendThread)
java.io.IOException: Connection to srvr0:9092 (id: 0 rack: null) failed.
...
[2020-01-16 12:21:45,596] INFO [Controller id=0] Processing automatic preferred replica leader election (kafka.controller.KafkaController)
[2020-01-16 12:21:45,597] TRACE [Controller id=0] Checking need to trigger auto leader balancing (kafka.controller.KafkaController)
[2020-01-16 12:21:45,601] DEBUG [Controller id=0] Preferred replicas by broker Map(0 -> Map(__confluent.support.metrics-0 -> Vector(0))) (kafka.controller.KafkaController)
[2020-01-16 12:21:45,605] DEBUG [Controller id=0] Topics not in preferred replica for broker 0 Map() (kafka.controller.KafkaController)
[2020-01-16 12:21:45,609] TRACE [Controller id=0] Leader imbalance ratio for broker 0 is 0.0 (kafka.controller.KafkaController)
[2020-01-16 12:21:45,616] WARN [RequestSendThread controllerId=0] Controller 0's connection to broker srvr0:9092 (id: 0 rack: null) was unsuccessful (kafka.controller.RequestSendThread)
java.io.IOException: Connection to srvr0:9092 (id: 0 rack: null) failed.
at org.apache.kafka.clients.NetworkClientUtils.awaitReady(NetworkClientUtils.java:71)
at kafka.controller.RequestSendThread.brokerReady(ControllerChannelManager.scala:295)
at kafka.controller.RequestSendThread.doWork(ControllerChannelManager.scala:249)
at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:89)
[2020-01-16 12:21:45,717] WARN [RequestSendThread controllerId=0] Controller 0's connection to broker srvr0:9092 (id: 0 rack: null) was unsuccessful (kafka.controller.RequestSendThread)
java.io.IOException: Connection to srvr0:9092 (id: 0 rack: null) failed.
...
[2020-01-16 12:22:10,944] INFO [ControllerEventThread controllerId=0] Shutting down (kafka.controller.ControllerEventManager$ControllerEventThread)
[2020-01-16 12:22:10,946] INFO [ControllerEventThread controllerId=0] Stopped (kafka.controller.ControllerEventManager$ControllerEventThread)
[2020-01-16 12:22:10,946] INFO [ControllerEventThread controllerId=0] Shutdown completed (kafka.controller.ControllerEventManager$ControllerEventThread)
[2020-01-16 12:22:10,947] DEBUG [Controller id=0] Resigning (kafka.controller.KafkaController)
[2020-01-16 12:22:10,948] DEBUG [Controller id=0] Unregister BrokerModifications handler for Set(0) (kafka.controller.KafkaController)
[2020-01-16 12:22:10,951] INFO [PartitionStateMachine controllerId=0] Stopped partition state machine (kafka.controller.ZkPartitionStateMachine)
[2020-01-16 12:22:10,953] INFO [ReplicaStateMachine controllerId=0] Stopped replica state machine (kafka.controller.ZkReplicaStateMachine)
[2020-01-16 12:22:10,955] INFO [RequestSendThread controllerId=0] Shutting down (kafka.controller.RequestSendThread)
[2020-01-16 12:22:10,956] TRACE [RequestSendThread controllerId=0] shutdownInitiated latch count reached zero. Shutdown called. (kafka.controller.RequestSendThread)
[2020-01-16 12:22:10,956] INFO [RequestSendThread controllerId=0] Stopped (kafka.controller.RequestSendThread)
[2020-01-16 12:22:10,956] INFO [RequestSendThread controllerId=0] Shutdown completed (kafka.controller.RequestSendThread)
[2020-01-16 12:22:10,960] INFO [Controller id=0] Resigned (kafka.controller.KafkaController)
Update 3:
Now, even worst... only zookeeper is starting. other services are failing to start...
Logs:
[root#srvr0 ~]# /opt/confluent/confluent-5.3.2/bin/confluent local start
The local commands are intended for a single-node development environment
only, NOT for production usage. https://docs.confluent.io/current/cli/index.html
Using CONFLUENT_CURRENT: /tmp/confluent.R3YJZ2UC
Starting zookeeper
zookeeper is [UP]
Starting kafka
-Kafka failed to start
kafka is [DOWN]
Cannot start Schema Registry, Kafka Server is not running. Check your deployment
Error: exit status 127
[root#srvr0 ~]#
Update 4:
confluent local start, zookeeper-server-start and kafka-server-start logs:
[root#srvr0 ~]# /opt/confluent/confluent-5.3.2/bin/confluent local start
Updates are available for confluent. To install them, please run:
$ confluent update
The local commands are intended for a single-node development environment
only, NOT for production usage. https://docs.confluent.io/current/cli/index.html
Using CONFLUENT_CURRENT: /tmp/confluent.xgVLokw7
Starting zookeeper
zookeeper is [UP]
Starting kafka
|Kafka failed to start
kafka is [DOWN]
Cannot start Schema Registry, Kafka Server is not running. Check your deployment
Error: exit status 127
[root#srvr0 ~]# /opt/confluent/confluent-5.3.2/bin/zookeeper-server-start
USAGE: /opt/confluent/confluent-5.3.2/bin/zookeeper-server-start [-daemon] zookeeper.properties
[root#srvr0 ~]# /opt/confluent/confluent-5.3.2/bin/kafka-server-start
USAGE: /opt/confluent/confluent-5.3.2/bin/kafka-server-start [-daemon] server.properties [--override property=value]*
server.properties hasn't been edited and its contents as follows:
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# see kafka.server.KafkaConfig for additional details and defaults
############################# Server Basics #############################
# The id of the broker. This must be set to a unique integer for each broker.
broker.id=0
############################# Socket Server Settings #############################
# The address the socket server listens on. It will get the value returned from
# java.net.InetAddress.getCanonicalHostName() if not configured.
# FORMAT:
# listeners = listener_name://host_name:port
# EXAMPLE:
# listeners = PLAINTEXT://your.host.name:9092
#listeners=PLAINTEXT://:9092
# Hostname and port the broker will advertise to producers and consumers. If not set,
# it uses the value for "listeners" if configured. Otherwise, it will use the value
# returned from java.net.InetAddress.getCanonicalHostName().
#advertised.listeners=PLAINTEXT://your.host.name:9092
# Maps listener names to security protocols, the default is for them to be the same. See the config documentation for more details
#listener.security.protocol.map=PLAINTEXT:PLAINTEXT,SSL:SSL,SASL_PLAINTEXT:SASL_PLAINTEXT,SASL_SSL:SASL_SSL
# The number of threads that the server uses for receiving requests from the network and sending responses to the network
num.network.threads=3
# The number of threads that the server uses for processing requests, which may include disk I/O
num.io.threads=8
# The send buffer (SO_SNDBUF) used by the socket server
socket.send.buffer.bytes=102400
# The receive buffer (SO_RCVBUF) used by the socket server
socket.receive.buffer.bytes=102400
# The maximum size of a request that the socket server will accept (protection against OOM)
socket.request.max.bytes=104857600
############################# Log Basics #############################
# A comma separated list of directories under which to store log files
log.dirs=/tmp/kafka-logs
# The default number of log partitions per topic. More partitions allow greater
# parallelism for consumption, but this will also result in more files across
# the brokers.
num.partitions=1
# The number of threads per data directory to be used for log recovery at startup and flushing at shutdown.
# This value is recommended to be increased for installations with data dirs located in RAID array.
num.recovery.threads.per.data.dir=1
############################# Internal Topic Settings #############################
# The replication factor for the group metadata internal topics "__consumer_offsets" and "__transaction_state"
# For anything other than development testing, a value greater than 1 is recommended for to ensure availability such as 3.
offsets.topic.replication.factor=1
transaction.state.log.replication.factor=1
transaction.state.log.min.isr=1
############################# Log Flush Policy #############################
# Messages are immediately written to the filesystem but by default we only fsync() to sync
# the OS cache lazily. The following configurations control the flush of data to disk.
# There are a few important trade-offs here:
# 1. Durability: Unflushed data may be lost if you are not using replication.
# 2. Latency: Very large flush intervals may lead to latency spikes when the flush does occur as there will be a lot of data to flush.
# 3. Throughput: The flush is generally the most expensive operation, and a small flush interval may lead to excessive seeks.
# The settings below allow one to configure the flush policy to flush data after a period of time or
# every N messages (or both). This can be done globally and overridden on a per-topic basis.
# The number of messages to accept before forcing a flush of data to disk
#log.flush.interval.messages=10000
# The maximum amount of time a message can sit in a log before we force a flush
#log.flush.interval.ms=1000
############################# Log Retention Policy #############################
# The following configurations control the disposal of log segments. The policy can
# be set to delete segments after a period of time, or after a given size has accumulated.
# A segment will be deleted whenever *either* of these criteria are met. Deletion always happens
# from the end of the log.
# The minimum age of a log file to be eligible for deletion due to age
log.retention.hours=168
# A size-based retention policy for logs. Segments are pruned from the log unless the remaining
# segments drop below log.retention.bytes. Functions independently of log.retention.hours.
#log.retention.bytes=1073741824
# The maximum size of a log segment file. When this size is reached a new log segment will be created.
log.segment.bytes=1073741824
# The interval at which log segments are checked to see if they can be deleted according
# to the retention policies
log.retention.check.interval.ms=300000
############################# Zookeeper #############################
# Zookeeper connection string (see zookeeper docs for details).
# This is a comma separated host:port pairs, each corresponding to a zk
# server. e.g. "127.0.0.1:3000,127.0.0.1:3001,127.0.0.1:3002".
# You can also append an optional chroot string to the urls to specify the
# root directory for all kafka znodes.
zookeeper.connect=localhost:2181
# Timeout in ms for connecting to zookeeper
zookeeper.connection.timeout.ms=6000
##################### Confluent Metrics Reporter #######################
# Confluent Control Center and Confluent Auto Data Balancer integration
#
# Uncomment the following lines to publish monitoring data for
# Confluent Control Center and Confluent Auto Data Balancer
# If you are using a dedicated metrics cluster, also adjust the settings
# to point to your metrics kakfa cluster.
#metric.reporters=io.confluent.metrics.reporter.ConfluentMetricsReporter
#confluent.metrics.reporter.bootstrap.servers=localhost:9092
#
# Uncomment the following line if the metrics cluster has a single broker
#confluent.metrics.reporter.topic.replicas=1
##################### Confluent Proactive Support ######################
# If set to true, and confluent-support-metrics package is installed
# then the feature to collect and report support metrics
# ("Metrics") is enabled. If set to false, the feature is disabled.
#
confluent.support.metrics.enable=true
# The customer ID under which support metrics will be collected and
# reported.
#
# When the customer ID is set to "anonymous" (the default), then only a
# reduced set of metrics is being collected and reported.
#
# Confluent customers
# -------------------
# If you are a Confluent customer, then you should replace the default
# value with your actual Confluent customer ID. Doing so will ensure
# that additional support metrics will be collected and reported.
#
confluent.support.customer.id=anonymous
############################# Group Coordinator Settings #############################
# The following configuration specifies the time, in milliseconds, that the GroupCoordinator will delay the initial consumer rebalance.
# The rebalance will be further delayed by the value of group.initial.rebalance.delay.ms as new members join the group, up to a maximum of max.poll.interval.ms.
# The default value for this is 3 seconds.
# We override this to 0 here as it makes for a better out-of-the-box experience for development and testing.
# However, in production environments the default value of 3 seconds is more suitable as this will help to avoid unnecessary, and potentially expensive, rebalances during application startup.
group.initial.rebalance.delay.ms=0
############################# Confluent Authorizer Settings #############################
# Uncomment to enable Confluent Authorizer with support for ACLs, LDAP groups and RBAC
#authorizer.class.name=io.confluent.kafka.security.authorizer.ConfluentServerAuthorizer
# Semi-colon separated list of super users in the format <principalType>:<principalName>
#super.users=
# Specify a valid Confluent license. By default free-tier license will be used
#confluent.license=
# Replication factor for the topic used for licensing. Default is 3.
confluent.license.topic.replication.factor=1
# Uncomment the following lines and specify values where required to enable RBAC
# Enable RBAC provider
#confluent.authorizer.access.rule.providers=ACL,RBAC
# Bootstrap servers for RBAC metadata. Must be provided if this broker is not in the metadata cluster
#confluent.metadata.bootstrap.servers=PLAINTEXT://127.0.0.1:9092
# Replication factor for the metadata topic used for authorization. Default is 3.
confluent.metadata.topic.replication.factor=1
# Listeners for metadata server
#confluent.metadata.server.listeners=http://0.0.0.0:8090
# Advertised listeners for metadata server
#confluent.metadata.server.advertised.listeners=http://127.0.0.1:8090
Please help me in resolving the issue!
It's not clear where srvr0:9092 is defined; I suggest reviewing your server.properties file to fix the connection strings.
You don't need to run confluent at all. You can follow the base Apache Kafka guides for running both Zookeeper and Kafka
zookeeper-server-start + kafka-server-start
Or you can use Confluent's APT/YUM repos rather than just extracting tarballs, then use systemctl to control services.
Or, using Docker is another way to get started quickly.
below is the error i am getting in the console while trying to start the kafka server with kafka-server-start command in command prompt .
ERROR Error while creating log for kafka_example-0 in dir
C:\Users\user11\Softwareskafka_2.12-2.3.0kafka_logs (kafka.server.LogDirFailureChannel)
java.io.IOException: The requested operation cannot be performed on a file with a user-mapped section open
at java.io.RandomAccessFile.setLength(Native Method)
at kafka.log.AbstractIndex.$anonfun$resize$1(AbstractIndex.scala:188)
at scala.runtime.java8.JFunction0$mcZ$sp.apply(JFunction0$mcZ$sp.java:23)
at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:253)
at kafka.log.AbstractIndex.resize(AbstractIndex.scala:174)
at kafka.log.AbstractIndex.$anonfun$trimToValidSize$1(AbstractIndex.scala:240)
at scala.runtime.java8.JFunction0$mcZ$sp.apply(JFunction0$mcZ$sp.java:23)
at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:253)
at kafka.log.AbstractIndex.trimToValidSize(AbstractIndex.scala:240)
INFO [ReplicaManager broker=0] Stopping serving replicas in dir
C:\Users\user11\Softwareskafka_2.12-2.3.0kafka_logs (kafka.server.ReplicaManager)
ERROR [ReplicaManager broker=0] Error while making broker the leader for partition Topic: kafka_example; Partition: 0; Leader: None; AllReplicas: ; InSyncReplicas: in dir None (kafka.server.ReplicaManager)
org.apache.kafka.common.errors.KafkaStorageException: Error while creating log for kafka_example-0 in dir C:\Users\user11\Softwareskafka_2.12-2.3.0kafka_logs
Caused by: java.io.IOException: The requested operation cannot be performed on a file with a user-mapped section open
at java.io.RandomAccessFile.setLength(Native Method)
at kafka.log.AbstractIndex.$anonfun$resize$1(AbstractIndex.scala:188)
at scala.runtime.java8.JFunction0$mcZ$sp.apply(JFunction0$mcZ$sp.java:23)
at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:253)
at kafka.log.AbstractIndex.resize(AbstractIndex.scala:174)
at kafka.log.AbstractIndex.$anonfun$trimToValidSize$1(AbstractIndex.scala:240)
at scala.runtime.java8.JFunction0$mcZ$sp.apply(JFunction0$mcZ$sp.java:23)
at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:253)
at kafka.log.AbstractIndex.trimToValidSize(AbstractIndex.scala:240)
at kafka.log.LogSegment.recover(LogSegment.scala:397)
at kafka.log.Log.recoverSegment(Log.scala:493)
at kafka.log.Log.recoverLog(Log.scala:608)
at kafka.log.Log.$anonfun$loadSegments$3(Log.scala:568)
INFO Replica loaded for partition --from-beginning-0 with initial high watermark 0 (kafka.cluster.Replica)
INFO [Partition --from-beginning-0 broker=0] --from-beginning-0 starts at Leader Epoch 0 from offset 0. Previous Leader Epoch was: -1 (kafka.cluster.Partition)
INFO [GroupMetadataManager brokerId=0] Scheduling loading of offsets and group metadata from __consumer_offsets-0 (kafka.coordinator.group.GroupMetadataManager)
INFO [ReplicaFetcherManager on broker 0] Removed fetcher for partitions Set(__consumer_offsets-0, --from-beginning-0, Kafka_Example-0) (kafka.server.ReplicaFetcherManager)
INFO [ReplicaAlterLogDirsManager on broker 0] Removed fetcher for partitions Set(__consumer_offsets-0, --from-beginning-0, Kafka_Example-0) (kafka.server.ReplicaAlterLogDirsManager)
INFO [ReplicaManager broker=0] Broker 0 stopped fetcher for partitions __consumer_offsets-0,--from-beginning-0,Kafka_Example-0 and stopped moving logs for partitions because they are in the failed log directory C:\Users\user11\Softwareskafka_2.12-2.3.0kafka_logs. (kafka.server.ReplicaManager)
INFO Stopping serving logs in dir C:\Users\user11\Softwareskafka_2.12-2.3.0kafka_logs (kafka.log.LogManager)
ERROR Shutdown broker because all log dirs in C:\Users\user11\Softwareskafka_2.12-2.3.0kafka_logs have failed (kafka.log.LogManager)
on my local using java 8 version and above subject mentioned is kafka version.
We are running Kafka (vesion kafka_2.11-0.10.1.0) in a 2 node cluster.
We have 2 producers (Java API) acting on different topics. Each topic has single partition.
The topic where we had this issue, has one consumer running.
This set up has been running fine for 3 months, and we saw this issue. All the suggested cases/solutions for this issue in other forums don't seem to apply for my scenario.
Exception at producer;
-2017-11-25T17:40:33,035 [kafka-producer-network-thread | producer-1] ERROR client.producer.BingLogProducerCallback - Encountered exception
in sending message ; > org.apache.kafka.common.errors.NotLeaderForPartitionException: This
server is not the leader for that topic-partition.
We haven't enabled retries for the messages, because this is transactional data and we want to maintain the order.
Producer config:
bootstrap.servers : server1ip:9092
acks :all
retries : 0
linger.ms :0
buffer.memory :10240000
max.request.size :1024000
key.serializer : org.apache.kafka.common.serialization.StringSerializer
value.serializer : org.apache.kafka.common.serialization.StringSerializer
We are connecting to server1 at both producer and consumer.
The controller log at server2 indicates there is some shutdown happened at during sametime, but I dont understand why this happened.
[2017-11-25 17:31:44,776] DEBUG [Controller 2]: topics not in
preferred replica Map() (kafka.controller.KafkaController) [2017-11-25
17:31:44,776] TRACE [Controller 2]: leader imbalance ratio for broker
2 is 0.000000 (kafka.controller.KafkaController) [2017-11-25
17:31:44,776] DEBUG [Controller 2]: topics not in preferred replica
Map() (kafka.controller.KafkaController) [2017-11-25 17:31:44,776]
TRACE [Controller 2]: leader imbalance ratio for broker 1 is 0.000000
(kafka.controller.KafkaController) [2017-11-25 17:34:18,314] INFO
[SessionExpirationListener on 2], ZK expired; shut down all controller
components and try to re-elect
(kafka.controller.KafkaController$SessionExpirationListener)
[2017-11-25 17:34:18,317] DEBUG [Controller 2]: Controller resigning,
broker id 2 (kafka.controller.KafkaController) [2017-11-25
17:34:18,317] DEBUG [Controller 2]: De-registering
IsrChangeNotificationListener (kafka.controller.KafkaController)
[2017-11-25 17:34:18,317] INFO [delete-topics-thread-2], Shutting down
(kafka.controller.TopicDeletionManager$DeleteTopicsThread) [2017-11-25
17:34:18,317] INFO [delete-topics-thread-2], Stopped
(kafka.controller.TopicDeletionManager$DeleteTopicsThread) [2017-11-25
17:34:18,318] INFO [delete-topics-thread-2], Shutdown completed
(kafka.controller.TopicDeletionManager$DeleteTopicsThread) [2017-11-25
17:34:18,318] INFO [Partition state machine on Controller 2]: Stopped
partition state machine (kafka.controller.PartitionStateMachine)
[2017-11-25 17:34:18,318] INFO [Replica state machine on controller
2]: Stopped replica state machine
(kafka.controller.ReplicaStateMachine) [2017-11-25 17:34:18,318] INFO
[Controller-2-to-broker-2-send-thread], Shutting down
(kafka.controller.RequestSendThread) [2017-11-25 17:34:18,318] INFO
[Controller-2-to-broker-2-send-thread], Stopped
(kafka.controller.RequestSendThread) [2017-11-25 17:34:18,319] INFO
[Controller-2-to-broker-2-send-thread], Shutdown completed
(kafka.controller.RequestSendThread) [2017-11-25 17:34:18,319] INFO
[Controller-2-to-broker-1-send-thread], Shutting down
(kafka.controller.RequestSendThread) [2017-11-25 17:34:18,319] INFO
[Controller-2-to-broker-1-send-thread], Stopped
(kafka.controller.RequestSendThread) [2017-11-25 17:34:18,319] INFO
[Controller-2-to-broker-1-send-thread], Shutdown completed
(kafka.controller.RequestSendThread) [2017-11-25 17:34:18,319] INFO
[Controller 2]: Broker 2 resigned as the controller
(kafka.controller.KafkaController) [2017-11-25 17:34:18,353] DEBUG
[IsrChangeNotificationListener] Fired!!!
(kafka.controller.IsrChangeNotificationListener) [2017-11-25
17:34:18,353] DEBUG [IsrChangeNotificationListener] Fired!!!
(kafka.controller.IsrChangeNotificationListener) [2017-11-25
17:34:18,354] INFO [BrokerChangeListener on Controller 2]: Broker
change listener fired for path /brokers/ids with children 1,2
(kafka.controller.ReplicaStateMachine$BrokerChangeListener)
[2017-11-25 17:34:18,355] DEBUG [DeleteTopicsListener on 2]: Delete
topics listener fired for topics to be deleted
(kafka.controller.PartitionStateMachine$DeleteTopicsListener)
[2017-11-25 17:34:18,362] INFO [AddPartitionsListener on 2]: Partition
modification triggered {"version":1,"partitions":{"0":[1]}} for path
/brokers/topics/ESQ
(kafka.controller.PartitionStateMachine$PartitionModificationsListener)
[2017-11-25 17:34:18,368] INFO [AddPartitionsListener on 2]: Partition
modification triggered {"version":1,"partitions":{"0":[1]}} for path
/brokers/topics/Test1
(kafka.controller.PartitionStateMachine$PartitionModificationsListener)
[2017-11-25 17:34:18,369] INFO [AddPartitionsListener on 2]: Partition
modification triggered {"version":1,"partitions":{"0":[2]}} for path
/brokers/topics/ImageQ
(kafka.controller.PartitionStateMachine$PartitionModificationsListener)
[2017-11-25 17:34:18,374] INFO [AddPartitionsListener on 2]: Partition
modification triggered
{"version":1,"partitions":{"8":[1,2],"4":[1,2],"9":[2,1],"5":[2,1],"6":[1,2],"1":[2,1],"0":[1,2],"2":[1,2],"7":[2,1],"3":[2,1]}}
for path /brokers/topics/NMS_NotifyQ
(kafka.controller.PartitionStateMachine$PartitionModificationsListener)
[2017-11-25 17:34:18,375] INFO [AddPartitionsListener on 2]: Partition
modification triggered {"version":1,"partitions":{"0":[1]}} for path
/brokers/topics/TempBinLogReqQ #
when starting the kafka broker i am getting some error:
I am giving last few lines of the error log:
INFO zookeeper state changed (SyncConnected) (org.I0Itec.zkclient.ZkClient)
[2016-05-12 01:07:01,759] INFO Log directory '/var/logs/service-bridge-logs' not found, creating it. (kafka.log.LogManager)
[2016-05-12 01:07:01,778] INFO Loading logs. (kafka.log.LogManager)
[2016-05-12 01:07:01,796] INFO Logs loading complete. (kafka.log.LogManager)
[2016-05-12 01:07:01,797] INFO Starting log cleanup with a period of 300000 ms. (kafka.log.LogManager)
[2016-05-12 01:07:01,806] INFO Starting log flusher with a default period of 9223372036854775807 ms. (kafka.log.LogManager)
[2016-05-12 01:07:01,874] INFO Awaiting socket connections on gns3-d.cloudapp.net:9092. (kafka.network.Acceptor)
[2016-05-12 01:07:01,875] INFO [Socket Server on Broker 2], Started (kafka.network.SocketServer)
[2016-05-12 01:07:02,042] INFO Will not load MX4J, mx4j-tools.jar is not in the classpath (kafka.utils.Mx4jLoader$)
[2016-05-12 01:07:02,168] INFO 2 successfully elected as leader (kafka.server.ZookeeperLeaderElector)
[2016-05-12 01:07:02,386] INFO Registered broker 2 at path /brokers/ids/2 with address 10.1.0.4:9092. (kafka.utils.ZkUtils$)
[2016-05-12 01:07:02,416] INFO [Kafka Server 2], started (kafka.server.KafkaServer)
[2016-05-12 01:07:02,529] INFO New leader is 2 (kafka.server.ZookeeperLeaderElector$LeaderChangeListener)
[2016-05-12 01:07:25,798] ERROR Closing socket for /40.122.64.23 because of error (kafka.network.Processor)
java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
at sun.nio.ch.IOUtil.read(IOUtil.java:197)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
at kafka.utils.Utils$.read(Utils.scala:380)
at kafka.network.BoundedByteBufferReceive.readFrom(BoundedByteBufferReceive.scala:54)
at kafka.network.Processor.read(SocketServer.scala:444)
at kafka.network.Processor.run(SocketServer.scala:340)
at java.lang.Thread.run(Thread.java:745)
while in zookeeper sidealso am getting few error:
INFO Established session 0x154a35b64f40000 with negotiated timeout 6000 for client /10.1.0.4:36673 (org.apache.zookeeper.server.ZooKeeperServer)
[2016-05-12 01:07:02,313] INFO Got user-level KeeperException when processing sessionid:0x154a35b64f40000 type:delete cxid:0x1d zxid:0x52 txntype:-1 reqpath:n/a Error Path:/admin/preferred_replica_election Error:KeeperErrorCode = NoNode for /admin/preferred_replica_election (org.apache.zookeeper.server.PrepRequestProcessor)
[2016-05-12 01:08:33,001] INFO Expiring session 0x154a35b64f40000, timeout of 6000ms exceeded (org.apache.zookeeper.server.ZooKeeperServer)
[2016-05-12 01:08:33,001] INFO Processed session termination for sessionid: 0x154a35b64f40000 (org.apache.zookeeper.server.PrepRequestProcessor)
[2016-05-12 01:08:33,017] INFO Closed socket connection for client /10.1.0.4:36673 which had sessionid 0x154a35b64f40000 (org.apache.zookeeper.server.NIOServerCnxn)
Any idea guys??
Thanks in advance..
As user avr pointed out, this is a known bug in Kafka 0.8.2.x
These are routine informational messages misclassified as ERROR.
As of Kafka v0.9.0.0, the loglevel has been corrected to WARN.