I a consuming Kafka messages from Kafka cli with following command
kafka-console-consumer.bat --bootstrap-server kafka:9092 --from-beginning --topic mytopic --consumer.config ....\config\consumer.properties
and I am getting following errors
[2022-09-23 15:59:33,175] WARN [Consumer clientId=console-consumer, groupId=group] Bootstrap broker kafka:9092 (id: -1 rack: null) disconnected (org.apache.kafka.clients.NetworkClient)
[2022-09-23 15:59:54,344] WARN [Consumer clientId=console-consumer, groupId=group] Connection to node -1 (kafka:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
[2022-09-23 15:59:54,345] WARN [Consumer clientId=console-consumer, groupId=group] Bootstrap broker kafka:9092 (id: -1 rack: null) disconnected (org.apache.kafka.clients.NetworkClient)
[2022-09-23 16:00:15,634] WARN [Consumer clientId=console-consumer, groupId=group] Connection to node -1 (kafka:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
I have updated server.properties with following configuration
port = 9092
advertised.host.name = kafka
KAFKA_CFG_LISTENERS=PLAINTEXT://kafka:9092
advertised.listeners=PLAINTEXT://kafka:9092
Related
[kafka consumer[kafka-producer][1]```
[root#kafka1 kafka_2.13-2.7.0]# bin/kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list kafka1:9092 --topic new-cluster-test --time -1 --offsets 1 | awk -F ":" '{sum += $3} END {print sum}'
[2021-05-18 02:33:16,390] WARN [Consumer clientId=GetOffsetShell, groupId=null] Bootstrap broker kafka1:9092 (id: -1 rack: null) disconnected (org.apache.kafka.clients.NetworkClient)
[2021-05-18 02:33:16,794] WARN [Consumer clientId=GetOffsetShell, groupId=null] Bootstrap broker kafka1:9092 (id: -1 rack: null) disconnected (org.apache.kafka.clients.NetworkClient)
[2021-05-18 02:33:17,198] WARN [Consumer clientId=GetOffsetShell, groupId=null] Bootstrap broker kafka1:9092 (id: -1 rack: null) disconnected (org.apache.kafka.clients.NetworkClient)
[2021-05-18 02:33:17,546] WARN [Consumer clientId=GetOffsetShell, groupId=null] Bootstrap broker kafka1:9092 (id: -1 rack: null) disconnected (org.apache.kafka.clients.NetworkClient)
[2021-05-18 02:33:17,824] WARN [Consumer clientId=GetOffsetShell, groupId=null] Bootstrap broker kafka1:9092 (id: -1 rack: null) disconnected (org.apache.kafka.clients.NetworkClient)
[2021-05-18 02:33:18,227] WARN [Consumer clientId=GetOffsetShell, groupId=null] Bootstrap broker kafka1:9092 (id: -1 rack: null) disconnected (org.apache.kafka.clients.NetworkClient)
^C
[root#kafka1 kafka_2.13-2.7.0]# bin/kafka-topics.sh --version
2.7.0 (Commit:448719dc99a19793)
[root#kafka1 kafka_2.13-2.7.0]#
[1]: https://i.stack.imgur.com/s93dH.png
I was using SASL_PLAINTEXT listeners protocol with krb5 and so it is not authenticating without config(client.properties) file on port 9092.
I have exported the JMX_PORT in kafka-server-start.sh and added dual listeners for kafka (SASL_PLAINTEXT and PLAINTEXT) on separate port(9094).
Restarted the Zookeeper and Kakfa, Now I'm able to get the count.
[root#kafka1 kafka_2.13-2.7.0]# bin/kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list kafka1:9094 --topic new-cluster-test --time -1 | awk -F ":" '{sum += $3} END {print sum}'
114
[root#kafka1 kafka_2.13-2.7.0]# bin/kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list kafka1:9094 --topic new-cluster-test --time -1
new-cluster-test:0:13
new-cluster-test:1:19
new-cluster-test:2:28
new-cluster-test:3:15
new-cluster-test:4:14
new-cluster-test:5:25
When I write to console:
kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test-first
to create a kafka consumer in VPS it gives me error like this:
"[2021-03-08 18:54:24,548] WARN [Consumer clientId=consumer-1, groupId=console-consumer-19666] Connection to node -1 (localhost/127.0.0.1:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)"
How to solve this error ?
*I checked all of the questions with same error.
**kafka and zookeeper are running.
***212.71.238.187 is my VPS ip
server.properties:
broker.id=0
listeners=PLAINTEXT://0.0.0.0:9092
advertised.listeners=PLAINTEXT://212.71.238.187:9092
zookeeper.connect=212.71.238.187:2181
I am working on Spring Batch and Apache Kafka Integration. Before posting the question I went over web : Is there a way to delete all the data from a topic or delete the topic before every run? to find out better solution, but did not find out.
I am using Kafka version 2.11.
I want to delete all data under the topic without stopping either Zookeeper or Kafka. How can we do that ?
Below commands causes lot of issues in windows
C:\kafka_2.11-2.3.1\bin\windows>kafka-topics.bat --zookeeper localhost:2181 --delete --topic customers
Topic customers is marked for deletion.
Note: This will have no impact if delete.topic.enable is not set to true.
C:\kafka_2.11-2.3.1\bin\windows>kafka-topics.bat --zookeeper localhost:2181 --delete --topic test
C:\kafka_2.11-2.3.1\bin\windows>kafka-console-consumer.bat --bootstrap-server localhost:9092 --topic customers --from-beginning
[2020-04-21 10:25:02,812] WARN [Consumer clientId=consumer-1, groupId=console-consumer-65075] Connection to node -1 (localhost/127.0.0.1:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
[2020-04-21 10:25:04,886] WARN [Consumer clientId=consumer-1, groupId=console-consumer-65075] Connection to node -1 (localhost/127.0.0.1:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
[2020-04-21 10:25:06,996] WARN [Consumer clientId=consumer-1, groupId=console-consumer-65075] Connection to node -1 (localhost/127.0.0.1:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
[2020-04-21 10:25:09,267] WARN [Consumer clientId=consumer-1, groupId=console-consumer-65075] Connection to node -1 (localhost/127.0.0.1:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
[2020-04-21 10:25:11,744] WARN [Consumer clientId=consumer-1, groupId=console-consumer-65075] Connection to node -1 (localhost/127.0.0.1:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
Processed a total of 0 messages
Terminate batch job (Y/N)?
^C
C:\kafka_2.11-2.3.1\bin\windows>
I am using Kafka version 2.11.
There is no Kafka 2.11. Your command prompt says kafka_2.11-2.3.1: hence, you are using Kafka 2.3.1. The 2.11 part is the Scala version that was used during compilation.
Note: This will have no impact if delete.topic.enable is not set to true.
Did you check your broker configs if delete.topic.enable is set to true? If yes, you should be able to delete a topic without stopping ZK or the brokers. Note though, that deleting topics is async, i.e., when you command returns the topic is not deleted yet and it will take some time until the command is executed.
I am using the Debezium Postgres connector. I have two tables in Postgres named 'publications' and 'comments'. kafka and zookeeper are running in docker containers as per the standard examples. The postgres is running locally. After using the debezium postgres connect, I have the following topics :
$ bin/kafka-topics.sh --list --zookeeper localhost:2181
__consumer_offsets
dbserver1.public.comments
dbserver1.public.publications
my_connect_configs
my_connect_offsets
my_connect_statuses
I would like to see a list of messages in the topic:
$ bin/kafka-console-consumer.sh --bootstrap-server kafka:9092 --topic dbserver1.public.publications
[2019-06-03 21:55:16,180] WARN [Consumer clientId=consumer-1,
groupId=console-consumer-5221] Connection to node -1
(kafka/23.202.231.166:9092) could not be established. Broker may not
be available. (org.apache.kafka.clients.NetworkClient) [2019-06-03
21:55:16,289] WARN [Consumer clientId=consumer-1,
groupId=console-consumer-5221] Connection to node -1
(kafka/23.202.231.166:9092) could not be established. Broker may not
be available. (org.apache.kafka.clients.NetworkClient) [2019-06-03
21:55:16,443] WARN [Consumer clientId=consumer-1,
groupId=console-consumer-5221] Connection to node -1
(kafka/23.202.231.166:9092) could not be established. Broker may not
be available. (org.apache.kafka.clients.NetworkClient) [2019-06-03
21:55:16,721] WARN [Consumer clientId=consumer-1,
groupId=console-consumer-5221] Connection to node -1
(kafka/23.202.231.166:9092) could not be established. Broker may not
be available. (org.apache.kafka.clients.NetworkClient) [2019-06-03
21:55:17,145] WARN [Consumer clientId=consumer-1,
groupId=console-consumer-5221] Connection to node -1
(kafka/23.202.231.166:9092) could not be established. Broker may not
be available. (org.apache.kafka.clients.NetworkClient) [2019-06-03
21:55:18,017] WARN [Consumer clientId=consumer-1,
groupId=console-consumer-5221] Connection to node -1
(kafka/23.202.231.166:9092) could not be established. Broker may not
be available. (org.apache.kafka.clients.NetworkClient) ^CProcessed a
total of 0 messages
$ bin/kafka-console-consumer.sh --bootstrap-server localhost:9092
--topic dbserver1.public.publications
[2019-06-03 21:55:16,180] WARN [Consumer clientId=consumer-1,
groupId=console-consumer-5221] Connection to node -1
(kafka/23.202.231.166:9092) could not be established. Broker may not
be available. (org.apache.kafka.clients.NetworkClient)
How do I specify the correct value for bootstrap-server? Thanks.
I am assuming you are trying to connect to kafka broker from an external server.
Since you have already mentioned your Kafka and Zookeeper instances are running from docker images. You need to identify your docker images external port corresponding to 9092 as well as its external IP address and you have to those along with --bootstrap-server parameter while executing command kafka-console-consumer.sh
If you are running the kafka-console-consumer.sh outside of docker then you should use localhost hostname. If inside the Docker container then make sure it is in container that sees kafka hostname.
I have installed Kafka and doing some basic testing. I am able to create topics using scripts provided under Kafka-broker/bin folder.
But when I am trying to produce message getting below WARNing every time I run this. And no message is getting generated. Please advice.
[root#node2 bin]# ./kafka-console-producer.sh --broker-list localhost:9092 --topic test_master
>testmsg1
[2019-05-15 06:25:19,092] WARN [Producer clientId=console-producer] Connection to node -1 could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
[2019-05-15 06:25:19,197] WARN [Producer clientId=console-producer] Connection to node -1 could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
[2019-05-15 06:25:19,349] WARN [Producer clientId=console-producer] Connection to node -1 could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
[2019-05-15 06:25:19,562] WARN [Producer clientId=console-producer] Connection to node -1 could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
[2019-05-15 06:25:20,017] WARN [Producer clientId=console-producer] Connection to node -1 could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
[2019-05-15 06:25:20,876] WARN [Producer clientId=console-producer] Connection to node -1 could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
[2019-05-15 06:25:21,987] WARN [Producer clientId=console-producer] Connection to node -1 could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
[2019-05-15 06:25:22,957] WARN [Producer clientId=console-producer] Connection to node -1 could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
[2019-05-15 06:25:23,818] WARN [Producer clientId=console-producer] Connection to node -1 could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
^Corg.apache.kafka.common.KafkaException: Producer closed while send in progress
at org.apache.kafka.clients.producer.KafkaProducer.doSend(KafkaProducer.java:826)
at org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:803)
at kafka.tools.ConsoleProducer$.send(ConsoleProducer.scala:75)
at kafka.tools.ConsoleProducer$.main(ConsoleProducer.scala:57)
at kafka.tools.ConsoleProducer.main(ConsoleProducer.scala)
Caused by: org.apache.kafka.common.KafkaException: Requested metadata update after close
at org.apache.kafka.clients.Metadata.awaitUpdate(Metadata.java:188)
at org.apache.kafka.clients.producer.KafkaProducer.waitOnMetadata(KafkaProducer.java:938)
at org.apache.kafka.clients.producer.KafkaProducer.doSend(KafkaProducer.java:823)
... 4 more
Open Server.xml of each broker of your cluster and make following changes
Change the listeners=PLAINTEXT://:9092 to listeners=PLAINTEXT://<our ip address>:9092
Just remove your local host and write the port
EXAMPLE:
{--broker-list 172.0.0.1:9092} will be changed to { --broker-list :9092 }