Kafka 0.10.1.0 Test Server: Request METADATA failed - apache-kafka

I'm getting basic errors after setting up a super simple single-instance Kafka VM. This is for tiny volume development testing.
This is using the latest Confluent Platform 3.1.1 which includes the almost latest Kafka 0.10.1.0.
FYI, a slightly newer bug patch Kafka 0.10.1.1 is out, but the next post 3.1.1 Confluent Platform binary that includes that isn't available quite yet.
I configure /etc/kafka/server.properties with (I'm using a static local IP for dev testing simplicity):
listeners=PLAINTEXT://192.168.50.20:9092
advertised.listeners=PLAINTEXT://192.168.50.20:9092
(is that right?)
Simple console admin commands are generating errors. This leads me to believe that there is something wrong with the basic setup/configuration.
~$ /usr/bin/kafka-consumer-groups --new-consumer --bootstrap-server localhost:9092 --list
Error while executing consumer group command Request METADATA failed on brokers List(localhost:9092 (id: -1 rack: null))
java.lang.RuntimeException: Request METADATA failed on brokers List(localhost:9092 (id: -1 rack: null))
at kafka.admin.AdminClient.sendAnyNode(AdminClient.scala:67)
at kafka.admin.AdminClient.findAllBrokers(AdminClient.scala:87)
at kafka.admin.AdminClient.listAllGroups(AdminClient.scala:96)
at kafka.admin.AdminClient.listAllGroupsFlattened(AdminClient.scala:117)
at kafka.admin.AdminClient.listAllConsumerGroupsFlattened(AdminClient.scala:121)
at kafka.admin.ConsumerGroupCommand$KafkaConsumerGroupService.list(ConsumerGroupCommand.scala:304)
at kafka.admin.ConsumerGroupCommand$.main(ConsumerGroupCommand.scala:66)
at kafka.admin.ConsumerGroupCommand.main(ConsumerGroupCommand.scala)
EDIT: The problem, thanks to Gondola_Ride, was that I specified the IP in listeners in server.properties. I could connect via that IP, but not via localhost. The solution was to use host 0.0.0.0 which is Kafka's convention for binding to all local TCP interfaces:
listeners=PLAINTEXT://0.0.0.0:9092
advertised.listeners=PLAINTEXT://192.168.50.20:9092

Try adding an entry in the /etc/hosts on the host where you are running this command for this host 192.168.50.20 and see if it works
Something like
127.0.0.1 localhost.localdomain localhost
OR
192.168.50.20 hostname hostname-alias
Then try using it in the command
OR
Try using ip address directly in the command instead of localhost

Related

unable to connect to kafka broker (via zookeeper) using Conduktor client

Able to connect successfully to local kafka broker/cluster running locally (dockerized) using Conduktor, but when trying to connect to Kafka cluster running on Unix VM, getting below error.
Error:
"The broker [...] is reachable but Kafka can't connect. Ensure you have access to the advertised listeners of the the brokers and the proper authorization"
Appreciate any assistance.
running locally (dockerized)
When running in docker, you need to ensure that the ports are accessible from outside of your container. To verify this, try doing a telnet <ip> <port> and check if you are able to connect.
Since the error message says, the broker is reachable, I suppose you would be able to successfully telnet to the broker.
Next, check your broker config called advertised.listeners. Here you need to mention your IP:Port combination where IP is what you will be giving in your client program i.e. Conduktor.
An example for that would be
advertised.listeners=PLAINTEXT://1.2.3.4:9092
and then restart your broker and reconnect. If you are using ssl then you need to provide some extra configuration. See Configuring Kafka brokers for more.
Try to add in /etc/hosts (Unix-like) or C:\Windows\System32\drivers\etc\hosts (windows-like) the Kafka server in such manner kafka_server_ip kafka_server_name_in_dns (e.g. 10.10.0.1 kafka).

Trouble with Apache Kafka to Allow External Connections

I'm just having a difficult time with Kafka right now, but I feel like I'm close.
I have two VMs on FreeNAS running locally. Both Running Ubuntu 18.04 LTS.
VM Graylog: 192.168.1.25. Running Graylog Server. Working well retrieving rsyslogs and apache from itself.
VM Kafka: 192.168.1.16. Running Kafka.
My goal is to have VM Graylog pull logs from VM Kafka, via a Graylog Kafka UDP input. The secondary goal is to replicate this, except tha the Kafka instance will sit on my VPS server feeding apache logs from a website. Of course, I want to test this in a dev environment first.
I am able to have my VM Kafka server successfully listen through this line of code:
/opt/kafka_2.13-2.6.0/bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic rsyslog_kafka --from-beginning
This is my 60-kafka.conf file:
module(load="omkafka")
template(name="json"
type="list"
option.json="on") {
constant(value="{")
constant(value="\"#timestamp\":\"") property(name="timereported" dateFormat="rfc33$
constant(value="\",\"#version\":\"1")
constant(value="\",\"message\":\"") property(name="msg")
constant(value="\",\"host\":\"") property(name="hostname")
constant(value="\",\"severity\":\"") property(name="syslogseverity-text")
constant(value="\",\"facility\":\"") property(name="syslogfacility-text")
constant(value="\",\"programname\":\"") property(name="programname")
constant(value="\",\"procid\":\"") property(name="procid")
constant(value="\"}\n")
}
action(
broker=["192.168.1.16:9092"]
type="omkafka"
topic="rsyslog_kafka"
template="json"
)
I'm using the default server.properties file which doesn't contain any listeners, just the defaults. I do understand I need to set the listeners and advertised.listeners.
I've attempted the following settings to no avail:
Attempt 1:
listeners = PLAINTEXT://localhost:9092
advertised.listeners=PLAINTEXT://192.168.1.16:9092
Attempt 2:
listeners = PLAINTEXT://127.0.0.1:9092
advertised.listeners=PLAINTEXT://192.168.1.16:9092
This after reloading both Kafka and Rsyslog and confirming their statuses are active.
Example errors when attempting to read messages.
Bunch of these
[2020-08-20 00:52:42,248] WARN [Consumer clientId=consumer-console-consumer-70205-1, groupId=console-consumer-70205] Connection to node -1 (localhost/127.0.0.1:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
Followed by an infinite amount of these:
[2020-08-20 00:48:50,598] WARN [Consumer clientId=consumer-console-consumer-11975-1, groupId=console-consumer-11975] Error while fetching metadata with correlation id 254 : {rsyslog_kafka=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)
I feel like I'm close. Perhaps there is something I'm just understanding. I've read lots of similar articles where they say just replace the IP addresses with your server. I feel like I've done that, with no success.
You need to set listeners to PLAINTEXT://0.0.0.0:9092 in order to bind externally.
The advertised listener ought to be set to an address that your consumers will be able to use to discover the cluster
Note: Docker Compose might be easier than VMs

Kafka consumer group script to see all consumer group not working

When I execute the below command in kafka
./kafka-consumer-groups.sh --bootstrap-server sample-address:9092 --list
I'm facing the below error
Error: Executing consumer group command failed due to org.apache.kafka.common.KafkaException: Failed to find brokers to send ListGroups
java.util.concurrent.ExecutionException: org.apache.kafka.common.KafkaException: Failed to find brokers to send ListGroups
at org.apache.kafka.common.internals.KafkaFutureImpl.wrapAndThrow(KafkaFutureImpl.java:45)
at org.apache.kafka.common.internals.KafkaFutureImpl.access$000(KafkaFutureImpl.java:32)
at org.apache.kafka.common.internals.KafkaFutureImpl$SingleWaiter.await(KafkaFutureImpl.java:89)
at org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:262)
at kafka.admin.ConsumerGroupCommand$ConsumerGroupService.listGroups(ConsumerGroupCommand.scala:132)
at kafka.admin.ConsumerGroupCommand$.main(ConsumerGroupCommand.scala:58)
at kafka.admin.ConsumerGroupCommand.main(ConsumerGroupCommand.scala)
Caused by: org.apache.kafka.common.KafkaException: Failed to find brokers to send ListGroups
at org.apache.kafka.clients.admin.KafkaAdminClient$22.handleFailure(KafkaAdminClient.java:2610)
at org.apache.kafka.clients.admin.KafkaAdminClient$Call.fail(KafkaAdminClient.java:614)
at org.apache.kafka.clients.admin.KafkaAdminClient$TimeoutProcessor.handleTimeouts(KafkaAdminClient.java:730)
at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.timeoutPendingCalls(KafkaAdminClient.java:798)
at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.run(KafkaAdminClient.java:1092)
at java.base/java.lang.Thread.run(Thread.java:835)
Caused by: org.apache.kafka.common.errors.TimeoutException: Timed out waiting for a node assignment.
In my case I noticed that we were using SSL
listeners=SSL://sample-address:9092
so I figured that I need to pass SSL properties in the command and it worked
bin/kafka-consumer-groups.sh --bootstrap-server sample-address:9092 --list --command-config /kafka/config/client-ssl.properties
client-ssl.properties
bootstrap.servers=sample-address:9092
security.protocol=SSL
ssl.truststore.location=/keys/truststore.jks
ssl.truststore.password=*****
ssl.keystore.location=/keys/keystore.jks
ssl.keystore.password=*****
After a lot of debugging, I have replicated this scenario and below solution working for me.
I have done changes in server.properties (that is mainly responsible for starting kafka server) instead of "localhost" pass "IP address" in listeners key.
Find detailed step below:-
These are configurations that you have to make sure while running a command.
Check for a correct IP address and port combination passed in command
bin/kafka-consumer-groups.sh --bootstrap-server 192.168.X.X:4848 --list
Main important point , configure listeners with IP address in server.properties correctly .
listeners=PLAINTEXT://**192.168.X.X:4848 --working**
listeners=PLAINTEXT://localhost:4848 --Not working
Post-change, restart kafka server.
Note:- This issue generally gets reproduce/comes in virtual box. After changing network setting like NAT to bridge.
I ran into the "Failed to find brokers to send ListGroups" problem but with the "Timed out waiting to send the call" exception.
In this case the problem was that the bootstrap server was not reachable from the machine where I was running the kafka-consumer-groups cli tool.
Since we have a VPC peering between our Kubernetes cluster in GCP and Confluent Cloud, I resolved the issue by executing the confluent cli tool inside the k8s cluster.

How to Setup a Public Kafka Broker Using a Dynamic DNS?

I configured a Kafka Cluster with 3 brokers using 3 Zookeepers along with each broker. Figure bellow presents a graphical representation of my cluster.
A producer and consumer test in the same network using the host 192.168.0.10 worked perfectly via kafka-console-producer and kafka-console-consumer commands.
Based on that context, when I try to produce some data via kafka-console-producer.sh --broker-list DYNAMIC_DNS_ADDR:30192,DYNAMIC_DNS_ADDR:30292,DYNAMIC_DNS_ADDR:30392 --topic twitter_tweets through the Internet, I am getting the following error:
[2018-12-10 09:59:20,772] ERROR Error when sending message to topic twitter_tweets with key: null, value: 16 bytes with error: (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)
org.apache.kafka.common.errors.TimeoutException: Expiring 1 record(s) for twitter_tweets-1: 1505 ms has passed since batch creation plus linger time
[2018-12-10 09:59:22,273] WARN [Producer clientId=console-producer] Connection to node 1 could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
Broker listeners are configured with the following properties:
listeners=PLAINTEXT://0.0.0.0:9092,SSL://0.0.0.0:9443
advertised.listeners=PLAINTEXT://192.168.0.241:9092,SSL://192.168.0.241:9443
Obviously, the IP address changed in each broker for the advertised.listeners property. I am using CentOS 6.10 and Kafka 2.0.1 for that setup. A telnet test worked. Another forward to a Kafka REST Proxy port is working via the Internet and listing all topics.
See https://rmoff.net/2018/08/02/kafka-listeners-explained/
You need two listeners—one responding to and advertising the internal addresses, one for the external one.
The key thing is that the listener that your client connects to will return the host address and port of that listener.
At the moment you're spoofing your external one to your internal one, and your external traffic is thus hitting the internal listener.
You need something like this (varying the IP/hostname of the aws_internal_listener as required per broker):
KAFKA_LISTENERS: aws_internal_listener://192.168.0.241:9092,external_listener://192.168.0.241:29092
KAFKA_ADVERTISED_LISTENERS: aws_internal_listener://192.168.0.241:9092,external_listener://DYNAMIC_DNS_ADDR:29092
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: aws_internal_listener:PLAINTEXT,external_listener:PLAINTEXT
KAFKA_INTER_BROKER_LISTENER_NAME: aws_internal_listener
Then your port forwarder for DYNAMIC_DNS_ADDR should redirect connections to 29092 on the AWS node. The key thing is that external connections should not end up at the listener port on the host matching the internal listener (which advertises an internal 192.168.0 address)
Use kafkacat -L -b DYNAMIC_DNS_ADDR:29092 to debug and validate your config, as described in the article here.

Unable to connect broker - kafka Tool

I am facing below error message when i was trying to connect and see the topic/consumer details of one of my kafka clusters we have.
we have 3 brokers in the cluster which I able to see but the topic and its partitions.
Note : I have kafka 1.0 and kafka tool version is 2.0.1
I had the same issue on my MacBook Pro. The tool was using "tshepo-mbp" as the hostname which it could not resolve. To get it to work I added 127.0.0.1 tshepo-mbp to the /etc/hosts file.
kafka tool is most likely using the hostname to connect to the broker and cannot reach it. You maybe connecting to the zookeeper host by IP address but make sure you can connect/ping the host name of the broker from the machine running the kafka tool.
If you cannot ping the broker either fix the network issues or as a workaround edit the host file on your client to let it know how to reach the broker by its name
This issue occurs if you have not set listeners and advertised.listeners property in server.properties file.
For Ex:
config/server.properties
...
listeners=PLAINTEXT://:9092
...
advertised.listeners=PLAINTEXT://<public-ip/host-name>:9092
...
To fix this issue, we need to change the server.properties file.
$ vim /usr/local/etc/kafka/server.properties
Here update the listeners value from
listeners=PLAINTEXT://:9092
to
listeners=PLAINTEXT://localhost:9092
source:https://medium.com/#Ankitthakur/apache-kafka-installation-on-mac-using-homebrew-a367cdefd273
For better visibility (even already commented the same in early days thread)
In my case, I got to know when I used Kafkatool from my local machine, tool tris to find out Kafka broker port which was blocked from my cluster admins for my local machine, that is the reason I was not able to connect.
Resolution:
Either ask the admin to open the port for intranet if they can, if they can not you can use tunnelling for your testing purpose or time being for your port.
Hope this would help a few.