I am using mac and install zookeeper and kafka through
brew install confluent-platform
By using the following commands,
zookeeper-server-start /usr/local/etc/kafka/zookeeper.properties
kafka-server-start /usr/local/etc/kafka/server.properties
connect-distributed /usr/local/etc/kafka/connect-distributed.properties
however connector shows the following messages such as
[2020-08-05 09:53:40,222] WARN [Producer clientId=inventory-connector2-dbhistory] Connection to node -1 (kafka/223.82.248.117:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient:756)
[2020-08-05 09:53:40,230] WARN [Producer clientId=inventory-connector2-dbhistory] Bootstrap broker kafka:9092 (id: -1 rack: null) disconnected (org.apache.kafka.clients.NetworkClient:1024)
[2020-08-05 09:53:40,427] WARN [Producer clientId=inventory-connector-dbhistory] Connection to node -1 (kafka/223.82.248.117:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient:756)
I started the kafka broker in the localhost, but as the message shows the broker 's address is 223.82.248.117:9092.
how would I fix it?
You would need to set advertised.listeners in server.properties to be localhost:9092
Then bootstrap.servers in connect-*.properties to point there as well
Related
I have two Kafka clusters on AWS MSK (in same environment and region). I have a KafkaConnect cluster setup on the destination cluster and have setup a mirror maker connector to run. The submission of the connector is fine and there are no errors.
When I try to check status of the connector, it says, RUNNING:
{"name":"mirror-maker-test-connector","connector":{"state":"RUNNING","worker_id":"<ip>:<port>"},"tasks":[task_list],"type":"source"}
I see the following exception:
[2022-01-12 19:46:33,772] DEBUG [Producer clientId=connector-producer-mirror-maker-test-connector-0] Connection with b-2.<broker_ip> disconnected (org.apache.kafka.common.network.Selector)
java.io.EOFException
at org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:120)
at org.apache.kafka.common.network.KafkaChannel.receive(KafkaChannel.java:452)
at org.apache.kafka.common.network.KafkaChannel.read(KafkaChannel.java:402)
at org.apache.kafka.common.network.Selector.attemptRead(Selector.java:674)
at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:576)
at org.apache.kafka.common.network.Selector.poll(Selector.java:481)
at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:551)
at org.apache.kafka.clients.producer.internals.Sender.runOnce(Sender.java:328)
at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:243)
at java.base/java.lang.Thread.run(Thread.java:829)
[2022-01-12 19:46:33,773] DEBUG [Producer clientId=connector-producer-mirror-maker-test-connector-0] Node -1 disconnected. (org.apache.kafka.clients.NetworkClient)
[2022-01-12 19:46:33,773] WARN [Producer clientId=connector-producer-mirror-maker-test-connector-0] Bootstrap broker b-2.<broker_ip>:9094 (id: -1 rack: null) disconnected (org.apache.kafka.clients.NetworkClient)
I am able to connect to the specified broker with netcat from within the Kafka Connect k8s pod.
Anyone faced this issue before?
I got it to work - had to add SSL properties for both consumer and producer when submitting the Mirror Maker connector.
"target.cluster.security.protocol": "SSL",
"target.cluster.ssl.truststore.location":"<certs_path>",
"target.cluster.ssl.truststore.password": "<password>"
"source.cluster.security.protocol": "SSL",
"source.cluster.ssl.truststore.location": "<certs_path>",
"source.cluster.ssl.truststore.password": "<password>"
When I write to console:
kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test-first
to create a kafka consumer in VPS it gives me error like this:
"[2021-03-08 18:54:24,548] WARN [Consumer clientId=consumer-1, groupId=console-consumer-19666] Connection to node -1 (localhost/127.0.0.1:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)"
How to solve this error ?
*I checked all of the questions with same error.
**kafka and zookeeper are running.
***212.71.238.187 is my VPS ip
server.properties:
broker.id=0
listeners=PLAINTEXT://0.0.0.0:9092
advertised.listeners=PLAINTEXT://212.71.238.187:9092
zookeeper.connect=212.71.238.187:2181
I am using the Debezium Postgres connector. I have two tables in Postgres named 'publications' and 'comments'. kafka and zookeeper are running in docker containers as per the standard examples. The postgres is running locally. After using the debezium postgres connect, I have the following topics :
$ bin/kafka-topics.sh --list --zookeeper localhost:2181
__consumer_offsets
dbserver1.public.comments
dbserver1.public.publications
my_connect_configs
my_connect_offsets
my_connect_statuses
I would like to see a list of messages in the topic:
$ bin/kafka-console-consumer.sh --bootstrap-server kafka:9092 --topic dbserver1.public.publications
[2019-06-03 21:55:16,180] WARN [Consumer clientId=consumer-1,
groupId=console-consumer-5221] Connection to node -1
(kafka/23.202.231.166:9092) could not be established. Broker may not
be available. (org.apache.kafka.clients.NetworkClient) [2019-06-03
21:55:16,289] WARN [Consumer clientId=consumer-1,
groupId=console-consumer-5221] Connection to node -1
(kafka/23.202.231.166:9092) could not be established. Broker may not
be available. (org.apache.kafka.clients.NetworkClient) [2019-06-03
21:55:16,443] WARN [Consumer clientId=consumer-1,
groupId=console-consumer-5221] Connection to node -1
(kafka/23.202.231.166:9092) could not be established. Broker may not
be available. (org.apache.kafka.clients.NetworkClient) [2019-06-03
21:55:16,721] WARN [Consumer clientId=consumer-1,
groupId=console-consumer-5221] Connection to node -1
(kafka/23.202.231.166:9092) could not be established. Broker may not
be available. (org.apache.kafka.clients.NetworkClient) [2019-06-03
21:55:17,145] WARN [Consumer clientId=consumer-1,
groupId=console-consumer-5221] Connection to node -1
(kafka/23.202.231.166:9092) could not be established. Broker may not
be available. (org.apache.kafka.clients.NetworkClient) [2019-06-03
21:55:18,017] WARN [Consumer clientId=consumer-1,
groupId=console-consumer-5221] Connection to node -1
(kafka/23.202.231.166:9092) could not be established. Broker may not
be available. (org.apache.kafka.clients.NetworkClient) ^CProcessed a
total of 0 messages
$ bin/kafka-console-consumer.sh --bootstrap-server localhost:9092
--topic dbserver1.public.publications
[2019-06-03 21:55:16,180] WARN [Consumer clientId=consumer-1,
groupId=console-consumer-5221] Connection to node -1
(kafka/23.202.231.166:9092) could not be established. Broker may not
be available. (org.apache.kafka.clients.NetworkClient)
How do I specify the correct value for bootstrap-server? Thanks.
I am assuming you are trying to connect to kafka broker from an external server.
Since you have already mentioned your Kafka and Zookeeper instances are running from docker images. You need to identify your docker images external port corresponding to 9092 as well as its external IP address and you have to those along with --bootstrap-server parameter while executing command kafka-console-consumer.sh
If you are running the kafka-console-consumer.sh outside of docker then you should use localhost hostname. If inside the Docker container then make sure it is in container that sees kafka hostname.
i started zookeeper and Kafka server in Linux machine. Both servers are started successfully. After that i created a "test" Topic. Everything working fine using console producer and console consumer. But when i try to send an event from a remote machine to Kafka server, it is failing. After some investigation i added the below configuration in server.properties
listeners=PLAINTEXT://0.0.0.0:9092
advertised.listeners=PLAINTEXT://xx.xx.xx.xx:9092
Then i got below continues Warning in Kafka logs
[2018-05-25 14:48:27,685] WARN [Controller id=0, targetBrokerId=0] Connection to node 0 could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
But exceptions in controller.log
[2018-05-25 14:48:27,583] WARN [RequestSendThread controllerId=0] Controller 0's connection to broker xx.xx.xx.xx:9092 (id: 0 rack: null) was unsuccessful (kafka.controller.Req
uestSendThread)
java.io.IOException: Connection to xx.xx.xx.xx:9092 (id: 0 rack: null) failed.
at org.apache.kafka.clients.NetworkClientUtils.awaitReady(NetworkClientUtils.java:70)
at kafka.controller.RequestSendThread.brokerReady(ControllerChannelManager.scala:271)
at kafka.controller.RequestSendThread.doWork(ControllerChannelManager.scala:225)
at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:82)
Please help to solve this issue
Thanks in advance
Could you please tell me about compatibility of Apache Kafka and Zookeeper (native Apache distributuins) with some Confluent's components. I have already installed in my environment Kafka and Zookeeper as a multinodes clusters. But now I need to add schema-registry, kafka-connect.
So I actually tried to deploy Confluent Schema registry from their official docker image. I logged in and was able to successfully telnet kafka broker on port 9093
root#schema-0:/usr/bin# telnet kafka-0.kafka-hs 9093
Trying 10.244.3.47...
Connected to kafka-0.kafka-hs.log-platform.svc.cluster.local.
Escape character is '^]'.
After I tried to do some tests:
# /usr/bin/kafka-avro-console-producer \
--broker-list localhost:9093 --topic bar \
--property value.schema='{"type":"record","name":"myrecord","fields" \
[{"name":"f1","type":"string"}]}'
Add some values:
{"f1": "value1"}
But no luck :(. Got next errors:
value.serializer = class org.apache.kafka.common.serialization.ByteArraySerializer
(org.apache.kafka.clients.producer.ProducerConfig)
[2018-01-28 11:23:23,561] INFO Kafka version : 1.0.0-cp1 (org.apache.kafka.common.utils.AppInfoParser)
[2018-01-28 11:23:23,561] INFO Kafka commitId : ec61c5e93da662df (org.apache.kafka.common.utils.AppInfoParser){"f1": "value1"}
[2018-01-28 11:23:36,233] WARN [Producer clientId=console-producer] Connection to node -1 could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
[2018-01-28 11:23:36,335] WARN [Producer clientId=console-producer] Connection to node -1 could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
[2018-01-28 11:23:36,486] WARN [Producer clientId=console-producer] Connection to node -1 could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
Entire system is spinning on Kubernetes
Confluent Platform is Apache Kafka, but with additional components (such as Schema Registry) bundled with it.
The error you're getting is related to the network configuration. You need to make sure that your Broker is available to other nodes, including Schema Registry. In your Schema Registry config you've specified broker-list localhost:9093 but this should be your Kafka broker. In addition as Dmitry Minkovsky mentions, make sure you've set the advertised listener in your broker. This article might help.