Steps to delete data inside the Kafka Topic on Windows? - apache-kafka

I am working on Spring Batch and Apache Kafka Integration. Before posting the question I went over web : Is there a way to delete all the data from a topic or delete the topic before every run? to find out better solution, but did not find out.
I am using Kafka version 2.11.
I want to delete all data under the topic without stopping either Zookeeper or Kafka. How can we do that ?
Below commands causes lot of issues in windows
C:\kafka_2.11-2.3.1\bin\windows>kafka-topics.bat --zookeeper localhost:2181 --delete --topic customers
Topic customers is marked for deletion.
Note: This will have no impact if delete.topic.enable is not set to true.
C:\kafka_2.11-2.3.1\bin\windows>kafka-topics.bat --zookeeper localhost:2181 --delete --topic test
C:\kafka_2.11-2.3.1\bin\windows>kafka-console-consumer.bat --bootstrap-server localhost:9092 --topic customers --from-beginning
[2020-04-21 10:25:02,812] WARN [Consumer clientId=consumer-1, groupId=console-consumer-65075] Connection to node -1 (localhost/127.0.0.1:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
[2020-04-21 10:25:04,886] WARN [Consumer clientId=consumer-1, groupId=console-consumer-65075] Connection to node -1 (localhost/127.0.0.1:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
[2020-04-21 10:25:06,996] WARN [Consumer clientId=consumer-1, groupId=console-consumer-65075] Connection to node -1 (localhost/127.0.0.1:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
[2020-04-21 10:25:09,267] WARN [Consumer clientId=consumer-1, groupId=console-consumer-65075] Connection to node -1 (localhost/127.0.0.1:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
[2020-04-21 10:25:11,744] WARN [Consumer clientId=consumer-1, groupId=console-consumer-65075] Connection to node -1 (localhost/127.0.0.1:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
Processed a total of 0 messages
Terminate batch job (Y/N)?
^C
C:\kafka_2.11-2.3.1\bin\windows>

I am using Kafka version 2.11.
There is no Kafka 2.11. Your command prompt says kafka_2.11-2.3.1: hence, you are using Kafka 2.3.1. The 2.11 part is the Scala version that was used during compilation.
Note: This will have no impact if delete.topic.enable is not set to true.
Did you check your broker configs if delete.topic.enable is set to true? If yes, you should be able to delete a topic without stopping ZK or the brokers. Note though, that deleting topics is async, i.e., when you command returns the topic is not deleted yet and it will take some time until the command is executed.

Related

Connection to node -1 (localhost/127.0.0.1:9092) could not be established. Broker may not be available. VPS connection

When I write to console:
kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test-first
to create a kafka consumer in VPS it gives me error like this:
"[2021-03-08 18:54:24,548] WARN [Consumer clientId=consumer-1, groupId=console-consumer-19666] Connection to node -1 (localhost/127.0.0.1:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)"
How to solve this error ?
*I checked all of the questions with same error.
**kafka and zookeeper are running.
***212.71.238.187 is my VPS ip
server.properties:
broker.id=0
listeners=PLAINTEXT://0.0.0.0:9092
advertised.listeners=PLAINTEXT://212.71.238.187:9092
zookeeper.connect=212.71.238.187:2181

kafka connect could not be established to the kafka broker

I am using mac and install zookeeper and kafka through
brew install confluent-platform
By using the following commands,
zookeeper-server-start /usr/local/etc/kafka/zookeeper.properties
kafka-server-start /usr/local/etc/kafka/server.properties
connect-distributed /usr/local/etc/kafka/connect-distributed.properties
however connector shows the following messages such as
[2020-08-05 09:53:40,222] WARN [Producer clientId=inventory-connector2-dbhistory] Connection to node -1 (kafka/223.82.248.117:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient:756)
[2020-08-05 09:53:40,230] WARN [Producer clientId=inventory-connector2-dbhistory] Bootstrap broker kafka:9092 (id: -1 rack: null) disconnected (org.apache.kafka.clients.NetworkClient:1024)
[2020-08-05 09:53:40,427] WARN [Producer clientId=inventory-connector-dbhistory] Connection to node -1 (kafka/223.82.248.117:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient:756)
I started the kafka broker in the localhost, but as the message shows the broker 's address is 223.82.248.117:9092.
how would I fix it?
You would need to set advertised.listeners in server.properties to be localhost:9092
Then bootstrap.servers in connect-*.properties to point there as well

Unable to produce message when leader is not available

I am unable to produce message when Leader is not available. I created new topic with replication factor 2 using below command
~/kafka/bin/kafka-topics.sh --create --zookeeper localhost:2181 --topic TestTopic202 --partitions 1 --replication-factor 2
Then
~/kafka/bin/kafka-topics.sh --describe --zookeeper localhost:2181 --topic TestTopic202
Output:
OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropri
ately using -XX:ParallelGCThreads=N
Topic:TestTopic202 PartitionCount:1 ReplicationFactor:2 Configs:
Topic: TestTopic202 Partition: 0 Leader: 1 Replicas: 1,2 Isr: 1,2
After creating topic I stopped broker-1 which is leader of this topic to test fault tolerance. I expected broker-2 will be elected as leader but I received broker down error message.
~/kafka/bin/kafka-console-producer.sh --broker-list 34.93.59.30:9092 --topic TestTopic202
OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreads=N
>Hi
After bringing down,
^C[kafka#hgtestsrv1 ~]$ ~/kafka/bin/kafka-console-producer.sh --broker-list localhost:9092 --topic TestTopic202
\OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreads=N
>Hi
[2019-09-03 12:25:47,305] WARN [Producer clientId=console-producer] Connection to node -1 (localhost/127.0.0.1:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
[2019-09-03 12:25:47,407] WARN [Producer clientId=console-producer] Connection to node -1 (localhost/127.0.0.1:9092) could not be established. Broker may not be ava
ilable. (org.apache.kafka.clients.NetworkClient)
[2019-09-03 12:25:47,511] WARN [Producer clientId=console-producer] Connection to node -1 (localhost/127.0.0.1:9092) could not be established. Broker may not be ava
ilable. (org.apache.kafka.clients.NetworkClient)
[2019-09-03 12:25:47,765] WARN [Producer clientId=console-producer] Connection to node -1 (localhost/127.0.0.1:9092) could not be established. Broker may not be ava
ilable. (org.apache.kafka.clients.NetworkClient)
[2019-09-03 12:25:48,222] WARN [Producer clientId=console-producer] Connection to node -1 (localhost/127.0.0.1:9092) could not be established. Broker may not be ava
ilable. (org.apache.kafka.clients.NetworkClient)
^Corg.apache.kafka.common.KafkaException: Producer closed while send in progress
at org.apache.kafka.clients.producer.KafkaProducer.doSend(KafkaProducer.java:862)
at org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:839)
at kafka.tools.ConsoleProducer$.send(ConsoleProducer.scala:75)
at kafka.tools.ConsoleProducer$.main(ConsoleProducer.scala:57)
at kafka.tools.ConsoleProducer.main(ConsoleProducer.scala)
Caused by: org.apache.kafka.common.KafkaException: Requested metadata update after close
at org.apache.kafka.clients.Metadata.awaitUpdate(Metadata.java:200)
at org.apache.kafka.clients.producer.KafkaProducer.waitOnMetadata(KafkaProducer.java:982)
at org.apache.kafka.clients.producer.KafkaProducer.doSend(KafkaProducer.java:859)
... 4 more
Can anyone tell me how to test fault tolerance?
My server.properties
default.replication.factor=2
offsets.topic.replication.factor=2
transaction.state.log.replication.factor=2
transaction.state.log.min.isr=1
Before producing a message to Kafka, Kafka fetches metadata regarding how many partitions are there for that topic and which broker is the leader for which partition etc.
Approach #1. Now, it is important to give your bootstrap servers (broker list) of the producer to the broker that is reachable. You said, you have brought broker-1 down, so you cannot include it as the only one in your Kafka producer bootstrap servers.
The following configuration needs to be changed.
bootstrap.servers=<IP>:<Port>
Set this to your broker-2 and see.
For your question of testing fault tolerance...
Your approach is right, but you must include multiple (if not all) brokers in your producer/consumer bootstrap.servers property.
You gave localhost:9092 Is this the broker which you brought down? Where is the other broker running? The running broker must be given to the producer.
Check to see if the other broker is running and is registered with the same zookeeper.
Update:
You gave 2 different IPs 34.93.59.30:9092 and next localhost:9092, I suspect that they are two different brokers on two different machines.
Approach #2. If it is the case, ensure that (as said before) your localhost:9092 broker is registered with the zookeeper of 34.93.59.30:9092 broker (It should mostly be registered with 34.93.59.30:2181 if you use the default ports).
So in the server.properties of your localhost:9092 (your second broker you should give the same zookeeper as your broker-1).
zookeeper.connect=34.93.59.30:2181
and foremost, ensure that your localhost can connect to that first.
Approach #3. If you still were not able to figure this out check the output of your describe topics.
~/kafka/bin/kafka-topics.sh --describe --zookeeper localhost:2181 --topic TestTopic202
after you brought the broker-1 down. It should show you the leader.

Read Docker Kafka messages populated by debezium postgres connect

I am using the Debezium Postgres connector. I have two tables in Postgres named 'publications' and 'comments'. kafka and zookeeper are running in docker containers as per the standard examples. The postgres is running locally. After using the debezium postgres connect, I have the following topics :
$ bin/kafka-topics.sh --list --zookeeper localhost:2181
__consumer_offsets
dbserver1.public.comments
dbserver1.public.publications
my_connect_configs
my_connect_offsets
my_connect_statuses
I would like to see a list of messages in the topic:
$ bin/kafka-console-consumer.sh --bootstrap-server kafka:9092 --topic dbserver1.public.publications
[2019-06-03 21:55:16,180] WARN [Consumer clientId=consumer-1,
groupId=console-consumer-5221] Connection to node -1
(kafka/23.202.231.166:9092) could not be established. Broker may not
be available. (org.apache.kafka.clients.NetworkClient) [2019-06-03
21:55:16,289] WARN [Consumer clientId=consumer-1,
groupId=console-consumer-5221] Connection to node -1
(kafka/23.202.231.166:9092) could not be established. Broker may not
be available. (org.apache.kafka.clients.NetworkClient) [2019-06-03
21:55:16,443] WARN [Consumer clientId=consumer-1,
groupId=console-consumer-5221] Connection to node -1
(kafka/23.202.231.166:9092) could not be established. Broker may not
be available. (org.apache.kafka.clients.NetworkClient) [2019-06-03
21:55:16,721] WARN [Consumer clientId=consumer-1,
groupId=console-consumer-5221] Connection to node -1
(kafka/23.202.231.166:9092) could not be established. Broker may not
be available. (org.apache.kafka.clients.NetworkClient) [2019-06-03
21:55:17,145] WARN [Consumer clientId=consumer-1,
groupId=console-consumer-5221] Connection to node -1
(kafka/23.202.231.166:9092) could not be established. Broker may not
be available. (org.apache.kafka.clients.NetworkClient) [2019-06-03
21:55:18,017] WARN [Consumer clientId=consumer-1,
groupId=console-consumer-5221] Connection to node -1
(kafka/23.202.231.166:9092) could not be established. Broker may not
be available. (org.apache.kafka.clients.NetworkClient) ^CProcessed a
total of 0 messages
$ bin/kafka-console-consumer.sh --bootstrap-server localhost:9092
--topic dbserver1.public.publications
[2019-06-03 21:55:16,180] WARN [Consumer clientId=consumer-1,
groupId=console-consumer-5221] Connection to node -1
(kafka/23.202.231.166:9092) could not be established. Broker may not
be available. (org.apache.kafka.clients.NetworkClient)
How do I specify the correct value for bootstrap-server? Thanks.
I am assuming you are trying to connect to kafka broker from an external server.
Since you have already mentioned your Kafka and Zookeeper instances are running from docker images. You need to identify your docker images external port corresponding to 9092 as well as its external IP address and you have to those along with --bootstrap-server parameter while executing command kafka-console-consumer.sh
If you are running the kafka-console-consumer.sh outside of docker then you should use localhost hostname. If inside the Docker container then make sure it is in container that sees kafka hostname.

Kafka-Broker not available after some time of message transfer

Connection to node 1 could not be established after _consumer_offset-49,
I am not able to solve the issue,till _consumer_offset-49 consumer is able to get message but after offset-49 , it show WARNING message i.e Connection to node 1 could not be established,broker not may not be available.
C:\kafka_2.11-2.0.0>.\bin\windows\kafka-console-producer.bat --broker-list localhost:9092 --topic test1
>hi
>hello
>hey
>whatsupp??
>how are u
>[2019-02-25 03:53:14,876] WARN [Producer clientId=console-producer] Connection to node 1 could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
[2019-02-25 03:53:15,982] WARN [Producer clientId=console-producer] Connection to node 1 could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
[2019-02-25 03:53:17,240] WARN [Producer clientId=console-producer] Connection to node 1 could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
I solved the issue, I just changed the Java version to Java-1.8-181 and it worked.