I have two machines localhost and 192.168.1.110 to run two independent single machine kafka.
(1)At localhost, I run:
➜ kafka_2.11-0.10.0.0 bin/kafka-console-producer.sh --broker-list 192.168.1.110:9092 --topic test
this is a message
[2016-08-24 18:15:27,441] ERROR Error when sending message to topic test with key: null, value: 2 bytes with error: (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)
org.apache.kafka.common.errors.TimeoutException: Batch containing 1 record(s) expired due to timeout while requesting metadata from brokers for test-0
Why message can't be sent to broker at 192.168.1.110?
could I use broker ip directly in consumer or producer?
If I could only use hostname, does this relate to advertised.host.name?
then how to set up advertised.host.name? does this host name should be globally resolvable(could I use /etc/hosts to resolve the host name?)
(2)
I edited /etc/hosts to let localhost point to 192.168.1.110,
then I run:
➜ kafka_2.11-0.10.0.0 bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test
and I could successfully sent messages to 192.168.1.110 and consumed the messages there.
(3)I edited /etc/hosts to let rrlocalhost point to 192.168.1.110,
then I run:
➜ kafka_2.11-0.10.0.0 bin/kafka-console-producer.sh --broker-list rrlocalhost:9092 --topic test
then I sent messages to rrlocalhost, there is the same error as in (1).
Definitely you can use ip address directly.
broker config advertised.host.name will be registered in zookeeper, and producer and consumer will fetch them as cluster metadata. If you config it using local nick name, producer and consumer will be trouble to communicate with it.
Related
I am trying to check the kafka consumer by consuming the data from a topic on a remote Kafka cluster. I am getting the following error when I use the kafka-console-consumer.sh:
ERROR Error processing message, terminating consumer process: (kafka.tools.ConsoleConsumer$)
java.lang.IllegalStateException: No entry found for connection 2147475658
at org.apache.kafka.clients.ClusterConnectionStates.nodeState(ClusterConnectionStates.java:330)
at org.apache.kafka.clients.ClusterConnectionStates.disconnected(ClusterConnectionStates.java:134)
at org.apache.kafka.clients.NetworkClient.initiateConnect(NetworkClient.java:885)
at org.apache.kafka.clients.NetworkClient.ready(NetworkClient.java:276)
at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.tryConnect(ConsumerNetworkClient.java:548)
at org.apache.kafka.clients.consumer.internals.AbstractCoordinator$FindCoordinatorResponseHandler.onSuccess(AbstractCoordinator.java:655)
at org.apache.kafka.clients.consumer.internals.AbstractCoordinator$FindCoordinatorResponseHandler.onSuccess(AbstractCoordinator.java:635)
at org.apache.kafka.clients.consumer.internals.RequestFuture$1.onSuccess(RequestFuture.java:204)
at org.apache.kafka.clients.consumer.internals.RequestFuture.fireSuccess(RequestFuture.java:167)
at org.apache.kafka.clients.consumer.internals.RequestFuture.complete(RequestFuture.java:127)
at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient$RequestFutureCompletionHandler.fireCompletion(ConsumerNetworkClient.java:575)
at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.firePendingCompletedRequests(ConsumerNetworkClient.java:389)
at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:297)
at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:236)
at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:215)
at org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureCoordinatorReady(AbstractCoordinator.java:231)
at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(ConsumerCoordinator.java:316)
at org.apache.kafka.clients.consumer.KafkaConsumer.updateAssignmentMetadataIfNeeded(KafkaConsumer.java:1214)
at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1179)
at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1164)
at kafka.tools.ConsoleConsumer$ConsumerWrapper.receive(ConsoleConsumer.scala:436)
at kafka.tools.ConsoleConsumer$.process(ConsoleConsumer.scala:104)
at kafka.tools.ConsoleConsumer$.run(ConsoleConsumer.scala:76)
at kafka.tools.ConsoleConsumer$.main(ConsoleConsumer.scala:54)
at kafka.tools.ConsoleConsumer.main(ConsoleConsumer.scala)
Processed a total of 0 messages
Here is the command that I use:
./bin/kafka-console-consumer.sh --bootstrap-server SSL://{IP}:{PORT},SSL://{IP}:{PORT},SSL://{IP}:{PORT} --consumer.config ./config/consumer.properties --topic MYTOPIC --group MYGROUP
Here is the ./config/consumer.properties file:
bootstrap.servers=SSL://{IP}:{PORT},SSL://{IP}:{PORT},SSL://{IP}:{PORT}
# consumer group id
group.id=MYGROUP
# What to do when there is no initial offset in Kafka or if the current
# offset does not exist any more on the server: latest, earliest, none
auto.offset.reset=earliest
#### Security
security.protocol=SSL
ssl.key.password=test1234
ssl.keystore.location=/opt/kafka/config/certs/keystore.jks
ssl.keystore.password=test1234
ssl.truststore.location=/opt/kafka/config/certs/truststore.jks
ssl.truststore.password=test1234
Do you have any idea what the problem is?
I have found the problem. It was a DNS problem at the end. I was reaching out the Kafka brokers by the IP addresses, but the broker replies with DNS name. After setting the DNS names on the consumer side, it started working again.
I had this problem (with consumers and producers) when running Kafka and Zookeeper as Docker containers.
The solution was to set advertised.listeners in the config/server.properties file of the Kafka brokers, so that it contains the IP address of the container, e.g.
advertised.listeners=PLAINTEXT://172.15.0.8:9092
See https://github.com/maxant/kafkaplayground/blob/master/start-kafka.sh for an example of a script used to start Kafka inside the container after setting up the properties file correctly.
It seems the Kafka cluster listener property is not configured in server.properties.
In the remote kafka cluster, this property should be uncommented with the proper host name.
listeners=PLAINTEXT://0.0.0.0:9092
In my case I was receiving that while trying to connect to my Kafka container, I had to pass the following:
-e KAFKA_LISTENERS=PLAINTEXT://0.0.0.0:9092
-e KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://localhost:9092
Hope it helps someone
Are you sure the remote kafka is running. I would suggest running nmap -p PORT HOST in order to verify the port is open (unless it is configured differently the port should be 9092). If that is ok, then you can use kafkacat which makes things easier. Create a consumer running kafkacat -b HOST:PORT -t YOUR_TOPIC -C -o beginning or create a producer running kafkacat -b HOST:PORT -t YOUR_TOPIC -P
I've a local Apache Kafka setup and there are total 2 broker (id - 0 and 1) on port 9092 and 9093.
I created a topic and published the messages using this command:
bin/kafka-console.-producer.sh --broker-list localhost:9092 --topic test
Then I consumed the messages on other terminal using the command:
bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test
Till now everything is fine.
But when i type command -
bin/kafka-console.-producer.sh --broker-list localhost:9093 --topic test
and write some messages it is showing in the 2nd terminal where I've typed this command -
bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test
Why port 9093 messages are publishing to 9092?
Your cluster contains of two brokers. It is not important which host you use for initial connection. Using kafka client you don't specify from which broker you consume or to which your produce messages. Those hostname are only to discover whole list of kafka brokers (cluster)
According to documentation:
https://kafka.apache.org/documentation/#producerconfigs
https://kafka.apache.org/documentation/#consumerconfigs
bootstrap.servers::
A list of host/port pairs to use for establishing the initial connection to the Kafka cluster. The client will make use of all servers irrespective of which servers are specified here for bootstrapping—this list only impacts the initial hosts used to discover the full set of servers.
I have been trying to run my producer and consumer api on Kafka cluster but it is not working.
1) Zookeeper is running on one GCP VM instance
2) Kafka is running on another GCP VM instance
Steps Followed:
Step 1) Running Zookeeper using the following command:
bin/zookeeper-server-start.sh config/zookeeper.properties
Zookeeper.properties:
#the directory where the snapshot is stored.
dataDir=/tmp/zookeeper
#the port at which the clients will connect
clientPort=2181
#disable the per-ip limit on the number of connections since this is a non-production config
maxClientCnxns=0
# In milliseconds
tickTime=2000
# In ticks
initLimit=10
syncLimit=5
maxClientCnxns=30
# All Zookeeper servers need to be aware of other Zookeepers part of the cluster
server.1=zook-1:2888:3888
Note : zook-1 is the name of the Zookeeper instance
Step 2)
Running Kafka on another GCP VM instance as :
bin/kafka-console-producer.sh --broker-li de-kafka-1:9092 --topic test
server.properties:
############################# Server Basics ############################# # The id of the broker. This must be set to a unique integer for each broker.
broker.id=0
############################ Zookeeper ############################# #
# root directory for all kafka znodes.
zookeeper.connect=zook-1:2181
# Timeout in ms for connecting to zookeeper
zookeeper.connection.timeout.ms=6000
host.name=de-kafka-1
Step 3)
Running producer api on the Kafka instance from Step 2 (using another SSH tunnel) as follows :
bin/kafka-console-producer.sh --broker-list de-kafka-1:9 092 --topic test
Note : Here de-kafka-1 is the name of the GCP Kafka VM instance I have tried and replaced de-kafka-1 with External IP of de-Kafka-1 instance, localhost, zook-1 (Name of Zookeeper VM instance), external IP of zook-1 vm instance
Error: OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreads=N Sending msg1 sending msg2
Error: error when sending message to topic test single node with key: null, value: 5 bytes with error (org.apache.kafka.clients.producer.internals.errorloggingcallback)org.apache.kafka.common.errors.timeoutexception: failed to update metadata after 60000 ms.
Step 4:
Running Consumer API on Kafka instance (using third SSH tunnel):
bin/kafka-console-consumer.sh --bootstrap-server de-kafka-1:2181 --topic test --from-beginning
Would appreciate if anyone could help on this. Thanks
Try to create topic before producing bin/kafka-topics.sh --create --zookeeper zook-1:2181 --replication-factor 1 --partitions 1 --topic test.
If its not working read about zkCli.sh script here - with this script you can check if the broker is registered in the zookeeper.
If you still have a problems, and you trying to produce from outside the kafka cluster, try to read about advertised.listeners property here.
The system on which my Kafka server is running has two NICs, one with a public IP (135.220.23.45) and the other with a private one (192.168.1.14). The private NIC is connected to a subnet composed of 7 machines in total (all with addresses 192.168.1.xxx). Kafka has been installed as a service using HDP and has been configured with zookeeper.connect=192.168.1.14:2181 and listeners=PLAINTEXT://192.168.1.14:6667. I have started a consumer on the system that hosts the kafka server using: [bin/kafka-console-consumer.sh --bootstrap-server 192.168.1.14:6667 --topic test --from-beginning].
When I start producers (using [bin/kafka-console-producer.sh --broker-list 192.168.1.14:6667 --topic test]) on any of the machines on the private subnet the messages are received normally by the consumer.
I would like to start producers on public systems and receive the messages by the consumer running on the kafka server. I believed that this could be achieved by IP masquerading and by forwarding all external requests to 135.220.23.45:15501 (I have chosen 15501 to receive kafka messages) to 192.168.1.14:6667. To that extend I setup this port forwarding rule on firewalld: [port=15501:proto=tcp:toport=6670:toaddr=192.168.1.14].
However, this doesn’t seem to work since when I start a producer on an external system with [bin/kafka-console-producer.sh --broker-list 135.220.23.45:15501 --topic] the messages cannot be received by the consumer.
I have tried different kafka config settings for listeners and advertised.listeners but none of them worked. Any help will be greatly appreciated.
You need to define different endpoints for your internal and external traffic in order for this to work. As it is currently configured, when you connect to 135.220.23.45:15501 Kafka would reply with "please talk to me on 192.168.1.14:6667 which is not reachable from the outside and everything from there on out fails.
With KIP-103 Kafka was extended to cater to these scenarios by letting you define multiple endpoints.
Full disclosure, I have not yet tried this out, but something along the following lines should at least get you started down the right road.
advertised.listeners=EXTERNAL://135.220.23.45:15501,INTERNAL://192.168.1.14:6667
inter.broker.listener.name=INTERNAL
listener.security.protocol.map=EXTERNAL:PLAINTEXT,INTERNAL:PLAINTEXT
Update:
I've tested this on a cluster of three ec2 machines out of interest. I've used the following configuration:
# internal ip: 172.31.61.130
# external ip: 184.72.211.109
listeners=INTERNAL://:9092,EXTERNAL_PLAINTEXT://:9094
advertised.listeners=INTERNAL://172.31.61.130:9092,EXTERNAL_PLAINTEXT://184.72.211.109:9094
listener.security.protocol.map=INTERNAL:PLAINTEXT,EXTERNAL_PLAINTEXT:PLAINTEXT
inter.broker.listener.name=INTERNAL
And that allowed me to send messages from both an internal machine as well as my laptop at home:
# Create topic
kafka-topics --create --topic testtopic --partitions 9 --replication-factor 3 --zookeeper 127.0.0.1:2181
# Produce messages from internal machine
[ec2-user#ip-172-31-61-130 ~]$ kafka-console-producer --broker-list 127.0.0.1:9092 --topic testtopic
>internal1
>internal2
>internal3
# Produce messages from external machine
➜ bin ./kafka-console-producer --topic testtopic --broker-list 184.72.211.109:9094
external1
external2
external3
# Check topic
[ec2-user#ip-172-31-61-130 ~]$ kafka-console-consumer --bootstrap-server 172.31.52.144:9092 --topic testtopic --from-beginning
external3
internal2
external1
external2
internal3
internal1
My problem is: how can I send data from a kafka producer to broker?
the schema below explains my network configuration :
I have a producer in VM which is located in server A, and my broker too is in VM which is located in Server B.
I use an SSH connection from my producer VM to the server B with a redirection port : ssh -L 9092:192.168.56.101:9092 xx#IP1
I use kafka console to test :
bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test
thanks
You need to set the --broker-list to wherever the broker resides. In your code, you are saying that I want to produce a message and send it to a broker that is on the localhost machine at port 9092. Try
bin/kafka-console-producer.sh --broker-list 192.168.56.101:9092 --topic test