NiFi in docker container fails to talk to kafka: TimoutException, kafkacat ist working just fine - apache-kafka
I have set up NiFi (1.11.4) & Kafka(2.5) via docker (docker-compose file below, actual NiFi flow definition https://github.com/geoHeil/streaming-reference).
When trying to follow up on basic getting started tutorials (such as https://towardsdatascience.com/big-data-managing-the-flow-of-data-with-apache-nifi-and-apache-kafka-af674cd8f926) which combine processors such as:
generate flowfile (CSV)
update attribute
PublishKafka2.0
I run into issues of timeoutException:
nifi_1 | 2020-06-10 11:15:47,311 ERROR [kafka-producer-network-thread | producer-2] o.a.n.p.k.pubsub.PublishKafkaRecord_2_0 PublishKafkaRecord_2_0[id=959f0e64-0172-1000-0000-0000650181a4] Failed to send StandardFlowFileRecord[uuid=944464e4-94ea-48dc-89fa-d19c34f163e7,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1591771086044-1, container=default, section=1], offset=227962, length=422],offset=0,name=c8dd1dd2-0ffe-4875-9d45-902ea331c210,size=422] to Kafka: org.apache.kafka.common.errors.TimeoutException: Expiring 64 record(s) for test-0: 30029 ms has passed since batch creation plus linger time
nifi_1 | org.apache.kafka.common.errors.TimeoutException: Expiring 64 record(s) for test-0: 30029 ms has passed since batch creation plus linger time
nifi_1 | 2020-06-10 11:15:47,311 ERROR [kafka-producer-network-thread | producer-2] o.a.n.p.k.pubsub.PublishKafkaRecord_2_0 PublishKafkaRecord_2_0[id=959f0e64-0172-1000-0000-0000650181a4] Failed to send StandardFlowFileRecord[uuid=944464e4-94ea-48dc-89fa-d19c34f163e7,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1591771086044-1, container=default, section=1], offset=227962, length=422],offset=0,name=c8dd1dd2-0ffe-4875-9d45-902ea331c210,size=422] to Kafka: org.apache.kafka.common.errors.TimeoutException: Expiring 64 record(s) for test-0: 30029 ms has passed since batch creation plus linger time
nifi_1 | org.apache.kafka.common.errors.TimeoutException: Expiring 64 record(s) for test-0: 30029 ms has passed since batch creation plus linger time
However, a:
kafkacat -C -b localhost:9092 -t test #starts listener
kafkacat -P -b localhost:9092 -t test #starts producer
pipes events just fine through the kafka instance.
The docker-compose file looks like:
version: "3"
services:
nifi:
image: apache/nifi:1.11.4
ports:
- 8080:8080 # Unsecured HTTP Web Port
environment:
- NIFI_WEB_HTTP_PORT=8080
- NIFI_CLUSTER_IS_NODE=true
- NIFI_CLUSTER_NODE_PROTOCOL_PORT=8082
- NIFI_ZK_CONNECT_STRING=zookeeper:2181
- NIFI_ELECTION_MAX_WAIT=1 min
links:
- broker
- zookeeper
volumes:
- ./for_nifi/conf:/opt/nifi/nifi-current/conf
zookeeper: #https://github.com/confluentinc/cp-all-in-one/blob/5.5.0-post/cp-all-in-one/docker-compose.yml#L5
image: confluentinc/cp-zookeeper:5.5.0
hostname: zookeeper
container_name: zookeeper
ports:
- "2181:2181"
environment:
ZOOKEEPER_CLIENT_PORT: 2181
ZOOKEEPER_TICK_TIME: 2000
broker:
image: confluentinc/cp-kafka:5.5.0
hostname: broker
container_name: broker
depends_on:
- zookeeper
ports:
- "29092:29092"
- "9092:9092"
environment:
KAFKA_BROKER_ID: 1
KAFKA_ZOOKEEPER_CONNECT: 'zookeeper:2181'
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://broker:29092,PLAINTEXT_HOST://localhost:9092
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
You're using the wrong port to connect to the broker. By connecting to 9092 you connect to the listener that advertises localhost:9092 to the client for subsequent connections. That's why it works when you use kafkacat from your local machine (because 9092 is exposed to your local machine)
If you use broker:29092 then the broker will give the client the correct address for the connection (i.e. broker:29092).
To understand more about advertised listeners see this blog
Related
Unable to establish connection between Zookeeper and Kafka on Apple M1 using docker-compose
I am trying to run Kafka + Zookeeper through docker-compose.yml `version: '3' services: zookeeper: image: zookeeper:3.4.9 hostname: zookeeper ports: - "2181:2181" environment: ZOO_MY_ID: 1 ZOO_PORT: 2181 # ZOO_SERVERS: server.1=zookeeper:2888:3888;2181 volumes: - ./data/zookeeper/data:/data - ./data/zookeeper/datalog:/datalog kafka1: image: confluentinc/cp-kafka:5.3.0 hostname: kafka1 ports: - "9091:9091" environment: KAFKA_ADVERTISED_LISTENERS: LISTENER_DOCKER_INTERNAL://kafka1:19091,LISTENER_DOCKER_EXTERNAL://${DOCKER_HOST_IP:-127.0.0.1}:9091 KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: LISTENER_DOCKER_INTERNAL:PLAINTEXT,LISTENER_DOCKER_EXTERNAL:PLAINTEXT KAFKA_INTER_BROKER_LISTENER_NAME: LISTENER_DOCKER_INTERNAL KAFKA_ZOOKEEPER_CONNECT: "zookeeper:2181" KAFKA_BROKER_ID: 1 KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1 volumes: - ./data/kafka1/data:/var/lib/kafka/data depends_on: - zookeeper However, Kafka is unable to connect with the ZooKeeper kafka101-kafka1-1 | [main-SendThread(zookeeper:2181)] INFO org.apache.zookeeper.ClientCnxn - Socket error occurred: zookeeper/172.19.0.2:2181: Connection refused kafka101-kafka1-1 | [main-SendThread(zookeeper:2181)] INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server zookeeper/172.19.0.2:2181. Will not attempt to authenticate using SASL (unknown error) I checked and confirmed that the Zoo keeper is running ps -ef | grep zookeeper 1 zookeepe 0:19 {java} /usr/bin/qemu-x86_64 /usr/lib/jvm/java-1.8-openjdk/jre/bin/java /usr/lib/jvm/java-1.8-openjdk/jre/bin/java -Dzookeeper.log.dir=. -Dzookeeper.root.logger=INFO,CONSOLE -cp /zookeeper-3.4.9/bin/../build/classes:/zookeeper-3.4.9/bin/../build/lib/*.jar:/zookeeper-3.4.9/bin/../lib/slf4j-log4j12-1.6.1.jar:/zookeeper-3.4.9/bin/../lib/slf4j-api-1.6.1.jar:/zookeeper-3.4.9/bin/../lib/netty-3.10.5.Final.jar:/zookeeper-3.4.9/bin/../lib/log4j-1.2.16.jar:/zookeeper-3.4.9/bin/../lib/jline-0.9.94.jar:/zookeeper-3.4.9/bin/../zookeeper-3.4.9.jar:/zookeeper-3.4.9/bin/../src/java/lib/*.jar:/conf: -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.local.only=false org.apache.zookeeper.server.quorum.QuorumPeerMain /conf/zoo.cfg The zoo configuration looks like this: clientPort=2181 dataDir=/data dataLogDir=/datalog tickTime=2000 initLimit=5 syncLimit=2 Telnet output directly from zookeeper is : zookeeper-3.4.9 # telnet localhost 2181 telnet: can't connect to remote host (127.0.0.1): Connection refused ping to zookeeper host in kafka1 > ping zookeeper PING zookeeper (172.19.0.2) 56(84) bytes of data. PING zookeeper (172.19.0.2) 56(84) bytes of data. PING zookeeper (172.19.0.2) 56(84) bytes of data. I am running on Apple M1 Chipset What else can I check and do here?
Try using confluentinc's zookeeper image: confluentinc/cp-zookeeper:latest I was in the exact same boat, also on the M1 chip and this worked for me. These are the two services from my docker-compose for ref: zoo: image: confluentinc/cp-zookeeper:latest restart: unless-stopped ports: - 2181:2181 environment: ZOO_MY_ID: 1 ZOO_PORT: 2181 ZOO_SERVERS: server.1=zoo:2888:3888 ZOOKEEPER_CLIENT_PORT: 2181 kafka: image: confluentinc/cp-kafka:latest restart: always ports: - 9092:9092 environment: KAFKA_ADVERTISED_LISTENERS: LISTENER_DOCKER_INTERNAL://kafka:19092,LISTENER_DOCKER_EXTERNAL://${DOCKER_HOST_IP:-127.0.0.1}:9092 KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: LISTENER_DOCKER_INTERNAL:PLAINTEXT,LISTENER_DOCKER_EXTERNAL:PLAINTEXT KAFKA_INTER_BROKER_LISTENER_NAME: LISTENER_DOCKER_INTERNAL KAFKA_ZOOKEEPER_CONNECT: 'zoo:2181' KAFKA_BROKER_ID: 1 KAFKA_LOG4J_LOGGERS: 'kafka.controller=INFO,kafka.producer.async.DefaultEventHandler=INFO,state.change.logger=INFO' KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1 depends_on: - zoo
Kafka Consumer is not receiving Messages on docker
I'm a begginer on kafka as well as docker, I have been doing a course and working with kafka producer and consumer but for some reason it is not working. When I do use of the producer the message are saved in the topic (I have already checked it) but when I try to get the message using the consumer it is not working and I have no idea why. It had worked previously but not anymore. The unique difference I have in this case is that I'm using the confluentinc image instead of the bitnami image. So, if anyone has any idea or solution I would really appreciate it. I share my compose and an screenshot so you can see it. version: "3.2" services: ############################################################### zookeeper: image: 'confluentinc/cp-zookeeper:latest' container_name: zookeeper environment: ZOOKEEPER_CLIENT_PORT: 2181 ZOOKEEPER_TICK_TIME: 2000 ports: - 2181:2181 ############################################################### broker: image: 'confluentinc/cp-kafka:latest' container_name: broker depends_on: - zookeeper ports: - 9092:9092 environment: KAFKA_BROKER_ID: 1 KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181 # Exposes 9092 for external connections to the broker # Use kafka:29092 for connections internal on the docker network # See https://rmoff.net/2018/08/02/kafka-listeners-explained/ for details KAFKA_LISTENERS: "PLAINTEXT://:29092,PLAINTEXT_HOST://:9092" KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://broker:29092,PLAINTEXT_HOST://localhost:9092 KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT KAFKA_DELETE_TOPIC_ENABLE: "true" Producer and Consumer All is running in my local machine.
Take a look at docker-compose logs broker... You should see a lot of Error processing create topic request CreatableTopic(name='__consumer_offsets', numPartitions=50, replicationFactor=3 Without a valid __consumer_offsets topic, no consumer will be able to run and commit offsets. Similarly, transactions won't work either (which are enabled by default in latest Kafka) Add these variables and re-create the containers KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1 KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1 KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
kafka-reassign-partitions --generate for topic __commit_offsets gives me strange result: partition replica only on one broker anyway
kafka-reassign-partitions --generate for topic __commit_offsets gives me strange result: partition replica only on one broker anyway, but expected 3 replicas for each partition. I do the following: test#kafka-1:~/Kafka-Docker$ sudo docker exec -it broker bash [appuser#broker ~]$ echo '{"version":1, "topics":[{"topic":"__consumer_offsets"}]}' > topics-to-move.json [appuser#broker ~]$ kafka-reassign-partitions --zookeeper $KAFKA_ZOOKEEPER_CONNECT --topics-to-move-json-file topics-to-move.json --broker-list "1,2,3" --generate Warning: --zookeeper is deprecated, and will be removed in a future version of Kafka. Current partition replica assignment {"version":1,"partitions":[{"topic":"__consumer_offsets","partition":0,"replicas":[1],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":1,"replicas":[1],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":2,"replicas":[1],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":3,"replicas":[1],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":4,"replicas":[1],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":5,"replicas":[1],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":6,"replicas":[1],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":7,"replicas":[1],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":8,"replicas":[1],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":9,"replicas":[1],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":10,"replicas":[1],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":11,"replicas":[1],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":12,"replicas":[1],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":13,"replicas":[1],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":14,"replicas":[1],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":15,"replicas":[1],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":16,"replicas":[1],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":17,"replicas":[1],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":18,"replicas":[1],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":19,"replicas":[1],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":20,"replicas":[1],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":21,"replicas":[1],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":22,"replicas":[1],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":23,"replicas":[1],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":24,"replicas":[1],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":25,"replicas":[1],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":26,"replicas":[1],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":27,"replicas":[1],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":28,"replicas":[1],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":29,"replicas":[1],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":30,"replicas":[1],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":31,"replicas":[1],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":32,"replicas":[1],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":33,"replicas":[1],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":34,"replicas":[1],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":35,"replicas":[1],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":36,"replicas":[1],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":37,"replicas":[1],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":38,"replicas":[1],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":39,"replicas":[1],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":40,"replicas":[1],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":41,"replicas":[1],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":42,"replicas":[1],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":43,"replicas":[1],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":44,"replicas":[1],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":45,"replicas":[1],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":46,"replicas":[1],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":47,"replicas":[1],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":48,"replicas":[1],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":49,"replicas":[1],"log_dirs":["any"]}]} Proposed partition reassignment configuration {"version":1,"partitions":[{"topic":"__consumer_offsets","partition":0,"replicas":[1],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":1,"replicas":[2],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":2,"replicas":[3],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":3,"replicas":[1],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":4,"replicas":[2],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":5,"replicas":[3],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":6,"replicas":[1],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":7,"replicas":[2],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":8,"replicas":[3],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":9,"replicas":[1],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":10,"replicas":[2],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":11,"replicas":[3],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":12,"replicas":[1],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":13,"replicas":[2],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":14,"replicas":[3],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":15,"replicas":[1],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":16,"replicas":[2],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":17,"replicas":[3],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":18,"replicas":[1],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":19,"replicas":[2],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":20,"replicas":[3],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":21,"replicas":[1],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":22,"replicas":[2],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":23,"replicas":[3],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":24,"replicas":[1],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":25,"replicas":[2],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":26,"replicas":[3],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":27,"replicas":[1],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":28,"replicas":[2],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":29,"replicas":[3],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":30,"replicas":[1],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":31,"replicas":[2],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":32,"replicas":[3],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":33,"replicas":[1],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":34,"replicas":[2],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":35,"replicas":[3],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":36,"replicas":[1],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":37,"replicas":[2],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":38,"replicas":[3],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":39,"replicas":[1],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":40,"replicas":[2],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":41,"replicas":[3],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":42,"replicas":[1],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":43,"replicas":[2],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":44,"replicas":[3],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":45,"replicas":[1],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":46,"replicas":[2],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":47,"replicas":[3],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":48,"replicas":[1],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":49,"replicas":[2],"log_dirs":["any"]}]} [appuser#broker ~]$ I see that zookeepers option is deprecated so I also tried this (with the same result): kafka-reassign-partitions --bootstrap-server kafka-1:9092 --topics-to-move-json-file topics-to-move.json --broker-list "1,2,3" --generate PROBLEM: As you may see, "Current partition replica assignment" has "replicas: [1]" only and "Proposed partition reassignment configuration" has "replicas: [1]" or "replicas: [2]" or "replicas: [3]" for each partition. But for "Proposed partition reassignment configuration" it expected to be something like "replicas: [1,2,3]" or "replicas: [2,3,1]" and so on for redudancy of __commit_offsets across 3 brokers. Why only one replica for each partiotion is bound to one broker only? Why 3 replicas for each partition are not spreaded across 3 brokers? Example of good topic (but it has only one partition) replicated across all 3 brokers, it looks this way: {"version":1,"partitions":[{"topic":"test_topic","partition":0,"replicas":[1,2,3],"log_dirs":["any","any","any"]}]} My setup: 3 VMs 192.168.1.11, .12 and.13 each has zookeeper and broker, zookeepers connected to quorum ok - I see only one leader through its REST API (for example http://192.168.1.11:4888/commands/stats same for .12 and .13 IPs) and that's good. Also tested switch off one of zookeepers and new leader elected in that case - works perfect. Example of docker-compose.yml for 192.168.1.11 VM is (other similar): version: '3.7' x-zoo: &zoo "kafka-1:2888:3888;kafka-2:2888:3888;kafka-3:2888:3888" x-kafkaZookeepers: &kafkaZookeepers "kafka-1:2181,kafka-2:2181,kafka-3:2181" x-kafkaBrokers: &kafkaBrokers "kafka-1:9092,kafka-2:9092,kafka-3:9092" services: zookeeper: image: confluentinc/cp-zookeeper:6.2.1 hostname: zookeeper container_name: zookeeper ports: - "2181:2181" - "2888:2888" - "3888:3888" - "4888:8080" extra_hosts: - "kafka-1:192.168.1.11" - "kafka-2:192.168.1.12" - "kafka-3:192.168.1.13" environment: ZOOKEEPER_SERVER_ID: 1 ZOOKEEPER_QUORUM_LISTEN_ON_ALL_IPS: 'true' ZOOKEEPER_CLIENT_PORT: 2181 ZOOKEEPER_PEER_PORT: 2888 ZOOKEEPER_LEADER_PORT: 3888 ZOOKEEPER_TICK_TIME: 2000 ZOOKEEPER_INIT_LIMIT: 5 ZOOKEEPER_SYNC_LIMIT: 2 ZOOKEEPER_SERVERS: *zoo volumes: - ./kafka-data/zookeeper:/var/lib/zookeeper/data - ./kafka-data/zookeeper-logs:/var/lib/zookeeper/log networks: - mynet broker: image: confluentinc/cp-kafka:6.2.1 hostname: broker container_name: broker depends_on: - zookeeper ports: - "9092:9092" extra_hosts: - "kafka-1:192.168.1.11" - "kafka-2:192.168.1.12" - "kafka-3:192.168.1.13" environment: KAFKA_BROKER_ID: 1 KAFKA_LISTENERS: OUTSIDE://0.0.0.0:9092 KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: OUTSIDE:PLAINTEXT KAFKA_ADVERTISED_LISTENERS: OUTSIDE://192.168.1.11:9092 KAFKA_INTER_BROKER_LISTENER_NAME: OUTSIDE KAFKA_ZOOKEEPER_CONNECT: *kafkaZookeepers KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 3 KAFKA_DEFAULT_REPLICATION_FACTOR: 3 KAFKA_LOG_MESSAGE_TIMESTAMP_TYPE: 'LogAppendTime' KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0 KAFKA_CONNECTIONS_MAX_IDLE_MS: 31536000000 # 1 year volumes: - ./kafka-data/kafka:/var/lib/kafka/data networks: - mynet networks: mynet: driver: bridge It was set up first with: KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1 KAFKA_DEFAULT_REPLICATION_FACTOR: 1 Then +2 other setups on .12 and .13 VMs added (of course with different ZOOKEEPER_SERVER_ID and KAFKA_BROKER_ID) and on each KAFKA_DEFAULT_REPLICATION_FACTOR and KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR changed from 1 to 3. Then new topics created OK with replication factor 3, also I was able easily change replication factor to 3 for my existing non-system topics, but except topic __commit_offsets of cource - it need to be reassigned using kafka-reassign-partitions command line tool.
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR changed from 1 to 3. The offsets topic only gets created on a brand new cluster. It will not retroactively get updated if you modify offsets.topic.replication.factor. new topics created OK with replication factor 3 Correct, however, any topic created before you increased default.replication.factor will also only have one replica. Reassign partition tool doesn't take into account your server properties. You either need a different tool or manually place the replicas yourself. - How to change the number of replicas of a Kafka topic?
Kafka Listener is not working! It is isolated in intranet
My Kafka node is hosted in Google Cloud Dataproc. However, we realized that the Kafka installed through default initialization script is set up in such a way that it only allows intranet access. It is completely isolated from the outside world. The producer outside the google cloud network can't publish the message to Kafka and the Kafka message can't chain to its extranet subscriber. Remark I have whitelisted the producer IP After read thru the other StackOverflow, blog post and documentation. I think it could due to advertised.listeners parts of Socket Server Settings in /usr/lib/kafka/server.properties. First solution I added advertised.listeners=PLAINTEXT://[External_IP]:19092 then sudo /etc/init.d/kafka-server restart OUTCOME However, when I trying to Kafkacat or telnet, it always failed. I also tested advertised.listeners with various port Second solution from https://rmoff.net/2018/08/02/kafka-listeners-explained/ ############################# Server Basics ############################# # The id of the broker. This must be set to a unique integer for each broker. broker.id=0 ############################# Socket Server Settings ############################# # The address the socket server listens on. It will get the value returned from # java.net.InetAddress.getCanonicalHostName() if not configured. # FORMAT: # listeners = listener_name://host_name:port # EXAMPLE: # listeners = PLAINTEXT://your.host.name:9092 # # Hostname and port the broker will advertise to producers and consumers. If not set, # it uses the value for "listeners" if configured. Otherwise, it will use the value # returned from java.net.InetAddress.getCanonicalHostName(). ->>>>>>> I added below listener config according to https://rmoff.net/2018/08/02/kafka-listeners-explained/ listeners=INTERNAL://0.0.0.0:9092,EXTERNAL://0.0.0.0:19092 listener.security.protocol.map=INTERNAL:PLAINTEXT,EXTERNAL:PLAINTEXT advertised.listeners=EXTERNAL://[External_IP]:19092,INTERNAL://[Internal_IP]:9092 inter.broker.listener.name=INTERNAL OUTCOME It's the same result as above, Not Working. Firewall Rules [Updated] This is my current firewall rules config. Am I doing a mistake? Can anyone help me to resolve this?
Here is what worked for my cluster: I've set the following properties from the second solution: listeners=INTERNAL://0.0.0.0:9092,EXTERNAL://0.0.0.0:19092 listener.security.protocol.map=INTERNAL:PLAINTEXT,EXTERNAL:PLAINTEXT advertised.listeners=EXTERNAL://[External_IP]:19092,INTERNAL://[Internal_IP]:9092 inter.broker.listener.name=INTERNAL I've created a firewall rule opening port 19092 to my personal development machine IP and applied it to the network. From my machine, I've tried to telnet the kafka server and I got: $ telnet [EXTERNAL-IP] 19092 Trying [EXTERNAL-IP]... Connected to [EXTERNAL-IP]. Escape character is '^]'. I then tried to use kafkacat, and got an error. Running in debug, I saw the error was because I have not set any topics: %7|1578351264.551|METADATA|rdkafka#producer-1| [thrd:main]: [EXTERNAL-IP]:19092/bootstrap: ===== Received metadata: application requested ===== %7|1578351264.551|METADATA|rdkafka#producer-1| [thrd:main]: [EXTERNAL-IP]:19092/bootstrap: ClusterId: jYxfi6zzR0euAovYyKCFZg, ControllerId: -1 %7|1578351264.551|METADATA|rdkafka#producer-1| [thrd:main]: [EXTERNAL-IP]:19092/bootstrap: 0 brokers, 0 topics %7|1578351264.551|METADATA|rdkafka#producer-1| [thrd:main]: [EXTERNAL-IP]:19092/bootstrap: No brokers or topics in metadata: should retry %7|1578351264.551|REQERR|rdkafka#producer-1| [thrd:main]: [EXTERNAL-IP]:19092/bootstrap: MetadataRequest failed: Local: Partial response: explicit actions Retry %7|1578351264.551|RETRY|rdkafka#producer-1| [thrd:[EXTERNAL-IP]:19092/bootstrap]: [EXTERNAL-IP]:19092/bootstrap: Retrying MetadataRequest (v2, 25 bytes, retry 1/2, prev CorrId 3) in 100ms Please notice that I've tried to connect to the kafka server from outside to the cluster. In the questions, the telnet and kafkacat are running on the same machine as the kafka server (kafka-tng-w-0).
Here is a sample docker-compose.yaml file. version: '2' services: zookeeper: image: strimzi/kafka:0.20.0-kafka-2.6.0 command: [ "sh", "-c", "bin/zookeeper-server-start.sh config/zookeeper.properties" ] ports: - "2181:2181" environment: LOG_DIR: /tmp/logs kafka: image: strimzi/kafka:0.20.0-kafka-2.6.0 command: [ "sh", "-c", "bin/kafka-server-start.sh config/server.properties --override listeners=$${KAFKA_LISTENERS} --override advertised.listeners=$${KAFKA_ADVERTISED_LISTENERS} --override zookeeper.connect=$${KAFKA_ZOOKEEPER_CONNECT}" ] depends_on: - zookeeper ports: - "9092:9092" environment: LOG_DIR: "/tmp/logs" # Dev GQ - Laptop KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://172.23.240.1:9092 # AWS Pre-Prod #KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://11.122.200.229:9092 KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:9092 KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181 And here is a sample Quarkus application.properties file with kafka bootstrap server configured as advertised listeners in docker-compose.yaml. # Configure the SmallRye Kafka connector # Dev GQ - Laptop mp.messaging.connector.smallrye-kafka.bootstrap.servers=172.23.240.1:9092 # AWS Pre-Prod #mp.messaging.connector.smallrye-kafka.bootstrap.servers=11.122.200.229:9092 quarkus.kafka.health.enabled=true # Configure the Kafka sink (we write to it) mp.messaging.outgoing.generated-price.connector=smallrye-kafka mp.messaging.outgoing.generated-price.topic=prices mp.messaging.outgoing.generated-price.value.serializer=org.apache.kafka.common.serialization.IntegerSerializer # Configure the Kafka source (we read from it) mp.messaging.incoming.prices.connector=smallrye-kafka mp.messaging.incoming.prices.topic=prices # ..... more codes
version: "3" services: zookeeper: image: wurstmeister/zookeeper ports: - "2181:2181" kafka: image: wurstmeister/kafka hostname: kafka ports: - "9093:9093" - "9092:9092" environment: TZ: CST-8 KAFKA_BROKER_ID: 3 KAFKA_ADVERTISED_HOST_NAME: kafka KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181 KAFKA_LISTENERS: INSIDE://:9092,OUTSIDE://:9093 KAFKA_ADVERTISED_LISTENERS: INSIDE://kafka:9092,OUTSIDE://${Your_External_IP}:9093 KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INSIDE:PLAINTEXT,OUTSIDE:PLAINTEXT KAFKA_INTER_BROKER_LISTENER_NAME: INSIDE volumes: - /var/run/docker.sock:/var/run/docker.sock links: - zookeeper
Controller 1's connection to broker was unsuccessful docker confluent
I'm following the setup of provisioning Kafka Cluster, Zookeeper, Control-center docker images provided by Confluent team in my local machine with the docker-compose.yml file below: docker-compose.yml redis: image: redis container_name: redis ports: - 6379:6379 mysql: image: mysql:5.6 container_name: mysql command: --default-authentication-plugin=mysql_native_password restart: always environment: MYSQL_ROOT_PASSWORD: rootpass MYSQL_DATABASE: MYSQL_USER: MYSQL_PASSWORD: ports: - 3306:3306 ksql-server: image: confluentinc/cp-ksql-server:5.2.1 hostname: ksql-server container_name: ksql-server links: - kafka ports: - "8088:8088" environment: KSQL_CONFIG_DIR: "/etc/ksql" KSQL_LOG4J_OPTS: "-Dlog4j.configuration=file:/etc/ksql/log4j-rolling.properties" KSQL_BOOTSTRAP_SERVERS: "kafka:29092" KSQL_HOST_NAME: ksql-server KSQL_APPLICATION_ID: "cp-all-in-one" KSQL_LISTENERS: "http://0.0.0.0:8088" KSQL_CACHE_MAX_BYTES_BUFFERING: 0 # KSQL_KSQL_SCHEMA_REGISTRY_URL: "http://schema-registry:8081" KSQL_PRODUCER_INTERCEPTOR_CLASSES: "io.confluent.monitoring.clients.interceptor.MonitoringProducerInterceptor" KSQL_CONSUMER_INTERCEPTOR_CLASSES: "io.confluent.monitoring.clients.interceptor.MonitoringConsumerInterceptor" zookeeper: image: confluentinc/cp-zookeeper:latest container_name: zookeeper environment: ZOOKEEPER_CLIENT_PORT: 2181 ZOOKEEPER_TICK_TIME: 2000 kafka: image: confluentinc/cp-kafka:latest container_name: kafka links: - zookeeper ports: - 9092:9092 - 29092:29092 environment: KAFKA_BROKER_ID: 1 KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181 KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:29092,PLAINTEXT_HOST://localhost:9092 KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1 control-center: image: confluentinc/cp-enterprise-control-center:5.2.1 hostname: control-center container_name: control-center links: - zookeeper - kafka - ksql-server ports: - 9021:9021 environment: CONTROL_CENTER_BOOTSTRAP_SERVERS: kafka:29092 CONTROL_CENTER_ZOOKEEPER_CONNECT: zookeeper:2181 CONTROL_CENTER_KSQL_URL: http://ksql-server:8088 CONTROL_CENTER_KSQL_ADVERTISED_URL: http://localhost:8088 CONTROL_CENTER_REPLICATION_FACTOR: 1 CONTROL_CENTER_INTERNAL_TOPICS_PARTITIONS: 1 CONTROL_CENTER_MONITORING_INTERCEPTOR_TOPIC_PARTITIONS: 1 CONFLUENT_METRICS_TOPIC_REPLICATION: 1 PORT: 9021 The setup worked well on my old laptop. But after changing to use new one, I cannot make it works. I tried to run docker logs control-center and see the logs below, Logs : [main] INFO org.apache.kafka.common.utils.AppInfoParser - Kafka version: 2.2.0-cp2 [main] INFO org.apache.kafka.common.utils.AppInfoParser - Kafka commitId: c4bca159dd111016 [kafka-admin-client-thread | adminclient-1] WARN org.apache.kafka.clients.NetworkClient - [AdminClient clientId=adminclient-1] Connection to node -1 (kafka/172.17.0.5:29092) could not be established. Broker may not be available. [kafka-admin-client-thread | adminclient-1] WARN org.apache.kafka.clients.NetworkClient - [AdminClient clientId=adminclient-1] Connection to node -1 (kafka/172.17.0.5:29092) could not be established. Broker may not be available. [kafka-admin-client-thread | adminclient-1] WARN org.apache.kafka.clients.NetworkClient - [AdminClient clientId=adminclient-1] Connection to node -1 (kafka/172.17.0.5:29092) could not be established. Broker may not be available. [kafka-admin-client-thread | adminclient-1] WARN org.apache.kafka.clients.NetworkClient - [AdminClient clientId=adminclient-1] Connection to node -1 (kafka/172.17.0.5:29092) could not be established. Broker may not be available. I checked the logs of kafka docker and see the errors [2019-05-18 11:39:28,721] WARN [RequestSendThread controllerId=1] Controller 1's connection to broker kafka:29092 (id: 1 rack: null) was unsuccessful (kafka.controller.RequestSendThread) java.lang.IllegalStateException: No entry found for connection 1 at org.apache.kafka.clients.ClusterConnectionStates.nodeState(ClusterConnectionStates.java:339) at org.apache.kafka.clients.ClusterConnectionStates.disconnected(ClusterConnectionStates.java:143) at org.apache.kafka.clients.NetworkClient.initiateConnect(NetworkClient.java:921) at org.apache.kafka.clients.NetworkClient.ready(NetworkClient.java:287) at org.apache.kafka.clients.NetworkClientUtils.awaitReady(NetworkClientUtils.java:65) at kafka.controller.RequestSendThread.brokerReady(ControllerChannelManager.scala:279) at kafka.controller.RequestSendThread.doWork(ControllerChannelManager.scala:233) at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:82) Below is the docker containers running CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 94e7a2e4e61a backend_collection-engine-consumer-generator "java -cp collection…" 4 minutes ago Up 4 minutes collection-engine-consumer-generator 791528e51a06 backend_collection-engine-executor "java -DsplitEvents=…" 4 minutes ago Up 4 minutes collection-engine-executor d2eadae2ce7d confluentinc/cp-kafka:latest "/etc/confluent/dock…" 8 minutes ago Up 8 minutes 0.0.0.0:9092->9092/tcp, 0.0.0.0:29092->29092/tcp kafka b1230959ceeb confluentinc/cp-zookeeper:latest "/etc/confluent/dock…" 8 minutes ago Up 8 minutes 2181/tcp, 2888/tcp, 3888/tcp zookeeper 0bcb7bc0e2d4 redis "docker-entrypoint.s…" 8 minutes ago Up 8 minutes 0.0.0.0:6379->6379/tcp redis 040fec54d7b8 mysql:5.6 "docker-entrypoint.s…" 8 minutes ago Up 8 minutes 0.0.0.0:3306->3306/tcp mysql I cannot even access into Control Center via http://localhost:9021 as normal.