Unable to consume or produce to a remote Kafka broker - apache-kafka

I have setup a simple droplet on Digital Ocean and am running a single Kafka and Zookeeper node which is started using a docker-compose file.
I am running into an issue with consuming or producing to the Kafka broker from outside of the Digital Ocean droplet.
This is what my docker-compose looks like,
version: '3.4'
services:
zookeeper:
image: confluentinc/cp-zookeeper:latest
environment:
ZOOKEEPER_CLIENT_PORT: 2181
ZOOKEEPER_TICK_TIME: 2000
volumes:
- /root/data/zookeeper/etc:/etc/zookeeper
- /root/data/zookeeper/data:/var/lib/zookeeper/data
container_name: "zookeeper"
network_mode: "host"
kafka:
image: confluentinc/cp-kafka:latest
depends_on:
- zookeeper
environment:
KAFKA_BROKER_ID: 1
KAFKA_ZOOKEEPER_CONNECT: PUBLIC_DIGITIAL_OCEAN_IP:2181
KAFKA_ADVERTISED_LISTENERS:PLAINTEXT://PUBLIC_DIGITIAL_OCEAN_IP:9093
KAFKA_LISTENER: PUBLIC_DIGITIAL_OCEAN_IP:9093
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
KAFKA_LOG4J_LOGGERS: "kafka.controller=WARN"
KAFKA_LOG4J_ROOT_LOGLEVEL: WARN
KAFKA_TOOLS_LOG4J_LOGLEVEL: ERROR
volumes:
- /root/data/kafka/etc:/etc/kafka
- /root/data/kafka/data:/var/lib/kafka/data
container_name: "kafka"
network_mode: "host"
I have tried different combinations with setting KAFKA_ADVERTISED_LISTENERS to use localhost, 0.0.0.0 and I am not having any success.
I can consume and produce if I enter the kafka container and use the CLI.
From what I have read, digital ocean does not have any firewall rules so the the ports are being exposed.
snippet from running netstat within the droplet
> netstat -tulpn | grep :2181
> tcp6 0 0 :::2181 :::* LISTEN 10522/java
> netstat -tulpn | grep :9093
> tcp6 0 0 :::9093 :::* LISTEN 13093/java
Any help is greatly appreciated!

The issue was my firewall rules on the droplet. running the commands;
sudo ufw allow 2181 && sudo ufw allow 9092 resolved my issue.

Related

Unable to establish connection between Zookeeper and Kafka on Apple M1 using docker-compose

I am trying to run Kafka + Zookeeper through docker-compose.yml
`version: '3'
services:
zookeeper:
image: zookeeper:3.4.9
hostname: zookeeper
ports:
- "2181:2181"
environment:
ZOO_MY_ID: 1
ZOO_PORT: 2181
# ZOO_SERVERS: server.1=zookeeper:2888:3888;2181
volumes:
- ./data/zookeeper/data:/data
- ./data/zookeeper/datalog:/datalog
kafka1:
image: confluentinc/cp-kafka:5.3.0
hostname: kafka1
ports:
- "9091:9091"
environment:
KAFKA_ADVERTISED_LISTENERS: LISTENER_DOCKER_INTERNAL://kafka1:19091,LISTENER_DOCKER_EXTERNAL://${DOCKER_HOST_IP:-127.0.0.1}:9091
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: LISTENER_DOCKER_INTERNAL:PLAINTEXT,LISTENER_DOCKER_EXTERNAL:PLAINTEXT
KAFKA_INTER_BROKER_LISTENER_NAME: LISTENER_DOCKER_INTERNAL
KAFKA_ZOOKEEPER_CONNECT: "zookeeper:2181"
KAFKA_BROKER_ID: 1
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
volumes:
- ./data/kafka1/data:/var/lib/kafka/data
depends_on:
- zookeeper
However, Kafka is unable to connect with the ZooKeeper
kafka101-kafka1-1 | [main-SendThread(zookeeper:2181)] INFO org.apache.zookeeper.ClientCnxn - Socket error occurred: zookeeper/172.19.0.2:2181: Connection refused
kafka101-kafka1-1 | [main-SendThread(zookeeper:2181)] INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server zookeeper/172.19.0.2:2181. Will not attempt to authenticate using SASL (unknown error)
I checked and confirmed that the Zoo keeper is running
ps -ef | grep zookeeper
1 zookeepe 0:19 {java} /usr/bin/qemu-x86_64 /usr/lib/jvm/java-1.8-openjdk/jre/bin/java /usr/lib/jvm/java-1.8-openjdk/jre/bin/java -Dzookeeper.log.dir=. -Dzookeeper.root.logger=INFO,CONSOLE -cp /zookeeper-3.4.9/bin/../build/classes:/zookeeper-3.4.9/bin/../build/lib/*.jar:/zookeeper-3.4.9/bin/../lib/slf4j-log4j12-1.6.1.jar:/zookeeper-3.4.9/bin/../lib/slf4j-api-1.6.1.jar:/zookeeper-3.4.9/bin/../lib/netty-3.10.5.Final.jar:/zookeeper-3.4.9/bin/../lib/log4j-1.2.16.jar:/zookeeper-3.4.9/bin/../lib/jline-0.9.94.jar:/zookeeper-3.4.9/bin/../zookeeper-3.4.9.jar:/zookeeper-3.4.9/bin/../src/java/lib/*.jar:/conf: -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.local.only=false org.apache.zookeeper.server.quorum.QuorumPeerMain /conf/zoo.cfg
The zoo configuration looks like this:
clientPort=2181
dataDir=/data
dataLogDir=/datalog
tickTime=2000
initLimit=5
syncLimit=2
Telnet output directly from zookeeper is :
zookeeper-3.4.9 # telnet localhost 2181
telnet: can't connect to remote host (127.0.0.1): Connection refused
ping to zookeeper host in kafka1
> ping zookeeper
PING zookeeper (172.19.0.2) 56(84) bytes of data.
PING zookeeper (172.19.0.2) 56(84) bytes of data.
PING zookeeper (172.19.0.2) 56(84) bytes of data.
I am running on Apple M1 Chipset
What else can I check and do here?
Try using confluentinc's zookeeper image: confluentinc/cp-zookeeper:latest
I was in the exact same boat, also on the M1 chip and this worked for me.
These are the two services from my docker-compose for ref:
zoo:
image: confluentinc/cp-zookeeper:latest
restart: unless-stopped
ports:
- 2181:2181
environment:
ZOO_MY_ID: 1
ZOO_PORT: 2181
ZOO_SERVERS: server.1=zoo:2888:3888
ZOOKEEPER_CLIENT_PORT: 2181
kafka:
image: confluentinc/cp-kafka:latest
restart: always
ports:
- 9092:9092
environment:
KAFKA_ADVERTISED_LISTENERS: LISTENER_DOCKER_INTERNAL://kafka:19092,LISTENER_DOCKER_EXTERNAL://${DOCKER_HOST_IP:-127.0.0.1}:9092
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: LISTENER_DOCKER_INTERNAL:PLAINTEXT,LISTENER_DOCKER_EXTERNAL:PLAINTEXT
KAFKA_INTER_BROKER_LISTENER_NAME: LISTENER_DOCKER_INTERNAL
KAFKA_ZOOKEEPER_CONNECT: 'zoo:2181'
KAFKA_BROKER_ID: 1
KAFKA_LOG4J_LOGGERS: 'kafka.controller=INFO,kafka.producer.async.DefaultEventHandler=INFO,state.change.logger=INFO'
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
depends_on:
- zoo

Unable to connect to Kafka(in docker) from Internet

I am attempting to make it so I can connect to Kafka from over the internet. I have opened Ports: 9092 and 2181 on my router.
I have had no luck at all! I am using OffsetExplorer and I am able to ping kafka from another network. The IP of the system that is is running on is: 10.0.1.104
I AM able to connect to kafka on the local network from another computer though.
Here is my Kafka Docker-Compose:
version: '3.7'
services:
zookeeper:
image: wurstmeister/zookeeper:3.4.6
environment:
JVMFLAGS: "-Djava.security.auth.login.config=/etc/zookeeper/zookeeper_jaas.conf"
volumes:
- ./zookeeper_jaas.conf:/etc/zookeeper/zookeeper_jaas.conf
ports:
- 2181:2181
kafka:
image: wurstmeister/kafka:2.13-2.8.1
depends_on:
- zookeeper
ports:
- 9092:9092
- 29092:29092
environment:
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
KAFKA_LISTENERS: INTERNAL://:9093,EXTERNAL://:9092
KAFKA_ADVERTISED_LISTENERS: INTERNAL://kafka:9093,EXTERNAL://10.0.1.104:9092
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INTERNAL:SASL_PLAINTEXT,EXTERNAL:SASL_PLAINTEXT
ALLOW_PLAINTEXT_LISTENER: 'yes'
KAFKA_AUTO_CREATE_TOPICS_ENABLE: 'true'
KAFKA_INTER_BROKER_LISTENER_NAME: INTERNAL
KAFKA_SASL_ENABLED_MECHANISMS: PLAIN
KAFKA_SASL_MECHANISM_INTER_BROKER_PROTOCOL: PLAIN
KAFKA_OPTS: "-Djava.security.auth.login.config=/etc/kafka/kafka_jaas.conf"
volumes:
- ./kafka_server_jaas.conf:/etc/kafka/kafka_jaas.conf
Connection attempts result in this on Kafka's output
Thank you very much!

Kafka Consumer is not receiving Messages on docker

I'm a begginer on kafka as well as docker, I have been doing a course and working with kafka producer and consumer but for some reason it is not working.
When I do use of the producer the message are saved in the topic (I have already checked it) but when I try to get the message using the consumer it is not working and I have no idea why.
It had worked previously but not anymore.
The unique difference I have in this case is that I'm using the confluentinc image instead of the bitnami image.
So, if anyone has any idea or solution I would really appreciate it.
I share my compose and an screenshot so you can see it.
version: "3.2"
services:
###############################################################
zookeeper:
image: 'confluentinc/cp-zookeeper:latest'
container_name: zookeeper
environment:
ZOOKEEPER_CLIENT_PORT: 2181
ZOOKEEPER_TICK_TIME: 2000
ports:
- 2181:2181
###############################################################
broker:
image: 'confluentinc/cp-kafka:latest'
container_name: broker
depends_on:
- zookeeper
ports:
- 9092:9092
environment:
KAFKA_BROKER_ID: 1
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
# Exposes 9092 for external connections to the broker
# Use kafka:29092 for connections internal on the docker network
# See https://rmoff.net/2018/08/02/kafka-listeners-explained/ for details
KAFKA_LISTENERS: "PLAINTEXT://:29092,PLAINTEXT_HOST://:9092"
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://broker:29092,PLAINTEXT_HOST://localhost:9092
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
KAFKA_DELETE_TOPIC_ENABLE: "true"
Producer and Consumer
All is running in my local machine.
Take a look at docker-compose logs broker...
You should see a lot of Error processing create topic request CreatableTopic(name='__consumer_offsets', numPartitions=50, replicationFactor=3
Without a valid __consumer_offsets topic, no consumer will be able to run and commit offsets. Similarly, transactions won't work either (which are enabled by default in latest Kafka)
Add these variables and re-create the containers
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1

NiFi in docker container fails to talk to kafka: TimoutException, kafkacat ist working just fine

I have set up NiFi (1.11.4) & Kafka(2.5) via docker (docker-compose file below, actual NiFi flow definition https://github.com/geoHeil/streaming-reference).
When trying to follow up on basic getting started tutorials (such as https://towardsdatascience.com/big-data-managing-the-flow-of-data-with-apache-nifi-and-apache-kafka-af674cd8f926) which combine processors such as:
generate flowfile (CSV)
update attribute
PublishKafka2.0
I run into issues of timeoutException:
nifi_1 | 2020-06-10 11:15:47,311 ERROR [kafka-producer-network-thread | producer-2] o.a.n.p.k.pubsub.PublishKafkaRecord_2_0 PublishKafkaRecord_2_0[id=959f0e64-0172-1000-0000-0000650181a4] Failed to send StandardFlowFileRecord[uuid=944464e4-94ea-48dc-89fa-d19c34f163e7,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1591771086044-1, container=default, section=1], offset=227962, length=422],offset=0,name=c8dd1dd2-0ffe-4875-9d45-902ea331c210,size=422] to Kafka: org.apache.kafka.common.errors.TimeoutException: Expiring 64 record(s) for test-0: 30029 ms has passed since batch creation plus linger time
nifi_1 | org.apache.kafka.common.errors.TimeoutException: Expiring 64 record(s) for test-0: 30029 ms has passed since batch creation plus linger time
nifi_1 | 2020-06-10 11:15:47,311 ERROR [kafka-producer-network-thread | producer-2] o.a.n.p.k.pubsub.PublishKafkaRecord_2_0 PublishKafkaRecord_2_0[id=959f0e64-0172-1000-0000-0000650181a4] Failed to send StandardFlowFileRecord[uuid=944464e4-94ea-48dc-89fa-d19c34f163e7,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1591771086044-1, container=default, section=1], offset=227962, length=422],offset=0,name=c8dd1dd2-0ffe-4875-9d45-902ea331c210,size=422] to Kafka: org.apache.kafka.common.errors.TimeoutException: Expiring 64 record(s) for test-0: 30029 ms has passed since batch creation plus linger time
nifi_1 | org.apache.kafka.common.errors.TimeoutException: Expiring 64 record(s) for test-0: 30029 ms has passed since batch creation plus linger time
However, a:
kafkacat -C -b localhost:9092 -t test #starts listener
kafkacat -P -b localhost:9092 -t test #starts producer
pipes events just fine through the kafka instance.
The docker-compose file looks like:
version: "3"
services:
nifi:
image: apache/nifi:1.11.4
ports:
- 8080:8080 # Unsecured HTTP Web Port
environment:
- NIFI_WEB_HTTP_PORT=8080
- NIFI_CLUSTER_IS_NODE=true
- NIFI_CLUSTER_NODE_PROTOCOL_PORT=8082
- NIFI_ZK_CONNECT_STRING=zookeeper:2181
- NIFI_ELECTION_MAX_WAIT=1 min
links:
- broker
- zookeeper
volumes:
- ./for_nifi/conf:/opt/nifi/nifi-current/conf
zookeeper: #https://github.com/confluentinc/cp-all-in-one/blob/5.5.0-post/cp-all-in-one/docker-compose.yml#L5
image: confluentinc/cp-zookeeper:5.5.0
hostname: zookeeper
container_name: zookeeper
ports:
- "2181:2181"
environment:
ZOOKEEPER_CLIENT_PORT: 2181
ZOOKEEPER_TICK_TIME: 2000
broker:
image: confluentinc/cp-kafka:5.5.0
hostname: broker
container_name: broker
depends_on:
- zookeeper
ports:
- "29092:29092"
- "9092:9092"
environment:
KAFKA_BROKER_ID: 1
KAFKA_ZOOKEEPER_CONNECT: 'zookeeper:2181'
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://broker:29092,PLAINTEXT_HOST://localhost:9092
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
You're using the wrong port to connect to the broker. By connecting to 9092 you connect to the listener that advertises localhost:9092 to the client for subsequent connections. That's why it works when you use kafkacat from your local machine (because 9092 is exposed to your local machine)
If you use broker:29092 then the broker will give the client the correct address for the connection (i.e. broker:29092).
To understand more about advertised listeners see this blog

Can't connect to postgres db from pgadmin (both running on docker)?

I'm running postgres and pgadmin4 on docker with docker-compose up on a fedora 28 OS and I'm having trouble creating a new db server from pgadmin's web console.
This is the docker-compose.yml file I'm using.
version: '3.0'
services:
db:
image: postgres:9.6
ports:
- 5432:5432/tcp
environment:
- POSTGRES_USER=admin
- POSTGRES_PASSWORD=admin
- POSTGRES_DB=mydb
pgadmin:
image: dpage/pgadmin4
ports:
- 5454:5454/tcp
environment:
- PGADMIN_DEFAULT_EMAIL=admin#mydomain.com
- PGADMIN_DEFAULT_PASSWORD=postgres
- PGADMIN_LISTEN_PORT=5454
What should I write in the Create new server > Connection tab > "Host name/address" field? If I type in localhost or 127.0.0.1 I get an error (Unable to connect, see screenshot1 and screenshot2). If I type db (the service's name as specified in the yml file), only then pgadmin accepts it and creates a db server with a postgres database called mydb.
Why? How do I find the ip that goes in the address field?
Furthermore, on Fedora28:
$ netstat -napt | grep LIST
(Not all processes could be identified, non-owned process info
will not be shown, you would have to be root to see it all.)
tcp 0 0 192.168.122.1:53 0.0.0.0:* LISTEN -
tcp 0 0 127.0.0.1:3350 0.0.0.0:* LISTEN -
tcp 0 0 127.0.0.1:631 0.0.0.0:* LISTEN -
tcp 0 0 0.0.0.0:3389 0.0.0.0:* LISTEN -
tcp6 0 0 :::5454 :::* LISTEN -
tcp6 0 0 ::1:631 :::* LISTEN -
tcp6 0 0 :::5432 :::* LISTEN -
$
I encountered this problem just recently too. There are two approaches I found:
1) See here. Basically, you just search for the IP address of the postgres container and use that IP address in pgadmin4:
$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
194d4a5f9dd0 dpage/pgadmin4 "/entrypoint.sh" 48 minutes ago Up 48 minutes 443/tcp, 0.0.0.0:8080->80/tcp docker-postgis_pgadmin_1
334d5bdc87f7 kartoza/postgis:11.0-2.5 "/bin/sh -c /docke..." 48 minutes ago Up 48 minutes (healthy) 0.0.0.0:5432->5432/tcp docker-postgis_db_1
In my case, the postgres container ID is 334d5bdc87f7. Then look for the IP address:
$ docker inspect 334d5bdc87f7 | grep IPAddress
"SecondaryIPAddresses": null,
"IPAddress": "",
"IPAddress": "172.18.0.2",
When I used 172.18.0.2 in pgadmin4, I connected to the database! Yey!
2) The 2nd approach is easier. Instead of using localhost or 127.0.0.1 or ::1, I used my IP address in my local network (e.g. in your case 192.168.122.1?). Afterwards, I connected to the postgres container!
From my reading of the docs and testing this myself you're doing it right using the database service name from the docker-compose yml file as the "Host name/address" field value in pgAdmin.
https://docs.docker.com/compose/networking/
In your "pgadmin" section I would use port values of 8080 or 80, why 5454?