toxiproxy for kafka in a docker-compose file - apache-kafka

Question

I'm trying to integrate toxy-proxy with a kafka cluster, but seems to not work.
The docker-compose file looks something like this:
kafka:
image: confluentinc/cp-kafka:latest
depends_on:
- zookeeper
environment:
KAFKA_BROKER_ID: 1
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2182
KAFKA_LISTENERS: PLAINTEXT://kafka:9092,PLAINTEXT_HOST://localhost:29092
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092,PLAINTEXT_HOST://localhost:29092
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
toxiproxy:
image: 'ghcr.io/shopify/toxiproxy:2.4.0'
ports:
- 8474:8474
- 29092:29092
Also my application is set up like this:
my_application:
image: image
environment:
KAFKA_BROKERS_URL: toxiproxy:8474
The problem is that my application does not connect to kafka. Not sure how should be done. I used this https://github.com/Shopify/sarama/blob/main/docker-compose.yml as an example.
Also I'd like to know how to call kafka from my localhost, so not from inside a docker container.
Basically this works now
kafka:
image: confluentinc/cp-kafka:latest
depends_on:
- zookeeper
ports:
- 29092:29092
environment:
KAFKA_BROKER_ID: 1
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2182
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092,PLAINTEXT_HOST://localhost:29092
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
toxiproxy:
container_name: toxiproxy
image: 'ghcr.io/shopify/toxiproxy:latest'
command: ['-config', '/config/toxiproxy.json','-host', '0.0.0.0']
volumes:
- ./config/toxiproxy.json:/config/toxiproxy.json
ports:
- target: 8474
published: 8474
protocol: tcp
mode: host
depends_on:
- kafka
Where the config file looks like this:
[
{
"name": "kafka_proxy",
"listen": "[::]:45390",
"upstream": "kafka:9092",
"enabled": true
}
]
However because after the application makes a first connection to Kafka it will ask for cluster configuration, and will be served hostnames according to ADVERTISED_LISTENERS, which will be set to the actual kafka container and not to my proxy.
This I still don't know how to do, to set the advertised_listeners to use the proxy as well.
I've tried setting it like this, but kafka stops working.
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://toxiproxy:45390