I'm trying a simplified example of using Kafka and the Kafka Exporter with Grafana. I have a docker-compose.yml similar to the following:
version: '2'
networks:
app-tier:
driver: bridge
services:
zookeeper:
image: 'bitnami/zookeeper:latest'
environment:
- 'ALLOW_ANONYMOUS_LOGIN=yes'
networks:
- app-tier
kafka:
image: 'bitnami/kafka:latest'
environment:
- KAFKA_CFG_ZOOKEEPER_CONNECT=zookeeper:2181
- ALLOW_PLAINTEXT_LISTENER=yes
networks:
- app-tier
kafka-exporter:
build: kafka-exporter
ports:
- "9308:9308"
networks:
- app-tier
entrypoint: ["run.sh"]
grafana:
image: grafana/grafana
ports:
- "3000:3000"
networks:
- app-tier
where run.sh is a wrapper script to keep retrying to run the exporter while Kafka is starting, see https://github.com/khpeek/kafka-exporter-example. The problem is that when I log into Grafana and try to add a Prometheus data source with URL http://kafka-exporter:9308/metrics, I get an error:
Error reading Prometheus: bad_response: readObjectStart: expect { or n, but found <, error found in #1 byte of ...|<html> |..., bigger context ...|<html> <head><title>Kafka Exporter</title>|...
Here is how it looks in the UI:
It seems like Grafana ignores the /metrics path and tries to scrape data from http://kafka-exporter:9308 directly, which indeed looks like the error message describes:
By contrast, the /metrics endpoint contains the actual metrics:
> curl localhost:9308/metrics
# HELP go_gc_duration_seconds A summary of the pause duration of garbage collection cycles.
# TYPE go_gc_duration_seconds summary
go_gc_duration_seconds{quantile="0"} 0
go_gc_duration_seconds{quantile="0.25"} 0
go_gc_duration_seconds{quantile="0.5"} 0
go_gc_duration_seconds{quantile="0.75"} 0
go_gc_duration_seconds{quantile="1"} 0
go_gc_duration_seconds_sum 0
go_gc_duration_seconds_count 0
# HELP go_goroutines Number of goroutines that currently exist.
# TYPE go_goroutines gauge
go_goroutines 20
...
Why am I getting this error? Shouldn't Grafana pick up the path?
To show the implementation of #usuario's answer, the docker-compose.yml needs a Prometheus service,
version: '2'
networks:
app-tier:
driver: bridge
services:
zookeeper:
image: 'bitnami/zookeeper:latest'
environment:
- 'ALLOW_ANONYMOUS_LOGIN=yes'
networks:
- app-tier
kafka:
image: 'bitnami/kafka:latest'
environment:
- KAFKA_CFG_ZOOKEEPER_CONNECT=zookeeper:2181
- ALLOW_PLAINTEXT_LISTENER=yes
networks:
- app-tier
kafka-exporter:
build: kafka-exporter
ports:
- "9308:9308"
networks:
- app-tier
entrypoint: ["run.sh"]
cli:
image: 'bitnami/kafka:latest'
environment:
- KAFKA_CFG_ZOOKEEPER_CONNECT=zookeeper:2181
- ALLOW_PLAINTEXT_LISTENER=yes
networks:
- app-tier
prometheus:
image: bitnami/prometheus:latest
ports:
- "9090:9090"
volumes:
- "./prometheus/prometheus.yml:/opt/bitnami/prometheus/conf/prometheus.yml"
networks:
- app-tier
grafana:
image: grafana/grafana
ports:
- "3000:3000"
networks:
- app-tier
where the prometheus/prometheus.yml configuration file is statically configured to scrape the Kafka Exporter:
global:
scrape_interval: 10s
scrape_timeout: 10s
evaluation_interval: 1m
scrape_configs:
- job_name: kafka-exporter
metrics_path: /metrics
honor_labels: false
honor_timestamps: true
sample_limit: 0
static_configs:
- targets: ['kafka-exporter:9308']
Now the Prometheus data source can be added and metrics from the Kafka Exporter such as kafka_consumergroup_lag can be viewed:
You cannot connect Grafana Prometheus datasource to an exporter directly. You need to set up a Prometheus server, scrape from that server the kafka exporter metrics and finally connect Grafana to the Prometheus server.
Related
When I run the graylog web service on localhost, I get it at 127.0.0.1:9000. I installed Docker on a server.
GRAYLOG_HTTP_EXTERNAL_URI
GRAYLOG_HTTP_BIND_ADDRESS
GRAYLOG_HTTP_PUBLISH_URI
I wrote the ip addresses, but graylog works in server local. What do I need to do to send data from Graylog outside?
version: '3'
services:
# MongoDB: https://hub.docker.com/_/mongo/
mongo:
image: mongo:5.0.13
networks:
- graylog
volumes:
- /var/lib/docker/volumes/mongo:/data/db
# Elasticsearch: https://www.elastic.co/guide/en/elasticsearch/reference/7.10/docker.html
elasticsearch:
image: docker.elastic.co/elasticsearch/elasticsearch-oss:7.10.2
environment:
- http.host=0.0.0.0
- transport.host=localhost
- network.host=0.0.0.0
- "ES_JAVA_OPTS=-Dlog4j2.formatMsgNoLookups=true -Xms1024m -Xmx1024m"
ulimits:
memlock:
soft: -1
hard: -1
deploy:
resources:
limits:
memory: 1.5G
volumes:
- /var/lib/docker/volumes/elk:/usr/share/elasticsearch/data
networks:
- graylog
# Graylog: https://hub.docker.com/r/graylog/graylog/
graylog:
image: graylog/graylog:5.0
environment:
- TZ=Europe/Istanbul
# CHANGE ME (must be at least 16 characters)!
- GRAYLOG_PASSWORD_SECRET=908bd4dee1
- Password=Y71
- GRAYLOG_ROOT_PASSWORD_SHA2=8c6976e5b541
- GRAYLOG_HTTP_EXTERNAL_URI=http://10.90.104.143:9000/ #example ip
- GRAYLOG_HTTP_BIND_ADDRESS=10.90.104.143:9000 #example ip
- GRAYLOG_HTTP_PUBLISH_URI=http://10.90.104.143:9000/ #example ip
#- GRAYLOG_TRAGRAYLOG_TRANSPORT_EMAIL_ENABLED: "true"
#- GRAYLOG_TRANSPORT_EMAIL_HOSTNAME: smtp
#- GRAYLOG_TRANSPORT_EMAIL_PORT: 25
#- GRAYLOG_TRANSPORT_EMAIL_USE_AUTH: "false"
#- GRAYLOG_TRANSPORT_EMAIL_USE_TLS: "false"
#- GRAYLOG_TRANSPORT_EMAIL_USE_SSL: "false"
volumes:
- graylog_data:/usr/share/graylog/data
- graylog_journal:/usr/share/graylog/journal
networks:
- graylog
restart: always
depends_on:
- mongo
- elasticsearch
ports:
# Graylog web interface and REST API
- 9000:9000
# Syslog TCP
- 1514:1514
# Syslog UDP
- 1514:1514/udp
# GELF TCP
- 12201:12201
# GELF UDP
- 12201:12201/udp
networks:
graylog:
driver: bridge
volumes:
graylog_data:
graylog_journal:
What settings should I add to send external data to Graylog?
I had the previous version (from last yr) of minio running well behind traefik and on portainer with no issues. I attempted an upgrade 4 days ago to the new version and it's been downhill since then. I get the login page but it won't accept credentials; Error:
{"code":500,"detailedMessage":"Post \"https://storage.example.com/\": dial tcp <ip>:443: i/o timeout","message":"invalid Login"}
I suspect it has something to do with the TLS certificate.
Below is my compose file, I've generated the certs as required; any assistance is is welcomed:
minio:
image: minio/minio
container_name: minio
restart: unless-stopped
command: server /data --certs-dir "./minio-data/certs" --address ":9000" --console-address ":9001"
networks:
- traefik-proxy2
expose:
- "9000"
- "9001"
volumes:
- ./minio-data:/data
environment:
- "MINIO_ROOT_USER=love"
- "MINIO_ROOT_PASSWORD=love1234"
- "MINIO_BROWSER_REDIRECT_URL=https://stash.example.com"
- "MINIO_SERVER_URL=https://storage.example.com"
labels:
- "traefik.enable=true"
- "traefik.http.services.minio.loadbalancer.server.port=9000"
- "traefik.http.routers.minio.rule=Host(`storage.example.com`)"
- "traefik.http.middlewares.minio-https-redirect.redirectscheme.scheme=https"
- "traefik.http.routers.minio.middlewares=minio-https-redirect"
- "traefik.http.routers.minio.entrypoints=https"
- "traefik.http.routers.minio.service=minio"
- "traefik.http.routers.minio.tls=true"
- "traefik.http.routers.minio.tls.certresolver=http"
- "traefik.http.services.minio-console.loadbalancer.server.port=9001"
- "traefik.http.routers.minio-console.rule=Host(`stash.example.com`)"
I was getting the same Invalid Login error: Post "http://minio.localhost/": dial tcp: lookup minio.localhost on 127.0.0.11:53: no such host.
The following docker-compose.yml works. The solution is NOT to use MINIO_DOMAIN or MINIO_SERVER_URL. See # comments at environment: variables.
version: "3.3"
services:
minio:
# Please use fixed versions :D
image: minio/minio:RELEASE.2021-10-06T23-36-31Z
networks:
- traefik-proxy2
volumes:
- minio-data:/data
command:
- server
- /data
- --console-address
- ":9001"
environment:
- MINIO_ROOT_USER=love
- MINIO_ROOT_PASSWORD=love1234
# Do NOT use MINIO_DOMAIN or MINIO_SERVER_URL with Traefik.
# All Routing is done by Traefik, just tell minio where to redirect to.
- MINIO_BROWSER_REDIRECT_URL=http://stash.localhost
deploy:
labels:
- traefik.enable=true
- traefik.docker.network=traefik-proxy2
- traefik.constraint-label=traefik-proxy2
- traefik.http.routers.minio.service=minio
- traefik.http.routers.minio.rule=Host(`storage.localhost`)
- traefik.http.services.minio.loadbalancer.server.port=9000
- traefik.http.routers.minio-console.service=minio-console
- traefik.http.routers.minio-console.rule=Host(`stash.localhost`)
- traefik.http.services.minio-console.loadbalancer.server.port=9001
volumes:
minio-data:
networks:
traefik-proxy2:
external: true
I am new to Prometheus and Docker Compose. I have a project structure with docker-compose.yml file that setup Prometheus and Grafana:
/prometheus-grafana/prometheus/docker-compose.yml
version: '3'
services:
prometheus:
image: prom/prometheus:v2.21.0
ports:
- 9000:9090
volumes:
- ./prometheus:/etc/prometheus
- prometheus-data:/prometheus
command: --web.enable-lifecycle --config.file=/etc/prometheus/prometheus.yml
grafana:
image: grafana/grafana:$GRAFANA_VERSION
environment:
GF_SECURITY_ADMIN_USER: $GRAFANA_ADMIN_USER
GF_SECURITY_ADMIN_PASSWORD: $GRAFANA_ADMIN_PASSWORD
ports:
- 3000:3000
volumes:
- grafana-storage:/var/lib/grafana
depends_on:
- prometheus
networks:
- internal
networks:
internal:
volumes:
prometheus-data:
grafana-storage:
I just added more config in /prometheus-grafana/prometheus/prometheus/prometheus.yml like this, the part that I added is the last 3 lines:
global:
scrape_interval: 30s
scrape_timeout: 10s
rule_files:
- alert.yml
scrape_configs:
- job_name: services
metrics_path: /metrics
static_configs:
- targets:
- 'prometheus:9090'
- 'idonotexists:564'
- job_name: myapp
scrape_interval: 10s
static_configs:
- targets:
- localhost:2112
I started Prometheus by running: docker-compose up -d, I got to Prometheus on http://localhost:9000/graph but I don't see the new config I added to prometheus.yml, in docker-compose.yml file, there is a line:
command: --web.enable-lifecycle --config.file=/etc/prometheus/prometheus.yml
Am I supposed to change this too refers to the path of my another /prometheus-grafana/prometheus/prometheus/prometheus.yml in my project instead of from /etc/prometheus/prometheus.yml as in reality, this file doesn't exist /etc/prometheus/prometheus.yml
Thank you in advance.
If you have prometheus configured with --config.file=/etc/prometheus/prometheus.yml it expects its config file to be at exact this position.
Since you have the the following mount points already configured:
volumes:
- ./prometheus:/etc/prometheus
- prometheus-data:/prometheus
you just need to put your config file into ./prometheus and name it prometheus.yml
I have a beginner question with Docker Compose. I am trying to extend the docker-compose-slim.yml example file from Zipkin GitHub repository.
I need to change it so that it can include a simple FastAPI app that I have written. Unfortunately, I cannot make them connect to each other. FastAPI gets rejected when it attempts to send a POST request to the Zipkin container, even though they are both connected to the same network with explicit links and port mapping defined in the YAML file. However, I am able to connect to both of them from the host, however.
Could you please tell me what I have done wrong?
Here is the error message:
Error emitting zipkin trace. ConnectionError(MaxRetryError("HTTPConnectionPool(host='127.0.0.1', port=9411): Max retries exceeded with url: /api/v2/spans (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fce354711c0>: Failed to es
tablish a new connection: [Errno 111] **Connection refused**'))"))
Here is the Docker Compose YAML file:
version: '2.4'
services:
zipkin:
image: openzipkin/zipkin-slim
container_name: zipkin
environment:
- STORAGE_TYPE=mem
ports:
# Port used for the Zipkin UI and HTTP Api
- 9411:9411
depends_on:
- storage
storage:
image: busybox:1.31.0
container_name: fake_storage
myfastapi:
build: .
ports:
- 8000:8000
links:
- zipkin
depends_on:
- zipkin
dependencies:
image: busybox:1.31.0
container_name: fake_dependencies
networks:
default:
name: foo_network
Here is the Dockerfile:
FROM python:3.8.5
ADD . /app
WORKDIR /app
RUN pip install -r requirements.txt
EXPOSE 8000
CMD ["uvicorn", "wsgi:app", "--host", "0.0.0.0", "--port", "8000"]
You must tell the containers the network "foo_network". The External flag says that the containers are not accessible from outside. Of course you don't have to bet, but I thought as an example it might be quite good.
And because of the "links" function look here Link
version: '2.4'
services:
zipkin:
image: openzipkin/zipkin-slim
container_name: zipkin
environment:
- STORAGE_TYPE=mem
ports:
# Port used for the Zipkin UI and HTTP Api
- 9411:9411
depends_on:
- storage
networks:
- foo_network
storage:
image: busybox:1.31.0
container_name: fake_storage
networks:
- foo_network
myfastapi:
build: .
ports:
- 8000:8000
links:
- zipkin
depends_on:
- zipkin
networks:
- foo_network
dependencies:
image: busybox:1.31.0
container_name: fake_dependencies
networks:
- foo_network
networks:
foo_network:
external: false
My config is a simple Docker Swarm (on Docker for Windows) with a Treafik container and a very simple Spring Boot 'echo' container.
I would like to forward a browser request for 'localhost/traefik' to the Traefik dashboard and 'localhost/echo/something' to a simple Sprint Boot echo application.
Expected to work:
localhost/treafik => gives a "Gateway timeout"
localhost/echo/something => gives a "Gateway timeout"
Expected not to work, but bypass Traefik (incorrectly):
localhost:8080 => shows the Traefik dashboard
localhost:8082/echo/something => { "status":"something" }
How can I improve the docker-compose.yml file to get the expected result? Can this have to do with 'networking'? When creating any network, even an overlay, gave error messages.
version: '3.7'
services:
traefik:
image: traefik:latest
command: --api --docker
restart: always
volumes:
- /var/run/docker.sock:/var/run/docker.sock
- //k/data/slackbot/traefik.toml:/traefik.toml
ports:
- "80:80"
- "8080:8080"
labels:
- "traefik.enable=true"
- "traefik.frontend.rule=Host:localhost;PathPrefixStrip:/traefik"
- "traefik.port=8080"
- "traefik.protocol=http"
- "traefik.backend=traefik"
slackbotsimple:
image: solvedshared/slackbotsimple:latest
ports:
- 8082:8080
volumes:
- /var/run/docker.sock:/var/run/docker.sock
labels:
- "traefik.enable=true"
- "traefik.frontend.rule=Host:localhost;PathPrefix:/echo"
- "traefik.port=8082"
- "traefik.protocol=http"
- "traefik.backend=slackbotsimple
The first issue was a networking issue. Solved, thanks to clever Jim.
The other issue was that I used the wrong port number. To route from the frontend '/echo' to the backend, the 'ports' section is NOT used. I could route directly to the exposed port of the slackbot application!
Take a look at the ports config: only 80 is available.
version: '3.7'
services:
traefik:
image: traefik:latest
command: --api --docker
restart: always
networks:
- slackbotnet
volumes:
- /var/run/docker.sock:/var/run/docker.sock
- //k/data/slackbot/traefik.toml:/traefik.toml
ports:
- "80:80"
labels:
- "traefik.enable=true"
- "traefik.docker.network=slackbotnet"
- "traefik.frontend.rule=Host:localhost;PathPrefixStrip:/traefik"
- "traefik.port=8080"
- "traefik.protocol=http"
- "traefik.backend=traefik"
slackbotsimple:
image: solvedshared/slackbotsimple:latest
networks:
- slackbotnet
volumes:
- /var/run/docker.sock:/var/run/docker.sock
labels:
- "traefik.enable=true"
- "traefik.docker.network=slackbotnet"
- "traefik.frontend.rule=Host:localhost;PathPrefix:/echo"
- "traefik.port=8080"
- "traefik.protocol=http"
- "traefik.backend=slackbotsimple"
networks:
slackbotnet:
name: slackbotnet