Connect celery-flower and prometheus in separate docker-compose files - docker-compose

I want to connect celery-flower and prometheus. I work following this instruction - https://github.com/mher/flower/blob/master/docs/prometheus-integration.rst
I have 2 docker-compose.yml.
app/docker-compose.yml
celery-flower:
...
container_name: celery-flower
command: celery -A configs flower -l INFO --url_prefix=flower --port=5555 --persisten=True
ports:
- "5555:5555"
networks:
- default
- flower
networks:
default:
flower:
external: true
observability/docker-compose.yml
prometheus:
...
volumes:
- $PWD/prometheus.yml:/etc/prometheus/prometheus.yml
command:
- '--config.file=/etc/prometheus/prometheus.yml'
- '--storage.tsdb.path=/prometheus'
- '--web.console.libraries=/etc/prometheus/console_libraries'
- '--web.console.templates=/etc/prometheus/consoles'
- '--web.enable-lifecycle'
expose:
- 9090
ports:
- 9090:9090
networks:
- observability
- flower
networks:
observability:
flower:
external: true
observability/prometheus.yml
scrape_configs:
- job_name: flower
static_configs:
- targets:
- celery-flower:5555
But i don't see celery-flower on the http://localhost:9090/graph
Both containers see the flower network (via docker inspect)
Ping from prometheus container
PING celery-flower (172.21.0.3): 56 data bytes
64 bytes from 172.21.0.3: seq=0 ttl=64 time=0.122 ms
64 bytes from 172.21.0.3: seq=1 ttl=64 time=0.113 ms
64 bytes from 172.21.0.3: seq=2 ttl=64 time=0.116 ms
64 bytes from 172.21.0.3: seq=3 ttl=64 time=0.093 ms
64 bytes from 172.21.0.3: seq=4 ttl=64 time=0.072 ms
64 bytes from 172.21.0.3: seq=5 ttl=64 time=0.076 ms
--- celery-flower ping statistics ---
6 packets transmitted, 6 packets received, 0% packet loss
round-trip min/avg/max = 0.072/0.098/0.122 ms
Connection
telnet celery-flower 5555
Connected to celery-flower

Incorrect command for celery-flower running (need to remove --url_prefix=flower)
app/docker-compose.yml
celery-flower:
restart: always
image: mher/flower:1.2.0
container_name: celery-flower
command: celery flower -l INFO --persisten=True
volumes:
- flower_data:/data
environment:
- CELERY_BROKER_URL=redis://redis:6379
ports:
- 5555:5555
networks:
- default
- flower

Related

Unable to establish connection between Zookeeper and Kafka on Apple M1 using docker-compose

I am trying to run Kafka + Zookeeper through docker-compose.yml
`version: '3'
services:
zookeeper:
image: zookeeper:3.4.9
hostname: zookeeper
ports:
- "2181:2181"
environment:
ZOO_MY_ID: 1
ZOO_PORT: 2181
# ZOO_SERVERS: server.1=zookeeper:2888:3888;2181
volumes:
- ./data/zookeeper/data:/data
- ./data/zookeeper/datalog:/datalog
kafka1:
image: confluentinc/cp-kafka:5.3.0
hostname: kafka1
ports:
- "9091:9091"
environment:
KAFKA_ADVERTISED_LISTENERS: LISTENER_DOCKER_INTERNAL://kafka1:19091,LISTENER_DOCKER_EXTERNAL://${DOCKER_HOST_IP:-127.0.0.1}:9091
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: LISTENER_DOCKER_INTERNAL:PLAINTEXT,LISTENER_DOCKER_EXTERNAL:PLAINTEXT
KAFKA_INTER_BROKER_LISTENER_NAME: LISTENER_DOCKER_INTERNAL
KAFKA_ZOOKEEPER_CONNECT: "zookeeper:2181"
KAFKA_BROKER_ID: 1
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
volumes:
- ./data/kafka1/data:/var/lib/kafka/data
depends_on:
- zookeeper
However, Kafka is unable to connect with the ZooKeeper
kafka101-kafka1-1 | [main-SendThread(zookeeper:2181)] INFO org.apache.zookeeper.ClientCnxn - Socket error occurred: zookeeper/172.19.0.2:2181: Connection refused
kafka101-kafka1-1 | [main-SendThread(zookeeper:2181)] INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server zookeeper/172.19.0.2:2181. Will not attempt to authenticate using SASL (unknown error)
I checked and confirmed that the Zoo keeper is running
ps -ef | grep zookeeper
1 zookeepe 0:19 {java} /usr/bin/qemu-x86_64 /usr/lib/jvm/java-1.8-openjdk/jre/bin/java /usr/lib/jvm/java-1.8-openjdk/jre/bin/java -Dzookeeper.log.dir=. -Dzookeeper.root.logger=INFO,CONSOLE -cp /zookeeper-3.4.9/bin/../build/classes:/zookeeper-3.4.9/bin/../build/lib/*.jar:/zookeeper-3.4.9/bin/../lib/slf4j-log4j12-1.6.1.jar:/zookeeper-3.4.9/bin/../lib/slf4j-api-1.6.1.jar:/zookeeper-3.4.9/bin/../lib/netty-3.10.5.Final.jar:/zookeeper-3.4.9/bin/../lib/log4j-1.2.16.jar:/zookeeper-3.4.9/bin/../lib/jline-0.9.94.jar:/zookeeper-3.4.9/bin/../zookeeper-3.4.9.jar:/zookeeper-3.4.9/bin/../src/java/lib/*.jar:/conf: -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.local.only=false org.apache.zookeeper.server.quorum.QuorumPeerMain /conf/zoo.cfg
The zoo configuration looks like this:
clientPort=2181
dataDir=/data
dataLogDir=/datalog
tickTime=2000
initLimit=5
syncLimit=2
Telnet output directly from zookeeper is :
zookeeper-3.4.9 # telnet localhost 2181
telnet: can't connect to remote host (127.0.0.1): Connection refused
ping to zookeeper host in kafka1
> ping zookeeper
PING zookeeper (172.19.0.2) 56(84) bytes of data.
PING zookeeper (172.19.0.2) 56(84) bytes of data.
PING zookeeper (172.19.0.2) 56(84) bytes of data.
I am running on Apple M1 Chipset
What else can I check and do here?
Try using confluentinc's zookeeper image: confluentinc/cp-zookeeper:latest
I was in the exact same boat, also on the M1 chip and this worked for me.
These are the two services from my docker-compose for ref:
zoo:
image: confluentinc/cp-zookeeper:latest
restart: unless-stopped
ports:
- 2181:2181
environment:
ZOO_MY_ID: 1
ZOO_PORT: 2181
ZOO_SERVERS: server.1=zoo:2888:3888
ZOOKEEPER_CLIENT_PORT: 2181
kafka:
image: confluentinc/cp-kafka:latest
restart: always
ports:
- 9092:9092
environment:
KAFKA_ADVERTISED_LISTENERS: LISTENER_DOCKER_INTERNAL://kafka:19092,LISTENER_DOCKER_EXTERNAL://${DOCKER_HOST_IP:-127.0.0.1}:9092
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: LISTENER_DOCKER_INTERNAL:PLAINTEXT,LISTENER_DOCKER_EXTERNAL:PLAINTEXT
KAFKA_INTER_BROKER_LISTENER_NAME: LISTENER_DOCKER_INTERNAL
KAFKA_ZOOKEEPER_CONNECT: 'zoo:2181'
KAFKA_BROKER_ID: 1
KAFKA_LOG4J_LOGGERS: 'kafka.controller=INFO,kafka.producer.async.DefaultEventHandler=INFO,state.change.logger=INFO'
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
depends_on:
- zoo

Docker compose GrayLog

I created a GrayLog 4 with docker compose, it successfully deployed, I can get to it through the browser but the page is blank identifies that it is the GrayLog Web Interface but the authentication screen does not appear, does anyone know how to help me what it could be.
version: '3'
services:
mongo:
image: mongo:4.2
# Elasticsearch: https://www.elastic.co/guide/en/elasticsearch/reference/7.10/docker.html
elasticsearch:
image: docker.elastic.co/elasticsearch/elasticsearch-oss:7.10.2
environment:
- http.host=0.0.0.0
- transport.host=localhost
- network.host=0.0.0.0
- "ES_JAVA_OPTS=-Xms512m -Xmx512m"
ulimits:
memlock:
soft: -1
hard: -1
deploy:
resources:
limits:
memory: 1g
# Graylog: https://hub.docker.com/r/graylog/graylog/
graylog:
image: graylog/graylog:4.0
environment:
# CHANGE ME (must be at least 16 characters)!
- GRAYLOG_PASSWORD_SECRET=somepasswordpepper
# Password: admin
- GRAYLOG_ROOT_PASSWORD_SHA2=8c6976e5b5410415bde908bd4dee15dfb167a9c873fc4bb8a81f6f2ab448a918
- GRAYLOG_HTTP_EXTERNAL_URI=http://127.0.0.1:9000/
entrypoint: /usr/bin/tini -- wait-for-it elasticsearch:9200 -- /docker-entrypoint.sh
restart: always
depends_on:
- mongo
- elasticsearch
ports:
# Graylog web interface and REST API
- 9000:9000
# Syslog TCP
- 1514:1514
# Syslog UDP
- 1514:1514/udp
# GELF TCP
- 12201:12201
# GELF UDP
- 12201:12201/udp
enter image description here
In your screenshot the IP address ends in a 6, but Graylog is bouund to 127.0.0.1. Set http_bind to 127.0.0.1 and http_publish or http_external to the interface IP that ends in 6.
ref: Graylog docs

graylog mongdb configuration with user password on docker-compose

I'a trying to set a mongodb user password whend makeing a docker-compose file for graylog.
But i can't set up the user and the password correctly.
here is my docker-compose.yml :
version: '2'
services:
# MongoDB: https://hub.docker.com/_/mongo/
mongodb:
image: mongo:4.2
volumes:
- mongo_data:/data/db
environment:
- MONGO_INITDB_ROOT_USERNAME=user
- MONGO_INITDB_ROOT_PASSWORD=password
- MONGO_INITDB_DATABASE=graylog
ports:
# mongodb
- 27017:27017
# Elasticsearch: https://www.elastic.co/guide/en/elasticsearch/reference/7.10/docker.html
elasticsearch:
image: docker.elastic.co/elasticsearch/elasticsearch-oss:7.10.2
volumes:
- es_data:/usr/share/elasticsearch/data
environment:
- http.host=0.0.0.0
- transport.host=localhost
- network.host=0.0.0.0
- "ES_JAVA_OPTS=-Xms512m -Xmx512m"
ulimits:
memlock:
soft: -1
hard: -1
mem_limit: 1g
# Graylog: https://hub.docker.com/r/graylog/graylog/
graylog:
image: graylog/graylog:4.1
volumes:
- graylog_data:/usr/share/graylog/data
environment:
# CHANGE ME (must be at least 16 characters)!
- GRAYLOG_PASSWORD_SECRET=somepasswordpepper
# Password: admin
- GRAYLOG_ROOT_PASSWORD_SHA2=8c6976e5b5410415bde908bd4dee15dfb167a9c873fc4bb8a81f6f2ab448a918
- GRAYLOG_HTTP_EXTERNAL_URI=http://127.0.0.1:9000/
- MONGODB_URI=mongodb://user:password#localhost:27017/graylog
- GRAYLOG_MONGODB_URI=mongodb://user:password#localhost:27017/graylog
- MONGO_INITDB_ROOT_USERNAME=user
- MONGO_INITDB_ROOT_PASSWORD=password
- GRAYLOG_MONGO_INITDB_ROOT_USERNAME=user
- GRAYLOG_MONGO_INITDB_ROOT_PASSWORD=password
entrypoint: /usr/bin/tini -- wait-for-it elasticsearch:9200 -- /docker-entrypoint.sh
links:
- mongodb:mongo
- elasticsearch
restart: always
depends_on:
- mongodb
- elasticsearch
ports:
# Graylog web interface and REST API
- 9000:9000
# Syslog TCP
- 1514:1514
# Syslog UDP
- 1514:1514/udp
# GELF TCP
- 12201:12201
# GELF UDP
- 12201:12201/udp
# Volumes for persisting data, see https://docs.docker.com/engine/admin/volumes/volumes/
volumes:
mongo_data:
driver: local
es_data:
driver: local
graylog_data:
driver: local
but i still have a :
graylog_1 | 2021-06-30 07:00:32,022 INFO :
org.mongodb.driver.cluster - Cluster description not yet available.
Waiting for 30000 ms before timing out graylog_1 | 2021-06-30
07:00:32,022 INFO : org.mongodb.driver.cluster - Exception in monitor
thread while connecting to server localhost:27017 graylog_1 |
com.mongodb.MongoSocketOpenException: Exception opening socket
graylog_1 | at
com.mongodb.internal.connection.SocketStream.open(SocketStream.java:70)
~[graylog.jar:?] graylog_1 | at
com.mongodb.internal.connection.InternalStreamConnection.open(InternalStreamConnection.java:128)
~[graylog.jar:?] graylog_1 | at
com.mongodb.internal.connection.DefaultServerMonitor$ServerMonitorRunnable.run(DefaultServerMonitor.java:117)
[graylog.jar:?] graylog_1 | at
java.lang.Thread.run(Thread.java:748) [?:1.8.0_292] graylog_1 |
Caused by: java.net.ConnectException: Connection refused (Connection
refused)
So any idea how to specify the user / password in the docker compose file ?
The error message says that the connection is refused (=the port is closed). This is because you have used localhost as the database address here:
- GRAYLOG_MONGODB_URI=mongodb://user:password#localhost:27017/graylog
Change localhost for mongodb:
- GRAYLOG_MONGODB_URI=mongodb://user:password#mongodb:27017/graylog
Under normal circumstances localhost inside a container refers to the container itself. Since graylog container does not have a mongodb instance inside it, it tells you that there is nothing listening on localhost:27017 in graylog container.
After this change graylog may tell you that it cannot connect due to bad credentials. This is because user created by MONGO_INITDB_ROOT_USERNAME is added to the admin database (not 'graylog'). I guess you need to create a user manually if you haven't already.
Also, these:
- MONGODB_URI=mongodb://user:password#localhost:27017/graylog
- MONGO_INITDB_ROOT_USERNAME=user
- MONGO_INITDB_ROOT_PASSWORD=password
- GRAYLOG_MONGO_INITDB_ROOT_USERNAME=user
- GRAYLOG_MONGO_INITDB_ROOT_PASSWORD=password
have no effect on graylog.

MongoDB Golang driver trying to connect to localhost instead of docker host

ERROR
app | 2020/07/14 13:19:00 server selection error: server selection timeout, current topology: { Type: Unknown, Servers: [{ Addr: localhost:27017, Type: Unknown, State: Connected, Average RTT: 0, Last error: connection() : dial tcp 127.0.0.1:27017: connect: connection refused }, ] }
I am using docker-compose to manage golang app and mongo db (without replica set) instances. docker-compose file contents:
version: '3.1'
services:
mongodb:
image: mongo
container_name: mongodb
restart: always
ports:
- 27017:27017
environment:
MONGO_INITDB_ROOT_USERNAME: root
MONGO_INITDB_ROOT_PASSWORD: local
command: mongod --bind_ip mongodb
app:
image: app:1.4
container_name: app
environment:
- MONGO_URL=mongodb
ports:
- 80:8080
depends_on:
- mongodb
restart: always
Running Docker Containers:
docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
d130336be71e app:1.4 "/app/main" 4 minutes ago Up 2 seconds 0.0.0.0:80->8080/tcp app
dc57dd2de645 mongo "docker-entrypoint.s…" 4 minutes ago Up 2 seconds 0.0.0.0:27017->27017/tcp mongodb
For Connecting Golang app with MongoDB
I am using official mongodb go driver (go.mongodb.org/mongo-driver/mongo) and my code is trying to connect to docker host (mongodb):
var dbUser string = "root"
var dbPassword string = "local"
dbURL := os.Getenv("MONGO_URL")
clientOptions := options.Client()
clientOptions.SetDirect(true).ApplyURI("mongodb://" + dbUser + ":" + dbPassword + "#" + dbURL +":27017/?connect=direct")
Troubleshooting
As part of trying to figure out what is wrong I looked at similar issues highlighted in stackoverflow and tried below options:
Tried to use connect=direct in connection URI
Tried to use SetDirect
Added --bindip as highlighted in my docker-compose file
I logged-in into the app container and tried to ping mongodb and it worked:
root#d130336be71e:/app# ping mongodb
PING mongodb (172.18.0.2) 56(84) bytes of data.
64 bytes from mongodb.app_default (172.18.0.2): icmp_seq=1 ttl=64 time=0.159 ms
64 bytes from mongodb.app_default (172.18.0.2): icmp_seq=2 ttl=64 time=0.200 ms
64 bytes from mongodb.app_default (172.18.0.2): icmp_seq=3 ttl=64 time=0.079 ms
64 bytes from mongodb.app_default (172.18.0.2): icmp_seq=4 ttl=64 time=0.087 ms
64 bytes from mongodb.app_default (172.18.0.2): icmp_seq=5 ttl=64 time=0.083 ms
^C
--- mongodb ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 60ms
rtt min/avg/max/mdev = 0.079/0.121/0.200/0.050 ms
After trying these options, I am seeking help from Golang community as I am fairly new to Golang.
Thanks in Advance.

How to pass container ip as ENV to other container in docker-compose file

this is my docker-compose file:
version: '3.0'
services:
app-web:
restart: always
build: ./web
environment:
PG_HOST: $(APP_DB_IP)
PG_PORT: 5432
ports:
- "8081:8080"
links:
- app-db
app-db:
build: ./db
expose:
- "5432"
volumes:
- /var/lib/postgresql/data
I want to pass to app-web the ip of app-db (Postgresql in this case) as ENV var so it can connect to the DB smoothly... any idea on how to achieve it?
You actually don't need to do any of this, since you're already using the links feature in Docker Compose. Just get rid of the PG_HOST variable and use the app-db hostname:
services:
app-web:
restart: always
build: ./web
environment:
PG_PORT: 5432
ports:
- "8081:8080"
links:
- app-db
Since you included the app-db entry under links, you can simply use app-db as a hostname in your app-web container. Docker will set up a hostname mapping in the app-web container that resolves the app-db hostname to the database container's IP address.
You can verify that by running the following, which will try to ping the app-db container from the app-web container:
docker-compose exec app-web bash -c "ping app-db"
This should show output from the ping command showing the resolved IP address of the app-db container, for example like this:
PING app-db (172.19.0.2): 56 data bytes
64 bytes from 172.19.0.2: icmp_seq=0 ttl=64 time=0.055 ms
64 bytes from 172.19.0.2: icmp_seq=1 ttl=64 time=0.080 ms
64 bytes from 172.19.0.2: icmp_seq=2 ttl=64 time=0.098 ms
Press ctrl+c to stop the ping command.
Like shown in the other answer, if you still want to pass in the hostname (which is probably a good idea, just in case you ever want to point to a different database), you can just use app-db as a value:
services:
app-web:
restart: always
build: ./web
environment:
PG_HOST: app-db
PG_PORT: 5432
ports:
- "8081:8080"
links:
- app-db
You can use app-db as name instead of ip, docker will automatically determine what the right ip. As stated in the Docker docs: A container can always discover other containers on the same stack using just the container name as hostname.
So in your example you can use:
environment:
PG_HOST: app-db
Source:https://docs.docker.com/docker-cloud/apps/service-links/#discovering-containers-on-the-same-service-or-stack