I am trying to setup docker compose with 3 services: 1 rabbitmq service and 2 java applications(1 producer and 1 consumer). When rabbitmq service starts, it has no queues by default. Only when produces send message to queue it actually creates it. In the same time, consumer requires queue which it has listen to. So here is the problem: when i run my docker compose, rabbitmq starts fine with my producer, but consumer can't find queue and fails. So, can i somehow set-up docker compose rabbitmq service to start and create queue by default?
version: '3.3'
services:
rabbitmq:
image: rabbitmq:management
container_name: rabbitmq
restart: always
environment:
RABBITMQ_DEFAULT_USER: help
RABBITMQ_DEFAULT_PASS: 51243
ports:
- "5672:5672"
- "15672:15672"
#Update
I found pretty good way to achieve what i want. I can start rabbitmq, create all what i need(queue) and export definitions.json. Then i need put it under /etc/rabbitmq/definitions.json
version: '3.3'
services:
rabbitmq:
image: rabbitmq:management
container_name: rabbitmq
restart: always
environment:
RABBITMQ_DEFAULT_USER: ete
RABBITMQ_DEFAULT_PASS: 1402
ports:
- "5672:5672"
- "15672:15672"
volumes:
- ./definitions.json:/etc/rabbitmq/definitions.json
But here is another problem: this definition file overrides my user that i set by environment variables.
Here is definition.json:
{
"rabbit_version": "3.8.9",
"rabbitmq_version": "3.8.9",
"product_name": "RabbitMQ",
"product_version": "3.8.9",
"users": [
{
"name": "eternal",
"password_hash": "wz1jzbGjNMZ115U7XhEUvF271uImnOfho2jpx2pOvLSY/Ssl",
"hashing_algorithm": "rabbit_password_hashing_sha256",
"tags": "administrator"
}
],
"vhosts": [
{
"name": "/"
}
],
"permissions": [
{
"user": "eternal",
"vhost": "/",
"configure": ".*",
"write": ".*",
"read": ".*"
}
],
"topic_permissions": [],
"parameters": [],
"global_parameters": [
{
"name": "cluster_name",
"value": "rabbit#58cb492cd4ee"
},
{
"name": "internal_cluster_id",
"value": "rabbitmq-cluster-id-DlZ_FZVpiFx93CVZXneG4A"
}
],
"policies": [],
"queues": [
{
"name": "qqq",
"vhost": "/",
"durable": false,
"auto_delete": false,
"arguments": {}
}
],
"exchanges": [],
"bindings": []
}
If i will remove users section and everything else and keep only queue section, then after docker-compose up, i can't log in into managment page. Because my login and password is not accepted.
Related
I have the following docker-compose file:
services:
pgdatabase:
image: postgres:13
environment:
- POSTGRES_USER=root
- POSTGRES_PASSWORD=root
- POSTGRES_DB=ny_taxi
volumes:
- "./data:/var/lib/postgresql/data:rw"
ports:
- "5432:5432"
pgadmin:
image: dpage/pgadmin4
environment:
- PGADMIN_DEFAULT_EMAIL=admin#admin.com
- PGADMIN_DEFAULT_PASSWORD=root
volumes:
- "./data_pgadmin:/var/lib/pgadmin"
ports:
- "8080:80"
I'm trying to connect to postgres using pgadmin but I'm getting the following error:
Unable to connect to server: could not translate host name "pgdatabase" to address: Name does not resolve
Running docker network ls I get:
NAME DRIVER SCOPE
bridge bridge local
docker-sql-pg_default bridge local
host host local
none null local
Then running docker network inspect docker-sql-pg_default I get
[
{
"Name": "docker-sql-pg_default",
"Id": "bfee2f08620b5ffc1f8e10d8bed65c4d03a98a470deb8b987c4e52a9de27c3db",
"Created": "2023-01-24T17:57:27.831702189Z",
"Scope": "local",
"Driver": "bridge",
"EnableIPv6": false,
"IPAM": {
"Driver": "default",
"Options": null,
"Config": [
{
"Subnet": "172.24.0.0/16",
"Gateway": "172.24.0.1"
}
]
},
"Internal": false,
"Attachable": false,
"Ingress": false,
"ConfigFrom": {
"Network": ""
},
"ConfigOnly": false,
"Containers": {
"8f53be84a95c9c0591df6cc6edb72d4ca070243c3c067ab2fb14c2094b23bcee": {
"Name": "docker-sql-pg-pgdatabase-1",
"EndpointID": "7f3ddb29b000bc4cfda9c54a4f13e0aa30f1e3f8e5cc1a8ba91cee840c16cd60",
"MacAddress": "02:42:ac:18:00:02",
"IPv4Address": "172.24.0.2/16",
"IPv6Address": ""
},
"bf2eb29b73fe9e49f4bef668a1f70ac2c7e9196b13350f42c28337a47fcd71f4": {
"Name": "docker-sql-pg-pgadmin-1",
"EndpointID": "b3a9504d75e11aa0f08f6a2b5c9c2f660438e23f0d1dd5d7cf4023a5316961d2",
"MacAddress": "02:42:ac:18:00:03",
"IPv4Address": "172.24.0.3/16",
"IPv6Address": ""
}
},
"Options": {},
"Labels": {
"com.docker.compose.network": "default",
"com.docker.compose.project": "docker-sql-pg",
"com.docker.compose.version": "2.13.0"
}
}
]
I tried to connect to the gateway IP 172.24.0.1 and the IP of postgres base 172.24.0.2 but I got timeout error. Why my network isn't running?
Basically I solved my problem using the following steps:
First I accessed the pg-admin container and I used ping to verify if the pgadmin could reach the postgres.
Since pgadmin was reaching postgres I used sudo netstat -tulpn | grep LISTEN to verify in my host machine the ports that are in use. surprisingly I have two instances of pgadmin running on 8080 (one bugged).
I used docker-compose down to stop the servers and used docker system prune to delete all images/containers...
I verified the used ports again and one pgadmin still running on 8080.
I used pidof to check the PID of running (bugged) pgadmin.
Then I used kill -9 to kill the proccess.
Last, I used docker-compose up -d and I was able to communicate pgadmin with postgres via pgadmin interface.
I have a mongodb container which i name e-learning
and i have a docker image which should connect to the mongodb container to update my database but it's not working i get this error:
Unknown, Last error: connection() error occurred during connection handshake: dial tcp 127.0.0.1:27017: connect: connection refused }
here's my docker build file
# syntax=docker/dockerfile:1
FROM golang:1.18
WORKDIR /go/src/github.com/worker
COPY go.mod go.sum main.go ./
RUN go mod download
RUN CGO_ENABLED=0 GOOS=linux go build -a -installsuffix cgo -o app .
FROM jrottenberg/ffmpeg:4-alpine
FROM alpine:latest
RUN apk --no-cache add ca-certificates
WORKDIR /root/
ENV LD_LIBRARY_PATH=/usr/local/lib
COPY --from=jrottenberg/ffmpeg / /
COPY app.env /root
COPY --from=0 /go/src/github.com/worker/app .
CMD ["./app"]
my docker compose file
version: "3.9"
services:
worker:
image: worker
environment:
- MONGO_URI="mongodb://localhost:27017/"
- MONGO_DATABASE=e-learning
- RABBITMQ_URI=amqp://user:password#rabbitmq:5672/
- RABBITMQ_QUEUE=upload
networks:
- app_network
external_links:
- e-learning
- rabbitmq
volumes:
- worker:/go/src/github.com/worker:rw
networks:
app_network:
external: true
volumes:
worker:
my docker inspect network
[
{
"Name": "app_network",
"Id": "f688edf02a194fd3b8a2a66076f834a23fa26cead20e163cde71ef32fc1ab598",
"Created": "2022-06-27T12:18:00.283531947+03:00",
"Scope": "local",
"Driver": "bridge",
"EnableIPv6": false,
"IPAM": {
"Driver": "default",
"Options": {},
"Config": [
{
"Subnet": "172.20.0.0/16",
"Gateway": "172.20.0.1"
}
]
},
"Internal": false,
"Attachable": false,
"Ingress": false,
"ConfigFrom": {
"Network": ""
},
"ConfigOnly": false,
"Containers": {
"2907482267e1f6e42544e5e8d852c0aac109ec523c6461e003572963e299e9b0": {
"Name": "rabbitmq",
"EndpointID": "4b46e091e4d5a79782185dce12cb2b3d79131b92d2179ea294a639fe82a1e79a",
"MacAddress": "02:42:ac:14:00:03",
"IPv4Address": "172.20.0.3/16",
"IPv6Address": ""
},
"8afd004a981715b8088af53658812215357e156ede03905fe8fdbe4170e8b13f": {
"Name": "e-learning",
"EndpointID": "1c12d592a0ef6866d92e9989f2e5bc3d143602fc1e7ad3d980efffcb87b7e076",
"MacAddress": "02:42:ac:14:00:02",
"IPv4Address": "172.20.0.2/16",
"IPv6Address": ""
},
"ad026f7e10c9c1c41071929239363031ff72ad1b9c6765ef5c977da76f24ea31": {
"Name": "video-transformation-worker-1",
"EndpointID": "ce3547014a6856725b6e815181a2c3383d307ae7cf7132e125c58423f335b73f",
"MacAddress": "02:42:ac:14:00:04",
"IPv4Address": "172.20.0.4/16",
"IPv6Address": ""
}
},
"Options": {},
"Labels": {}
}
]
Change MONGO_URI="mongodb://localhost:27017/" to MONGO_URI="mongodb://e-learning:27017/" (working on the assumption that e-learning is the mongo container).
Within a container attached to a bridge network (the default) localhost (127.0.0.1) is the container itself. So your app container is trying to access the database at port 27017 on itself (not on the host or on the db container). The easiest solution is to use the automatic DNS resolution between containers that docker provides.
I added extra hosts and changed my mongo uri to host.docker.internal
and it solved my problems
version: "3.9"
services:
worker:
image: worker
environment:
- MONGO_URI="mongodb://host.docker.internal:27017/"
- MONGO_DATABASE=e-learning
- RABBITMQ_URI=amqp://user:password#rabbitmq:5672/
- RABBITMQ_QUEUE=upload
networks:
- app_network
external_links:
- e-learning
- rabbitmq
volumes:
- worker:/go/src/github.com/worker:rw
extra_hosts:
- "host.docker.internal:host-gateway"
networks:
app_network:
external: true
volumes:
worker:
#Docker compose file
version: "3.4" # optional since v1.27.0
services:
flaskblog:
build:
context: flaskblog
dockerfile: Dockerfile
image: flaskblog
restart: unless-stopped
environment:
APP_ENV: "prod"
APP_DEBUG: "False"
APP_PORT: 5000
MONGODB_DATABASE: flaskdb
MONGODB_USERNAME: flaskuser
MONGODB_PASSWORD:
MONGODB_HOSTNAME: mongodbuser
volumes:
- appdata:/var/www
depends_on:
- mongodb
networks:
- frontend
- backend
mongodb:
image: mongo:4.2
#container_name: mongodb
restart: unless-stopped
command: mongod --auth
environment:
MONGO_INITDB_ROOT_USERNAME: admin
MONGO_INITDB_ROOT_PASSWORD:
MONGO_INITDB_DATABASE: flaskdb
MONGODB_DATA_DIR: /data/db2
MONDODB_LOG_DIR: /dev/null
volumes:
- mongodbdata:/data/db2
#- ./mongo-init.js:/docker-entrypoint-initdb.d/mongo-init.js:ro
networks:
- backend
networks:
backend:
driver: bridge
volumes:
mongodbdata:
driver: local
appdata:
driver: local
#Flask container file
FROM python:3.8-slim-buster
#python:3.6-stretch
LABEL MAINTAINER="Shekhar Banerjee"
ENV GROUP_ID=1000 \
USER_ID=1000 \
SECRET_KEY="4" \
EMAIL_USER="dankml.com" \
EMAIL_PASS="nzw"
RUN mkdir /home/logix3
WORKDIR /home/logix3
ADD . /home/logix3/
RUN pip install -r requirements.txt
RUN pip install gunicorn
RUN groupadd --gid $GROUP_ID www
RUN useradd --create-home -u $USER_ID --shell /bin/sh --gid www www
USER www
EXPOSE 5000/tcp
#CMD ["python","run.py"]
CMD [ "gunicorn", "-w", "4", "--bind", "127.0.0.1:5000", "run:app"]
These are my docker-compose file and Dockerfile for an application . The container for MongoDB and Flask are working fine individually, but when I try to run them together, I do not get any response on the localhost, There are no error messages in the logs or the containers
Curl in the system gives this :
$ curl -i http://127.0.0.1:5000
curl: (7) Failed to connect to 127.0.0.1 port 5000: Connection refused
Can anyone suggest how do I debug this ??
**Only backend network is functional
[
{
"Name": "mlspace_backend",
"Id": "e7e37cad0058330c99c55ffea2b98281e0c6526763e34550db24431b30030b77",
"Created": "2022-05-15T22:52:25.0281001+05:30",
"Scope": "local",
"Driver": "bridge",
"EnableIPv6": false,
"IPAM": {
"Driver": "default",
"Options": null,
"Config": [
{
"Subnet": "172.20.0.0/16",
"Gateway": "172.20.0.1"
}
]
},
"Internal": false,
"Attachable": true,
"Ingress": false,
"ConfigFrom": {
"Network": ""
},
"ConfigOnly": false,
"Containers": {
"12c50f5f70d18b4c7ffc076177b59ff063a8ff81c4926c8cae0bf9e74dc2fc83": {
"Name": "mlspace_mongodb_1",
"EndpointID": "8278f672d9211aec9b539e14ae4eeea5e8f7aaef95448f44aab7ec1a8c61bb0b",
"MacAddress": "02:42:ac:14:00:02",
"IPv4Address": "172.20.0.2/16",
"IPv6Address": ""
},
"75cde7a34d6b3c4517c536a06349579c6c39090a93a017b2c280d987701ed0cf": {
"Name": "mlspace_flaskblog_1",
"EndpointID": "20489de8841d937f01768170a89f74c37ed049d241b096c8de8424c51e58704c",
"MacAddress": "02:42:ac:14:00:03",
"IPv4Address": "172.20.0.3/16",
"IPv6Address": ""
}
},
"Options": {},
"Labels": {
"com.docker.compose.network": "backend",
"com.docker.compose.project": "mlspace",
"com.docker.compose.version": "1.23.2"
}
}
]
I checked the logs of the containers. no errors found, flask engine seem to be running but the site wont run , even curl won't given any output
this is my docker-compose-single-broker.yml file.
version: '2'
services:
zookeeper:
image: wurstmeister/zookeeper
ports:
- "2181:2181"
networks:
my-network:
ipv4_address: 172.18.0.100
kafka:
image: wurstmeister/kafka
ports:
- "9092:9092"
environment:
KAFKA_ADVERTISED_HOST_NAME: 172.18.0.101
KAFKA_CREATE_TOPICS: "test:1:1"
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
volumes:
- /var/run/docker.sock:/var/run/docker.sock
depends_on:
- zookeeper
networks:
my-network:
ipv4_address: 172.18.0.101
networks:
my-network:
name: ecommerce-network # 172.18.0.1 ~
and I executed the command.
docker-compose -f docker-compose-single-broker.yml up -d
I check my network by the command.
docker network inspect ecommerce-network
[
{
"Name": "ecommerce-network",
"Id": "f543bd92e299455454bd1affa993d1a4b7ca2c347d576b24d8f559d0ac7f07c2",
"Created": "2021-05-23T12:42:01.804785417Z",
"Scope": "local",
"Driver": "bridge",
"EnableIPv6": false,
"IPAM": {
"Driver": "default",
"Options": {},
"Config": [
{
"Subnet": "172.18.0.0/16",
"Gateway": "172.18.0.1"
}
]
},
"Internal": false,
"Attachable": false,
"Ingress": false,
"ConfigFrom": {
"Network": ""
},
"ConfigOnly": false,
"Containers": {
"cad97d79a92ea8e0f24b000c8255c2db1ebc64865fab3d7cda37ff52a8755f14": {
"Name": "kafka-docker_kafka_1",
"EndpointID": "4c867d9d5f4d28e608f34247b102f1ff2811a9bbb2f78d30b2f55621e6ac6187",
"MacAddress": "02:42:ac:12:00:65",
"IPv4Address": "172.18.0.101/16",
"IPv6Address": ""
},
"f7df5354b9e114a1a849ea9d558d8543ca5cb02800c5189d9f09ee1b95a517d6": {
"Name": "kafka-docker_zookeeper_1",
"EndpointID": "b304581db258dd3da95e15fb658cae0e40bd38440c1f845b09936d9b69d4fb23",
"MacAddress": "02:42:ac:12:00:64",
version: '2'
"IPv4Address": "172.18.0.100/16",
"IPv6Address": ""
}
},
"Options": {},
"Labels": {}
}
]
and I entered kafka container. I executed the command to look up the topic list.
however, I couldn't get the topic list even though I waited indefinitely.
this is my kafka container's logs.
What should I do to solve this problem?
Unclear why you think you'll need IP addresses. Remove those
For example, KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181 works out of the box with Docker networking
The KAFKA_ADVERTISED_HOST_NAME can simply be localhost if you don't plan on connecting outside of that container, otherwise, it can be the Docker service name you've set of kafka
And you don't need to mount the Docker socket
Related - Connect to Kafka running in Docker
For local development purposes, you could check out my containerized Kafka that uses only single image for both Zookeeper and Kafka if you interested in it
You could find it via
https://github.com/howiesynguyen/Java-basedExamples/tree/main/DockerContainerizedKafka4Dev
Or
https://hub.docker.com/repository/docker/howiesynguyen/kafka4dev
I tested it with one of my examples of Spring Cloud Stream and it worked for me. Hopefully someone can find it helpful
Just in case it would be useful to anybody.
I had pretty similar setup as question's author did and in my case Kafka didn't see Zookeeper (but I was using different image - from Debezium). In my case I figured out that environment variable KAFKA_ZOOKEEPER_CONNECT should be named just ZOOKEEPER_CONNECT.
Solution that worked to me (network can be omitted, it's not necessary):
version: "3.9"
services:
zookeeper:
image: debezium/zookeeper
ports:
- 2181:2181
- 2888:2888
- 3888:3888
networks:
- main
kafka:
image: debezium/kafka
ports:
- 9092:9092
environment:
ZOOKEEPER_CONNECT: zookeeper
depends_on:
- zookeeper
networks:
- main
networks:
main:
I'm using a sqlite3 database file with my ghost blog with docker-compose. However when I start it the config file correctly copies over my database and theme to /content but it doesn't load in the browser. Instead it's loading the original database and theme from /versions/3.32.1/content.
I can get it working correctly when I cp the db file and theme folder to /versions/3.32.1/content and change the permissions of the db file but I'd like this to happen automatically. How can I adjust the config file or docker-compose to do this?
Here is my config.development.json:
{
"url": "http://ghost:2368",
"server": {
"port": 2368,
"host": "0.0.0.0"
},
"database": {
"client": "sqlite3",
"connection": {
"filename": "content/data/threadlet.db"
},
"debug": false
},
"paths": {
"contentPath": "content/"
},
"privacy": {
"useRpcPing": false,
"useUpdateCheck": true
},
"useMinFiles": false,
"caching": {
"theme": {
"maxAge": 0
},
"admin": {
"maxAge": 0
}
}
}
and my ghost setup in docker-compose:
ghost:
image: ghost:latest
container_name: ghost
restart: always
ports:
- 2368:2368
env_file:
- ".env"
environment:
# DATABASE_URL:
url: "http://ghost:2368"
NODE_ENV: development
volumes:
- ./ghost/config.${NODE_ENV}.json:/var/lib/ghost/config.${NODE_ENV}.json
- ./ghost/content:/var/lib/ghost/content