Using Podman with docker-compose, how to get multiple replicas of a service - docker-compose

Question

EDIT: weird DNS behavior was some kind of transient issue, and now RHEL/podman works the same way as Ubuntu/podman. I can't reproduce the issue, which makes most part (not 100% though) of this question moot.
I am trying to use Podman and docker-compose to create a compose stack with multiple replicas of backend container, and having a hard time with it.
I use Podman because I have to (comes from Red Hat platform), and picked docker-compose because it is familiar and I use in local dev host, too. I know there are alternatives (podman-compose etc). I learned that Podman 4.1 supported Docker Compose so this sounded like a good candidate.
As an example I have docker-compose.yml with one frontend container and 3 backend containers:
version: "3"
services:
frontend:
image: "nginx:latest"
ports:
- "3000:80"
depends_on:
- backend
backend:
image: "tomcat:latest"
ports:
- "8180-8280:8080"
scale: 3
Note: this stack is just an example. It's purpose is only to highlight the networking aspects of multi-replica docker-compose . I could use something else than nginx:latest, and using e.g. Traefik can solve some of the problems...but sometimes you wish to connect directly from one container to a service with multiple container replicas.
Docker & docker-compose (Ubuntu)
Running this on host which has docker and docker-compose is straightforward.
Backend containers get assigned random ports from range 8180-8280.
$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
98941117708c nginx:latest "/docker-entrypoint.…" 25 seconds ago Up 22 seconds 0.0.0.0:3000->80/tcp, :::3000->80/tcp example-docker-compose-frontend-1
3d749e25eaba tomcat:latest "catalina.sh run" 26 seconds ago Up 23 seconds 0.0.0.0:8193->8080/tcp, :::8193->8080/tcp example-docker-compose-backend-1
854ba8f60cb3 tomcat:latest "catalina.sh run" 26 seconds ago Up 23 seconds 0.0.0.0:8192->8080/tcp, :::8192->8080/tcp example-docker-compose-backend-2
e57e32181e8e tomcat:latest "catalina.sh run" 26 seconds ago Up 23 seconds 0.0.0.0:8194->8080/tcp, :::8194->8080/tcp example-docker-compose-backend-3
Logging into frontend container, service name backend resolves to all 3 backend containers
dig backend
; <<>> DiG 9.16.27-Debian <<>> backend
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 31602
;; flags: qr rd ra; QUERY: 1, ANSWER: 3, AUTHORITY: 0, ADDITIONAL: 0
;; QUESTION SECTION:
;backend. IN A
;; ANSWER SECTION:
backend. 600 IN A 172.19.0.3
backend. 600 IN A 172.19.0.2
backend. 600 IN A 172.19.0.4
curl backend:8080 works
Podman & docker-compose (RHEL 8)
Podman came preinstalled, I added docker-compose (standalone) and podman-docker:
# curl -SL https://github.com/docker/compose/releases/download/v2.10.2/docker-compose-linux-x86_64 -o /usr/local/bin/docker-compose
# chmod a+x /usr/local/bin/docker-compose
# sudo yum install podman-docker
And activated rootless podman socket so that podman and docker-compose can talk to each other:
# systemctl --user enable podman.socket
# systemctl --user start podman.socket
# systemctl --user status podman.socket
# export DOCKER_HOST=unix:///run/user/$UID/podman/podman.sock
I also switched network backend to netavark, DNS did not work without that change
$ podman info |grep -i networkbackend
networkBackend: netavark
1. Port ranges are not supported
Podman does not like ports: - "8180-8280:8080" due to this bug: https://github.com/containers/podman/issues/15111
[+] Running 1/0
⠿ Network example-docker-compose_default Created 0.0s
⠋ Container example-docker-compose-backend-3 Creating 0.0s
⠋ Container example-docker-compose-backend-1 Creating 0.0s
⠋ Container example-docker-compose-backend-2 Creating 0.0s
Error response from daemon: make cli opts(): strconv.Atoi: parsing "8180-8280": invalid syntax
2. Without port range, address already in use conflict
Changed docker-compose.yml to remove port range
ports:
- "8080:8080"
This results in port conflict, all 3 backend containers try to bind to 8080
Catalina.startup.Catalina.start Server startup in [190] milliseconds
Error response from daemon: rootlessport listen tcp 0.0.0.0:8080: bind: address already in use
3. Scale = 1
Let's try with just one backend container. System starts up
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
217c3e726431 docker.io/library/tomcat:latest catalina.sh run 7 seconds ago Up 6 seconds ago 0.0.0.0:8080->8080/tcp example-docker-compose-backend-1
9c3f86676bde docker.io/library/nginx:latest nginx -g daemon o... 6 seconds ago Up 6 seconds ago 0.0.0.0:3000->80/tcp example-docker-compose-frontend-1
DNS from frontend server looks weird. What are all these different backend IP addresses 10.89.0.3 - 10.89.0.12? Only the last of them works when I curl 10.89.0.x. Still, curl backend:8080 works fine?
4. Scale up from command line
I remove ports and scale from docker-compose.yml and start compose stack with scale=3 option:
version: "3"
services:
frontend:
image: "nginx:latest"
ports:
- "3000:80"
depends_on:
- backend
backend:
image: "tomcat:latest"
$ docker-compose up --scale backend=3
[+] Running 4/4
⠿ Container example-docker-compose-backend-2 Recreated 0.3s
⠿ Container example-docker-compose-backend-3 Recreated 0.2s
⠿ Container example-docker-compose-backend-1 Recreated 0.3s
⠿ Container example-docker-compose-frontend-1 Recreated 0.3s
Attaching to example-docker-compose-backend-1, example-docker-compose-backend-2, example-docker-compose-backend-3, example-docker-compose-frontend-1
Now compose stack starts nicely with 3 backend containers
$ docker ps
Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg.
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
b81c44246ae7 docker.io/library/tomcat:latest catalina.sh run 22 minutes ago Up 22 minutes ago example-docker-compose-backend-3
814dc5d307f7 docker.io/library/tomcat:latest catalina.sh run 22 minutes ago Up 22 minutes ago example-docker-compose-backend-2
fb0a5090a456 docker.io/library/tomcat:latest catalina.sh run 22 minutes ago Up 22 minutes ago example-docker-compose-backend-1
c0219d7fded4 docker.io/library/nginx:latest nginx -g daemon o... 22 minutes ago Up 22 minutes ago 0.0.0.0:3000->80/tcp example-docker-compose-frontend-1
DNS from frontend has even more entries
# dig backend
; <<>> DiG 9.16.27-Debian <<>> backend
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 29526
;; flags: qr rd ad; QUERY: 1, ANSWER: 11, AUTHORITY: 0, ADDITIONAL: 1
;; WARNING: recursion requested but not available
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
; COOKIE: 410008265c2b0edb (echoed)
;; QUESTION SECTION:
;backend. IN A
;; ANSWER SECTION:
backend. 86400 IN A 10.89.0.3
backend. 86400 IN A 10.89.0.4
backend. 86400 IN A 10.89.0.6
backend. 86400 IN A 10.89.0.7
backend. 86400 IN A 10.89.0.15
backend. 86400 IN A 10.89.0.16
backend. 86400 IN A 10.89.0.18
backend. 86400 IN A 10.89.0.19
backend. 86400 IN A 10.89.0.20
backend. 86400 IN A 10.89.0.21
backend. 86400 IN A 10.89.0.22
And curl backend:8080 does not work (not sure which port I should use now)
Questions
What's going on here?
Can I achieve a setup of 3 backend containers, so that DNS name backend would resolve to those, with podman & docker-compose?
Podman seems to support docker-compose (or vice versa), but only to a degree. Is there some documentation which tells what docker-compose features are supported on Podman, and which are not?

Answer 1

Podman is my container runtime of choice...unless I'm working with docker-compose, in which case I have found it to be "close but not quite" in terms of its docker API support.
However, for what you're trying to do, you could replace Nginx with Traefik, and let Traefik handle the load balancing. Traefik is a dynamic proxy that uses the Docker API and container labelling to discover containers and configure the proxy rules.
For example:
version: "3"
services:
frontend:
image: "docker.io/traefik:v2.8"
ports:
- "3000:80"
- "127.0.0.1:3080:8080"
command:
- --api.insecure=true
- --providers.docker
volumes:
- /run/user/$UID/podman/podman.sock:/var/run/docker.sock
backend:
labels:
traefik.http.routers.backend.rule: Host(`localhost`)
image: "quay.io/larsks/demoserver:latest"
scale: 3
Here we're mapping all requests for Host: localhost to to our backend containers. This is just for the purposes of a demonstration (since I'll be running curl localhost/...); a more realistic configuration would use specific hostnames, or paths, etc. You can read more in the Routing configuration section of Traefik's Docker documentation, and also in the general Router documentation.
With this configuration, we see the following containers running:
$ podman ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
c29de1d137e2 docker.io/library/traefik:v2.8 --api.insecure=tr... 5 seconds ago Up 6 seconds ago 0.0.0.0:3000->80/tcp, 127.0.0.1:3080->8080/tcp demoserver_frontend_1
f4fac24cb494 quay.io/larsks/demoserver:latest /usr/local/bin/st... 5 seconds ago Up 6 seconds ago demoserver_backend_2
d9d388202be2 quay.io/larsks/demoserver:latest /usr/local/bin/st... 5 seconds ago Up 5 seconds ago demoserver_backend_1
5a3e6330739d quay.io/larsks/demoserver:latest /usr/local/bin/st... 5 seconds ago Up 5 seconds ago demoserver_backend_3
And we can see that requests on port 3000 cycle between the available backends. Running this script:
for i in {1..10}; do
curl http://localhost:3000/hostname
done
Produces as output:
5a3e6330739d
d9d388202be2
f4fac24cb494
5a3e6330739d
d9d388202be2
f4fac24cb494
5a3e6330739d
d9d388202be2
f4fac24cb494
5a3e6330739d