Decided to try and really understand the docker-compose.yaml file for airflow. At the beginning of the file there is this piece of code.
x-airflow-common:
&airflow-common
image: ${AIRFLOW_IMAGE_NAME:-apache/airflow:2.2.5}
What I'm gathering is that x-airflow-common is defining a variable and that the the &airflow-common says that "any image in this file that points to *airflow-common look here. That is why further down we see.
<<: *airflow-common
Which says "look in this docker-compose file for the image declared in airflow-common" then when it runs the command against the image: scheduler, celery worker etc. The airflow image sees those commands and knows what type of image to spin up for that container.
Hoping someone can confirm/correct my assumptions or point me to good documentation for this. I've been searching the past two days, but have been unable to locate anything that "dissects" this file.
This uses a feature in Docker Compose called YAML anchors. It allows you to create a sort of template block, and then create other services that are based on that template, but replace certain settings in it.
This section on the Compose specification docs can probably explain it better than I can.
Related
Use Case
The docker-compose.yml defines multiple services which represent the full application stack. In development mode, we'd like to dynamically exclude certain services, so that we can run them from an IDE.
As of Docker compose 1.28 it is possible to assign profiles to services as documented here but as far as I have understood, it only allows specifying which services shall be started, not which ones shall be excluded.
Another way I could imagine is to split "excludable" services into their own docker-compose.yml file but all of this seems kind of tedious to me.
Do you have a better way to exclude services?
It seems we both overlooked a certain very important thing about profiles and that is:
Services without a profiles attribute will always be enabled
So if you run docker-compose up with a file like this:
version: "3.9"
services:
database:
image: debian:buster
command: echo database
frontend:
image: debian:buster
command: echo frontend
profiles: ['frontend']
backend:
image: debian:buster
command: echo backend
profiles: ['backend']
It will start the database container only. If you run it with docker-compose --profile backend up it will bring database and backend containers. To start everything you need to docker-compose --profile backend --profile frontend up or use a single profile but several times.
That seems to me as the best way to make docker-compose not to run certain containers. You just need to mark them with a profile and it's done. I suggest you give the profiles reference a second chance as well. Apart from some good examples it explains how the feature interacts with service dependencies.
How to run docker-compose across different lifecycle environments (say dev, qa, staging, production).
Sometimes a larger VM is being shared by multiple developers, so would like to start the containers with appropriate developer specific suffixes (say dev1, dev2, dev3 ..). Should port customization be handled manually via the environment file (i.e. .env file)
This is an unusual use case for docker-compose, but I'll leave some tips anyway! :)
There's two different ways to name stuff you start with docker-compose. One is to name the service that you specify under the main services: key of your docker-compose.yml file. By default, individual running containers will be assigned names indicating what project they are from (by default, the name of the directory from which your docker-compose file is in), what service they run (this is what's specified under your services: key), and which instance of that service they are (this number changes if eg. you're using replicas). Eg. default container names for a service named myservice specified in a compose file ~/my_project/docker/docker-compose.yml will have a name like docker_myservice_1 (or _2, _3, etc if more than one container is supposed to run).
You can use environment variables to specify a lot of key-value pairs in docker-compose files, but you can't conditionally specify the service name - service keys are only allowed to have alphanumeric characters in them and compose files can't look like eg:
version: "3"
services:
${ENVVAR}:
image: ubuntu:20.04
However, you can override the container naming scheme by using the container_name field in your docker-compose file (documentation for usage here). Maybe a solution you could use looks like this:
version: "3"
services:
myservice:
image: ubuntu:20.04
container_name: ${DEVELOPER_ENVVAR?err}
this will require a developer to specify DEVELOPER_ENVVAR at runtime, either by exporting it in their shell or by running docker-compose like DEVELOPER_ENVVAR=myservice_dev1 docker-compose up. Note that using container_name is incompatible with using replicas to run multiple containers for the same service - the names have to be unique for those running containers, so you'll either have to define separate services for each name, or give up on using container_name.
However, you're in a pickle if you expect multiple developers to be able to run containers with different names using the same compose file in the same directory. That's because when starting a service, docker-compose has a Recreating step where, if there's already containers implementing that service running, they'll wait for that container to finish. Ultimately, I think this is for the best - if multiple developers were trying to run the exact same compose project at once, should a developer have control over other developers' running containers? Probably not, right?
If you want multiple developers to be able to run services at once in the same VM, I think you probably want to do two things:
first, (and you may well have already done this! but it's still a good reminder) make sure that this is a good idea. Are there going to be resource contention issues (eg. for port-forwarding) that make different running instances of your project conflict? For many Docker services, there are going to be, but there probably won't be for eg. images that are meant to be run in a swarm.
second, have different compose files checked out in different directories, so that there are separate compose projects for each developer. To use .env files one way one obvious option is to just maintain separate copies, one per developer directory. If, for your use case, it's unsatisfactory to maintain one copy of .env per developer this way, you could use symlinks named .env (or whatever your env file is named) to the same file somewhere else on the VM.
After you've done this, you'll be able to tell from the container names who is running what.
If none of these are satisfactory, you might want to consider, eg. using one VM per developer, or maybe even considering using a different container management system than docker-compose.
I have done very similar automation and I've used Ansible to create "docker compose" config on the fly.
So based on input-Environment , the ansible playbook will create the relevant docker-compose file. So basically I have a docker-compose template in my git repository with values that are dynamic and ansible playbook populates them etc.
and also you can use ansible to trigger such creation or automation one after another
A similar sample has been posted at ansible_docker_splunk repository.
Basically the whole project is to automate end-to-end docker cluster from CSV file
I am having an issue getting an image to build in Azure Devops from a docker-compose file.
It appears that the first issue is that the image does not build.
This is, I believe, causing the push step to fail, as there is no created image, it is just running an existing image.
What can I do to "force" the process to build an image off of this to pass into our repo? Here is our current docker compose file
version: '3.4'
services:
rabbit:
image: rabbitmq:3.6.16-management
labels:
NAME: "rabbit"
environment:
- "RabbitMq/Host=localhost"
ports:
- "15672:15672"
- "5672:5672"
container_name: rabbit
restart: on-failure:5
Here's the build and push steps (truncating the top which doesn't really matter)
Build:
Push:
I spent a fair amount of time fighting with this today (similar issue, anyway). I do not believe the image being non-local is necessarily your problem.
Looks like you are using the "Docker Compose" Task in Azure DevOps for your Build. I had the same problem - I could get it to build fine, but couldn't ever seem to "pipe" the result to the Push Task. I suspect one could add another Task between them to resolve this, but there is an easier way.
Instead, trying using the "Docker" Task to do your build. Without really changing anything else, I was able to make that work and the "Push" step next in line was happy as could be.
I'm trying to customise OpenMaptiles-Server running under Docker. I've NO docker exposure. I've read the docs and they suggest there is a config file for Docker, but what its name is, where it lives seems to be assumed knowledge.
Is there a blog that explains this for absolute novices as the documentation from Klokantech is not very helpful if you have not used these technologies before.
Can somebody let me know where the configuration file lives, what its name is supposed to be and how I get rid of the error about unconfigured /data directory? thanks
Assuming you mean the docker image at klokantech/openmaptiles-server, the method for configuration is as follows:
Pull the image.
Run the image - expose a local port you can connect to, and bind-mount a local path to the image at /data
Run a browser and connect to the container host at the specified port. A 'first time install' interface will come up allowing you select what regions you want to download mbtile data for. Note that you can only run 1 mbtile per server - it will render 'blank' (tan) tiles for everything outside that region. Allow the container server to download the tiles.
Inspect the local path that was mounted to the container at /data. You should now see a .mbtiles and a config.json file. Save those off somewhere for posterity.
In the actual location you intend to now run the container, deploy the image, the mbtiles, and the config.json. Put the mbtiles and config.json in the same local directory, and bind mount those to /data when you run the openmap-tileserver. It will pick up on the existing mbtiles and config, skipping the install step, and boot straight into serving the tiles.
mem_limit is no longer supported in version 3 of docker-compose.yml file. The documentation tells that I should use the deploy.resources key instead but also that this part will only be effective with swarm or docker stack.
cpu_shares, cpu_quota, cpuset, mem_limit, memswap_limit: These have been replaced by the resources key under deploy. Note that deploy configuration only takes effect when using docker stack deploy, and is ignored by docker-compose.
... as written in the docs.
How do I set memory/cpu limits with docker-compose with v3 format of the yml file?
I was wondering the same thing and found this:
https://github.com/docker/compose/issues/4513
So in short it's just not possible to do that, you have to use the version 2.1 of the docker-compose format to be able to specify limits that are not ignored by docker-compose up
you can try docker-compose --compatibility up which is CLI flag that convert v3 files to their v2 equivalent, with deploy options translated when possible.
I was searching for this issue a while ago. I have found my answer here. At first, I tried to implement this functionality without using docker stack, but that did not work.
Here is the piece of code which you would use to limit container's CPU/memory consumption. For additional attributes, you can search the documentation of docker.
deploy:
replicas: 5
resources:
limits:
cpus: "0.1"
memory: 50M
Compose file does not recognize deploy attributes, unless you deploy the application as a stack.
This is not the case anymore. According to the new documentation here: https://docs.docker.com/compose/compose-file/compose-file-v3/#deploy
It states that resources are repspected by docker compose.
I can confirm this now.