mem_limit is no longer supported in version 3 of docker-compose.yml file. The documentation tells that I should use the deploy.resources key instead but also that this part will only be effective with swarm or docker stack.
cpu_shares, cpu_quota, cpuset, mem_limit, memswap_limit: These have been replaced by the resources key under deploy. Note that deploy configuration only takes effect when using docker stack deploy, and is ignored by docker-compose.
... as written in the docs.
How do I set memory/cpu limits with docker-compose with v3 format of the yml file?
I was wondering the same thing and found this:
https://github.com/docker/compose/issues/4513
So in short it's just not possible to do that, you have to use the version 2.1 of the docker-compose format to be able to specify limits that are not ignored by docker-compose up
you can try docker-compose --compatibility up which is CLI flag that convert v3 files to their v2 equivalent, with deploy options translated when possible.
I was searching for this issue a while ago. I have found my answer here. At first, I tried to implement this functionality without using docker stack, but that did not work.
Here is the piece of code which you would use to limit container's CPU/memory consumption. For additional attributes, you can search the documentation of docker.
deploy:
replicas: 5
resources:
limits:
cpus: "0.1"
memory: 50M
Compose file does not recognize deploy attributes, unless you deploy the application as a stack.
This is not the case anymore. According to the new documentation here: https://docs.docker.com/compose/compose-file/compose-file-v3/#deploy
It states that resources are repspected by docker compose.
I can confirm this now.
Related
Decided to try and really understand the docker-compose.yaml file for airflow. At the beginning of the file there is this piece of code.
x-airflow-common:
&airflow-common
image: ${AIRFLOW_IMAGE_NAME:-apache/airflow:2.2.5}
What I'm gathering is that x-airflow-common is defining a variable and that the the &airflow-common says that "any image in this file that points to *airflow-common look here. That is why further down we see.
<<: *airflow-common
Which says "look in this docker-compose file for the image declared in airflow-common" then when it runs the command against the image: scheduler, celery worker etc. The airflow image sees those commands and knows what type of image to spin up for that container.
Hoping someone can confirm/correct my assumptions or point me to good documentation for this. I've been searching the past two days, but have been unable to locate anything that "dissects" this file.
This uses a feature in Docker Compose called YAML anchors. It allows you to create a sort of template block, and then create other services that are based on that template, but replace certain settings in it.
This section on the Compose specification docs can probably explain it better than I can.
How to run docker-compose across different lifecycle environments (say dev, qa, staging, production).
Sometimes a larger VM is being shared by multiple developers, so would like to start the containers with appropriate developer specific suffixes (say dev1, dev2, dev3 ..). Should port customization be handled manually via the environment file (i.e. .env file)
This is an unusual use case for docker-compose, but I'll leave some tips anyway! :)
There's two different ways to name stuff you start with docker-compose. One is to name the service that you specify under the main services: key of your docker-compose.yml file. By default, individual running containers will be assigned names indicating what project they are from (by default, the name of the directory from which your docker-compose file is in), what service they run (this is what's specified under your services: key), and which instance of that service they are (this number changes if eg. you're using replicas). Eg. default container names for a service named myservice specified in a compose file ~/my_project/docker/docker-compose.yml will have a name like docker_myservice_1 (or _2, _3, etc if more than one container is supposed to run).
You can use environment variables to specify a lot of key-value pairs in docker-compose files, but you can't conditionally specify the service name - service keys are only allowed to have alphanumeric characters in them and compose files can't look like eg:
version: "3"
services:
${ENVVAR}:
image: ubuntu:20.04
However, you can override the container naming scheme by using the container_name field in your docker-compose file (documentation for usage here). Maybe a solution you could use looks like this:
version: "3"
services:
myservice:
image: ubuntu:20.04
container_name: ${DEVELOPER_ENVVAR?err}
this will require a developer to specify DEVELOPER_ENVVAR at runtime, either by exporting it in their shell or by running docker-compose like DEVELOPER_ENVVAR=myservice_dev1 docker-compose up. Note that using container_name is incompatible with using replicas to run multiple containers for the same service - the names have to be unique for those running containers, so you'll either have to define separate services for each name, or give up on using container_name.
However, you're in a pickle if you expect multiple developers to be able to run containers with different names using the same compose file in the same directory. That's because when starting a service, docker-compose has a Recreating step where, if there's already containers implementing that service running, they'll wait for that container to finish. Ultimately, I think this is for the best - if multiple developers were trying to run the exact same compose project at once, should a developer have control over other developers' running containers? Probably not, right?
If you want multiple developers to be able to run services at once in the same VM, I think you probably want to do two things:
first, (and you may well have already done this! but it's still a good reminder) make sure that this is a good idea. Are there going to be resource contention issues (eg. for port-forwarding) that make different running instances of your project conflict? For many Docker services, there are going to be, but there probably won't be for eg. images that are meant to be run in a swarm.
second, have different compose files checked out in different directories, so that there are separate compose projects for each developer. To use .env files one way one obvious option is to just maintain separate copies, one per developer directory. If, for your use case, it's unsatisfactory to maintain one copy of .env per developer this way, you could use symlinks named .env (or whatever your env file is named) to the same file somewhere else on the VM.
After you've done this, you'll be able to tell from the container names who is running what.
If none of these are satisfactory, you might want to consider, eg. using one VM per developer, or maybe even considering using a different container management system than docker-compose.
I have done very similar automation and I've used Ansible to create "docker compose" config on the fly.
So based on input-Environment , the ansible playbook will create the relevant docker-compose file. So basically I have a docker-compose template in my git repository with values that are dynamic and ansible playbook populates them etc.
and also you can use ansible to trigger such creation or automation one after another
A similar sample has been posted at ansible_docker_splunk repository.
Basically the whole project is to automate end-to-end docker cluster from CSV file
I'm trying to i) troubleshoot a simple onyx-kafka job not writing to a topic. More details are given here. And you can try it out in this sample project. I think the reason is because there's only one kafka node.
So I tried ii) launching kafka confluentinc/cp-kafka:3.3.1 (with zookeeper confluentinc/cp-zookeeper:3.3.1) running docker (17.09.0-ce, build afdb6d4) in swarm mode. But then I get this error.
Warning: space is running low in /dev/shm (shm) threshold=167,772,160 usable=58,716,160
A) With docker compose, I could just configure the mounted /dev/shm disk to have more capacity with the configuration option shm_size: 1G (see here).
Is there an equivalent for docker swarm? I just need to control the size of the disk volume. But I don’t see such an option in docker compose’s deploy configuration (see here). And the docker service --mount option doesn't seem to solve the problem. Any options here?
B) I don't quite grok the Memory volume mount described here. So don't know if that's a workable option.
C) Thanks to Andrew Mulholland (#itwasntandy) from the devops-engineers Slack channel, I can fake it by passing in --mount type=tmpfs,dst=/dev/shm,tmpfs-size=768000000 as options to a call to docker service create ... (see here).
But the offical documentation says that the tmpfs option is not supported for docker stack deploy. This optiononly applies to docker compose's service configuration. It doesn't apply to docker's swarm mode.
Summary. So is there any way to get that option into my existing docker-compose.yml file, in the swarm configuration?
I’d love to be able to run docker stack deploy --compose-file docker-compose.yml my-app. Details are here. Sample project is here. Ultimately, I want to:
i) run a docker stack, with a service which has onyx-kafka write to a topic (see here). And for that, I think I need to...
ii) launch docker (17.09.0-ce, build afdb6d4) in swarm mode, with the images kafka confluentinc/cp-kafka:3.3.1 (with zookeeper
confluentinc/cp-zookeeper:3.3.1). But I'm running into this memory error issue.
I'm not sure I completely understand the role of Docker in the process of development and deployment.
Say, I create a Dockerfile with nginx, some database and something else which creates a container and runs fine.
I drop it somewhere in the cloud and execute it to install and configure all the dependencies and environment settings.
Next, I have a repository with a web application which I want to run inside the container I created and deployed in the first 2 steps. I regularly work on it and push the changes.
Now, how do I integrate the web application into the container?
Do I put it as a dependency inside the Dockerfile I create in the 1st step and recreate the container each time from scratch?
Or, do I deploy the container once but have procedures inside Dockerfile that install utils that pull the code from repo by command or via hooks?
What if a container is running but I want to change some settings of, say, nginx? Do I add these changes into Dockerfile and recreate the image?
In general, what's the role of Docker in the daily app development routine? Is it used often if the infrastructure is running fine and only code is changing?
I think there is no singl "use only this" answer - as you already outlined, there are different viable concepts available.
Deployment to staging/production/pre-production
a)
Do I put it as a dependency inside the Dockerfile I create in the 1st step and recreate the container each time from scratch?
This is for sure the most docker`ish way and aligns fully with he docker-philosophy. It is highly portable, reproducible and suites anything, from one container to "swarm" thousands of. E.g. this concept has no issue suddenly scaling horizontally when you need more containers, lets say due to heavy traffic / load.
It also aligns with the idea that only the configuration/data should be dynamic in a docker container, not code / binaries /artifacts
This strategy should be chosen for production use, so when not as frequent deployments happen. If you care about downtimes during container-rebuilds (on upgrade), there are good concepts to deal with that too.
We use this for production and pre-production intances.
b)
Or, do I deploy the container once but have procedures inside
Dockerfile that install utils that pull the code from repo by command
or via hooks?
This is a more common practice for very frequent deployment. You can go the pull ( what you said ) or the push (docker cp / ssh scp) concept, while i guess the latter is preferred in this kind of environment.
We use this for any kind strategy for staging instances, which basically should reflect the current "codebase" and its status. We also use this for smoke-tests and CI, but depending on the application. If the app actually changes its dependencies a lot and a clean build requires a rebuild with those to really ensure stuff is tested as it is supposed to, we actually rebuild the image during CI.
Configuration management
1.
What if a container is running but I want to change some settings of,
say, nginx? Do I add these changes into Dockerfile and recreate the
image?
I am not using this as c) since this is configuration management, not applications deployment and the answer to this can be very complicated, depending on your case. In general, if redeployment needs configuration changes, it depends on your configuration management, if you can go with b) or always have to go a).
E.g. if you use https://github.com/markround/tiller with consul as the backend, you can push the configuration changes into consul, regenerating the configuration with tiller, while using consul watch -prefix /configuration tiller as a watch-task to react on those value changes.
This enables you to go b) and fix the configuration
You can also use https://github.com/markround/tiller and on deployment, e.g. change ENV vars or some kind of yml file ( tiller supports different backends ), and call tiller during deployment yourself. This most probably needs you to have ssh or you ssh on the host and use docker cp and docker exec
Development
In development, you generally reuse your docker-compose.yml file you use for production, but overload it with docker-compose-dev.yml to e.g. mount your code-folder, set RAILS_ENV=development, reconfigurat / mount some other configurations like xdebug or more verbose nginx loggin, whatever you need. You can also add some fake MTA-services like fermata and so on
docker-compose -f docker-compose.yml -f docker-compose-dev.yml up
docker-compose-dev.yml only overloads some values, it does not redefine it or duplicate it.
Depending on how powerful your configuration management is, you can also do a pre-installation during development stack up.
We actually use scaffolding for that, we use https://github.com/xeger/docker-compose and after running it, we use docker exec and docker cp to preinstall a instance or stage something. Some examples are here https://github.com/EugenMayer/docker-sync/wiki/7.-Scripting-with-docker-sync
If you are developing under OSX and you face performance issues due to OSXFS / code shares, you probably want to have a look at http://docker-sync.io ( i am biased though )
I have a repository which builds three different images:
powerpy-base
powerpy-web
powerpy-worker
Both powerpy-web and powerpy-worker inherit from powerpy-base using the FROM keyword in their Dockerfile.
I'm using Docker Compose in the project to run a Redis and RabbitMQ container. Is there a way for me to tell Docker Compose that I'd like to build the base image first and then the web and worker images?
You can use depends_on to enforce an order, however that order will also be applied during "runtime" (docker-compose up), which may not be correct.
If you're only using compose to build images it should be fine.
You could also split it into two compose files. a docker-compose.build.yml which has depends_on for build, and a separate one for running the images as services.
These is a related issue: https://github.com/docker/compose/issues/295
About run containers:
It was bug before, but they fixed it since docker-compose v1.10.
https://blog.docker.com/2016/02/docker-1-10/
Start linked containers in correct order when restarting daemon: This is a little thing, but if you’ve run into it you’ll know what a headache it is. If you restarted a daemon with linked containers, they sometimes failed to start up if the linked containers weren’t running yet. Engine will now attempt to start up containers in the correct order.
About build:
You need to build base image first.