Question
Is there a way to define a variable within a docker-compose's config yaml file?
My usecase FYI
Using "extends" (docker.com) I'm thinking to use a base yaml and configure it from many child yaml files by passing a variable.
One limitation is that I cannot make the operational commands complicated at all (intended users are not developers) so I want the command to be a single-line, and cannot be too long. So defining variable in the operation command (e.g. docker-compose -f ... up is fine, but adding VAR=foo at the beginning of it is too long and complicated for me, particularly I might want to define multiple variables).
What I've tried but didn't work
So far I thought of using docker-compose's environment variable features (note that I don't care if the variable I want to set is environment variable or something else. I just gave a try as the features are already there).
NG: Using "environment"
version: '3.3'
services:
service_a:
image: ubuntu:focal
environment:
ENV_A: "env_a"
command: >
/bin/bash -c "echo In docker container: $ENV_A"
Then have env_a, the content of ENV_A, printed. But it didn't get printed:
$ docker-compose -f a.yaml up
WARNING: The ENV_A variable is not set. Defaulting to a blank string.
:
service_a_1 | In docker container:
tmp_service_a_1 exited with code 0
NG: Using "env_file"
$ cat a.env
ENV_A=env_a
$ cat b.yaml
version: '3.3'
services:
service_b:
image: ubuntu:focal
env_file: ./a.env
command: >
/bin/bash -c "echo In docker container: $ENV_A"
Result: NG.
$ docker-compose -f b.yaml up
WARNING: The ENV_A variable is not set. Defaulting to a blank string.
:
service_b_1 | In docker container:
Operation environment
$ docker-compose --version
docker-compose version 1.29.2, build 5becea4c
$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 18.04.6 LTS
:
First of all, extends is not supported on compose 3.x. This is at the top of the link you sent. So you may not want to rely on that.
As for environment variables, only variables set in the .env file are available to compose. Variables set in env_file or environment settings within the compose file will be passed into the container, but will NOT be available for use in compose itself.
You're asking compose to interpolate $ENV_A, but you haven't set it on the command line (like you described at the top, VAR=foo docker...) or in the .env, so it can't find a value.
More in this thread on GitHub.
Related
There are many way to set command in pod.
1.
command: ["/bin/sh","-c","sleep 1000"]
2.
command: ["/bin/sh"]
args: ["-c","sleep 1000"]
3.
args: ["/bin/sh","-c","sleep 1000"]
Are they the same or different?
These are all almost the same; except that if the underlying image has a Docker ENTRYPOINT then the args: are passed to it as arguments in the third case, without command:.
In core Docker, there are two ways to specify the main container command, the Dockerfile ENTRYPOINT and CMD directives. Either or both of these can be overridden when the container starts, but resetting the ENTRYPOINT value always clears CMD. If both are present then the CMD is passed as arguments to the ENTRYPOINT.
Kubernetes has the same concept and uses the same underlying image structure, but its names are different:
Kubernetes command: overrides Docker ENTRYPOINT (and resets CMD)
Kubernetes args: overrides Docker CMD
So in the first case setting command: overrides the container entrypoint and resets the container command to an empty list; in the second you directly set both parts. In both cases the two lists are combined together and you get an identical 3-word command.
In the last case args: only overrides the container command part. If the image has a Dockerfile ENTRYPOINT then this list of arguments is passed as arguments to the entrypoint process.
Which one is "right" depends on how the image is constructed:
Some images use CMD to specify the main command to run and don't have ENTRYPOINT; in this case any of the forms you show will work.
CMD the-main-program
# override with either Kubernetes command: or args:
Some images use ENTRYPOINT to specify the main command to run (and depending on how they're constructed may completely ignore CMD). In this case you must use one of the first two forms with command:.
ENTRYPOINT ["the-main-command"]
CMD ["--argument", "value"]
# override with Kubernetes command: and optionally args:
Some images use CMD to specify the main container command, and ENTRYPOINT as a wrapper script to do first-time setup. In this case you need the third form that sets only args: or else the entrypoint script will get skipped.
ENTRYPOINT ["docker-entrypoint.sh"]
CMD ["the-main-command"]
# override with Kubernetes args: only
I've read several tutorials about this and it seems like this code should work. I want to use RUN in my Dockerfile to write environment variables to a config file. The Dockerfile is simply:
FROM php:7.4-apache
RUN echo "I am ${USERNAME}" > /home/dockerfile.log
I am starting it with this docker-compose.yml
version: "3"
services:
web:
build:
context: .
environment:
- USERNAME=larry
After starting with docker-compose up -d the log file just says I Am so the environment variable is empty. What's wrong here? If I run docker-compose config I can see the variable shown correctly there.
The Run command is executed at build time (I.e. when building the image ) while the environment variable in your docker-compose is added to the container at run time. So when the Run command is executed the environment variable has not been populated yet. If you want to pass a parameter at build time, I would suggest using a build argument instead. In your compose file use arg instead of environment
I am trying to create absolute paths in my docker-compose using variable substitution for the path up to the current directory. I pass these paths as environment variables in my container.
More precisely I try to pass it via the cd command and not via the PWD env var. It is supposed to be possible declaring the env var in the .env (https://github.com/docker/compose/issues/4081).
Unfortunately I cannot get the substitution to work.
I tried the different approaches:
with the declaration of the var as a environment variable in a .env OR directly in the environment list
Running with powershell and command prompt
using %cd% and $pwd with both shells
escape $ and %, doubling them down with extra $ or %
Relevant piece of the docker-compose:
version: "3.2"
services:
jupyter:
environment:
- WORKING_DIR=%cd% # OR $(pwd) OR $VAR
and when I use the .env:
VAR=%cd% or ${pwd}
When I 'set' my environment variables in my container, it just shows the command I used (%cd%, etc) as if they were strings.
I do not quite understand where is the docker-compose formated/executed: in the shell I use to call docker-compose, in the linux image where my docker daemon runs, etc ?
EDIT:
I tried on a MacOS version and I still have the problem ! (using adequate bash variable substitution)
Not a complete solution, but perhaps something you can script:
If you specify DOCUMENT_ROOT as a variable in the docker compose file
environment:
- WORKING_DIR=${DOCUMENT_ROOT}
And have an .env file in your working directory that contains
DOCUMENT_ROOT=./
, then you can use the local .env file by running
docker-compose -f /path/to/docker-compose.yml up
https://docs.docker.com/compose/compose-file/#env_file
EDIT: The problem is the COMPOSE_CONVERT_WINDOWS_PATHS environment variable isn't getting set. I tried adding it to the Dockerfile and to a .env file but its not being set. I can't set any new environment variables. Its like its building it from a cached Dockerfile.
I was running docker-composer up after deleting the container, but this didn't recreate a new container, even when adding --force-recreate, it created the container from the old Dockerfile. So I ran docker build ./ instead, and it built a new container, it reinstalled all the packages, and it said that it was setting the new environment variables, but in the finished container, when I check kinematic, I see the new environment variables aren't set. But all of the old environment variables that were set when I built the container for the first time, they're all set. I don't know why this is happening, why I can't create a new container from the updated Dockerfile.
I've found a number of threads about this error, it happens when trying to mount a volume in Windows docker-tools. When I run docker-compose up, I get this error:
ERROR: for web Cannot create container for service web: Invalid bind mount spec "C:\\path\\to\\project:/app:rw": Invalid volume specification: 'C:\path\to\project:/app:rw'
[31mERROR[0m: Encountered errors while bringing up the project.
In the docker-compose file I have this:
volumes:
- ./:/app
I tried setting this environment variable:
COMPOSE_CONVERT_WINDOWS_PATHS=1
to 1, as that worked for some people in another thread but its not working in this case. I tried adding an absolute path in linux like format:
volumes:
- /c/path/to/project:/app
and I get this error:
ERROR: for web Cannot create container for service web: create \c\path\to\project: "\\c\\path\\to\\project" includes invalid characters for a local volume name, only "[a-zA-Z0-9][a-zA-Z0-9_.-]" are allowed
Notice it reversed the forward slashes. Changing:
ENV COMPOSE_CONVERT_WINDOWS_PATHS=0
to 0 like that didn't change anything.
If I give a Windows format absolute directory, I get the first error again. Both of those errors, I get them if I point to a directory that doesn't exist.
Has anyone here figured out a way to resolve this?
So, if this problem exists, you should create new Windows environment variable called COMPOSE_CONVERT_WINDOWS_PATHS and set it to 1. Or you can create .env file in the path docker-compose.yml is placed with following content:
COMPOSE_CONVERT_WINDOWS_PATHS=1
..
cat >.env
COMPOSE_CONVERT_WINDOWS_PATHS=1
It looks like your COMPOSE_CONVERT_WINDOWS_PATHS=1 environment variable is not read by docker cli (powershell). you can try to run Get-childItem Env: command from your powershell console to get the list of environment variable which has been set in your powershell console. the output is like below:
Name Value
---- -----
COMPOSE_CONVERT_WINDOWS_PATHS 1
COMSPEC C:\WINDOWS\system32\cmd.exe
DOCKER_CERT_PATH C:\Users\devsa\.docker\machine\machines\default
DOCKER_HOST tcp://192.168.99.100:2376
DOCKER_TLS_VERIFY 1
...
If you don't see the COMPOSE_CONVERT_WINDOWS_PATHS=1 variable in your powershell console, it means the variable has not been set yet. To set the environment variable via powershell console, try this command:
$env:COMPOSE_CONVERT_WINDOWS_PATHS=1
it will set the environment variable in your power shell console check this powershell documentation for full documentation. Try to run the Get-childItem Env: command again to check whether the variable has been already set or not. if the variable has been set, you can try to run docker-compose up -d again.
After setting the environment variable to COMPOSE_CONVERT_WINDOWS_PATHS=1 try below setting in docker-compose
- ./app-demo:/dockermount under volumes
It worked for me
I am having troubles in passing environment variables to my custom image via the compose command option:
My compose file:
---
version: '2'
services:
myservice:
image: mycustomimage_lms
environment:
CONF_HOME: /opt/apps-java/
APP_ENV: dev
UUID: me1
command: -Dconfig.home=${CONF_HOME} -Dcomponent.name=LMS -Denv=${APP_ENV} -Duser.dir=/tmp/ -DLMS_UUID=${UUID} -jar /opt/apps-java/my.jar
ports:
- "9060"
volumes:
- ./:/opt/apps-java/
- ./:/var/logs/apps-logs/
- ./:/tmp/data
My image is just a custom jre image which has an entrypoint set to a shell script that accepts jvm arguments. My run.sh that is called from enrtypoint
#!/bin/sh
export JAVA_HOME="/usr/java/latest/"
exec $JAVA_HOME/bin/java $#
I need to pass values to command at runtime since I can then use my image for a lot of other jars and just changing parameters to my image.
This is what i get:
$> docker-compose up
WARNING: The CONF_HOME variable is not set. Defaulting to a blank string.
WARNING: The APP_ENV variable is not set. Defaulting to a blank string.
WARNING: The UUID variable is not set. Defaulting to a blank string.
I have also
gone through couple of answers such as :
Docker Compose - Command using Container Environment Variable
and
Docker-compose environment variables
but could not get it working. Any directions please?
The variables are being read by Compose when the file is parsed. But setting environment only provides values to the container, not to the file parsing.
If you're trying to pass those variables into the container, you need to escape them in the command using an extra $
-Dconfig.home=$${CONF_HOME} -Dcomponent.name=LMS -Denv=$${APP_ENV} -Duser.dir=/tmp/ -DLMS_UUID=$${UUID
If you're just trying to use variables in the Compose file, you need to put those variables into an .env file.
See https://docs.docker.com/compose/compose-file/compose-file-v3/#variable-substitution for the full documentation
Escaping the $ so the variable is not substituted immediately but later in the container environment is the trick, as the accepted answer does in the the docker-compose.yml.
I just wanted to add that in case you pass a command to the docker-compose call, you need to escape the character in the shell, i.e. with \
docker-compose run --rm myservice "-Dconfig.home=\${CONF_HOME} -Dcomponent.name=LMS -Denv=\${APP_ENV} -Duser.dir=/tmp/ -DLMS_UUID=\${UUID} -jar /opt/apps-java/my.jar"