Basic string variable substitution in docker-compose files (3.8) - docker-compose

Take my example of two services:
services:
nginx:
ports:
- 443:443
volumes:
- "CONFIG_DIRECTORY/nginx/nginx.conf:/etc/nginx/nginx.conf"
- "CONFIG_DIRECTORY/certs:/etc/ssl/certs"
web:
command: ["node", "index.js"]
volumes:
- "CONFIG_DIRECTORY/certs:/var/client/config/ssl/certs"
- "CONFIG_DIRECTORY/process:/var/client/process"
I'd really like to be able to substitute a string such as /home/garnettm/development/config directly into the indicated CONFIG_DIRECTORY locations in the above strings.
Is there any way to do this other than the many currently available environment variable substitution process options?
A .env file for example would allow you to do this using an already defined variable and the $VARIABLE syntax.

The only form of string interpolation Compose supports at all is environment variable substitution. This uses shell-style $VARIABLE syntax, with very limited options for if the variable isn't set, ${VARIABLE:-default value} or ${VARIABLE:?error message}.
There is no way to declare these variables in the Compose file itself. Compose does support putting variable values in a .env file, so they don't necessarily need to be set as actual environment variables. You do not need to mention this file in a Compose env_file: directive, Compose reads it on its own to set environment variables to do these substitutions.
# .env
CONFIG_DIRECTORY=/home/garnettm/development/config
# docker-compose.yml
volumes:
- "$CONFIG_DIRECTORY/nginx/nginx.conf:/etc/nginx/nginx.conf"
For this specific setting, on the left-hand side of volumes: and other places that refer to host directories, relative paths are interpreted relative to the location of the Compose file (the first one, if you have multiple docker-compose -f options). A very common setup is to put all of the state and source code in the same directory tree, and to put the docker-compose.yml file at the root of that tree. In that case the relative directory . will almost always be right
volumes:
- "./nginx/nginx.conf:/etc/nginx/nginx.conf"
YAML anchors are occasionally used to provide common blocks of settings, but you can't do string interpolation using anchors, only supply an entire YAML scalar node. They won't be helpful here.

Related

How to reference values inside a Docker env file inside the same file

This is my connection string inside my docker env file:
DB_URL=mongodb://${MONGO_INITDB_ROOT_USERNAME}:${MONGO_INITDB_ROOT_PASSWORD}#XX.XXXX.XXX.XXXX:27017/david?authSource=admin
And inside my env file I have this:
MONGO_INITDB_ROOT_USERNAME=root
MONGO_INITDB_ROOT_PASSWORD=example
I do I reference these values in the connection string above in the same file, or I need to hard code them. Currently I am using the format ${.....} but not sure it's the right way.
I have this in the env file:
MONGO_INITDB_ROOT_USERNAME=root
MONGO_INITDB_ROOT_PASSWORD=example
...
DB_URL=mongodb://${MONGO_INITDB_ROOT_USERNAME}:${MONGO_INITDB_ROOT_PASSWORD}#XX.XXXX.XXX.XXXX:27017/david?authSource=admin

What is the hyphen after the ENV variable DOCKER_REGISTRY for in a docker-compose file

In my docker-compose files, my image tag is like so
image: ${DOCKER_REGISTRY-}webapi
However the ENV variable is just DOCKER_REGISTRY, so what's the hyphen for?
Also, now I'm wanting to push these images to ECR, when I set the DOCKER_REGISTRY to my ECR instance, there is no /, between the DOCKER_REGISTRY and webapi part, so I guess I need to change it to look like image: ${DOCKER_REGISTRY-}/webapi? But maybe only for my non-dev docker-compose files?
The - is part of the syntax for variable substitution, which is based on the standard described here: https://pubs.opengroup.org/onlinepubs/009604599/utilities/xcu_chap02.html#tag_02_06_02
${SOME_VAR-<default value if variable is not set>}
In this case, no default value is set, so it's redundant.
Don't add a /. it's not a path, it's the name of the image.

docker swam - secrets from file not resolving tilde

Using secrets from docker-compose on my dev machine works. But the remote server via ssh just says open /home/johndoe/~/my-secrets/jenkinsuser.txt: no such file or directory.
secret definition in stack.yml:
secrets:
jenkinsuser:
file: ~/my-secrets/jenkinsuser.txt
Run with:
docker stack deploy -c stack.yml mystack
The documentation does not mention any gotchas about ~ path. I am not going to put the secret files inside . as all examples do, because that directory is version controlled.
Am I missing some basics about variable expansion, or differences between docker-compose and docker swarm?
Your ~ character in your path is considered as literal. Use $HOME wich is treated as a variable in your string path.
Tilde character work only if it is unquoted. In your remote environment, the SWARM yaml parser consider your path as a string, where the prefix-tilde is read as a normal character (see prefix-tilde).

Is there any clean way to append an env_file to an autogenerated docker file?

I am planning to self-host Bitwarden using Ansible. During the execution of the ansible playbook, a file hierarchy and a docker-compose file is generated on the remote based on config file parameters and other. At the end, this docker file is executed to spawn the containers.
My personal server infrastructure already servers an nginx-proxy combined with letsencrypt. In order to use this service, I need to add several specific environmental parameters to the "main container" in this docker-compose file. I collected them in an external file and now I need to add a this external file to the docker file. I want to do it dynamically. Until now, I am using the ansible task:
- name: append nginx.env as env file to automatically generated docker file
shell: 'sed -i "\$a\ \ \ \ \ \ - ../env/nginx.env" /srv/bitwarden/bwdata/docker/docker-compose.yml'
to transform:
...
*OTHER CONTAINERS*
...
nginx:
...
*UNIMPORTANT PARAMETERS*
...
env_file:
- ../env/uid.env(EOF)
into
...
*OTHER CONTAINERS*
...
nginx:
...
*UNIMPORTANT PARAMETERS*
...
env_file:
- ../env/uid.env
- ../env/nginx.env(EOF)
But I am unhappy with this sed solution because Bitwarden could modify this auto-generated file (appending to the end would eventually fail.. and I also can't really use any regEx because they could change the structure of single components of the file etc...).
Is there any built-in safe way by ansible to achieve this exact behavior?
EDIT: I am looking for something like:match <container_name> append to env_file <../env/nginx.env>

Concourse CI shell commands being quoted

We are trying to copy multiple jar files in a Concourse CI package.yml file as part of a run step:
run:
path:
args:
- -exc
- |
...
cp project/target/*.jar build-output/.
But Concourse is adding single quotes to the source file, so that it's looking for a file named 'project/target/*.jar', and of course it's not finding it.
+ cp 'project/target/*.jar' build-output/.
cp: can't stat 'project/target/*.jar'; no such file or directory
I even tried putting double quotes around the jar file name, hoping that it might keep Concourse from changing it, but it made no difference.
We want to use file globbing so that we can use this generically, so that we don't need to know the file names ahead of time. Is there any way we can get this to work?
Bizarre. Try specifying the shell:
run:
path: sh <== missing `sh` or the shell you have available in the image
args:
- -exc
- |
...
cp project/target/*.jar build-output/ <== no `.`
Mhhh, actually maybe the quotes are misleading, there is nothing below project/target :-) Try with
run:
path: sh <== missing `sh` or the shell you have available in the image
args:
- -exc
- |
...
# is anything here ?
ls -1 project/target
cp project/target/*.jar build-output/ <== no `.`
set -x mode quotes all arguments; it doesn't mean that it actually quoted the argument. I think you had a red herring, but I'm glad you got it working.