Mounting host volumes in development container - visual-studio-code

I have created a development container with some custom dependency compilation. After a few hours building it is time to run my first tests. My tests require loading files that are on the host disk (pretrained models and some data), in a separate folder. Running with tools like docker compose or docker command line I can share files of the host with the container by specifying volumes. The development container does the same with the code folder, but how can I specify additional volumes for development containers.

You can use Multi Compose files for that:
Say you have a docker-compose.yml file where you have a base configuration for all your environments (dev, prod, etc.):
web:
image: example/my_web_app:latest
ports:
- "1234:1234"
And then you have other environment files like docker-compose.env.yml where you add or override configuration based on the environment.
With your example, you'd have a docker-compose.dev.yml where you'd add the additional volumes required for your development containers:
web:
volumes:
- 'path/to/host/folder:/path/to/container/folder'
During development, you would then launch your stack with:
docker-compose -f docker-compose.yml -f docker-compose.dev.yml up -d
Note: make sure to check how properties are handled when overriding files here, and especially volumes in your case.
Edit: now to use several docker-compose.yml files in your devcontainer.json, use the dockerComposeFile property. Here you'll remove the "build" property and supply a list of docker-compose files:
...
"dockerComposeFile": [
"path/to/docker-compose.yml",
"path/to/docker-compose.dev.yml"
],
"service": "web",
"shutdownAction": "stopCompose",
...
/!\ Note that the order of the files is important since later files override previous ones.

Related

Copy files to host from project folder and use them inside my container

I have a docker-compose file to build three different images from 3 different Dockerfiles.
In my project structure, I have some files that I want to copy to a specific folder in my host machine, because those files will be used by one of the containers.
I don´t want those folders to be inside of my container, because anytime I change something on one of the files, I need to build the image again.
Instead of using command line to copy the files from my project to the host machine, how how can it be done automatically when I run docker-compose build?
Unfortunately docker-compose has no such concept as pre- or post-scripts. The simplest way of achieving your goal would be creating a shell script which would copy the files and call docker-compose. If for some reason you are limited to calling docker-compose itself, you can create additional "setup" container to copy the files before starting other containers:
version: "3"
services:
# setup container copies files from the project directory PROJECT_DIR
# to another directory HOST_DIR on the host machine
setup:
image: alpine:latest
volumes:
- ${PROJECT_DIR}:/project
- ${HOST_DIR}:/host
command: >
sh -c "cp -R /project/* /host/"
# service container depends on the setup container and uses files from
# the host machine directory HOST_DIR mounted to CONTAINER_DIR
service:
image: alpine:latest
depends_on:
- setup
volumes:
- ${HOST_DIR}:${CONTAINER_DIR}
command: >
sh -c "ls ${CONTAINER_DIR}"

docker-compose: how to map a file from a container into a volume

I'm trying to share a file from one container to another other. An essential detail is that (the machine running) my docker host does not have explicit access to the file: It pulls a git repo and doesn't know about the internal file organization of the repo (with the single exception of the docker-compose file). Hence, the approach of standard mapping <host-path>:<container-path> is not applicable; e.g. this is not possible for me: How to mount a single file in a volume
Below is the docker-compose file, stripped for increased readability. Say service_1 has the file /app/awesome.txt. We then want to mount it into service_2 as /opt/awesome.txt.
# docker-compose.yml
version: '3'
services:
service_1:
volumes:
- shared_vol:/public
# how to make service_1 map 'awesome.txt' into /public ?
service_2:
volumes:
- shared_vol/awesome.txt:/opt/awesome.txt
volumes:
shared_vol:
Working solutions that I have, but are not fond of,
running a script/cmd within service_1, copying the file into the shared volume: causes race-condition as service_2 needs the file upon startup
introducing a third service, which the other two depends_on, and does nothing but put the file in the shared volume
Any help, tips or guidance is most appreciated!
can you just do something like this
version: '3.5'
volumes:
xxx:
services:
service_1:
...
volumes:
- xxx:/app
service_2:
...
volumes:
- xxx:/opt

postgresql persist data: which is better named volume or bind mount

Option 1: (named container. the volume is identified by its name. It store its data in the /var/lib/docker/volumes/nameofthevolume)
# create the volume in advance
$ docker volume create test_vol
Option: 2 (here name of the volume bind-test does not matter, what matter is which local path /home/user/test it mounts to, which is persistant. Rather than /var/lib/docker/volume/somevolumename /home/user/somedatafolder makes more readability. Cons: we have to ensure that the /home/user/somedatafolder exists.)
# inside a docker-compose file
...
volumes:
bind-test:
driver: local
driver_opts:
type: none
o: bind
device: /home/user/test
or:
version: '3'
services:
myservice:
volumes:
- ./path:/volume/path
The downside of bind mounts is that it places files that are managed by containers, with the uid/gid from the container, inside a path likely used by other users on the host, often with a different uid/gid on the host. The result is permission issues either on the host or inside the container. You need to align uid/gid's between the two to avoid this.
At the end of the day, there isn't a big difference between bind mount and Docker named volumes.
I tend to prefer keeping persistent data from Docker services in Docker volumes. You can then use tools like docker system df -v to inspect what your application uses.
As for exporting the data, you can use docker cp
docker cp someContainer:/somedir/ .

When are docker-compose on-the-fly volumes reused vs. recreated?

I have a docker-compose.yml like this:
version: '2'
services:
app:
build: .
volumes:
- /usr/src/app
If I do docker-compose up, then any changes I make to the /usr/src/app are persisted across runs. I can control+C and then docker-compose up, and the contents are still there.
But if I do docker-compose run app ls -la /usr/src/app, then the path is always empty.
My goal is that I'd like to have that volume 1) automatically created on the fly for me, 2) specific to this docker-compose project (since I'll have many others), and 3) persist across docker-compose up/run/etc.
I think one way around this is to use named volumes, which will automatically pull the name of my docker-compose project.
But with on-the-fly containers, is this the expected behavior? They persist automatically for docker-compose up, and are recreated from scratch for each docker-compose run?
Also, is there any documentation that makes clear the lifetime of on-the-fly containers?
Thanks!

How to run a command once in Docker compose

So I'm working on a docker compose file to deploy my Go web server. My server uses mongo, so I added a data volume container and the mongo service in docker compose.
Then I wrote a Dockerfile in order to build my Go project, and finally run it.
However, there is another step that must be done. Once my project has been compiled, I have to run the following command:
./my-project -setup
This will add some necessary information to the database, and the information only needs to be added once.
I can't however add this step on the Dockerfile (in the build process) because mongo must already be started.
So, how can I achieve this? Even if I restart the server and then run again docker-compose up I don't want this command to be executed again.
I think I'm missing some Docker understanding, because I don't actually understand everything about data volume containers (are they just stopped containers that mount a volume?).
Also, if I restart the server, and then run docker-compose up, which commands will be run? Will it just start the same container that was now stopped with the given CMD?
In any case, here is my docker-compose.yml:
version: '2'
services:
mongodata:
image: mongo:latest
volumes:
- /data/db
command: --break-mongo
mongo:
image: mongo:latest
volumes_from:
- mongodata
ports:
- "28001:27017"
command: --smallfiles --rest --auth
my_project:
build: .
ports:
- "6060:8080"
depends_on:
- mongo
- mongodata
links:
- mongo
And here is my Dockerfile to build my project image:
FROM golang
ADD . /go/src/my_project
RUN cd /go/src/my_project && go get
RUN go install my_project
RUN my_project -setup
ENTRYPOINT /go/bin/my_project
EXPOSE 8080
I suggest to add an entrypoint-script to your container; in this entrypoint-script, you can check if the database has been initialized, and if it isn't, perform the required steps.
As you noticed in your question, the order in which services / containers are started should not be taken for granted, so it's possible your application container is started before the database container, so the script should take that into account.
As an example, have a look at the official WordPress image, which performs a one-time initialization of the database in it's entrypoint-script. The script attempts to connect to the database (and retries if the database cannot be contacted (yet)), and checks if initialization is needed; https://github.com/docker-library/wordpress/blob/df190dc9c5752fd09317d836bd2bdcd09ee379a5/apache/docker-entrypoint.sh#L146-L171
NOTE
I notice you created a "data-only container" to attach your volume to. Since docker 1.9, docker has volume management, including naming volumes. Because of this, you no longer need to use "data-only" containers.
You can remove the data-only container from your compose file, and change your mongo service to look something like this;
mongo:
image: mongo:latest
volumes:
- mongodata:/data/db
ports:
- "28001:27017"
command: --smallfiles --rest --auth
This should create a new volume, named mongodata if it doesn't exist, or re-use the existing volume with that name. You can list all volumes using docker volume ls and remove a volume with docker volume rm <some-volume> if you no longer need it
You could try to use ONBUILD instruction:
The ONBUILD instruction adds to the image a trigger instruction to be executed at a later time, when the image is used as the base for another build. The trigger will be executed in the context of the downstream build, as if it had been inserted immediately after the FROM instruction in the downstream Dockerfile.
Any build instruction can be registered as a trigger.
This is useful if you are building an image which will be used as a base to build other images, for example an application build environment or a daemon which may be customized with user-specific configuration.
For example, if your image is a reusable Python application builder, it will require application source code to be added in a particular directory, and it might require a build script to be called after that. You can’t just call ADD and RUN now, because you don’t yet have access to the application source code, and it will be different for each application build. You could simply provide application developers with a boilerplate Dockerfile to copy-paste into their application, but that is inefficient, error-prone and difficult to update because it mixes with application-specific code.
The solution is to use ONBUILD to register advance instructions to run later, during the next build stage.
Here’s how it works:
When it encounters an ONBUILD instruction, the builder adds a trigger to the metadata of the image being built. The instruction does not otherwise affect the current build.
At the end of the build, a list of all triggers is stored in the image manifest, under the key OnBuild. They can be inspected with the docker inspect command.
Later the image may be used as a base for a new build, using the FROM instruction. As part of processing the FROM instruction, the downstream builder looks for ONBUILD triggers, and executes them in the same order they were registered. If any of the triggers fail, the FROM instruction is aborted which in turn causes the build to fail. If all triggers succeed, the FROM instruction completes and the build continues as usual.
Triggers are cleared from the final image after being executed. In other words they are not inherited by “grand-children” builds.
In docker-compose you can define:
restart: no
To run the container only once, which is useful for example for db-migration containers.
Your application need some initial state for working. It means that you should:
Check if required state already exists
Depends on first step result init state or not
You can write program for checking current database state (here I will use bash script but it can be every other language program):
RUN if $(./check.sh); then my_project -setup; fi
In my case if script will return 0 (success exit status) then setup command will be called.