i have the following dockerfile for an elixir+phoenix app
FROM elixir:latest as build_base
RUN apt-get -y update
RUN apt-get -y install inotify-tools curl
ARG TARGETARCH
RUN if [ ${TARGETARCH} = arm64 ]; then \
curl -L -o /tmp/dart-sass.tar.gz https://github.com/sass/dart-sass/releases/download/1.54.5/dart-sass-1.54.5-linux-${TARGETARCH}.tar.gz \
;else \
curl -L -o /tmp/dart-sass.tar.gz https://github.com/sass/dart-sass/releases/download/1.54.5/dart-sass-1.54.5-linux-x64.tar.gz \
;fi
RUN tar -xvf /tmp/dart-sass.tar.gz -C /tmp
RUN mv /tmp/dart-sass/sass /usr/local/bin/sass
RUN mkdir -p /app
WORKDIR /app
COPY mix.* ./
RUN mix local.hex --force
RUN mix archive.install hex phx_new --force
RUN mix local.rebar --force
RUN mix deps.clean --all
RUN mix deps.get
RUN mix --version
RUN mix deps.compile
COPY assets assets
COPY vendor vendor
COPY lib lib
COPY config config
COPY priv priv
COPY test test
RUN mix compile
the docker-compose file looks like the following
services:
web:
build:
context: .
dockerfile: Dockerfile
target: build_base
volumes:
- ./:/app
ports:
- "80:80"
command: mix phx.server
I'm trying to run docker-compose as part of the build step in buildkite, this is an extract of the step in buildkite
- label: "run web"
key: "web"
commands:
- mix phx.server
plugins:
- docker-compose#v4.9.0:
run: web
config: docker-compose.yml
however when running web i see everything happens properly including the package installation, however when running the application i see the following error
web_1 | Unchecked dependencies for environment dev:
web_1 | * telemetry_metrics (Hex package)
web_1 | the dependency is not available, run "mix deps.get"
and the list goes on and on, this works fine on my local machine, its only when running on buildkite. does anyone have any idea on how to fix this ?
Related
I have a small python app developed with docker containers.
My setup is:
Dockerfile
FROM python:3
ARG http_proxy
ARG https_proxy
ENV http_proxy ${http_proxy}
ENV https_proxy ${https_proxy}
ENV VIRTUAL_ENV=/opt/venv
ENV PATH="$VIRTUAL_ENV/bin:$PATH"
RUN apt-get update
RUN apt install -y vim screen
RUN \
echo 'alias py="/opt/venv/bin/python"' >> /root/.bashrc && \
echo 'alias ls="ls --color=auto"' >> /root/.bashrc && \
echo 'PS1="\u#\h:\[\e[33m\]\w\[\e[0m\]\$ "' >> /root/.bashrc
RUN python3 -m venv $VIRTUAL_ENV
WORKDIR /app
COPY requirements.txt ./
RUN pip install --no-cache-dir -r requirements.txt
docker-compose.yml
version: '3.8'
x-dev:
&proxy_conf
http_proxy: "${HTTP_PROXY}"
https_proxy: "${HTTPS_PROXY}"
services:
py_service:
container_name: ${APP_NAME}
build:
context: .
dockerfile: Dockerfile
args: *proxy_conf
image: ${APP_NAME}_img
volumes:
- '.:/app'
restart: always
command: tail -f /dev/null
.env
HTTP_PROXY=<http_proxy_server_here>
HTTPS_PROXY=<https_proxy_server_here>
APP_NAME=python_app
The problem is if the proxy server has changed i need to rebuild the image and i don't want that(as a last result maybe i will do it).
What i'm trying to do is change the proxy environment variables inside the container but i don't find the file where the env is stored.
The container OS version is:
[root#5b1b77079e10 ~ >>>] $ cat /etc/os-release
PRETTY_NAME="Debian GNU/Linux 10 (buster)"
NAME="Debian GNU/Linux"
VERSION_ID="10"
VERSION="10 (buster)"
VERSION_CODENAME=buster
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"
You should only need to recreate containers, not rebuild the image. I assume you are doing something like this to get everything up initially:
docker-compose build
docker-compose up -d
Then I assume you are updating you .env file, once you do that you should be able to just do the following for your container to pick up the change:
docker-compose down
docker-compose up -d
You should not need to do a docker-compose build again.
I have the following docker file:
FROM mcr.microsoft.com/dotnet/core/sdk:3.1-buster AS build
WORKDIR /src
COPY ["myapp.database/src/myapp.database.csproj", "myapp.database/"]
COPY ["myapp.database/src/NuGet.config", "myapp.database/"]
RUN dotnet restore "myapp.database/myapp.database.csproj"
COPY myapp.database/src myapp.database
WORKDIR /src/myapp.database
RUN dotnet build "myapp.database.csproj" -c Release -o /app/build
FROM build AS publish
RUN dotnet publish "myapp.database.csproj" -c Release -o /app/publish
FROM build AS final
RUN groupadd -g 500 dotnetuser && \
useradd -r -u 500 -g dotnetuser dotnetuser
RUN dotnet tool install --global dotnet-ef --version 3.1.0
ENV PATH="${PATH}:/root/.dotnet/tools"
WORKDIR /app
COPY --from=publish /app/publish .
ENTRYPOINT ["dotnet", "myapp.database.dll"]
I create a user "dotnetuser" with uid as 500. I specify the following in my docker-compose:
version: '3.4'
services:
postgres:
image: postgres:12.1-alpine
ports:
- "5432:5432"
environment:
POSTGRES_USER: myapp
POSTGRES_PASSWORD: myapp
POSTGRES_DB: myapp
volumes:
- postgresvolume:/var/lib/postgresql/data
networks:
- dockercomposedatabase_default
myapp.database:
depends_on:
- postgres
user: "500"
build:
context: ..
dockerfile: myapp.database/build/Dockerfile
environment:
DOTNET_CLI_HOME: "/tmp/DOTNET_CLI_HOME"
networks:
- dockercomposedatabase_default
volumes:
postgresvolume:
external: false
networks:
dockercomposedatabase_default:
external: true
However, I can only run EF commands from my container if I run the container as root.
If I run as dotnetuser, then I get the following error:
Could not execute because the specified command or file was not found.
Possible reasons for this include:
* You misspelled a built-in dotnet command.
* You intended to execute a .NET Core program, but dotnet-ef does not exist.
* You intended to run a global tool, but a dotnet-prefixed executable with this name could not be found on the PATH.
I've tried various ways to get the dotnetuser to run ef commands as non-root, but without any success :(
Even if I install the tools to the dotnetuser home path, I still get issues with permissions.
How can I run dotnet ef database update if I run the docker as non-root?
It turns out that what I wanted to do wasn't possible. That is because EF will try to build even if you ship the compiled binaries. On immutable images, you won't be able to execute the EF tools. https://github.com/dotnet/efcore/issues/13339
However, there is a workaround I found on this Blog mentioned on the github issue: https://mattmillican.com/blog/ef-core-migrations-on-linux
Of course in my case, things were a little different. I changed my docker and created a migration.sh script.
FROM mcr.microsoft.com/dotnet/core/sdk:3.1-buster AS build
WORKDIR /src
COPY ["myapp.database/src/myapp.database.csproj", "myapp.database/"]
COPY ["myapp.database/src/NuGet.config", "myapp.database/"]
COPY ["myapp.database/src/migration.sh", "myapp.database/"]
RUN dotnet restore "myapp.database/myapp.database.csproj"
COPY myapp.database/src myapp.database
WORKDIR /src/myapp.database
RUN dotnet build "myapp.database.csproj" -c Release -o /app/build
FROM build AS publish
RUN dotnet publish "myapp.database.csproj" -c Release -o /app/publish
RUN chmod +x /app/publish/migration.sh
FROM build AS final
RUN groupadd -g 500 dotnetuser && \
useradd -r -u 500 -g dotnetuser dotnetuser
RUN mkdir /tools
RUN dotnet tool install dotnet-ef --tool-path /tools --version 3.1.0
ENV PATH="${PATH}:/tools:/app"
WORKDIR /app
COPY --chown=dotnetuser:dotnetuser --from=publish /app/publish .
RUN rm -r /src
RUN chown dotnetuser:dotnetuser -R /tools
RUN chown dotnetuser:dotnetuser -R /app
USER dotnetuser
ENTRYPOINT ["migration.sh"]
I used my migration.sh script to execute the migration and then execute my myapp.database.dll to run my fixtures.
#!/bin/sh
# MIGRATION_NAME is an environmental variable for specifying the which migration to migrate to. It will allow you to migrate forward and back. Don't set the environmental value if you want to migrate to the latest migration.
# This path is important but may change between upgrades!!!
EF_DLL_PATH=/tools/.store/dotnet-ef/3.1.0/dotnet-ef/3.1.0/tools/netcoreapp3.1/any/tools/netcoreapp2.0/any/ef.dll
# standard compiled files for dotnet
DEPS_FILE=myapp.database.deps.json
RUNTIME_CONFIG=myapp.database.runtimeconfig.json
# assembly name
PROJECT_NAME=myapp.database
cd /app
echo "Executing the migration script..."
dotnet exec --depsfile ${DEPS_FILE} --runtimeconfig ${RUNTIME_CONFIG} "${EF_DLL_PATH}" database update ${MIGRATION_NAME} --context MyAppDbContext --assembly ${PROJECT_NAME}.dll --startup-assembly ${PROJECT_NAME}.dll --root-namespace ${PROJECT_NAME} || {
echo "Could not execute the migration!"
exit 1
}
echo "migration to ${MIGRATION} complete!"
echo "Run the fixtures..."
dotnet myapp.database.dll
echo "Fixtures applied!"
Hope this can helps someone.
I'm trying to create a cache for the following Github action:
name: dockercompose
on: push
jobs:
test:
runs-on: ubuntu-18.04
steps:
- uses: actions/checkout#v1
- name: Cache Docker Compose
id: cache-docker
uses: actions/cache#v1
with:
path: fhe_app/
key: cache-docker
- name: Build the stack
run: docker-compose up -d
working-directory: fhe_app/
whit the following Dockerfile:
FROM tensorflow/tensorflow:nightly-py3
# set work directory
WORKDIR /usr/src/app
# set environment variables
ENV PYTHONDONTWRITEBYTECODE 1
ENV PYTHONUNBUFFERED 1
# install dependencies
RUN python3 -m pip install --upgrade pip
COPY local_requirements.txt /usr/src/app/local_requirements.txt
RUN \
apt-get update && \
apt-get -y install python3 postgresql-server-dev-10 gcc python3-dev musl-dev netcat
RUN python3 -m pip install -r local_requirements.txt
# copy entrypoint.sh
COPY entrypoint.sh /usr/src/app/entrypoint.sh
RUN chmod +x entrypoint.sh
# copy project
COPY . /usr/src/app/
# run entrypoint.sh
ENTRYPOINT ["/usr/src/app/entrypoint.sh"]
When pushing to Github, instead of a success message, I get:
Cache Docker Compose
Cache not found for input keys: cache-docker.
And:
Post Cache Docker Compose
Post job cleanup.
/bin/tar -cz -f /home/runner/work/_temp/e13e2694-e020-476d-888e-cb29cb9184b6/cache.tgz -C /home/runner/work/fhe_server/fhe_server/fhe_app .
/bin/tar: ./app: file changed as we read it
##[warning]The process '/bin/tar' failed with exit code 1
I've other yml files not using Docker that are caching properly, so the overall structure of the yml should be fine. Is this the right way to cache docker-compose?
This is my circle.yml:
version: 2
jobs:
build:
working_directory: /app
docker:
- image: docker:stable-git
steps:
- checkout
- setup_remote_docker
- run:
name: Install dependencies
command: |
apk add --no-cache py-pip bash
pip install docker-compose
- run:
name: Start service containers and run tests
command: |
docker-compose -f docker-compose.test.yml up -d db es redis
docker-compose run web bash -c "cd myDir && ./manage.py test"
This works fine in that it brings up my service containers (db, es, redis) and I build a new image for my web container. However, my working code is not inside the freshly built image (so "cd myDir" always fails).
I figure the following lines in my Dockerfile should make my code available when it's built but it appears that it doesn't work like that:
ENV APPLICATION_ROOT /app/
RUN mkdir -p $APPLICATION_ROOT
WORKDIR $APPLICATION_ROOT
ADD . $APPLICATION_ROOT
What am I doing wrong and how can I make my code available inside my test container?
Thanks,
Use COPY, Your Dockerfile should look something like this.
FROM image
COPY . /opt/app
WORKDIR "/opt/app"
(More commands)
ENTRYPOINT
When I run the command psql --version within the railsApp container, I get 9.4.12 and when I run the same within the postgres container, I get 9.6.2. How can I get the versions to match?
I am getting the following error when I try to do a migration on Rails App which does a pg_dump sql import.
pg_dump: server version: 9.6.2; pg_dump version: 9.4.12
pg_dump: aborting because of server version mismatch
rails aborted!
Here's my Docker-compose.yml file:
version: "2.1"
services:
railsApp:
build:
context: ./
ports:
- "3000:3000"
links:
- postgres
volumes:
- .:/app
postgres:
image: postgres:9.6
ports:
- "5432:5432"
volumes:
- ./.postgres:/var/lib/postgresql
The Dockerfile:
FROM ruby:2.3.3
# setup /app as our working directory
RUN mkdir /app
WORKDIR /app
# Replace shell with bash so we can source files
RUN rm /bin/sh && ln -s /bin/bash /bin/sh
# Set debconf to run non-interactively
RUN echo 'debconf debconf/frontend select Noninteractive' | debconf-set-selections
# Install base dependencies
FROM ruby:2.3.3
# setup /app as our working directory
RUN mkdir /app
WORKDIR /app
# Replace shell with bash so we can source files
RUN rm /bin/sh && ln -s /bin/bash /bin/sh
# Set debconf to run non-interactively
RUN echo 'debconf debconf/frontend select Noninteractive' | debconf-set-selections
# Install base dependencies
RUN apt-get update && apt-get install -y -q --no-install-recommends \
apt-transport-https \
build-essential \
ca-certificates \
curl \
git \
libssl-dev \
python \
rsync \
software-properties-common \
wget \
postgresql-client \
graphicsmagick \
&& rm -rf /var/lib/apt/lists/*
# Install node and npm with nvm
RUN curl https://raw.githubusercontent.com/creationix/nvm/v0.33.0/install.sh | bash
ENV NVM_DIR=/root/.nvm
ENV NODE_VERSION v7.2.1
ENV NODE_PATH $NVM_DIR/versions/node/$NODE_VERSION
ENV PATH $NODE_PATH/bin:./node_modules/.bin:$PATH
RUN source $NVM_DIR/nvm.sh \
&& nvm install $NODE_VERSION \
&& nvm alias default $NODE_VERSION \
&& nvm use default
# Install our ruby dependencies
ADD Gemfile Gemfile.lock /app/
RUN bundle install
# copy the rest of our code over
ADD . /app
ENV RAILS_ENV development
ENV SECRET_KEY_BASE a6bdc5f788624f00b68ff82456d94bf81bb50c2e114b2be19af2e6a9b76f9307b11d05af4093395b0471c4141b3cd638356f888e90080f8ae60710f992beba8f
# Expose port 3000 to the Docker host, so we can access it from the outside.
EXPOSE 3000
# Set the default command to run our server on port 3000
CMD ["rails", "server", "-p", "3000", "-b", "0.0.0.0"]
The same issue for me, I used an alternative way to take a dump,
First I access the terminal and run the pd_dump inside docker and copied the file from docker to host.
Below are the commands
docker exec -it <container-id> /bin/bash # accessing docker terminal
pg_dump > ~/dump # taking dump inside the docker
docker cp <container-id>:/root/dump ~/dump #copying the dump files to host
Hope the above solution helps.
The easiest approach is to use the correct postgres version in the docker-compose. Change:
postgres:
image: postgres:9.6
To:
postgres:
image: postgres:9.4.2
All available versions here.