In my Kubernetes cluster I am running GitLab-ee 15.8.0 with a GitLab Runner. This runner is configured for a kubernetes executor and I have mounted the /var/run/docker.sock to this runner in the configmap. When running a pipeline which brings up a docker-compose-test.yml, I can see that all pods that exist in kubernetes are starting to crash and are getting restarted. After that I can see that the pipeline is still in the Running state, but nor runner is working on it. The last command the runner executed in the pipeline was: docker-compose -f docker-compose-test.yml up -d.
I expected the pipeline to just bring up the docker containers and run the Laravel tests using the database container and the application container, but instead it messes up the Nginx-Ingress resource.
I am running GitLab-ee:15.8.0 with the gitlab-runner version 15.8.2
Here is the gitlab-ci.yml:
image: docker:20.10.16
services:
- docker:20.10.16-dind
variables:
DOCKER_COMPOSE_CMD: "docker-compose -f docker-compose-test.yml"
stages:
- test
- build
test:
stage: test
script:
- docker-compose --version
- $DOCKER_COMPOSE_CMD down --volumes --remove-orphans
- $DOCKER_COMPOSE_CMD up -d
- $DOCKER_COMPOSE_CMD exec -T -e APP_ENV=testing laravel-api-test ./scripts/wait-for.sh database-test:54321 -t 60 -- echo "Database connection established"
- $DOCKER_COMPOSE_CMD exec -T -e APP_ENV=testing laravel-api-test php artisan passport:keys
- $DOCKER_COMPOSE_CMD exec -T -e APP_ENV=testing laravel-api-test php artisan migrate
- $DOCKER_COMPOSE_CMD exec -T -e APP_ENV=testing laravel-api-test sh -c "vendor/bin/phpunit ./tests $PARAMETERS --coverage-text --colors=never --stderr"
- $DOCKER_COMPOSE_CMD down --volumes --remove-orphans
# only:
# - tags
build:
stage: build
script:
- export IMAGE_TAG=$(echo "$CI_COMMIT_TAG" | awk -F '/' '{print $NF}')
- docker build -t laravel-api:"$IMAGE_TAG" .
- docker login -u "$CONTAINER_REGISTRY_USERNAME" -p "$CONTAINER_REGISTRY_PASSWORD" "$CONTAINER_REGISTRY_URL"
- docker push laravel-api:"$IMAGE_TAG"
only:
- tags
And this is the docker-compose-test.yml that seems to mess things up:
version: "3.7"
services:
laravel-api-test:
build:
args:
user: laravel
uid: 1000
context: .
dockerfile: docker/development/Dockerfile
working_dir: /var/www/
volumes:
- ./:/var/www
ports:
- ${APP_PORT}:9000
networks:
- application
database-test:
image: postgres:15.1-alpine
ports:
- 54321:5432
environment:
POSTGRES_PASSWORD: ${DB_PASSWORD}
POSTGRES_USER: ${DB_USERNAME}
networks:
- application
networks:
application:
driver: bridge
The last thing that is probably relevant is the gitlab-runner config:
apiVersion: v1
kind: ConfigMap
metadata:
name: gitlab-runner-config
namespace: gitlab-runner
data:
config.toml: |-
concurrent = 4
[[runners]]
name = "Runner_1"
url = "https://gitlab.project.com/ci"
token = "my-token"
executor = "kubernetes"
[runners.kubernetes]
namespace = "gitlab-runner"
privileged = true
poll_timeout = 600
cpu_request = "1"
service_cpu_request = "200m"
[[runners.kubernetes.volumes.host_path]]
name = "docker"
mount_path = "/var/run/docker.sock"
host_path = "/var/run/docker.sock"
Finally this the output from the pipeline after it crashed:
Running with gitlab-runner 15.8.2 (4d1ca121)
on Runner_1 eNNz4y9k, system ID: r_y3jEhmF8fN58
Preparing the "kubernetes" executor
00:00
Using Kubernetes namespace: gitlab-runner
Using Kubernetes executor with image docker:20.10.16 ...
Using attach strategy to execute scripts...
Preparing environment
00:04
Waiting for pod gitlab-runner/runner-ennz4y9k-project-117-concurrent-0f24cx to be running, status is Pending
Running on runner-ennz4y9k-project-117-concurrent-0f24cx via gitlab-runner-56cd6f4bb5-zrbd9...
Getting source from Git repository
00:01
Fetching changes with git depth set to 20...
Initialized empty Git repository in /builds/Clients/opus-volvere/laravel-api/.git/
Created fresh repository.
Checking out 3890412c as main...
Skipping Git submodules setup
Executing "step_script" stage of the job script
$ docker-compose --version
Docker Compose version v2.6.0
$ $DOCKER_COMPOSE_CMD down --volumes --remove-orphans
Container laravel-api-database-test-1 Stopping
Container laravel-api-laravel-api-test-1 Stopping
Container laravel-api-database-test-1 Stopping
Container laravel-api-laravel-api-test-1 Stopping
Container laravel-api-database-test-1 Stopped
Container laravel-api-database-test-1 Removing
Container laravel-api-laravel-api-test-1 Stopped
Container laravel-api-laravel-api-test-1 Removing
Container laravel-api-laravel-api-test-1 Removed
Container laravel-api-database-test-1 Removed
Network laravel-api_application Removing
Network laravel-api_application Removed
$ $DOCKER_COMPOSE_CMD up -d
#1 [internal] load build definition from Dockerfile
#1 transferring dockerfile: 827B done
#1 DONE 0.1s
#2 [internal] load .dockerignore
#2 transferring context: 88B done
#2 DONE 0.1s
I am not really sure where to look, with log files or anything, so some help regarding the debugging of this issue is really appreciated...
As far as I can see, the only starts when I try to launch the docker compose. I already built the image in the pipeline and that worked like it should, but it start to go wrong when I actually try to run the containers. Maybe that helps? This is just a really annoying problem that isn't my real expertise or anything so I am reading, learning and trying a lot :(
I followed this tutorial on how to add a gitlab runner to kubernetes. Maybe is has something to do with the fact that it tries to create a new pod for the pipeline, because the tuotial I sent says:
The second is a ServiceAccount, Role, and RoleBinding to give the
Runner the privileges to add new Pods to the Namespace.
Again, I am not familiar with all this stuff, so for me its also a shot in the dark, but I really want this fixed so I can continue working on this project
What could cause this GitLab pipeline to crash my entire kubernetes?
Never expose the host container runtime to workload inside the cluster.
This can lead to the situation that the GitLab runner "cleans up" and removes the containers that operate your cluster components.
In addition to that you get tied to a specific container runtime which should be an implementation detail of your cluster.
As an alternative you can use docker-in-docker for the GitLab runner for example.
I have the following compose file:
version: '3'
services:
worker:
ports:
- "8089:8089"
image: locustio/locust
volumes:
- ./:/mnt/locust
command: --config=/mnt/locust/devndb.conf --users 1 --spawn-rate 1 --run-time 10s --host http://devndb-ci:8081 --headless -f /mnt/locust/rest/normId_by_external_identifier.py
When I run the container the --host argument is not evaluated in my locust file.
When I run the locust file directly without a container it works:
locust --headless --users 10 --spawn-rate 1 -H http://devndb-ci:8081 --run-time 10s -f rest/normId_by_external_identifier.py
Is this a bug in the locust docker image?
I am expecting the locust container to take the --host argument into account and build the REST URLs accordingly.
I have a working docker based setup - peer(s), orderers and explorer (db & app) which I am aiming to deployed on GCP - Kubernetes.
For the peer(s) and orderer I have used the docker images and created kubernetes yaml file with (StatefulSet, Service, NodePort and Ingress) to deploy on Kubernetes.
For Explorer I have the below docker-compose file which depends on my local connection-profile and crypto files.
I am struggling to deploy explorer on kubernetes and looking for advice on the approach
I have tried to convert docker-compose using Kompose - but face issues while translating network and health-check tags.
I have tried to create a single docker-image (Dockerfile - multiple FROM tags) from hyperledger/explorer-db:latest and hyperledger/explorer:latest but again specifying network becomes an issue.
Any suggestions or examples on how Explorer can be deployed in the cluster ??
Thanks
Explorer Docker Compose
version: '2.1'
volumes:
pgdata:
walletstore:
networks:
mynetwork.com:
external:
name: my-netywork
services:
explorerdb.mynetwork.com:
image: hyperledger/explorer-db:latest
container_name: explorerdb.mynetwork.com
hostname: explorerdb.mynetwork.com
environment:
- DATABASE_DATABASE=fabricexplorer
- DATABASE_USERNAME=hppoc
- DATABASE_PASSWORD=password
healthcheck:
test: "pg_isready -h localhost -p 5432 -q -U postgres"
interval: 30s
timeout: 10s
retries: 5
volumes:
- pgdata:/var/lib/postgresql/data
networks:
- mynetwork.com
explorer.mynetwork.com:
image: hyperledger/explorer:latest
container_name: explorer.mynetwork.com
hostname: explorer.mynetwork.com
environment:
- DATABASE_HOST=explorerdb.mynetwork.com
- DATABASE_DATABASE=fabricexplorer
- DATABASE_USERNAME=hppoc
- DATABASE_PASSWD=password
- LOG_LEVEL_APP=info
- LOG_LEVEL_DB=info
- LOG_LEVEL_CONSOLE=debug
- LOG_CONSOLE_STDOUT=true
- DISCOVERY_AS_LOCALHOST=false
volumes:
- ./config.json:/opt/explorer/app/platform/fabric/config.json
- ./connection-profile:/opt/explorer/app/platform/fabric/connection-profile
- ../config/crypto-config:/tmp/crypto
- walletstore:/opt/explorer/wallet
ports:
- 8080:8080
depends_on:
explorerdb.mynetwork.com:
condition: service_healthy
networks:
- mynetwork.com
Explorer Dockerfile - multiple froms
# Updated to Fabric 2.x
#1. Docker file for setting up the Orderer
# FROM hyperledger/fabric-orderer:1.4.2
FROM hyperledger/explorer-db:latest
ENV DATABASE_DATABASE=fabricexplorer
ENV DATABASE_USERNAME=hppoc
ENV DATABASE_PASSWORD=password
FROM hyperledger/explorer:latest
COPY ./config/explorer/. /opt/explorer/
COPY ./config/crypto-config/. /tmp/crypto
ENV DATABASE_HOST=explorerdb.xxx.com
ENV DATABASE_DATABASE=fabricexplorer
ENV DATABASE_USERNAME=hppoc
ENV DATABASE_PASSWD=password
ENV LOG_LEVEL_APP=info
ENV LOG_LEVEL_DB=info
ENV LOG_LEVEL_CONSOLE=debug
ENV LOG_CONSOLE_STDOUT=true
ENV DISCOVERY_AS_LOCALHOST=false
ENV DISCOVERY_AS_LOCALHOST=false
# ENV EXPLORER_APP_ROOT=${EXPLORER_APP_ROOT:-dist}
# ENV ${EXPLORER_APP_ROOT}/main.js name - hyperledger-explorer
ENTRYPOINT ["tail", "-f", "/dev/null"]
There are 2 groups of required steps for this setup. One I tested is:
1.Create a K8s cluster
2.Connect your cluster with the cloud shell
3.Clone this repository
git clone https://github.com/acloudfan/HLF-K8s-Cloud.git
4.Setup the storage class
cd HLF-K8s-Cloud/gcp kubectl apply -f . This will setup the storage class
5.Launch the Acme Orderer
cd .. kubectl apply -f ./k8s-acme-orderer.yaml Check the logs for 'acme-orderer-0' to ensure there is no error
6.Launch the Acme Peer
kubectl apply -f ./k8s-acme-peer.yaml Check the logs for 'acme-peer-0' to ensure there is no error
7.Setup the Channel & Join acme peer to it.
kubectl exec -it acme-peer-0 /bin/bash ./submit-channel-create.sh
./join-channel.sh
Ensure that peer has joined the channel
peer channel list
exit
8.Launch the budget Peer and join it to the channel
kubectl apply -f ./k8s-budget-peer.yaml Wait for the container to launch & check the logs for errors
kubectl exec -it budget-peer-0 /bin/bash ./fetch-channel-block.sh ./join-channel.sh
Ensure that peer has joined the channel
peer channel list
exit ** At this point your K8s Fabric Network is up **
Validate the network
1.Install & Instantiate the test chaincode
kubectl exec -it acme-peer-0 /bin/bash
./cc-test.sh install ./cc-test.sh instantiate
2.Invoke | Query the chaincode to see the changes in values of a/b
./cc-test.sh query ./cc-test.sh invoke
3.Check the values inside the Budget peer
kubectl exec -it acme-peer-0 /bin/bash
./cc-test.sh install
./cc-test.sh query The query should return the same values as you see in acme-peer Execute invoke/query in both peers to validate
Plus, you can visit the following threads to see option 2 and more references on the proper steps to set up your environment Production Network with GKE, HLF-K8s-Cloud, Hyperledger Fabric blockchain deployment on Google Kubernetes Engine and hyperledger/fabric-peer.
As the title says, i want to setup Airflow that would run on a cluster (1 master, 2 nodes) using Docker swarm.
Current setup:
Right now i have Airflow setup that uses the CeleryExecutor that is running on single EC2.
I have a Dockerfile that pulls Airflow's image and pip install -r requirements.txt.
From this Dockerfile I'm creating a local image and this image is used in the docker-compose.yml that spins up the different services Airflow need (webserver, scheduler, redis, flower and some worker. metadb is Postgres that is on a separate RDS).
The docker-compose is used in docker swarm mode ie. docker stack deploy . airflow_stack
Required Setup:
I want to scale the current setup to 3 EC2s (1 master, 2 nodes) that the master would run the webserver, schedule, redis and flower and the workers would run in the nodes.
After searching and web and docs, there are a few things that are still not clear to me that I would love to know
from what i understand, in order for the nodes to run the workers, the local image that I'm building from the Dockerfile need to be pushed to some repository (if it's really needed, i would use AWS ECR) for the airflow workers to be able to create the containers from that image. is that correct?
syncing volumes and env files, right now, I'm mounting the volumes and insert the envs in the docker-compose file. would these mounts and envs be synced to the nodes (and airflow workers containers)? if not, how can make sure that everything is sync as airflow requires that all the components (apart from redis) would have all the dependencies, etc.
one of the envs that needs to be set when using a CeleryExecuter is the broker_url, how can i make sure that the nodes recognize the redis broker that is on the master
I'm sure that there are a few more things that i forget, but what i wrote is a good start.
Any help or recommendation would be greatly appreciated
Thanks!
Dockerfile:
FROM apache/airflow:2.1.3-python3.9
USER root
RUN apt update;
RUN apt -y install build-essential;
USER airflow
COPY requirements.txt requirements.txt
COPY requirements.airflow.txt requirements.airflow.txt
RUN pip install --upgrade pip;
RUN pip install --upgrade wheel;
RUN pip install -r requirements.airflow.txt
RUN pip install -r requirements.txt
EXPOSE 8793 8786 8787
docker-compose.yml:
version: '3.8'
x-airflow-celery: &airflow-celery
image: local_image:latest
volumes:
-some_volume
env_file:
-some_env_file
services:
webserver:
<<: *airflow-celery
command: airflow webserver
restart: always
ports:
- 80:8080
healthcheck:
test: [ "CMD-SHELL", "[ -f /opt/airflow/airflow-webserver.pid ]" ]
interval: 10s
timeout: 30s
retries: 3
scheduler:
<<: *airflow-celery
command: airflow scheduler
restart: always
deploy:
replicas: 2
redis:
image: redis:6.0
command: redis-server --include /redis.conf
healthcheck:
test: [ "CMD", "redis-cli", "ping" ]
interval: 30s
timeout: 10s
retries: 5
ports:
- 6379:6379
environment:
- REDIS_PORT=6379
worker:
<<: *airflow-celery
command: airflow celery worker
deploy:
replicas: 16
flower:
<<: *airflow-celery
command: airflow celery flower
ports:
- 5555:5555
Sounds like you are heading in the right direction (with one general comment at the end though).
Yes, you need to push image to container registry and refer to it via public (or private if you authenticate) tag. The tag in this case is usally the registry/name:tag. For example you can see one of the CI images of Airlfow here: https://github.com/apache/airflow/pkgs/container/airflow%2Fmain%2Fci%2Fpython3.9 - the purpose is a bit different (we use it for our CI builds) but the mechanism is the same: you build it locally, tag with the "registry/image:tag" docker build . --tag registry/image:tag and run docker push registry/image:tag.
Then whenever you refer to it from your docker compose, via registry/image:tag, docker compose/swarm will pull the right image. Just make sure you make unique TAGs when you build your images to know which image you push (and account for future images).
Env files should be fine and they will distribute across the instances, but locally mounted volumes will not. You either need to have some shared filesystem (like NFS, maybe EFS if you use AWS) where the DAGs are stored, or use some other synchronization method to distribute the DAGs. It can be for example git-sync - which has very nice properties especially if you use Git to store the DAG files, or baking DAGs into the image (which requires to re-push images when they change). You can see different options explained in our Helm Chart https://airflow.apache.org/docs/helm-chart/stable/manage-dags-files.html
You cannot use localhost you need to set it to a specific host and make sure your broker URL is reachable from all instances. This can be done either by assining specific IP address/DNS name to your 'broker' instance and opening up the right ports in firewalls (make sure you control where you can reach thsoe ports from) and maybe even employing some load-balancing.
I do not know DockerSwarm well enough how difficult or easy it is to set it all up, but nonestly, that's kind of a lot of work - it seems - to do it all manually.
I would strongly, really strongly encourage you to use Kubernetes and the Helm Chart which Airlfow community develops: https://airflow.apache.org/docs/helm-chart/stable/index.html . There a lot of issues and necessary configurations either solved in the K8S (scaling, shared filesystems - PVs, networking and connectiviy, resource management etc. etc.) or by our Helm (Git-Sync side containers, broker configuration etc.)
I run Airflow CeleryExecutor on Docker Swarm.
So assuming that you have Docker Swarm set up on your nodes, here are a few things you can do:
Map shared volumes to NFS folders like this (same for plugins and logs, or anything else you need to share)
volumes:
dags:
driver_opts:
type: "none"
o: "bind"
device: "/nfs/airflow/dags"
I personally use Docker Secrets to handle my webserver password, database password, etc. (similarly, I use Docker configs to pass in my celery and webserver config)
secrets:
postgresql_password:
external: true
fernet_key:
external: true
webserver_password:
external: true
To have Airflow read the secrets, I added a simple bash script that gets added to the entrypoint.sh script. So in my stack file I do not need to hardcode any passwords, but if the DOCKER-SECRET string is available, then it will look in /run/secrets/ (I think I used this as an example when setting it up https://gist.github.com/bvis/b78c1e0841cfd2437f03e20c1ee059fe)
In my entrypoint script I add the script that expands Docker Secrets:
source /env_secrets_expand.sh
x-airflow-variables: &airflow-variables
AIRFLOW__CORE__EXECUTOR: CeleryExecutor
...
AIRFLOW__WEBSERVER__SECRET_KEY: DOCKER-SECRET->webserver_secret_key
This is how the postgres image is set up as well, without environment variables:
services:
postgres:
image: postgres:11.5
secrets:
- source: postgresql_password
target: /run/secrets/postgresql_password
environment:
- POSTGRES_USER=airflow
- POSTGRES_DB=airflow
- POSTGRES_PASSWORD_FILE=/run/secrets/postgresql_password
You can obviously use Swarm labels or hostnames to determine which nodes a certain service should run
scheduler:
<<: *airflow-common
environment: *airflow-variables
command: scheduler
deploy:
replicas: 2
mode: replicated
placement:
constraints:
- node.labels.type == worker
restart_policy:
condition: on-failure
delay: 5s
max_attempts: 3
window: 120s
logging:
driver: fluentd
options:
tag: docker.airflow.scheduler
fluentd-async-connect: "true"
And for Celery workers, I have my default queue and then a special queue which is pinned to a single node for historical reasons (clients have white listed this specific IP address so I need to ensure that tasks only run on that node). So my entrypoint runs exec airflow celery "$#" -q "$QUEUE_NAME", and my stack file is like this:
worker_default:
<<: *airflow-common
environment:
<<: *airflow-variables
QUEUE_NAME: default
command: worker
deploy:
replicas: 3
mode: replicated
placement:
constraints:
- node.labels.type == worker
worker_nodename:
<<: *airflow-common
environment:
<<: *airflow-variables
QUEUE_NAME: nodename
command: worker
deploy:
replicas: 1
mode: replicated
placement:
constraints:
- node.hostname == nodename
I use Gitlab CI/CD to deploy my DAGs/plugins whenever I merge to main, and to build the images and deploy the services if I update the Dockerfile or other certain files. I have been running Airflow this way for a few years now (2017 or 2018) but I do plan on switching to Kubernetes eventually since that seems like the more standard approach.
I have following docker-compose.yaml for a private docker registry to be run on minikube:
version: '3'
services:
registry:
restart: always
image: registry:2
command: ["/bin/sh", "-ec","sleep 1000"]
ports:
- 443:443
environment:
REGISTRY_HTTP_TLS_CERTIFICATE: /certs/domain.crt
REGISTRY_HTTP_TLS_KEY: /certs/domain.key
REGISTRY_AUTH: htpasswd
REGISTRY_AUTH_HTPASSWD_PATH: /auth/htpasswd
REGISTRY_AUTH_HTPASSWD_REALM: Registry Realm
volumes:
- /home/usr/registry/data:/var/lib/registry
- /home/usr/registry/certs:/certs
- /home/usr/registry/auth:/auth
When I do kompose up, the registry should be up and running. But doing docker login localhost:443 only gives me a connection refused error. If I run
docker run -d --restart=always --name registry -v `pwd`/auth:/auth -v `pwd`/certs:/certs -v `pwd`/certs:/certs -e REGISTRY_AUTH=htpasswd -e REGISTRY_AUTH_HTPASSWD_REALM="Registry Realm" -e REGISTRY_AUTH_HTPASSWD_PATH=/auth/htpasswd -e REGISTRY_HTTP_ADDR=0.0.0.0:443 -e REGISTRY_HTTP_TLS_CERTIFICATE=/certs/domain.crt -e REGISTRY_HTTP_TLS_KEY=/certs/domain.key -p 443:443 registry:2
everything works fine and I can log into my private registry.
The reason this is important is that I have a webapp-image that is in the private registry and should be pulled by kubernetes (minikube) from it. However, I always get a CrashLoopBackoff error which I deem due to the fact that the registry cannot be run from kubernetes and incidentally not be accessed by it. What am I getting wrong?
The solution is to set up a registry in minikube and port-forward to from localhost so the image gets pushed onto the minikube regsitry.