I am playing around with Azure DevOps container jobs and service containers. My use case is as follows, I (unfortunately) have to do everything on Private Hosted Build agents.
I am running my job as a container job in Container A.
I have specific software installed (Fortify), which uses commandline, on Container B
Basically I want one of the steps running on container A to be run in Container B (to do the fortify scan, using the code from the workspace). Of course I could do it in a separate job, but I'd prefer to do it in the same job.
Any ideas if this is possible at the moment?
Thanks
Cool, I just read that this feature will be available in the sprint 163 release!
https://learn.microsoft.com/en-us/azure/devops/release-notes/2020/sprint-163-update
resources:
containers:
- container: python
image: python:3.8
- container: node
image: node:13.2
jobs:
- job: example
container: python
steps:
- script: echo Running in the job container
- script: echo Running on the host
target: host
- script: echo Running in another container, in restricted commands mode
target:
container: node
commands: restricted
You can use the Step target to choose which container or host the step will running at.
For example:
resources:
containers:
- container: pycontainer
image: python:3.8
steps:
- task: SampleTask#1
target: host
- task: AnotherTask#1
target: pycontainer
Related
I'm trying to run a container that runs a program until it finishes, as a step in Azure DevOps pipeline Job.
From documentation it looks that what's needed is a service container.
My pipeline yaml is:
trigger:
- main
resources:
containers:
- container: mycontainer
image: mycontainer:latest
endpoint: myregistry
pool:
vmImage: ubuntu-latest
services:
syncice: mycontainer
steps:
- script: |
ls
printenv
When the container is docker run locally the program shows output, but from DevOps Job no output is showing.
How to start the container and see output in Job?
As the title says, i want to setup Airflow that would run on a cluster (1 master, 2 nodes) using Docker swarm.
Current setup:
Right now i have Airflow setup that uses the CeleryExecutor that is running on single EC2.
I have a Dockerfile that pulls Airflow's image and pip install -r requirements.txt.
From this Dockerfile I'm creating a local image and this image is used in the docker-compose.yml that spins up the different services Airflow need (webserver, scheduler, redis, flower and some worker. metadb is Postgres that is on a separate RDS).
The docker-compose is used in docker swarm mode ie. docker stack deploy . airflow_stack
Required Setup:
I want to scale the current setup to 3 EC2s (1 master, 2 nodes) that the master would run the webserver, schedule, redis and flower and the workers would run in the nodes.
After searching and web and docs, there are a few things that are still not clear to me that I would love to know
from what i understand, in order for the nodes to run the workers, the local image that I'm building from the Dockerfile need to be pushed to some repository (if it's really needed, i would use AWS ECR) for the airflow workers to be able to create the containers from that image. is that correct?
syncing volumes and env files, right now, I'm mounting the volumes and insert the envs in the docker-compose file. would these mounts and envs be synced to the nodes (and airflow workers containers)? if not, how can make sure that everything is sync as airflow requires that all the components (apart from redis) would have all the dependencies, etc.
one of the envs that needs to be set when using a CeleryExecuter is the broker_url, how can i make sure that the nodes recognize the redis broker that is on the master
I'm sure that there are a few more things that i forget, but what i wrote is a good start.
Any help or recommendation would be greatly appreciated
Thanks!
Dockerfile:
FROM apache/airflow:2.1.3-python3.9
USER root
RUN apt update;
RUN apt -y install build-essential;
USER airflow
COPY requirements.txt requirements.txt
COPY requirements.airflow.txt requirements.airflow.txt
RUN pip install --upgrade pip;
RUN pip install --upgrade wheel;
RUN pip install -r requirements.airflow.txt
RUN pip install -r requirements.txt
EXPOSE 8793 8786 8787
docker-compose.yml:
version: '3.8'
x-airflow-celery: &airflow-celery
image: local_image:latest
volumes:
-some_volume
env_file:
-some_env_file
services:
webserver:
<<: *airflow-celery
command: airflow webserver
restart: always
ports:
- 80:8080
healthcheck:
test: [ "CMD-SHELL", "[ -f /opt/airflow/airflow-webserver.pid ]" ]
interval: 10s
timeout: 30s
retries: 3
scheduler:
<<: *airflow-celery
command: airflow scheduler
restart: always
deploy:
replicas: 2
redis:
image: redis:6.0
command: redis-server --include /redis.conf
healthcheck:
test: [ "CMD", "redis-cli", "ping" ]
interval: 30s
timeout: 10s
retries: 5
ports:
- 6379:6379
environment:
- REDIS_PORT=6379
worker:
<<: *airflow-celery
command: airflow celery worker
deploy:
replicas: 16
flower:
<<: *airflow-celery
command: airflow celery flower
ports:
- 5555:5555
Sounds like you are heading in the right direction (with one general comment at the end though).
Yes, you need to push image to container registry and refer to it via public (or private if you authenticate) tag. The tag in this case is usally the registry/name:tag. For example you can see one of the CI images of Airlfow here: https://github.com/apache/airflow/pkgs/container/airflow%2Fmain%2Fci%2Fpython3.9 - the purpose is a bit different (we use it for our CI builds) but the mechanism is the same: you build it locally, tag with the "registry/image:tag" docker build . --tag registry/image:tag and run docker push registry/image:tag.
Then whenever you refer to it from your docker compose, via registry/image:tag, docker compose/swarm will pull the right image. Just make sure you make unique TAGs when you build your images to know which image you push (and account for future images).
Env files should be fine and they will distribute across the instances, but locally mounted volumes will not. You either need to have some shared filesystem (like NFS, maybe EFS if you use AWS) where the DAGs are stored, or use some other synchronization method to distribute the DAGs. It can be for example git-sync - which has very nice properties especially if you use Git to store the DAG files, or baking DAGs into the image (which requires to re-push images when they change). You can see different options explained in our Helm Chart https://airflow.apache.org/docs/helm-chart/stable/manage-dags-files.html
You cannot use localhost you need to set it to a specific host and make sure your broker URL is reachable from all instances. This can be done either by assining specific IP address/DNS name to your 'broker' instance and opening up the right ports in firewalls (make sure you control where you can reach thsoe ports from) and maybe even employing some load-balancing.
I do not know DockerSwarm well enough how difficult or easy it is to set it all up, but nonestly, that's kind of a lot of work - it seems - to do it all manually.
I would strongly, really strongly encourage you to use Kubernetes and the Helm Chart which Airlfow community develops: https://airflow.apache.org/docs/helm-chart/stable/index.html . There a lot of issues and necessary configurations either solved in the K8S (scaling, shared filesystems - PVs, networking and connectiviy, resource management etc. etc.) or by our Helm (Git-Sync side containers, broker configuration etc.)
I run Airflow CeleryExecutor on Docker Swarm.
So assuming that you have Docker Swarm set up on your nodes, here are a few things you can do:
Map shared volumes to NFS folders like this (same for plugins and logs, or anything else you need to share)
volumes:
dags:
driver_opts:
type: "none"
o: "bind"
device: "/nfs/airflow/dags"
I personally use Docker Secrets to handle my webserver password, database password, etc. (similarly, I use Docker configs to pass in my celery and webserver config)
secrets:
postgresql_password:
external: true
fernet_key:
external: true
webserver_password:
external: true
To have Airflow read the secrets, I added a simple bash script that gets added to the entrypoint.sh script. So in my stack file I do not need to hardcode any passwords, but if the DOCKER-SECRET string is available, then it will look in /run/secrets/ (I think I used this as an example when setting it up https://gist.github.com/bvis/b78c1e0841cfd2437f03e20c1ee059fe)
In my entrypoint script I add the script that expands Docker Secrets:
source /env_secrets_expand.sh
x-airflow-variables: &airflow-variables
AIRFLOW__CORE__EXECUTOR: CeleryExecutor
...
AIRFLOW__WEBSERVER__SECRET_KEY: DOCKER-SECRET->webserver_secret_key
This is how the postgres image is set up as well, without environment variables:
services:
postgres:
image: postgres:11.5
secrets:
- source: postgresql_password
target: /run/secrets/postgresql_password
environment:
- POSTGRES_USER=airflow
- POSTGRES_DB=airflow
- POSTGRES_PASSWORD_FILE=/run/secrets/postgresql_password
You can obviously use Swarm labels or hostnames to determine which nodes a certain service should run
scheduler:
<<: *airflow-common
environment: *airflow-variables
command: scheduler
deploy:
replicas: 2
mode: replicated
placement:
constraints:
- node.labels.type == worker
restart_policy:
condition: on-failure
delay: 5s
max_attempts: 3
window: 120s
logging:
driver: fluentd
options:
tag: docker.airflow.scheduler
fluentd-async-connect: "true"
And for Celery workers, I have my default queue and then a special queue which is pinned to a single node for historical reasons (clients have white listed this specific IP address so I need to ensure that tasks only run on that node). So my entrypoint runs exec airflow celery "$#" -q "$QUEUE_NAME", and my stack file is like this:
worker_default:
<<: *airflow-common
environment:
<<: *airflow-variables
QUEUE_NAME: default
command: worker
deploy:
replicas: 3
mode: replicated
placement:
constraints:
- node.labels.type == worker
worker_nodename:
<<: *airflow-common
environment:
<<: *airflow-variables
QUEUE_NAME: nodename
command: worker
deploy:
replicas: 1
mode: replicated
placement:
constraints:
- node.hostname == nodename
I use Gitlab CI/CD to deploy my DAGs/plugins whenever I merge to main, and to build the images and deploy the services if I update the Dockerfile or other certain files. I have been running Airflow this way for a few years now (2017 or 2018) but I do plan on switching to Kubernetes eventually since that seems like the more standard approach.
I manage to use them fine as long as I don't need to pass custom arguments.
Lets say I want to use an official Docker image: somePublicImage:1.2.3; then the following works fine:
stages:
- stage: Build
jobs:
- job: BuildTestPack
displayName: 'Build, test & pack'
timeoutInMinutes: 5
cancelTimeoutInMinutes: 2
services:
someService:
image: somePublicImage:1.2.3
ports:
- 4223:4222
There's an option to configure the container with --foo bar
How do I define this in a Azure build pipeline?
I've tried:
command
options
arguments
entrypoint
Service containers must define a CMD or ENTRYPOINT. The pipeline will docker run the provided container without additional arguments.
Check the link below:
https://learn.microsoft.com/en-us/azure/devops/pipelines/process/service-containers?view=azure-devops&tabs=yaml
Seems like you need to create a "custom" resource container first. E.g
resources:
containers:
- container: myThing
image: somePublicImage:1.2.3
ports:
- 4223:4222
volumes:
- /docker_vol_config:/config
command: '--foo bar'
which then can be used as a service:
stages:
- stage: Build
jobs:
- job: BuildTestPack
displayName: 'Build, test & pack'
timeoutInMinutes: 5
cancelTimeoutInMinutes: 2
services:
myThing:myThing
I have a small cloudbuild.yaml file where I build a Docker image, push it to Google container registry (GCR) and then apply the changes to my Kubernetes cluster. It looks like this:
steps:
- name: 'gcr.io/cloud-builders/docker'
entrypoint: 'bash'
args: [
'-c',
'docker pull gcr.io/$PROJECT_ID/frontend:latest || exit 0'
]
- name: "gcr.io/cloud-builders/docker"
args:
[
"build",
"-f",
"./services/frontend/prod.Dockerfile",
"-t",
"gcr.io/$PROJECT_ID/frontend:$REVISION_ID",
"-t",
"gcr.io/$PROJECT_ID/frontend:latest",
".",
]
- name: "gcr.io/cloud-builders/docker"
args: ["push", "gcr.io/$PROJECT_ID/frontend"]
- name: "gcr.io/cloud-builders/kubectl"
args: ["apply", "-f", "kubernetes/gcp/frontend.yaml"]
env:
- "CLOUDSDK_COMPUTE_ZONE=europe-west3-a"
- "CLOUDSDK_CONTAINER_CLUSTER=cents-ideas"
- name: "gcr.io/cloud-builders/kubectl"
args: ["rollout", "restart", "deployment/frontend-deployment"]
env:
- "CLOUDSDK_COMPUTE_ZONE=europe-west3-a"
- "CLOUDSDK_CONTAINER_CLUSTER=cents-ideas"
The build runs smoothly, until the last step. args: ["rollout", "restart", "deployment/frontend-deployment"]. It has the following log output:
Already have image (with digest): gcr.io/cloud-builders/kubectl
Running: gcloud container clusters get-credentials --project="cents-ideas" --zone="europe-west3-a" "cents-ideas"
Fetching cluster endpoint and auth data.
kubeconfig entry generated for cents-ideas.
Running: kubectl rollout restart deployment/frontend-deployment
error: unknown command "restart deployment/frontend-deployment"
See 'kubectl rollout -h' for help and examples.
Allegedly, restart is an unknown command. But it works when I run kubectl rollout restart deployment/frontend-deployment manually.
How can I fix this problem?
Looking at the Kubernetes release notes, the kubectl rollout restart commmand was introduced in the v1.15 version. In your case, it seems Cloud Build is using an older version where this command wasn't implemented yet.
After doing some test, it appears Cloud Build uses a kubectl client version depending on the cluster's server version. For example, when running the following build:
steps:
- name: "gcr.io/cloud-builders/kubectl"
args: ["version"]
env:
- "CLOUDSDK_COMPUTE_ZONE=<cluster_zone>"
- "CLOUDSDK_CONTAINER_CLUSTER=<cluster_name>"
if the cluster's master version is v1.14, Cloud Build uses a v1.14 kubectl client and returns the same unknown command "restart" error message. When master's version is v1.15, Cloud Build uses a v1.15 kubectl client and the command runs successfully.
So about your case, I suspect your cluster "cents-ideas" master version is <1.15 which would explain the error you're getting. As per why it works when you run the command manually (I understand locally), I suspect your kubectl may be authenticated to another cluster with master version >=1.15.
In Concourse CI, by default, the underlying container for a job's task is instantiated and run with user root.
If the container used for my task needs to be executed with a different user (e.g. postgres), how can I do that in Concourse?
Concourse tasks provide a user parameter to explicitly set the user to run its container as.
See http://concourse-ci.org/running-tasks.html#task-run-user .
Here is a sample Concourse pipeline to demonstrate the use of that parameter:
---
jobs:
- name: check-container-user
plan:
- do:
- task: container-user-postgres
config:
platform: linux
image_resource:
type: docker-image
source:
repository: postgres
tag: "latest"
run:
user: postgres
path: sh
args:
- -exc
- |
whoami
echo "Container running with postgres user"