I am assessing the migration of my current Airflow deployment from Celery executor to Kubernetes (K8s) executor to leverage the dynamic allocation of resources and the isolation of tasks provided by pods.
It is clear to me that we can use the native KubernetesPodOperator to run tasks on a K8s cluster via the K8s executor. However I couldn't find info about the compatibility between the K8s executor with other operators, such as bash and athena.
Here is the question is it possible to run a bash (or any other) operator on a K8s powered Airflow or I should migrate all my tasks to the KubernetesPodOperator?
Thanks!
Kubernetes executor will work with all operators.
Using the kubernetes executor will create a worker pod for every task instead of using the celery worker as the celery executor will.
Using the KubernetesPodOperator will pull any specific image to launch a pod and execute your task.
So if you are to use the KubernetesPodOperator with the KubernetesExecutor, Airflow will launch a worker pod for your task, and that task will launch a pod and monitor its execution. 2 pods for 1 task.
If you use a BashOperator with the KubernetesExecutor, Airflow will launch a worker pod and execute bash commands on that worker pod. 1 pod for 1 task.
Related
I've created a Flink cluster using Session mode on native K8s using the command:
$ ./bin/kubernetes-session.sh -Dkubernetes.cluster-id=my-first-flink-cluster
based on these instructions.
I am able to submit jobs to the cluster and view the Flink UI. However, I noticed that Flink creates a taskmanager pod only when the job is submitted and deletes it right after the job is finished. Previously I tried the same using YARN based deployment on Google Dataproc and with that method the cluster had a taskmanager always running which reduced job start time.
Hence, is there a way to keep a taskmanager pod always running using K8s Flink deployment such that job start time is reduced?
the intention of the native k8s support provided by Flink is to have this active resource allocation (i.e. task slots through new TaskManager instances) in case it is needed. In addition to that, it will allow the shutdown of TaskManager pods if they are not used anymore. That's the behavior you're observing.
What you're looking for is the standalone k8s support. Here, Flink does not try to start new TaskManager pods. The ResourceManager is passive, i.e. it only considers the TaskManagers that are registered. Some outside process (or a user) has to manage TaskManager pods instead. This might lead to jobs failing if there are not enough task slots available.
Best,
Matthias
We decided to run Airflow on Kubernetes. We would like to make use of the power of Kubernetes, but in a balanced way.
We have some very small tasks in our DAGs, for example create a directory. The KubernetesExecutor spins up a pod for every task, this takes long and therefore is overkill for many short tasks.
My question is, is it possible to configure Airflow to spin up a Kubernetes pod for a whole DAG, instead of a pod per task? (Preferably without Celery)
I do not think it is possible to use one pod per DAG, because KubernetesExecutor is designed to request a pod per task:
When a DAG submits a task, the KubernetesExecutor requests a worker pod from the Kubernetes API. The worker pod then runs the task, reports the result, and terminates.
Maybe combining multiple smaller tasks into one is a way to go.
https://airflow.apache.org/docs/apache-airflow/stable/executor/celery_kubernetes.html
The CeleryKubernetes Executor allows you to use the immediate resources of a celery worker or spin up a pod for a task. I haven’t configured this setup but it seems to match your use case.
I'm new to nextflow. We would like to build our workflow using nextflow and have nextflow deploy the workflow to a large mulit-institution Kubernetes cluster that we use.
In this cluster we don't have admin permission, we have a namespace we work in. Also, pods in our cluster have limited resources, but jobs have unlimited resources.
In looking at the documentation for nextflow + kubernetes and it says that the workflow runs under a Kubernetes pod, which raises red flags for me because of the limitation on pods in our cluster.
Is there a way to execute nextflow workflows as a kubernetes jobs instead of a pod? What are my options in this area?
There might be a new feature with nextflow 22.04. Quoting Ben Sherman's nextflow k8s best practices page:
In Nextflow v22.04 and later, the k8s executor can be configured to use Jobs instead of Pods directly.
Based on a conversation on https://gitter.im/nextflow-io/nextflow, Nextflow cannot run a Job, it only supports spawning Pods. One pod is spawned for workflow control, and that Pod spawns pods for the individual tasks.
I am running Airflow with CeleryExecutor and they are package and deployed with helm charts. I have one worker that I can scale up/down by specifying the number of replicas in the yaml file. This worker is running as StatefulSet and by default it uses the airflow queue. Is it possible to create another StatefulSet chart for another worker and specify the queue name? This way for the DAG definition files, I can specify the queue to use as well.
In your use case it would be better to use Kubernetes Executor instead of splitting the worker management logic between StatefulSets.
The kubernetes executor is introduced in Apache Airflow 1.10.0. The
Kubernetes executor will create a new pod for every task instance.
You can find more details about it in this documentation.
And also I recommend this blog.
I hope it helps.
When running airflow in a single node, can I configure the celeryexecutor without a Celery Backend (redis, rabbitmq, zookeeper) etc.
Currently, since there is no backend infrastcucture available, running webserver and scheduler in single node (Localexecutor)