Airflow Kubernetes / DAGs in different clusters - kubernetes

Question

We have a few applications which span multiple AWS regions. So, rather than having multiple deployments of Airflow to handle our ETL tasks (one in each region), we would like to figure out if there is a way to have workers in different regions/clusters/namespaces.
Our Airflow deployment runs in EKS, so I'm guessing this would maybe be a setting in the KubernetesPodOperator if at all. I also don't see a way to specify a cluster via the DAG but I'm hoping some of the geniuses here may have an idea.
Thanks in advance,
Bill

Answer 1

At the company I work for, we use the KubernetesPodOperator to run in different namespaces.
KubernetesPodOperator has a parameter named 'namespace'. Thanks to this parameter, it can be determined in which namespace the job will run.
write_xcom = KubernetesPodOperator(
namespace='default',
image='alpine',
cmds=["sh", "-c", "mkdir -p /airflow/xcom/;echo '[1,2,3,4]' > /airflow/xcom/return.json"],
name="write-xcom",
do_xcom_push=True,
is_delete_operator_pod=True,
in_cluster=True,
task_id="write-xcom",
get_logs=True,
)
When I searched about working in a different cluster, I saw that KubernetesPodOperator has a parameter named 'config_file' in the last stable version. This value is set to '~/.kube/config' by default.
Link: https://airflow.apache.org/docs/apache-airflow-providers-cncf-kubernetes/stable/operators.html
I haven't tried it before, but it may be possible to work in different clusters with the 'config_file' parameter by creating different config files. I will be following for different better solutions.
I came across an example solution as below
from datetime import datetime, timedelta
from airflow import DAG
from airflow.configuration import conf
from airflow.providers.cncf.kubernetes.operators.kubernetes_pod import KubernetesPodOperator
default_args = {
'owner': 'airflow',
'depends_on_past': False,
'start_date': datetime(2022, 1, 1),
'email_on_failure': False,
'email_on_retry': False,
'retries': 1,
'retry_delay': timedelta(minutes=5),
}
namespace = conf.get('kubernetes', 'NAMESPACE')
# This will detect the default namespace locally and read the
# environment namespace when deployed to Astronomer.
if namespace =='default':
config_file = '/usr/local/airflow/include/.kube/config'
in_cluster = False
else:
in_cluster = True
config_file = None
dag = DAG('example_kubernetes_pod', schedule_interval='#once', default_args=default_args)
with mountain:
KubernetesPodOperator(
namespace=namespace,
image="hello-world",
labels={"<pod-label>": "<label-name>"},
name="airflow-test-pod",
task_id="task-one",
in_cluster=in_cluster, # if set to true, will look in the cluster, if false, looks for file
cluster_context="docker-desktop", # is ignored when in_cluster is set to True
config_file=config_file,
is_delete_operator_pod=True,
get_logs=True,
)
For details, you can refer to this link: https://docs.astronomer.io/software/kubepodoperator-local