Google Cloud Composer pulls stale image from Google Container Registry - kubernetes

Question

I'm trying to run an Airflow task through Google Cloud Composer with the KubernetesPodOperator in an environment built from an image in a private Google Container Registry. The Container Registry and Cloud Composer instances are under the same project and everything worked fine until I updated the image the DAG refers too.
When I update the image in the Container Registry, Cloud Composer keeps using a stale image.
Concretely, in the code below
import datetime
import airflow
from airflow.contrib.operators import kubernetes_pod_operator
YESTERDAY = datetime.datetime.now() - datetime.timedelta(days=1)
# Create Airflow DAG the the pipeline
with airflow.DAG(
'my_dag',
schedule_interval=datetime.timedelta(days=1),
start_date=YESTERDAY) as dag:
my_task = kubernetes_pod_operator.KubernetesPodOperator(
task_id='my_task',
name='my_task',
cmds=['echo 0'],
namespace='default',
image=f'gcr.io/<my_private_repository>/<my_image>:latest')
if I update the image gcr.io/<my_private_repository>/<my_image>:latest in the Container Registry, Cloud Composer keeps using the stale image that is not present anymore in the Container Registry and throws an error.
Is this a bug?
Thanks a lot!

Answer 1

As mentioned in the documentation for KubernetesPodOperator, the default value for image_pull_policy is 'IfNotPresent'. You need to configure your Pod Spec to pull image image always.
Simplest way to do it is setting the image_pull_policy to 'Always'.
Few more ways are mentioned in K8s Container Images documentation.