Desired GKE pod not found , google cloud composer - kubernetes

Question

I am using Google cloud composer ,and created composer environment.Composer environment is ready(has green tick), now I am trying to set variables used in DAG python code using google cloud shell.
command to set variables:
gcloud composer environments run test-environment \
--location us-central1 variables -- \
--set gcp_project xxx-gcp
Exact error message:
ERROR: (gcloud.composer.environments.run) Desired GKE pod not found. If the environment was recently started, please wait and retry.
I tried following things as part of investigation, but got same error each time.
I have created a new environment using UI and not google shell commands.
I checked pods in kubernetes engine and all are green , did not see any issue.
I verified composer API, Billing kubernetes, all required API's are enabled.
I have 'Editor' role assigned.
added screenshot I saw first time some failures
Error with exit code 1
google troubleshooting guide describe: If the exit code is 1, the container crashed because the application crashed.

Answer 1

This is a side effect of Composer version 1.6.0 if you are using a google-cloud-sdk that is too old, because it now launches pods in namespaces other than default. The error you see is a result of looking for Kubernetes pods in the default namespace and failing to find them.
To fix this, run gcloud components update. If you cannot yet update, a workaround to execute Airflow commands is to manually SSH to a pod yourself and run airflow. To start, obtain GKE cluster credentials:
$ gcloud container clusters get-credentials $COMPOSER_GKE_CLUSTER_NAME
Once you have the credentials, you should find which namespace the pods are running in (which you can also find using Cloud Console):
$ kubectl get namespaces
NAME STATUS AGE
composer-1-6-0-airflow-1-9-0-6f89fdb7 Active 17h
default Active 17h
kube-public Active 17h
kube-system Active 17h
You can then SSH into any scheduler/worker pod, and run commands:
$ kubectl exec \
--namespace=$NAMESPACE \
-it airflow-worker-569bc59df5-x6jhl airflow list_dags -r
You can also open a shell if you prefer:
$ kubectl exec \
--namespace=$NAMESPACE \
-it airflow-worker-569bc59df5-x6jhl bash
airflow#airflow-worker-569bc59df5-x6jhl:~$ airflow list_dags -r
The failed airflow-database-init-job jobs are unrelated and will not cause problems in your Composer environment.