Kubectl pods are not created - kubernetes

Question

Trying to create pods in a cluster in GKE. There is a docker container containing some python code with a sidecar container to access the sql database. The deployment worked perfectly previously, however after a few weeks I tried to redeploy with kubectl apply -f file_name.yaml.
The pods got temporarily created with a 'Pending' status and disappeared after 15 seconds. Happens every time. I am unable to access logs. kubectl get pods returns nothing after 15 seconds as well.
Not sure where to go from here... Any help would be appreciated!
apiVersion: v1
kind: Pod
metadata:
annotations:
kubernetes.io/limit-ranger: 'LimitRanger plugin set: cpu request for container
pyxy-web-v1'
creationTimestamp: "2020-05-14T00:38:09Z"
labels:
run: pyxy-web-v1
name: pyxy-web-v1
namespace: default
resourceVersion: "1215073"
selfLink: /api/v1/namespaces/default/pods/pyxy-web-v1
uid: *omitted
spec:
containers:
- image: gcr.io/my-project-{*omitted}/pyxy-web:latest
imagePullPolicy: Always
name: pyxy-web-v1
ports:
- containerPort: 8080
protocol: TCP
env:
- name: DB_USER
valueFrom:
secretKeyRef:
name: cloudsql-db-credentials
key: *omitted
- name: DB_PASS
valueFrom:
secretKeyRef:
name: cloudsql-db-credentials
key: *omitted
resources:
requests:
cpu: 100m
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /var/run/secrets/kubernetes.io/serviceaccount
name: default-token-94bct
readOnly: true
- name: cloudsql-proxy
image: gcr.io/cloudsql-docker/gce-proxy:1.16
command: ["/cloud_sql_proxy",
"-instances=my-project-{*omitted}:us-central1:routing-app-v1=tcp:3306",
# If running on a VPC, the Cloud SQL proxy can connect via Private IP. See:
# https://cloud.google.com/sql/docs/mysql/private-ip for more info.
# "-ip_address_types=PRIVATE",
"-credential_file=/secrets/cloudsql/credentials.json"]
# [START cloudsql_security_context]
securityContext:
runAsUser: 2 # non-root user
allowPrivilegeEscalation: false
volumeMounts:
- name: cloudsql-instance-credentials
mountPath: /secrets/cloudsql
readOnly: true
dnsPolicy: ClusterFirst
enableServiceLinks: true
nodeName: gke-pyxy-cluster-default-pool-{*omitted}
priority: 0
restartPolicy: Always
schedulerName: default-scheduler
securityContext: {}
serviceAccount: default
serviceAccountName: default
terminationGracePeriodSeconds: 180
tolerations:
- effect: NoExecute
key: node.kubernetes.io/not-ready
operator: Exists
tolerationSeconds: 300
- effect: NoExecute
key: node.kubernetes.io/unreachable
operator: Exists
tolerationSeconds: 300
volumes:
- name: default-token-94bct
secret:
defaultMode: 420
secretName: default-token-94bct
- name: cloudsql-instance-credentials
secret:
secretName: cloudsql-instance-credentials
During the 15 seconds-long pending period the kubectl describe pods returns the following.
Name: pyxy-web-v1
Namespace: default
Priority: 0
Node: gke-pyxy-cluster-default-pool-{*omitted}/
Labels: run=pyxy-web-v1
Annotations: kubernetes.io/limit-ranger: LimitRanger plugin set: cpu request for container cloudsql-proxy
Status: Pending
IP:
IPs: <none>
Containers:
pyxy-web-v1:
Image: gcr.io/my-project-{*omitted}/pyxy-web:latest
Port: 8080/TCP
Host Port: 0/TCP
Requests:
cpu: 100m
Environment:
DB_USER: <set to the key '*omitted' in secret 'cloudsql-db-credentials'> Optional: false
DB_PASS: <set to the key '*omitted' in secret 'cloudsql-db-credentials'> Optional: false
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-94bct (ro)
cloudsql-proxy:
Image: gcr.io/cloudsql-docker/gce-proxy:1.16
Port: <none>
Host Port: <none>
Command:
/cloud_sql_proxy
-instances=my-project-{*omitted}:us-central1:routing-app-v1=tcp:3306
-credential_file=/secrets/cloudsql/credentials.json
Requests:
cpu: 100m
Environment: <none>
Mounts:
/secrets/cloudsql from cloudsql-instance-credentials (ro)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-94bct (ro)
Volumes:
default-token-94bct:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-94bct
Optional: false
cloudsql-instance-credentials:
Type: Secret (a volume populated by a Secret)
SecretName: cloudsql-instance-credentials
Optional: false
QoS Class: Burstable
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events: <none>
However after this time, it returns
'No resources found in default namespace.'

Answer 1

Answer
The Pod spec had a Node Name for a Node that was no longer in the cluster (due to an upgrade). That is to say the pod.spec.nodeName was erroneous.
From kubectl explain pod.spec:
nodeName <string>
NodeName is a request to schedule this pod onto a specific node. If it is
non-empty, the scheduler simply schedules this pod onto that node, assuming
that it fits resource requirements.
During the ~15 second window the Pod was in Pending state, the following error log pointed to the solution:
Error from server (NotFound): pods "gke-pyxy-cluster-default-pool-94aa0302-pm35" not found