Flink not starting TaskManagers on Kubernetes, job reached global-terminal state - kubernetes

Question

I have deployed a Flink cluster to Kubrnetes and I only see the JobManagers running.
I had Flink running on another Kubernetes cluster where I did a SavePoint using the FlinkDeployment from the Flink Operator. The Savepoint was saved correctly. I then deployed the Flink app to the new Kubernetes cluster and patched the savepointLocationPath in FlinkDeployment.
The Flink pods now log this error
│ WARN org.apache.flink.runtime.dispatcher.StandaloneDispatcher - Ignoring JobGraph submission 'Windchill ESI Post Processing' because the job already reached a globally-terminal state (i.e. FAILED, CANCELED, FINISHED) in a previous execution.
...
│ io.fabric8.kubernetes.client.extended.leaderelection.resourcelock.LockException: Unable to update ConfigMapLock
...
│ Caused by: io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: PUT at: https://10.0.0.1/api/v1/namespaces/post-processing-int2/configmaps/post-processing-cluster-c │
│ onfig-map. Message: Operation cannot be fulfilled on configmaps "post-processing-cluster-config-map": the object has been modified; please apply your changes to the latest version and tr │
│ y again. Received status: Status(apiVersion=v1, code=409, details=StatusDetails(causes=[], group=null, kind=configmaps, name=post-processing-cluster-config-map, retryAfterSeconds=null, u │
│ id=null, additionalProperties={}), kind=Status, message=Operation cannot be fulfilled on configmaps "post-processing-cluster-config-map": the object has been modified; please apply your │
│ changes to the latest version and try again, metadata=ListMeta(_continue=null, remainingItemCount=null, resourceVersion=null, selfLink=null, additionalProperties={}), reason=Conflict, st │
│ atus=Failure, additionalProperties={}).
The ConfgiMap mentioned in the error is present.
My question is how do I start a new TaskManager now? I have numberOfTaskSlots: 4 set. I tried shelling into a JobManager pod and running bin/taskmanager.sh start but this just started a process in the pod which doesn't seem correct to me. I then stopped it.
I am expecting to see a new TaskManager pod start up. Thank you

Answer 1

The clue was in the first line of the logs
WARN org.apache.flink.runtime.dispatcher.StandaloneDispatcher - Ignoring JobGraph submission 'Windchill ESI Post Processing' because the job already reached a globally-terminal state (i.e. FAILED, CANCELED, FINISHED) in a previous execution
My mistake started with this command
kubectl patch flinkdeployment/<name-of-flink-deployment> --type=merge -p '{"spec": {"job": {"state": "suspended", "upgradeMode": "savepoint"}}}'
The problem was the upgradeMode. That should not have been edited and left as last-state. Last state tells the Flink deployment to start from where it left off using the HA state (which is state stored in Azure Blob Storage in my case). savepoint will put the deployment in a FINISHED state and will not start a new TaskManager upon deployment.
Below is the correct edit
kubectl patch flinkdeployment/<name-of-flink-deployment> --type=merge -p '{"spec": {"job": {"state": "suspended"}}}'