What is the difference between a resourceVersion and a generation? - kubernetes

In Kubernetes object metadata, there are the concepts of resourceVersion and generation. I understand the notion of resourceVersion: it is an optimistic concurrency control mechanism—it will change with every update. What, then, is generation for?

resourceVersion changes on every write, and is used for optimistic concurrency control
in some objects, generation is incremented by the server as part of persisting writes affecting the spec of an object.
some objects' status fields have an observedGeneration subfield for controllers to persist the generation that was last acted on.

In a Deployment context:
In Short
resourceVersion is the version of a k8s resource, while generation is the version of the deployment which you can use to undo, pause and so on using kubectl cli.
Source code for kubectl rollout: https://github.com/kubernetes/kubectl/blob/master/pkg/cmd/rollout/rollout.go#L50
The Long Version
resourceVersion
K8s server saves all the modifications to any k8s resource. Each modification has a version which is called resourceVersion.
k8s languages libraries provide a way to receive in real-time events of ADD, DELETE, MODIFY events of any resource. You also have BOOKMARK event, but let's leave that for a moment aside.
On any modification operation, you receive the new k8s resource with updated resourceVersion. You can use this resourceVersion and start a watch starting from this resourceVersion, so you won't miss any events between the time the k8s server sent you back the first response, until the watch has started.
K8s doesn't preserve the history for every resource for ever. I think that it will save for 5m, but I'm not sure exactly.
resourceVersion will change after any modification of the object.
The reason of it's existence is to avoid concurrency problems where multiple clients try to modify the same k8s resource. This pattern is pretty common also in databases and you can find more info about it:
Optimistic concurrency control (https://en.wikipedia.org/wiki/Optimistic_concurrency_control)
https://www.programmersought.com/article/1104647506/
observedGeneration
You didn't talk about it in your question but thats important piece of information we need to clarify before moving on to generation.
It is the version of the replicaSet which this deployment is currently tracking on.
When the deployment is still creating for the first time, this value won't exist (good discussion on this can be found here: https://github.com/kubernetes/kubernetes/issues/47871).
This value can be found under status:
....
apiVersion: apps/v1
kind: Deployment
.....
status:
availableReplicas: 1
conditions:
- lastTransitionTime: "2021-02-07T19:04:17Z"
lastUpdateTime: "2021-02-07T19:04:17Z"
message: Deployment has minimum availability.
reason: MinimumReplicasAvailable
status: "True"
type: Available
- lastTransitionTime: "2021-02-07T19:04:15Z"
lastUpdateTime: "2021-02-07T19:17:09Z"
message: ReplicaSet "deployment-bcb437a4-59bb9f6f69" has successfully progressed.
reason: NewReplicaSetAvailable
status: "True"
type: Progressing
observedGeneration: 3. <<<--------------------
readyReplicas: 1
replicas: 1
updatedReplicas: 1
p.s, from this article: https://thenewstack.io/kubernetes-deployments-work/
observedGeneration is equal to the deployment.kubernetes.io/revision annotation. It is the observedGeneration.
It looks correct because deployment.kubernetes.io/revision does not exist when the deployment is first created and not yet ready, and also it has the same value as observedGeneration when the deployment is updated.
generation
It represents the version of the "new" replicaSet which this deployment should track on.
When a deployment is created for the first time, the value of this will be equal to 1. When the observedGeneration will be set to 1, it means that replicate set is ready (This question is not about how to know if a deployment was successful (or not) so I'm not getting into what is "ready", which is some terminology I created for this answer - but be sure that there are additional conditions to check if a deployment was successful or not).
Same goes for any change in the deployment k8s resource which will trigger re-deployment. the generation value will be incremented by 1, and then it will take some time until observedGeneration will be equal to generation value.
More info on observedGeneration and generation in the context of kuebctl rollout status (to check if a deployment "finished") from kubectl source code:
https://github.com/kubernetes/kubectl/blob/a2d36ec6d62f756e72fb3a5f49ed0f720ad0fe83/pkg/polymorphichelpers/rollout_status.go#L75
if deployment.Generation <= deployment.Status.ObservedGeneration {
cond := deploymentutil.GetDeploymentCondition(deployment.Status, appsv1.DeploymentProgressing)
if cond != nil && cond.Reason == deploymentutil.TimedOutReason {
return "", false, fmt.Errorf("deployment %q exceeded its progress deadline", deployment.Name)
}
if deployment.Spec.Replicas != nil && deployment.Status.UpdatedReplicas < *deployment.Spec.Replicas {
return fmt.Sprintf("Waiting for deployment %q rollout to finish: %d out of %d new replicas have been updated...\n", deployment.Name, deployment.Status.UpdatedReplicas, *deployment.Spec.Replicas), false, nil
}
if deployment.Status.Replicas > deployment.Status.UpdatedReplicas {
return fmt.Sprintf("Waiting for deployment %q rollout to finish: %d old replicas are pending termination...\n", deployment.Name, deployment.Status.Replicas-deployment.Status.UpdatedReplicas), false, nil
}
if deployment.Status.AvailableReplicas < deployment.Status.UpdatedReplicas {
return fmt.Sprintf("Waiting for deployment %q rollout to finish: %d of %d updated replicas are available...\n", deployment.Name, deployment.Status.AvailableReplicas, deployment.Status.UpdatedReplicas), false, nil
}
return fmt.Sprintf("deployment %q successfully rolled out\n", deployment.Name), true, nil
}
return fmt.Sprintf("Waiting for deployment spec update to be observed...\n"), false, nil
I must say that I'm not sure when a observedGeneration can be higher than generation. Maybe folks can help me out in the comments.
To sum it all up: Illustration from this great article: https://thenewstack.io/kubernetes-deployments-work/
More Info:
https://kubernetes.slack.com/archives/C2GL57FJ4/p1612651711106700?thread_ts=1612650049.105300&cid=C2GL57FJ4
Some more information about Bookmarks Events (Which are related to resourceVersion: What k8s bookmark solves?
How do you rollback deployments in Kubernete: https://learnk8s.io/kubernetes-rollbacks

Related

Cert-Manager: renewing certificate not working

Folks, am trying to renew certificates for a wildcard domain, and am seeing the following errors when looking at the logs on the certmanager pod, and at the error in the certificaterequest
Message: Waiting on certificate issuance from order
production/certmanager-xxxxxxxxx-pp9n2-3392968554: "pending"
production/cert-manager-877fd747c-4nf2f[cert-manager]: E0817 21:32:34.447585 1
controller.go:166] cert-manager/challenges "msg"="re-queuing item due to error
processing" "error"="failed to change Route 53 record set: InvalidChangeBatch: [RRSet
with DNS name _acme-challenge.xxxxxx.com., type TXT, SetIdentifier
\"xxxxxxx\" cannot be created because a non
multivalue answer rrset exists with the same name and type.]"
"key"="production/certmanager-xxxx-pp9n2-3392968554-1376642102"
Do I need to update the TXT record in DNS? Currently it is set to a different value than the SetIdentifier value from the output above.
Also noticing a strange error in the log. The pod name mention is incorrect, there is a different pod by another name running:
production/cert-manager-877fd747c-4nf2f[cert-manager]: E0817 21:45:46.379332 1
controller.go:208] cert-manager/challenges "msg"="challenge in work queue no longer
exists" "error"="challenge.acme.cert-manager.io \"certmanager-idrive-ssl-srvw4-
3392968554-1376642102\" not found"
Thanks!

How to wait for tekton pipelinRun conditions

I have the following code within a gitlab pipeline which results in some kind of race condition:
kubectl apply -f pipelineRun.yaml
tkn pipelinerun logs -f pipeline-run
The tkn command immediately exits, since the pipelineRun object is not yet created. There is one very nice solution for this problem:
kubectl apply -f pipelineRun.yaml
kubectl wait --for=condition=Running --timeout=60s pipelinerun/pipeline-run
tkn pipelinerun logs -f pipeline-run
Unfortunately this is not working as expected, since Running seems to be no valid condition for a pipelineRun object. So my question is: what are the valid conditions of a pipelineRun object?
I didn't search too far and wide, but it looks like they only have two condition types imported from the knative.dev project?
https://github.com/tektoncd/pipeline/blob/main/vendor/knative.dev/pkg/apis/condition_types.go#L32
The link above is for the imported condition types from the pipeline source code of which it looks like Tekton only uses "Ready" and "Succeeded".
const (
// ConditionReady specifies that the resource is ready.
// For long-running resources.
ConditionReady ConditionType = "Ready"
// ConditionSucceeded specifies that the resource has finished.
// For resource which run to completion.
ConditionSucceeded ConditionType = "Succeeded"
)
But there may be other imports of this nature elsewhere in the project.
Tekton TaskRuns and PipelineRun only use a condition of type Succeeded.
Example:
conditions:
- lastTransitionTime: "2020-05-04T02:19:14Z"
message: "Tasks Completed: 4, Skipped: 0"
reason: Succeeded
status: "True"
type: Succeeded
The different status and messages available for the Succeeded condition are available in the documentation:
TaskRun: https://tekton.dev/docs/pipelines/taskruns/#monitoring-execution-status
PipelineRun: https://tekton.dev/docs/pipelines/pipelineruns/#monitoring-execution-status
As a side note, there is an activity timeout available in the API. That timeout is not surfaced to the CLI options though. You could create a tkn feature request for that.

Does Fabric8io K8s java client support patch() or rollingupdate() using YAML snippets?

I am trying to program the patching/rolling upgrade of k8s apps by taking deployment snippets as input. I use patch() method to apply the snippet onto an existing deployment as part of rollingupdate using fabric8io's k8s client APIS.. Fabric8.io kubernetes-client version 4.10.1
I'm also using some loadYaml helper methods from kubernetes-api 3.0.12.
Here is my sample snippet - adminpatch.yaml file:
kind: Deployment
spec:
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 1
maxUnavailable: 0
template:
spec:
containers:
- name: ${PATCH_IMAGE_NAME}
image: ${PATCH_IMAGE_URL}
imagePullPolicy: Always
I'm sending the above file content (with all the placeholders replaced) to patchDeployment() method as string.
Here is my call to fabric8 patch() method:
public static String patchDeployment(String deploymentName, String namespace, String deploymentYaml) {
try {
Deployment deploymentSnippet = (Deployment) getK8sObject(deploymentYaml);
if(deploymentSnippet instanceof Deployment) {
logger.debug("Valid deployment object.");
Deployment deployment = getK8sClient().apps().deployments().inNamespace(namespace).withName(deploymentName)
.rolling().patch(deploymentSnippet);
System.out.println(deployment.toString());
return getLastConfig(deployment.getMetadata(), deployment);
}
} catch (Exception Ex) {
Ex.printStackTrace();
}
return "Failed";
}
It throws the below exception:
> io.fabric8.kubernetes.client.KubernetesClientException: Failure
> executing: PATCH at:
> https://10.44.4.126:6443/apis/apps/v1/namespaces/default/deployments/patch-demo.
> Message: Deployment.apps "patch-demo" is invalid: spec.selector:
> Invalid value:
> v1.LabelSelector{MatchLabels:map[string]string{"app":"nginx",
> "deployment":"3470574ffdbd6e88d426a77dd951ed45"},
> MatchExpressions:[]v1.LabelSelectorRequirement(nil)}: field is
> immutable. Received status: Status(apiVersion=v1, code=422,
> details=StatusDetails(causes=[StatusCause(field=spec.selector,
> message=Invalid value:
> v1.LabelSelector{MatchLabels:map[string]string{"app":"nginx",
> "deployment":"3470574ffdbd6e88d426a77dd951ed45"},
> MatchExpressions:[]v1.LabelSelectorRequirement(nil)}: field is
> immutable, reason=FieldValueInvalid, additionalProperties={})],
> group=apps, kind=Deployment, name=patch-demo, retryAfterSeconds=null,
> uid=null, additionalProperties={}), kind=Status,
> message=Deployment.apps "patch-demo" is invalid: spec.selector:
> Invalid value:
> v1.LabelSelector{MatchLabels:map[string]string{"app":"nginx",
> "deployment":"3470574ffdbd6e88d426a77dd951ed45"},
> MatchExpressions:[]v1.LabelSelectorRequirement(nil)}: field is
> immutable, metadata=ListMeta(_continue=null, remainingItemCount=null,
> resourceVersion=null, selfLink=null, additionalProperties={}),
> reason=Invalid, status=Failure, additionalProperties={}).
I also tried the original snippet(with labels and selectors) with kubectl patch deployment <DEPLOYMENT_NAME> -n <MY_NAMESPACE> --patch "$(cat adminpatch.yaml) and this applies the same snippet fine.
I could not get much documentation on fabric8io k8s client patch() java API. Any help will be appreciated.
With latest improvements in Fabric8 Kubernetes Client, you can do it both via patch() and rolling() API apart from using createOrReplace() which is mentioned in older answer.
Patching JSON/Yaml String using patch() call:
As per latest release v5.4.0, Fabric8 Kubernetes Client does support patch via raw string. It can be either YAML or JSON, see PatchTest.java. Here is an example using raw JSON string to update image of a Deployment:
try (KubernetesClient kubernetesClient = new DefaultKubernetesClient()) {
kubernetesClient.apps().deployments()
.inNamespace(deployment.getMetadata().getNamespace())
.withName(deployment.getMetadata().getName())
.patch("{\"spec\":{\"template\":{\"spec\":{\"containers\":[{\"name\":\"patch-demo-ctr-2\",\"image\":\"redis\"}]}}}}");
}
Rolling Update to change container image:
However, if you just want to do rolling update; You might want to use rolling() API instead. Here is how it would look like for updating image of an existing Deployment:
try (KubernetesClient client = new DefaultKubernetesClient()) {
// ... Create Deployment
// Update Deployment for a single container Deployment
client.apps().deployments()
.inNamespace(namespace)
.withName(deployment.getMetadata().getName())
.rolling()
.updateImage("gcr.io/google-samples/hello-app:2.0");
}
Rolling Update to change multiple images in multi-container Deployment:
If you want to update Deployment with multiple containers. You would need to use updateImage(Map<String, String>) method instead. Here is an example of it's usage:
try (KubernetesClient client = new DefaultKubernetesClient()) {
Map<String, String> containerToImageMap = new HashMap<>();
containerToImageMap.put("nginx", "stable-perl");
containerToImageMap.put("hello", "hello-world:linux");
client.apps().deployments()
.inNamespace(namespace)
.withName("multi-container-deploy")
.rolling()
.updateImage(containerToImageMap);
}
Rolling Update Restart an existing Deployment
If you need to restart your existing Deployment you can just use the rolling().restart() DSL method like this:
try (KubernetesClient client = new DefaultKubernetesClient()) {
client.apps().deployments()
.inNamespace(namespace)
.withName(deployment.getMetadata().getName())
.rolling()
.restart();
}
Here is the related bug in Fabric8io rolling API: https://github.com/fabric8io/kubernetes-client/issues/1868
As of now, one way I found to achieve patching with fabri8io APIs is to :
Get the running deployment object
add/replace containers in it with new containers
use the createOrReplace() API to redeploy the deployment object
But your patch understandably could be more than just an update to the containers field. In that case, processing each editable field becomes messy.
I went ahead with using the official K8s client's patchNamespacedDeployment() API to implement patching. https://github.com/kubernetes-client/java/blob/356109457499862a581a951a710cd808d0b9c622/examples/src/main/java/io/kubernetes/client/examples/PatchExample.java

How to determine if a job is failed

How can I programatically determine if a job has failed for good and will not retry any more? I've seen the following on failed jobs:
status:
conditions:
- lastProbeTime: 2018-04-25T22:38:34Z
lastTransitionTime: 2018-04-25T22:38:34Z
message: Job has reach the specified backoff limit
reason: BackoffLimitExceeded
status: "True"
type: Failed
However, the documentation doesn't explain why conditions is a list. Can there be multiple conditions? If so, which one do I rely on? Is it a guarantee that there will only be one with status: "True"?
JobConditions is similar as PodConditions. You may read about PodConditions in official docs.
Anyway, To determine a successful pod, I follow another way. Let's look at it.
There are two fields in Job Spec.
One is spec.completion (default value 1), which says,
Specifies the desired number of successfully finished pods the
job should be run with.
Another is spec.backoffLimit (default value 6), which says,
Specifies the number of retries before marking this job failed.
Now In JobStatus
There are two fields in JobStatus too. Succeeded and Failed. Succeeded means how many times the Pod completed successfully and Failed denotes, The number of pods which reached phase Failed.
Once the Success is equal or bigger than the spec.completion, the job will become completed.
Once the Failed is equal or bigger than the spec.backOffLimit, the job will become failed.
So, the logic will be here,
if job.Status.Succeeded >= *job.Spec.Completion {
return "completed"
} else if job.Status.Failed >= *job.Spec.BackoffLimit {
return "failed"
}
If so, which one do I rely on?
You might not have to choose, considering commit dd84bba64
When a job is complete, the controller will indefinitely update its conditions
with a Complete condition.
This change makes the controller exit the
reconcilation as soon as the job is already found to be marked as complete.
As https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.26/#jobstatus-v1-batch says:
The latest available observations of an object's current state. When a
Job fails, one of the conditions will have type "Failed" and status
true. When a Job is suspended, one of the conditions will have type
"Suspended" and status true; when the Job is resumed, the status of
this condition will become false. When a Job is completed, one of the
conditions will have type "Complete" and status true. More info:
https://kubernetes.io/docs/concepts/workloads/controllers/jobs-run-to-completion/

Spinnaker pipeline failing when deployment strategy is Recreate

When the deployment strategy is changed from "Rolling update" to "Recreate", I am facing the below error
Failure executing: PATCH at: https://3x.xxx.2x1.xxx/apis/extensions/v1beta1/namespaces/default/deployments/xxxxxx. Message: Deployment.apps "xxxxxx" is invalid: spec.strategy.rollingUpdate: Forbidden: may not be specified when strategy type is 'Recreate'. Received status: Status(apiVersion=v1, code=422, details=StatusDetails(causes=[StatusCause(field=spec.strategy.rollingUpdate, message=Forbidden: may not be specified when strategy type is 'Recreate', reason=FieldValueForbidden, additionalProperties={})], group=apps, kind=Deployment, name=xxxxxx, retryAfterSeconds=null, uid=null, additionalProperties={}), kind=Status, message=Deployment.apps "xxxxxx" is invalid: spec.strategy.rollingUpdate: Forbidden: may not be specified when strategy type is 'Recreate', metadata=ListMeta(resourceVersion=null, selfLink=null, additionalProperties={}), reason=Invalid, status=Failure, additionalProperties={}).
Any help on this? I am using Spinnaker 1.6.0
There are many tickets on GitHub related to that problem: Kubernetes, Cert-manager, Spinnaker. And in each one you can find the same answer - it is not possible to switch the update strategy of already created resources.
So, the only way is to create a new deployment with a new strategy due to the implementation of the updating process in Kubernetes.