I am writing a Kubernetes controller.
Someone creates a custom resource via kubectl apply -f custom-resource.yaml. My controller notices the creation, and then creates a Deployment that pertains to the custom resource in some way.
I am looking for the proper way to set up the Deployment's ownerReferences field such that a deletion of the custom resource will result in a deletion of the Deployment. I understand I can do this like so:
ownerReferences:
- kind: <kind from custom resource>
apiVersion: <apiVersion from custom resource>
uid: <uid from custom resource>
controller: <???>
I'm unclear on whether this is case where I would set controller to true.
The Kubernetes reference documentation says (in its entirety):
If true, this reference points to the managing controller.
Given that a controller is running code, and an owner reference actually references another Kubernetes resource via matching uid, name, kind and apiVersion fields, this statement is nonsensical: a Kubernetes object reference can't "point to" code.
I have a sense that the documentation author is trying to indicate that—using my example—because the user didn't directly create the Deployment herself, it should be marked with some kind of flag indicating that a controller created it instead.
Is that correct?
The follow on question here is of course: OK, what behavior changes if controller is set to false here, but the other ownerReference fields are set as above?
ownerReferences has two purposes:
Garbage collection: Refer to the answer of ymmt2005. Essentially all owners are considered for GC. Contrary to the accepted answer the controller field has no impact on GC.
Adoption: The controller field prevents fighting over resources which are to be adopted. Consider a replica set. Usually, the replica set controller creates the pods. However, if there is a pod which matches the label selector it will be adopted by the replica set. To prevent two replica sets fighting over the same pod, the latter is given a unique controller by setting the controller to true. If a resource already has a controller it will not be adopted by another controller. Details are in the design proposal.
TLDR: The field controller is only used for adoption and not GC.
According to the source code of Kubernetes, the object will be garbage collected only after all objects in ownerReferences field are deleted.
https://github.com/kubernetes/apimachinery/blob/15d95c0b2af3f4fcf46dce24105e5fbb9379af5a/pkg/apis/meta/v1/types.go#L240-L247
// List of objects depended by this object. If ALL objects in the list have
// been deleted, this object will be garbage collected. If this object is managed by a controller,
// then an entry in this list will point to this controller, with the controller field set to true.
// There cannot be more than one managing controller.
According to the documentation:
Sometimes, Kubernetes sets the value of ownerReference automatically. For example, when you create a ReplicaSet, Kubernetes automatically sets the ownerReference field of each Pod in the ReplicaSet. In 1.6, Kubernetes automatically sets the value of ownerReference for objects created or adopted by ReplicationController, ReplicaSet, StatefulSet, DaemonSet, and Deployment.
You can also specify relationships between owners and dependents by manually setting the ownerReference field.
Basically, a Deployment is on the top of the ownership hierarchy, and ownerReference is not set to it automatically. Therefore, you can manually add ownerReference to your Deployment to create the reference to your Foo resource.
You asked:
The follow on question here is of course: OK, what behavior changes if controller is set to false here, but the other ownerReference fields are set as above?
OwnerReference is used by a Garbage Collector. The role of the Kubernetes Garbage Collector is to delete certain objects that once had an owner, but no longer have it.
Here is the link to the description of the OwnerReference structure on Github. As you mentioned, if controller: true, the reference points to the managing controller, in other words, the owner. And also, it is the instruction for Garbage Collector's behavior related to the object and its owner. If controller: false, Garbage Collector manages the object as an object without an owner, for example, allows to delete it freely.
For more information, you can visit the following links:
- Garbage Collection
- Deletion and Garbage Collection of Kubernetes Objects
Related
I have a table in a database that gets frequently updated. The table holds host/port information for Egressing. I want to create a custom operator or a custom controller to sync this information over to Kubernetes, specifically Istio ServiceEntry’s.
Is a custom operator the way to go about this? How can I keep this table and kubernetes in sync – through constant polling?
In Kubernetes and Operator-sdk, we can define CRD (Custom Resource Definition) and CR (Custom Resource). In my operator controller, when a CR is initialized, then the controller reconcillation create a new Deployment and service.
When we delete a CR object, then the associated resources (such as Deployment or service) will be deleted as well at the same time. I understand it should be done by CR Finalizer. But, in Operator-SDK and my controller code, I never see any code to register or add Finalizer for CR, is there any default behavior for Operator-Sdk?
Could anybody point how it work for the case - "while deleting CR, the associated Deployment and Service have deleted as well"? Which part in controller is responsible for that?
Deletion of associated resources is not part of a controller. It's done by Kubernetes's garbage collector.
Basically, garbage collector using OwnerReference objects to find orphaned resources and delete them. Most likely, you set OwnerReference by calling controllerutil.SetControllerReference method somewhere in your code.
When a kubernetes object has parent objects, it is mentioned under "ownerReferences". For example when i printed a pod spec in yaml format, i see ownerReferences mentioned as follows:
ownerReferences:
- apiVersion: apps/v1
blockOwnerDeletion: true
controller: true
kind: StatefulSet
name: statefuleset-name
uid: <uuid>
....
I see that ownerReferences is a list. Does anyone know when the ownerReferences will have more than one entries. I am not able to imagine a object having more than one owner.
If I understand you correctly it is possible in some circumstances.
In this blog you can see an example of multiple ownerReferences. The blog explains garbage collection in K8s and shows that Multiple ownerReferences are possible:
Yes, you heard that right, now postgres-namespace can be owned by more
than one database object.
I hope it helps.
You can have your own use-case and respective CRDs and there can be a requirement of associating an object with multiple owners.
Just taking a very basic example, consider there is a School, with multiple teachers and multiple students, if all 3 are different CRDs then the student may have OwnerReference of kind School with the school name and OwnerReference of kind Teacher with teacher name.
By the way, cluster-api uses multiple ownerReferences in few of it's CRDs.
I have a CF template with a custom resource (backed by a lambda function).
I would like this custom resource to be deleted and the newly created when I update my stack.
How can I achieve this?
To make this practical, let's assume I have a custom resource which simply produces a random string.
More or less as described here:
https://www.itonaut.com/2018/01/03/generate-passwords-in-aws-cloudformation-template/
So I have a resource like:
RandomSuffix:
Type: AWS::CloudFormation::CustomResource
DeletionPolicy: Delete
Properties:
Length: 3
ServiceToken: !ImportValue
'Fn::Sub': 'cf-utils-RandomStringFunction-Arn'
In general, this works, I get some random string as RandomSuffix.Value.
However, after an update, this values stays the same.
Is there any way to force deletion and recreation of my custom resource during update?
You need to change it's name. e.g. RandomSuffix=>RandomSuffix20191210
I created a custom resource definition (CRD) and its controller in my cluster, now I can create custom resources, but how do I validate update requests to the CR? e.g., only certain fields can be updated.
The Kubernetes docs on Custom Resources has a section on Advanced features and flexibility (never mind that validating requests should be considered a pretty basic feature 😉). For validation of CRDs, it says:
Most validation can be specified in the CRD using OpenAPI v3.0 validation. Any other validations supported by addition of a Validating Webhook.
The OpenAPI v3.0 validation won't help you accomplish what you're looking for, namely ensuring immutability of certain fields on your custom resource, it's only helpful for stateless validations where you're looking at one instance of an object and determining if it's valid or not, you can't compare it to a previous version of the resource and validate that nothing has changed.
You could use Validating Webhooks. It feels like a heavyweight solution, as you will need to implement a server that conforms to the Validating Webhook contract (responding to specific kinds of requests with specific kinds of responses), but you will have the required data at least to make the desired determination, e.g. knowing that it's an UPDATE request and knowing what the old object looked like. For more details, see here. I have not actually tried Validating Webhooks, but it feels like it could work.
An alternative approach I've used is to store the user-provided data within the Status subresource of the custom resource the first time it's created, and then always look at the data there. Any changes to the Spec are ignored, though your controller can notice discrepancies between what's in the Spec and what's in the Status, and embed a warning in the Status telling the user that they've mutated the object in an invalid way and their specified values are being ignored. You can see an example of that approach here and here. As per the relevant README section of that linked repo, this results in the following behaviour:
The AVAILABLE column will show false if the UAA client for the team has not been successfully created. The WARNING column will display a warning if you have mutated the Team spec after initial creation. The DIRECTOR column displays the originally provided value for spec.director and this is the value that this team will continue to use. If you do attempt to mutate the Team resource, you can see your (ignored) user-provided value with the -o wide flag:
$ kubectl get team --all-namespaces -owide
NAMESPACE NAME DIRECTOR AVAILABLE WARNING USER-PROVIDED DIRECTOR
test test vbox-admin true vbox-admin
If we attempt to mutate the spec.director property, here's what we will see:
$ kubectl get team --all-namespaces -owide
NAMESPACE NAME DIRECTOR AVAILABLE WARNING USER-PROVIDED DIRECTOR
test test vbox-admin true API resource has been mutated; all changes ignored bad-new-director-name