Issue while using the kubernetes annotations - kubernetes

I've read documentation of kubernetes annotations.
But I couldn't find basic example about using this annotations. For Example;
I have a deployment yaml like below:
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
annotations:
test_value: "test"
name: nginx-deployment
labels:
app: nginx
spec:
replicas: 1
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: nginx:1.13
ports:
- containerPort: 80
How can I use this annotation named test_value and where.
Best Regards...

Just as Labels, Annotations are key-value pairs which represent metadata that is attached to a Kubernetes object.
But contrary to Labels, which are internally utilized to find a collection of objects which satisfy specific conditions, the purpose of Annotations is simply to attach relevant metadata, which should not be used as a filter to identify those objects.
What if we wanted to describe whose person was responsible for generating a specific .yaml file?
We could attach such information to the Kubernetes's object, so that when we need to know who created such object, we can simply run kubectl describe ...
Another useful example, could be to add an annotation to a Deployment before a rollout, explaining what modifications occurred on the new version of the Deployment object. That information could be retrieved later while checking the history of your deployment versions.
But as you have realized with the Ingress example, with Annotations we can also perform advanced configuration on such objects. This is not limited only to Ingress, and for instance you can also provide configuration for running Prometheus on a Kubernetes cluster. You can check the details here.

As mentioned in Kubernetes Documentation, labels have a limited purpose of selecting objects and finding collections of objects that satisfy certain conditions. That put some limitation on the information you can store in labels. (Valid label values must be 63 characters or less and must be empty or begin and end with an alphanumeric character ([a-z0-9A-Z]) with dashes (-), underscores (_), dots (.), and alphanumerics between.)
However, annotations are not used to filter objects, so, you can put in annotation big/small structured/unstructured data that can contain characters, you’re not permitted to use in labels. Tools and libraries can retrieve annotations and use it to add some features to your cluster.
Here are some examples of information that could be recorded in annotations:
Fields managed by a declarative configuration layer. Attaching these fields as annotations distinguishes them from default values set by clients or servers, and from auto-generated fields and fields set by auto-sizing or auto-scaling systems.
Build, release, or image information like timestamps, release IDs, git branch, PR numbers, image hashes, and registry address.
Pointers to logging, monitoring, analytics, or audit repositories.
Client library or tool information that can be used for debugging purposes: for example, name, version, and build information.
User or tool/system provenance information, such as URLs of related objects from other ecosystem components.
Lightweight rollout tool metadata: for example, config or checkpoints.
Phone or pager numbers of persons responsible, or directory entries that specify where that information can be found, such as a team website.
Options for Ingress object, (nginx,gce)

Well, you are right Annotations is like Labels. But I saw that We could customize to config with Annotations for example:
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: cafe-ingress-with-annotations
annotations:
nginx.org/proxy-connect-timeout: "30s"
nginx.org/proxy-read-timeout: "20s"
nginx.org/client-max-body-size: "4m"
spec:
rules:
- host: cafe.example.com
http:
paths:
- path: /tea
backend:
serviceName: tea-svc
servicePort: 80
- path: /coffee
backend:
serviceName: coffee-svc
servicePort: 80
Nginx config can customize according to given Annotation. So how to do this. I couldn't find a tutorial.

I'll first give some background regarding annotations.
Annotations Vs Labels
Annotations are quiet different then labels.
Labels:
You use labels to group resources that you want to refer as a whole.
For example pods with the app=run, env=staging could be exposed by a service with a label selector that matches those labels or managed by a deployment or a daemon set.
Annotations:
Annotations have a few different usages like providing description and adding support for fields that are not part of the K8S API.
While labels should be short, annotations can contain much larger sets of data and can reach up to 256KB.
Annotations use cases examples
You can see below a few examples of how annotations are being used by the various providers / tools that interacts with your cluster.
1 ) Used internally by K8S - below are the annotations that are added to the API-server pod:
kubernetes.io/config.hash: 7c3646d2bcee38ee7dfb851711571ba3
kubernetes.io/config.mirror: 7c3646d2bcee38ee7dfb851711571ba3
kubernetes.io/config.seen: "2020-10-22T01:26:12.671011852+03:00"
kubernetes.io/config.source: file
2 ) If you provision a cluster with kubeadm - this will be added to the API-server pod:
annotations:
kubeadm.kubernetes.io/kube-apiserver.advertise-address.endpoint: 10.246.38.137:6443
3 ) If you run on amazon-eks you can see that the following annotation is added to your workloads - this is for backward compatibility - read more in here):
annotations:
kubernetes.io/psp: eks.privileged
4 ) There are cases when 3rd party tools like aws-alb-ingress-controller that requires you to pass (mandatory) configuration via annotations (because those fields are not supported by the K8S api):
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: aws-alb-ingress
namespace: default
annotations:
kubernetes.io/ingress.class: alb
alb.ingress.kubernetes.io/scheme: internet-facing
alb.ingress.kubernetes.io/tags: Role=Backend , Environment=prod , Name=eros-ingress-alb
alb.ingress.kubernetes.io/listen-ports: '[{"HTTP": 80},{"HTTPS": 443}]'
alb.ingress.kubernetes.io/security-groups : sg-0e3455g455
alb.ingress.kubernetes.io/backend-protocol : HTTP
alb.ingress.kubernetes.io/target-type: instance
alb.ingress.kubernetes.io/healthcheck-path:
alb.ingress.kubernetes.io/success-codes: "200"
alb.ingress.kubernetes.io/certificate-arn:
In your case
Ask yourself what is the reason for adding the annotations.
Then make sure you use a unique prefix for your key in order to avoid collusions.
If you're not sure how to add an annotation to a yaml you can add it manually:
$kubectl annotate pod <pod-name> unique.prefix/for-my-key="value"
And then run $kubectl get po <pod-name> -o yaml to view the annotation that you added manually and copy the yaml to your VCS.

Related

Kubernetes ingress with backend serviceName regex/wildcard or selector

Hi all we have a Flink application with blue-green deployment which we get using the Flink operator.
Flinkk8soperator for Apache Flink. The operator spins up the following three K8s services after a deployment:
my-flinkapp-14hdhsr (Top level service)
my-flinkapp-green
my-flinkapp-blue
The idea is that one of the two among blue green would be active and would have pods (either blue or green).
And a selector to the the active one would be stored in the top level myflinkapp-14hdhsr service with the selector flink-application-version=blue. Or green. As follows:
Labels: flink-app=my-flinkapp
flink-app-hash=14hdhsr
flink-application-version=blue
Annotations: <none>
Selector: flink-app=my-flinkapp,flink-application-version=blue,flink-deployment-type=jobmanager,
I have an ingress defined as follows which I want to use to point to the top level service.
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
annotations:
nginx.ingress.kubernetes.io/rewrite-target: /$2
nginx.ingress.kubernetes.io/configuration-snippet: |
proxy_set_header Accept-Encoding "";
sub_filter '<head>' '<head> <base href="/happy-flink-ui/">';
nginx.ingress.kubernetes.io/use-regex: "true"
nginx.ingress.kubernetes.io/auth-url: "https://$host/oauth2/auth"
nginx.ingress.kubernetes.io/auth-signin: "https://$host/oauth2/start?rd=$escaped_request_uri"
name: flink-secure-ingress-my-flink-app
namespace: happy-flink-flink
spec:
rules:
- host: flinkui-myapp.foo.com
http:
paths:
- path: /happy-flink-ui(/|$)(.*)
backend:
serviceName: my-flinkapp-14hdhsr // This works but....
servicePort: 8081
The issue I am facing is that the top level service keeps changing the hash at the end as the flink operator changes it at every deployment.. eg. myflinkapp-89hddew .etc.
So I cannot have a static service name in the ingress definition.
So I am wondering if an ingress can choose a service based on a selector or a regular expression of the service name which can account for the top level app service name plus the hash at the end.
The flink-app-hash (i.e the hash part of the service name - 14hdsr) is also part of the Labels in the top level service. Anyway I could leverage that?
Wondering if default backend could be applied here?
Have folks using Flink operator solved this a different way?
Unfortunately that's not possible OOB to use any kind of regex/wildcards/jsonpath/variables/references inside .backend.serviceName inside Ingres controller.
You are able to use regex only in path: Ingress Path Matching
And its not going to be implemented soon: Allow variable references in backend spec is in the hanged stage.
Would be very interesting to hear any possible solution for this. It was previously discussed on stack, however without any progress: https://stackoverflow.com/a/60810435/9929015

service selector vs deployment selector matchlabels

I understand that services use a selector to identify which pods to route traffic to by thier labels.
apiVersion: v1
kind: Service
metadata:
name: svc
spec:
ports:
- name: tcp
protocol: TCP
port: 443
targetPort: 443
selector:
app: nginx
Thats all well and good.
Now what is the difference between this selector and the one of the spec.selector from the deployment. I understand that it is used so that the deployment can match and manage its pods.
I dont understand however why i need the extra matchLabels declaration and cant just do it like in the service. Whats the use of this semantically?
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx
spec:
selector:
matchLabels:
app: nginx
replicas: 1
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: nginx
Thanks in advance
In the Service's spec.selector, you can identify which pods to route traffic to only by their labels.
On the other hand, in the Deployment's spec.selector you have two options to decide on which node the pods will be scheduled on, which are: matchExpressions, matchLabels.
How Deployment uses spec.selector
When a Deployment is changed, a new ReplicaSet is created. The ReplicaSet is responsible to manage the Pods. It uses the spec.selector to know what Pods it should manage.
Example:
If the replicas: 1 is changed in the Deployment to e.g. replicas: 2 a new ReplicaSet is created, and it observes the Pods using spec.selector to match Pods with matching labels. It only see 1 replica initially, but its desired state is now replicas: 2 so it is responsible for creating additionally one Pod from the template in the Deployment.
Selector syntax
There is two ways to declare the labels under the spec.selector in `Deployment.
matchLabels - you declare the labels
matchExpressions - you write an expression for labels
See kubectl explain deployment.spec.selector for full explanation of spec.selector alternatives.
Labels and Selectors
Labels and Selectors is a generic concept in Kubernetes and is used in multiple places. Another example is how you can filter what resources you want to see or use with kubectl. E.g. you can select the Pods for an app with:
kubectl get pod -l=app=myappname
(if your Pods is labelled with app: myappname.
why i need the extra matchLabels declaration and cant just do it like in the service. Whats the use of this semantically?
Because service spec only support equality-based selectors and the deployment is a newer resource that supports two syntax (equality-based and set-based).
The API currently supports two types of selectors: equality-based and set-based. A label selector can be made of multiple requirements which are comma-separated. In the case of multiple requirements, all must be satisfied so the comma separator acts as a logical AND (&&) operator.
Reference
The Service spec uses just the "equality-based" label selector syntax.
Newer resources, such as Job, Deployment, ReplicaSet, and DaemonSet, support set-based requirements...
Reference
My understanding is that earlier the only supported syntax was the equality-based one, like we have on the service spec, and that now, when the resource you are using supports the new syntax, you are required to use matchLabels or matchExpressions.

Rancher Metadata / Confd Equivalent in Kubernetes

On our current Rancher environment, we dynamically configure an Nginx configuration based on calls to the Rancher metadata, using labels on the containers to determine if that container is included in the Nginx routing.
We use confd with a Rancher backend to accomplish this metadata check and to dynamically change/reload a new Nginx configuration.
We have started working on migrating to Kubernetes (AWS EKS). Is there an equivalent to this confd/Rancher available for Kubernetes ?
Due to some technical reasons and time scoping reasons, we can't replace this nginx with an ingress equivalent at this time, so are looking into using annotations or labels on services/pods to keep a dynamic configuration capability.
To add a little further detail on what we eventually found thanks to Scott Anderson's answer.
Using the nginx custom template technique, we were able to dynamically configure the nginx configuration by using annotations in the Ingress resources and referencing them in the nginx custom template.
With an Ingress resource metadata defined as:
metadata:
name: foo
annotations:
kubernetes.io/ingress.class: "nginx-legacy"
mycompany/bar: "path_to_service"
Within the custom Nginx template (location block), to see if the annotation is present:
{{if index $ing.Annotations "mycompany/bar"}}
To get a value from an annotation:
{{$bar:= index $ing.Annotations "mycompany/bar"}}
Hosted Kubernetes providers don't generally give you direct access to the backing etcd so your best and desired option is to mine the Kubernetes API for the resources you are interested in and generate your configuration.
This is precisely what an ingress controller does - watches for changes in Kubernetes resources and generates configuration for a load balancer like nginx.
One of the nginx controllers lets you completely replace the template that it uses with one of your own.
The NGINX template is located in the file /etc/nginx/template/nginx.tmpl.
Using a Volume it is possible to use a custom template. This includes using a Configmap as source of the template
volumeMounts:
- mountPath: /etc/nginx/template
name: nginx-template-volume
readOnly: true
volumes:
- name: nginx-template-volume
configMap:
name: nginx-template
items:
- key: nginx.tmpl
path: nginx.tmpl
You could deploy a version of the nginx ingress controller under a custom class like nginx-legacy so it doesn't try to expose services that are expecting the normal nginx ingress.
To do this, the option --ingress-class must be changed to a value unique for the cluster within the definition of the replication controller.
spec:
template:
spec:
containers:
- name: nginx-ingress-legacy-controller
args:
- /nginx-ingress-controller
- '--election-id=ingress-controller-leader-internal'
- '--ingress-class=nginx-legacy'
- '--configmap=ingress/nginx-ingress-internal-controller'
And then annotate legacy services to assign resources to that ingress using:
metadata:
name: foo
annotations:
kubernetes.io/ingress.class: "nginx-legacy"

In Kubernetes, is it possible to reference a pod by its metadata fields in a service selector?

I am writing sample YAML files that will serve as examples to bring my coworkers on Kubernetes. This is what my very first "Pod + Service" example looks like so far:
apiVersion: v1
kind: Pod
metadata:
name: foobar
labels:
app: foobar
spec:
containers:
image: foobar
name: foobar
ports:
- containerPort: 5000
protocol: TCP
---
apiVersion: v1
kind: Service
metadata:
name: foobar
spec:
ports:
- port: 80
protocol: TCP
targetPort: 5000
selector:
app: foobar
type: LoadBalancer
My point being to provide an example as short and as simple as possible, but still fully functional, I was wondering if the service's spec.selector could use the pod name rather than having to define a label which just repeats the name of the pod.
I have searched quite a bit on Kubernetes GitHub issues, StackOverflow and others, but genuinely did not find any answer other than the Labels and Selectors documentation page which seems to imply, by mostly referring to "label selectors", that only labels can be used in all selectors.
I also tried some random things like spec.selector.name = foobar or, inspired by other features of Kubernetes, spec.selector.metadata.name = foobar, but none of those work, obviously (the service can not find any endpoint).
Also, I've come across a lot of manifests that keep repeating that kind of structure for all their declared resources:
name: foobar
labels:
name: foobar
I really feel that it would make a lot of sense to be able to use any metadata field in selectors. Am I right to assume that it is not currently possible? If so, what is the rationale?
The service is backed by one or more pods that all provide same functionality. Services are automatically configured to load balance the traffic to pods matching the label query. A random algorithm is used and currently is the only option.
When you want to provide the pod name to bind services instead of labels, then the pod name will be unique entity in kubernetes cluster and there will be only single pod with that name. Hence one service will bound with only one pod and effectively losing the services ability to be in front of one or more pods serving the same functionality.

How to configure a Kubernetes Multi-Pod Deployment

I would like to deploy an application cluster by managing my deployment via k8s Deployment object. The documentation has me extremely confused. My basic layout has the following components that scale independently:
API server
UI server
Redis cache
Timer/Scheduled task server
Technically, all 4 above belong in separate pods that are scaled independently.
My questions are:
Do I need to create pod.yml files and then somehow reference them in deployment.yml file or can a deployment file also embed pod definitions?
K8s documentation seems to imply that the spec portion of Deployment is equivalent to defining one pod. Is that correct? What if I want to declaratively describe multi-pod deployments? Do I do need multiple deployment.yml files?
Pagids answer has most of the basics. You should create 4 Deployments for your scenario. Each deployment will create a ReplicaSet that schedules and supervises the collection of PODs for the Deployment.
Each Deployment will most likely also require a Service in front of it for access. I usually create a single yaml file that has a Deployment and the corresponding Service in it. Here is an example for an nginx.yaml that I use:
apiVersion: v1
kind: Service
metadata:
annotations:
service.alpha.kubernetes.io/tolerate-unready-endpoints: "true"
name: nginx
labels:
app: nginx
spec:
type: NodePort
ports:
- port: 80
name: nginx
targetPort: 80
nodePort: 32756
selector:
app: nginx
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: nginxdeployment
spec:
replicas: 3
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginxcontainer
image: nginx:latest
imagePullPolicy: Always
ports:
- containerPort: 80
Here some additional information for clarification:
A POD is not a scalable unit. A Deployment that schedules PODs is.
A Deployment is meant to represent a single group of PODs fulfilling a single purpose together.
You can have many Deployments work together in the virtual network of the cluster.
For accessing a Deployment that may consist of many PODs running on different nodes you have to create a Service.
Deployments are meant to contain stateless services. If you need to store a state you need to create StatefulSet instead (e.g. for a database service).
You can use the Kubernetes API reference for the Deployment and you'll find that the spec->template field is of type PodTemplateSpec along with the related comment (Template describes the pods that will be created.) it answers you questions. A longer description can of course be found in the Deployment user guide.
To answer your questions...
1) The Pods are managed by the Deployment and defining them separately doesn't make sense as they are created on demand by the Deployment. Keep in mind that there might be more replicas of the same pod type.
2) For each of the applications in your list, you'd have to define one Deployment - which also makes sense when it comes to difference replica counts and application rollouts.
3) you haven't asked that but it's related - along with separate Deployments each of your applications will also need a dedicated Service so the others can access it.
additional information:
API server use deployment
UI server use deployment
Redis cache use statefulset
Timer/Scheduled task server maybe use a statefulset (If your service has some state in)