I have a number of repeated values in my kubernetes yaml file and I wondering if there was a way I could store variables somewhere in the file, ideally at the top, that I can reuse further down
sort of like
variables:
- appName: &appname myapp
- buildNumber: &buildno 1.0.23
that I can reuse further down like
labels:
app: *appname
tags.datadoghq.com/version:*buildno
containers:
- name: *appname
...
image: 123456.com:*buildno
if those are possible
I know anchors are a thing in yaml I just couldn't find anything on setting variables
You can't do this in Kubernetes manifests, because you need a processor to manipulate the YAML files. Though you can share the anchors in the same YAML manifest like this:
---
apiVersion: v1
kind: ConfigMap
metadata:
name: &cmname myconfig
namespace: &namespace default
labels:
name: *cmname
deployedInNamespace: *namespace
data:
config.yaml: |
[myconfig]
example_field=1
This will result in:
apiVersion: v1
data:
config.yaml: |
[myconfig]
example_field=1
kind: ConfigMap
metadata:
creationTimestamp: "2023-01-25T10:06:27Z"
labels:
deployedInNamespace: default
name: myconfig
name: myconfig
namespace: default
resourceVersion: "147712"
uid: 4039cea4-1e64-4d1a-bdff-910d5ff2a485
As you can see the labels name && deployedInNamespace have the values resulted from the anchor evaluation.
Based on your use case description, what you would need is going the Helm chart path and template your manifests. You can then leverage helper functions and easily customize when you want these fields. From my experience, when you have an use case like this, Helm is the way to go, because it will help you customize everything within your manifests when you decide to change something else.
I guess there is a similar question with answer.
Please check below
How to reuse an environment variable in a YAML file?
I've faced a strange behaviour with K8s pods running in AWS EKS cluster (version 1.14). The services are deployed via Helm 3 charts. The case is that pod receives more environment variables than expected.
The pod specification says that variables should be populated from a config map.
apiVersion: v1
kind: Pod
metadata:
name: apigw-api-gateway-59cf5bfdc9-s6hrh
namespace: development
spec:
containers:
- env:
- name: JAVA_OPTS
value: -server -XX:MaxRAMPercentage=75.0 -XX:+UseContainerSupport -XX:+HeapDumpOnOutOfMemoryError
- name: GATEWAY__REDIS__HOST
value: apigw-redis-master.development.svc.cluster.local
envFrom:
- configMapRef:
name: apigw-api-gateway-env # <-- this is the map
# the rest of spec is hidden
The config map apigw-api-gateway-env has this specification:
apiVersion: v1
data:
GATEWAY__APP__ADMIN_LOPUSH: ""
GATEWAY__APP__CUSTOMER_LOPUSH: ""
GATEWAY__APP__DISABLE_RATE_LIMITS: "true"
# here are other 'GATEWAY__' envs
JMX_AUTH: "false"
JMX_ENABLED: "true"
# here are other 'JMX_' envs
kind: ConfigMap
metadata:
name: apigw-api-gateway-env
namespace: development
If I request a list of environment variables, I can find values from a different service. These values are not specified in the config map of the 'apigw' application; they are stored in a map for a 'lopush' application. Here is a sample.
/ # env | grep -i lopush | sort | head -n 4
GATEWAY__APP__ADMIN_LOPUSH=<hidden>
GATEWAY__APP__CUSTOMER_LOPUSH=<hidden>
LOPUSH_GAME_ADMIN_MOBILE_PORT=tcp://172.20.248.152:5050
LOPUSH_GAME_ADMIN_MOBILE_PORT_5050_TCP=tcp://172.20.248.152:5050
I've also noticed that this behaviour is somehow relative to the order in which the services were launched. That could be just because some config maps didn't exist at that moment. It seems for now like the pod receives variables from all config maps in the current namespace.
Did any one faced this issue before? Is it possible, that there are other criteria which force K8s to populate environment from other maps?
If you mean the _PORT stuff, that's for compatibility with the old Docker Container Links system. All services in the namespace get automatically set up that way to make it easier to move things from older Docker-based systems.
So a typical k8s deployment file that I'm woking on looks like this:
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
labels:
...
name: ${service-name}
spec:
replicas: 1
strategy:
...
template:
metadata:
...
spec:
serviceAccountName: test
...
the goal is to create multiple services who have access to the same serviceAccount.
This structure works fine when test exists in
kubectl get serviceaccount
The question is how can I set serviceAccountName to default serviceAccount if test does not exist in the namespace (for any reason)?
I don't wanna fail the deployment
I essentially need to have something like
serviceAccountName: {test:-default}
P.S. clearly I can assign a variable to serviceAccountName and parse the yaml file from outside, but wanted to see if there's a better option
As long as you want run this validation inside the cluster, the only way would be to use MutatingAdmissionWebhook.
This will intercepts requests matching the rules defined in MutatingWebhookConfiguration before presisting into etcd. MutatingAdmissionWebhook executes the mutation by sending admission requests to webhook server. Webhook server is just plain http server that adhere to the API.
Thus, you can validate if the service account exists and set default sa if it's not.
Here is an example of the weebhook, which validates and sets custom labels.
More info about Admission Controller Webhooks
I'm deploying a single yaml file containing two manifests using the Spinnaker Kubernetes Provider V2 (Manifest deployer). Inside the Deployment I have a custom annotation that references the ConfigMap:
# ConfigMap
apiVersion: v1
kind: ConfigMap
metadata:
name: my-config-map
data:
foo: bar
---
# Deployment
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: my-deployment
spec:
template:
metadata:
annotations:
my-config-map-reference: my-config-map
[...]
Upon deployment, Spinnaker applies versioning to the ConfigMap, which is then deployed as my-config-map-v000.
I'd like to be able to retrieve the full name inside my custom annotation, but since Spinnaker replaces automatically the configMap references with the appropriate versioned values only in specific entrypoints ( https://github.com/spinnaker/clouddriver/blob/master/clouddriver-kubernetes/src/main/groovy/com/netflix/spinnaker/clouddriver/kubernetes/v2/artifact/ArtifactReplacerFactory.java ) in this case this does not work.
According to Spinnaker documentation ( https://www.spinnaker.io/reference/artifacts/in-kubernetes-v2/#why-not-pipeline-expressions ) I may be able to write a Pipeline Expression to retrieve the full name, but I wasn't able to do so.
How can I set the full ConfigMap name inside the annotation?
Spinnaker can inject artifacts from the currently executing pipeline into your manifests as they are deployed
Refer to this guide for the instructions on how to Binding artifacts in manifests
However, as mentioned here, there's NO resource mapping for annotation, so it should be user-supplied only as a parameter for your manifest.
In the future, certain relationships between resources will be recorded and annotated by Spinnaker
I've read documentation of kubernetes annotations.
But I couldn't find basic example about using this annotations. For Example;
I have a deployment yaml like below:
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
annotations:
test_value: "test"
name: nginx-deployment
labels:
app: nginx
spec:
replicas: 1
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: nginx:1.13
ports:
- containerPort: 80
How can I use this annotation named test_value and where.
Best Regards...
Just as Labels, Annotations are key-value pairs which represent metadata that is attached to a Kubernetes object.
But contrary to Labels, which are internally utilized to find a collection of objects which satisfy specific conditions, the purpose of Annotations is simply to attach relevant metadata, which should not be used as a filter to identify those objects.
What if we wanted to describe whose person was responsible for generating a specific .yaml file?
We could attach such information to the Kubernetes's object, so that when we need to know who created such object, we can simply run kubectl describe ...
Another useful example, could be to add an annotation to a Deployment before a rollout, explaining what modifications occurred on the new version of the Deployment object. That information could be retrieved later while checking the history of your deployment versions.
But as you have realized with the Ingress example, with Annotations we can also perform advanced configuration on such objects. This is not limited only to Ingress, and for instance you can also provide configuration for running Prometheus on a Kubernetes cluster. You can check the details here.
As mentioned in Kubernetes Documentation, labels have a limited purpose of selecting objects and finding collections of objects that satisfy certain conditions. That put some limitation on the information you can store in labels. (Valid label values must be 63 characters or less and must be empty or begin and end with an alphanumeric character ([a-z0-9A-Z]) with dashes (-), underscores (_), dots (.), and alphanumerics between.)
However, annotations are not used to filter objects, so, you can put in annotation big/small structured/unstructured data that can contain characters, you’re not permitted to use in labels. Tools and libraries can retrieve annotations and use it to add some features to your cluster.
Here are some examples of information that could be recorded in annotations:
Fields managed by a declarative configuration layer. Attaching these fields as annotations distinguishes them from default values set by clients or servers, and from auto-generated fields and fields set by auto-sizing or auto-scaling systems.
Build, release, or image information like timestamps, release IDs, git branch, PR numbers, image hashes, and registry address.
Pointers to logging, monitoring, analytics, or audit repositories.
Client library or tool information that can be used for debugging purposes: for example, name, version, and build information.
User or tool/system provenance information, such as URLs of related objects from other ecosystem components.
Lightweight rollout tool metadata: for example, config or checkpoints.
Phone or pager numbers of persons responsible, or directory entries that specify where that information can be found, such as a team website.
Options for Ingress object, (nginx,gce)
Well, you are right Annotations is like Labels. But I saw that We could customize to config with Annotations for example:
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: cafe-ingress-with-annotations
annotations:
nginx.org/proxy-connect-timeout: "30s"
nginx.org/proxy-read-timeout: "20s"
nginx.org/client-max-body-size: "4m"
spec:
rules:
- host: cafe.example.com
http:
paths:
- path: /tea
backend:
serviceName: tea-svc
servicePort: 80
- path: /coffee
backend:
serviceName: coffee-svc
servicePort: 80
Nginx config can customize according to given Annotation. So how to do this. I couldn't find a tutorial.
I'll first give some background regarding annotations.
Annotations Vs Labels
Annotations are quiet different then labels.
Labels:
You use labels to group resources that you want to refer as a whole.
For example pods with the app=run, env=staging could be exposed by a service with a label selector that matches those labels or managed by a deployment or a daemon set.
Annotations:
Annotations have a few different usages like providing description and adding support for fields that are not part of the K8S API.
While labels should be short, annotations can contain much larger sets of data and can reach up to 256KB.
Annotations use cases examples
You can see below a few examples of how annotations are being used by the various providers / tools that interacts with your cluster.
1 ) Used internally by K8S - below are the annotations that are added to the API-server pod:
kubernetes.io/config.hash: 7c3646d2bcee38ee7dfb851711571ba3
kubernetes.io/config.mirror: 7c3646d2bcee38ee7dfb851711571ba3
kubernetes.io/config.seen: "2020-10-22T01:26:12.671011852+03:00"
kubernetes.io/config.source: file
2 ) If you provision a cluster with kubeadm - this will be added to the API-server pod:
annotations:
kubeadm.kubernetes.io/kube-apiserver.advertise-address.endpoint: 10.246.38.137:6443
3 ) If you run on amazon-eks you can see that the following annotation is added to your workloads - this is for backward compatibility - read more in here):
annotations:
kubernetes.io/psp: eks.privileged
4 ) There are cases when 3rd party tools like aws-alb-ingress-controller that requires you to pass (mandatory) configuration via annotations (because those fields are not supported by the K8S api):
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: aws-alb-ingress
namespace: default
annotations:
kubernetes.io/ingress.class: alb
alb.ingress.kubernetes.io/scheme: internet-facing
alb.ingress.kubernetes.io/tags: Role=Backend , Environment=prod , Name=eros-ingress-alb
alb.ingress.kubernetes.io/listen-ports: '[{"HTTP": 80},{"HTTPS": 443}]'
alb.ingress.kubernetes.io/security-groups : sg-0e3455g455
alb.ingress.kubernetes.io/backend-protocol : HTTP
alb.ingress.kubernetes.io/target-type: instance
alb.ingress.kubernetes.io/healthcheck-path:
alb.ingress.kubernetes.io/success-codes: "200"
alb.ingress.kubernetes.io/certificate-arn:
In your case
Ask yourself what is the reason for adding the annotations.
Then make sure you use a unique prefix for your key in order to avoid collusions.
If you're not sure how to add an annotation to a yaml you can add it manually:
$kubectl annotate pod <pod-name> unique.prefix/for-my-key="value"
And then run $kubectl get po <pod-name> -o yaml to view the annotation that you added manually and copy the yaml to your VCS.