In the terraform aws_eks_node_group resource I can't set :
labels = {
"node-role.kubernetes.io/others" = "other"
}
as AWS complains labels key should not contains kubernetes.io.
Error: error creating EKS Node Group (my-cluster:others): InvalidParameterException: Label cannot contain reserved labels kubernetes.io/
{
ClusterName: "my-cluster",
Message_: "Label cannot contain reserved labels kubernetes.io/",
NodegroupName: "others"
}
Also, EC2 instances spawned have no name and I have no clue on how to specify a Name for my instances based on their node group.
Any idea on how to achieve this ?
As per the documentation you can't use specific labels, regarding labels:
The kubernetes.io/ and k8s.io/ prefixes are reserved for Kubernetes core components. Valid label values must be 63 characters or less and must be empty or begin and end with an alphanumeric character ([a-z0-9A-Z]) with dashes (-), underscores (_), dots (.), and alphanumerics between.
Regarding your specific label. There are many issues since k8s 1.15 or 1.16 where some change in the core kubernetes don’t allow that label. one detailed issue
As for naming of EC2 instances created by EKS Node Group. Currently, there is no way to pass "Name" tag. This question is a duplicate of this where you can also find the answer how to name your instances for time being.
node-role.kubernetes.io and kubernetes.io - this is DIFFERENT prefixes
Related
While checking the values of yaml files for a helm chart, one often encounters
changeme passed as a value. E.g.:
rabbitmq.conf: |-
##username and password
default_user={{.Values.rabbitmq.username}}
default_pass=CHANGEME
or:
config:
accumuloSite:
instance.secret: "changeme"
userManagement:
rootPassword: "changeme"
What is the meaning of "changeme"?
Is it just a word that needs to be replaced? If so, what will happen if it is not? A security hole, or hopefully an error?
Or is it a keyword that lets the system to replace this with a secure password? If so, how does the system know what type of password to produce?
In either case, how does the chart connect this value with other places where this value might be needed? ( e.g. if this is a password another -dependent to the first- service needs, how is the manually assigned / derived password propagated to the second service? )
(*mainly interested about helm v3 if this is important)
I'd almost always expect this to be just a placeholder that needs to be filled in. In many cases YAML can wind up having inconsistent types if a value is actually absent, so it can be useful to have some value in the chart's values.yaml, but for things like passwords there's not a "right default value" you could include.
Nothing will automatically replace these for you or warn if you're using the default values. Nothing bad will obviously happen if you do deploy with these values, but I'm sure changeme is up there with passw0rd on the short list of default passwords to try if you're actively trying to break into a system.
If you were writing your own chart, you could also test if a value is present using required and explain what's missing, and this approach might be more secure than having a well-known default password.
I use kubectl to list Kubernetes custom resources of a kind mykind with an additional table column LABEL that contains the value of a label a.b.c.com/key if present:
kubectl get mykind -o=custom-columns=LABEL:.metadata.labels.'a\.b\.c\.com/key'
This works, i.e., the label value is properly displayed.
Subsequently, I wanted to add a corresponding additional printer column to the custom resource definition of mykind:
- description: Label value
jsonPath: .metadata.labels.'a\.b\.c\.com/key'
name: LABEL
type: string
Although the additional column is added to kubectl get mykind, it is empty and no label value is shown (in contrast to above kubectl command). My only suspicion were problems with escaping of the special characters - but no variation helped.
Are you aware of any difference between the JSON path handling in kubectl and additional printer columns? I expected strongly that they are exactly the same.
mdaniel's comment works!
- description: Label value
jsonPath: '.metadata.labels.a\.b\.c\.com/key'
name: LABEL
type: string
You need to use \. instead of . and use single quotes ' '. It doesn't work with double quotes for the reasons I don't understand
I want to have a panel in Grafana which displays what pods are currently running in a host.
For the host variable I have the following query (the job variable is just label_values(node_uname_info, job).):
label_values(node_uname_info{job="$job"}, instance)
This gives me an array of sockets: host_ip:port
I can get the pod names from kube_pod_info{job="$job", host_ip="$host_ip"}, but in order to get the IP I need to remove the port part of the socket:
label_replace(node_uname_info{job="$job", instance="$node"}, "host_ip", "$1", "instance", "(.*):.*")
I haven't found how to use the new host_ip label in the pod query to eventually get all the pod label values of kube_pod_info. I don't want to put the label_replace in Prometheus to avoid data duplication - is there a way to use the new host_ip label in the pod query?
Edit:
I added the host_ip variable with the regex as shan1024 showed in his answer and changed the panel's query to:
sum by (pod) (kube_pod_info{job="$job", host_ip="$host_ip"})
Then I changed the panel's visualization to table and added column styles to Time and Value (chose type Hidden). This allows me to display the host's running pods in a list-like fashion.
This is actually quite easy to do in Grafana and no need to change labels in Prometheus. You just need to add a regex in the instance variable (when we add a regex with a capturing group, the value(s) of the 1st captured group will be the value(s) of the variable).
e.g.
Variable definition without Regex (you get host_ip:port)-
Variable definition with Regex (you only get host_ip)-
Then you can add a new variable with value kube_pod_info{ host_ip="$instance" } to get all pods in the selected host.
AKS = 1.17.9
Prometheus = 2.16.0
kube-state-metrics = 1.8.0
My use case: I want to alert when 1 of my persistent volumes are not in a "Bound" phase and only when this falls within a predefined set of namespaces.
This got me to my first attempt at joining Prometheus metrics - so, please bear with me : )
I opted to use the following to obtain the pv phase:
kube_persistentvolume_status_phase{phase="Bound",job="kube-state-metrics"}
Renders:
kube_persistentvolume_status_phase{instance="10.147.5.110:8080",job="kube-state-metrics",persistentvolume="pvc-33197ae6-d42a-777e-b8ca-efbd66a8750d",phase="Bound"} 1
kube_persistentvolume_status_phase{instance="10.147.5.110:8080",job="kube-state-metrics",persistentvolume="pvc-165d5006-erd4-481e-8acc-eed4a04a3bce",phase="Bound"} 1
This worked well, except for the fact that it does not include the namespace.
So I managed to determine the persistentvolumeclaim namespaces with this:
kube_persistentvolumeclaim_info{namespace=~"monitoring|vault"}
Renders:
kube_persistentvolumeclaim_info{instance="10.147.5.110:8080",job="kube-state-metrics",namespace="vault",persistentvolumeclaim="vault-file",storageclass="default",volumename="pvc-33197ae6-d42a-777e-b8ca-efbd66a8750d"} 1
kube_persistentvolumeclaim_info{instance="10.147.5.110:8080",job="kube-state-metrics",namespace="monitoring",persistentvolumeclaim="prometheus-prometheus-db-prometheus-prometheus-0",storageclass="default",volumename="pvc-165d5006-erd4-481e-8acc-eed4a04a3bce"} 1
So my idea was to join these sets with the matching values in the following fields:
(kube_persistentvolume_status_phase)persistentvolume
on
(kube_persistentvolumeclaim_info)volumename
BUT, if I understood it correctly you are only able to join two metrics sets on labels that match exactly (text and their values). I hence opted for the "instance" and "job" labels as these were common on both sides and matching.
kube_persistentvolume_status_phase{phase!="Bound",job="kube-state-metrics"} * on(instance,job) group_left(namespace) kube_persistentvolumeclaim_info{namespace=~"monitoring|vault"}
Renders:
Error executing query: found duplicate series for the match group {instance="10.147.5.110:8080" , job="kube-state-metrics"} on the right hand-side of the operation: [{__name__="kube_persistentvolumeclaim_info", instance="10.147.5.110:8080", job="kube-state-metrics", namespace="monitoring", persistentvolumeclaim="alertmanager-prometheusam-db-alertmanager-prometheusam-0", storageclass="default", volumename="pvc-b8406fb8-3262-7777-8da8-151815e05d75"}, {__name__="kube_persistentvolumeclaim_info", instance="10.147.5.110:8080", job="kube-state-metrics", namespace="vault", persistentvolumeclaim="vault-file", storageclass="default", volumename="pvc-33197ae6-d42a-777e-b8ca-efbd66a8750d"}];many-to-many matching not allowed: matching labels must be unique on one side
So in all fairness, the query does communicate well on what the problem is - so I attempted to solve this with the "ignoring" option - attempting to keep only the matching labels and values (instance and job) and "excluding/ignoring" the non-matching ones on both sides. This did not work either - resulting in a parsing error. Which in turn nudged me to take a step back and reassess what I am doing.
I am just a bit concerned that I am perhaps barking up the wrong tree here.
My question is: Is this at all possible and if so how? or is there perhaps another, more prudent way to achieve this?
Thanks in advance!
I have several metrics with the label "service". I want to get a list of all the "service" levels that begin with "abc" and end with "xyz". These will be the values of a grafana template variable.
This is that I have tried:
label_values(service) =~ "abc.*xyz"
However this produces a error Template variables could not be initialized: parse error at char 13: could not parse remaining input "(service_name) "...
Any ideas on how to filter the label values?
This should work (replacing up with the metric you mention):
label_values(up{service=~"abc.*xyz"}, service)
Or, in case you actually need to look across multiple metrics (assuming that for some reason some metrics have some service label values and other metrics have other values):
label_values({__name__=~"metric1|metric2|metric3", service=~"abc.*xyz"}, service)