Using https://github.com/mikefarah/yq v4
Given a sample.yaml file like this:
spec:
chart:
name: my-chart
version: 0.0.1
---
spec:
chart:
name: something-else
version: 0.0.2
I want to update the version value but only for the instance of .spec.chart.version where the sibling .spec.chart.name element == my-chart. The result would need to output the entire yaml file so that I can edit the YAML file inline.
If I use a select like
yq e -i 'select(.spec.chart.name == "my-chart") | .spec.chart.version = "1.0.0"' sample.yaml
The second instance of .spec has been removed:
spec:
chart:
name: my-chart
version: 1.0.0
Any advice?
yq '(.spec.chart | select(.name == "my-chart") | .version) = "1.0"' file.yaml
Explanation:
First we want to find all the (potential) nodes we want to update, in this case, it's .spec.chart.
Next we filter out those node to be only the ones we're interested in, select(.name == "my-chart")
Now we select the field we want to update .version
Importantly, the whole LHS is in brackets, so those matching version nodes are passed to the update (=) function. If you don't do this, then the filter happens before (and separately to) the update - and so you only see filtered results.
Disclaimer: I wrote yq
Related
I have a k8s yaml file with below block
spec:
replicas: 1
strategy:
type: Recreate
and I want to add below block after "spec:"
selector:
matchLabels:
app: test-app
The file is huge and has many "spec:" fields, so it should be added at the first match.
Final file content should look like this :
spec:
selector:
matchLabels:
app: test-app
replicas: 1
strategy:
type: Recreate
I came up with this working solution using yq with correct indentation but it appends at the end of the file, Its painful to maintain and reading similar 100's of files.
yq -i -y '.spec += {selector:{matchLabels:{app:"test-app"}}}' filename.yaml
Any answers with tools like sed or awk are welcome.
Here you go
$ yq --yaml-output '.spec |= ({selector: {matchLabels: {app: "test-app"}}} + .)' </tmp/your-yaml-file.yaml
spec:
selector:
matchLabels:
app: test-app
replicas: 1
strategy:
type: Recreate
Since you mentioned you have hundreds of files and each has many spec elements, it's unclear if this will solve your actual problem but hopefully it can be of help. Good luck!
I'm not familiar with yq, but I know it supports limited JSON I/O. Here's a solution to the structural problem with jq:
.spec |= ({selector: {matchLabels: {app: "test-app"}}} + .)
Maybe worth a shot in native yq?
Sample pipeline (untested):
yq r -j k8s.yaml | jq "$script" | yq r --prettyPrint
There are also these jq yamlifiers by the incorrigible Jeff Mercado.
You want a pickup-truck solution. I'm suggesting an Earth-mover instead.
(ATTENTION: Following requires Node.JS, Java-8 and command-line Git installed)..
npm install -g commander#2.20.0
npm install -g #asux.org/cli-npm
export NODE_PATH=`npm root -g`
asux
The above does installation.
For what you want..
Create a /tmp/batch-file.txt containing the following lines.
## This is a comment. No temporary files are created by this.
saveTo !ORIGINALINPUT
yaml read spec
saveTo !SAVED
useAsInput !ORIGINALINPUT
yaml delete 'spec/*'
yaml insert spec #/tmp/HugeSelectorFile.yaml
yaml insert spec !SAVED
Run the command:
asux yaml batch #/tmp/batch-file.txt -i ./YOURORIGINAL.yaml -o ./NEW.yaml
ASSUMPTIONS:
1) Your original YAML file is ./YOURORIGINAL.yaml
2) You want a new file called ./NEW.yaml
3) Your "huge selector" file is called /tmp/HugeSelectorFile.yaml (see 2nd-last lime in batch.txt above)
NOTE: The '#' character prefixing file-names is by-design (as without that '#' character, it means you are passing in JSON/YAML in-line within the command-line).
More can be found at https://github.com/org-asux/org-ASUX.github.io/wiki/Welcome-to-WIKI-for-org.ASUX
I want to pass a certificate to the helm chart and currently I am passing using --set-file global.dbValues.dbcacertificate=./server.crt but instead i want to pass the file in values file of helm chart.
The Values.yaml file reads
global:
dbValues:
dbcacertificate: <Some Way to pass the .crt file>
According to the relevant documentation, one must pre-process a file that is external to the chart into a means that can be provided via --set or --values, since .Files.Get cannot read file paths that are external to the chart bundle.
So, given the following example template templates/secret.yaml containing:
apiVersion: v1
kind: Secret
data:
dbcacertificate: {{ .Values.dbcacertificate | b64enc }}
one can use shell interpolation as:
helm template --set dbcacertificate="$(cat ./server.crt)" .
or, if shell interpolation is not suitable for your circumstances, you can pre-process the certificate into a yaml compatible format and feed it in via --values:
$ { echo "dbcacertificate: |"; sed -e 's/^/ /' server.crt; } > ca-cert.yaml
$ helm template --values ./ca-cert.yaml .
I have a values.yaml where I need to mention multiple ports like the following:
kafkaClientPort:
- 32000
- 32001
- 32002
In yaml for statefulset, I need to get value using ordinal number.
So for kf-0, I need to put first element of kafkaClientPort; and for kf-1, second element and so on.
I am trying like the following:
args:
- "KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://$(MY_NODE_NAME):{{ index .Values.kafkaClientPort ${HOSTNAME##*-} }}"
But it is showing an error.
Please advise what is the best way to access dynamically values.yaml value.
The trick here is that Helm template doesn't know anything about ordinal in your stateful set. If you look at the Kafka Helm Chart, you see that they are using a base port 31090 and then they add the ordinal number but that substitution is in place 'after' the template is created. Something like this in your values:
"advertised.listener": |-
PLAINTEXT://kafka.cluster.local:$((31090 + ${KAFKA_BROKER_ID}))
and then in the template file, the use a bash export under command with a printf which is an alias for fmt.Sprintf. Something like this in your case:
command:
- sh
- -exc
- |
unset KAFKA_PORT && \
export KAFKA_BROKER_ID=${HOSTNAME##*-} && \
export "KAFKA_ADVERTISED_LISTENERS={{ printf "%s" $advertised.listener }} \\
...
I'm trying to save the contents of a configmap to a file on my local hard drive. Kubectl supports selecting with JSONPath but I can't find the expression I need to select just the file contents.
The configmap was created using the command
kubectl create configmap my-configmap --from-file=my.configmap.json=my.file.json
When I run
kubectl describe configmap my-configmap
I see the following output:
Name: my-configmap
Namespace: default
Labels: <none>
Annotations: <none>
Data
====
my.file.json:
----
{
"key": "value"
}
Events: <none>
The furthest I've gotten so selecting only the file contents is this:
kubectl get configmap my-configmap -o jsonpath="{.data}"
Which outputs
map[my.file.json:{
"key": "value"
}]
The output that I want is
{
"key": "value"
}
What is the last piece of the JSONPath puzzle?
There’s an open issue at the Kubernetes GitHub repo with a list of things that needs to be fixed in regards to kubectl (and JSONpath), one of them are issue 16707 jsonpath template output should be json.
Edit:
How about this:
kubectl get cm my-configmap -o jsonpath='{.data.my\.file\.json}'
I just realized i had answered another question related (kind of) to this one. The above command should output what you had in mind!
If you have the ability to use jq, then you can use the following approach to e.g. "list" all config maps by selector, and extract the files:
readarray -d $'\0' -t a < <(kubectl get cm -l grafana=dashboards -o json | jq -cj '.items[] | . as $cm | .data | to_entries[] | [ ($cm.metadata.name + "-" + .key), .value ][]+"\u0000"') ; count=0; while [ $count -lt ${#a[#]} ]; do echo "${a[$((count + 1))]}" > ${a[$count]}; count=$(( $count + 2)); done
This uses kubectl (using -l for a label selector) to get all configmaps. Next it pipes them through jq, creating key value pairs with a null byte termination (the key also contains the name of the configmap, this way I ensured that duplicate file names are not an issue). Then it reads this into a bash array, iterating over the array in steps of 2. Creating files with the content.
This also works file config map values that contain newlines.
I'm building a helm chart for my application, and I'm using stable/nginx-ingress as a subchart. I have a single overrides.yml file that contains (among other overrides):
nginx-ingress:
controller:
annotations:
external-dns.alpha.kubernetes.io/hostname: "*.{{ .Release.Name }}.mydomain.com"
So, I'm trying to use the release name in the overrides file, and my command looks something like: helm install mychart --values overrides.yml, but the resulting annotation does not do the variable interpolation, and instead results in something like
Annotations: external-dns.alpha.kubernetes.io/hostname=*.{{ .Release.Name }}.mydomain.com
I installed the subchart by using helm fetch, and I'm under the (misguided?) impression that it would be best to leave the fetched thing as-is, and override values in it - however, if variable interpolation isn't available with that method, I will have to put my values in the subchart's values.yaml.
Is there a best practice for this? Is it ok to put my own values in the fetched subchart's values.yaml? If I someday helm fetch this subchart again, I'll have to put those values back in by hand, instead of leaving them in an untouched overrides file...
Thanks in advance for any feedback!
I found the issue on github -- it is not supported yet:
https://github.com/kubernetes/helm/issues/2133
Helm 3.x (Q4 2019) now includes more about this, but for chart only, not for subchart (see TBBle's comment)
Milan Masek adds as a comment:
Thankfully, latest Helm manual says how to achieve this.
The trick is:
enclosing variable in " or in a yaml block |-, and
then referencing it in a template as {{ tpl .Values.variable . }}
This seems to make Helm happy.
Example:
$ cat Chart.yaml | grep appVersion
appVersion: 0.0.1-SNAPSHOT-d2e2f42
$ cat platform/shared/t/values.yaml | grep -A2 image:
image:
tag: |-
{{ .Chart.AppVersion }}
$ cat templates/deployment.yaml | grep image:
image: "{{ .Values.image.repository }}:{{ tpl .Values.image.tag . }}"
$ helm template . --values platform/shared/t/values.betradar.yaml | grep image
image: "docker-registry.default.svc:5000/namespace/service:0.0.1-SNAPSHOT-d2e2f42"
imagePullPolicy: Always
image: busybox
Otherwise there is an error thrown..
$ cat platform/shared/t/values.yaml | grep -A1 image:
image:
tag: {{ .Chart.AppVersion }}
1 $ helm template . --values platform/shared/t/values.yaml | grep image
Error: failed to parse platform/shared/t/values.yaml: error converting YAML to JSON: yaml: invalid map key: map[interface {}]interface {}{".Chart.AppVersion":interface {}(nil)}
For Helm subchart, TBBle adds to issue 2133
#MilanMasek 's solution won't work in general for subcharts, because the context . passed into tpl will have the subchart's values, not the parent chart's values.
!<
It happens to work in the specific example this ticket was opened for, because .Release.Name should be the same in all the subcharts.
It won't work for .Chart.AppVersion as in the tpl example.
There was a proposal to support tval in #3252 for interpolating templates in values files, but that was dropped in favour of a lua-based Hook system which has been proposed for Helm v3: #2492 (comment)
That last issue 2492 include workarounds like this one:
You can put a placeholder in the text that you want to template and then replace that placeholder with the template that you would like to use in yaml files in the template.
For now, what I've done in the CI job is run helm template on the values.yaml file.
It works pretty well atm.
cp values.yaml templates/
helm template $CI_BUILD_REF_NAME ./ | sed -ne '/^# Source:
templates\/values.yaml/,/^---/p' > values.yaml
rm templates/values.yaml
helm upgrade --install ...
This breaks if you have multiple -f values.yml files, but I'm thinking of writing a small helm wrapper that runs essentially runs that bash script for each values.yaml file.
fsniper illustrates again the issue:
There is a use case where you would need to pass deployment name to dependency charts where you have no control.
For example I am trying to set podAffinity for zookeeper. And I have an application helm chart which sets zookeeper as a dependency.
In this case, I am passing pod antiaffinity to zookeeper via values. So in my apps values.yaml file I have a zookeeper.affinity section.
If I had the ability to get the release name inside the values yaml I would just set this as default and be done with it.
But now for every deployment I have to override this value, which is a big problem.
Update Oct. 2022, from issue 2133:
lazychanger proposes
I submitted a plugin to override values.yaml with additional templates.
See lazychanger/helm-viv: "Helm-variable-in-values" and its example.