I am migrating a script from yq 3 to yq 4 and cannot get one thing working.
I have the following YAML and want to add a list item after targetNamespaces:
apiVersion: operators.coreos.com/v1
kind: OperatorGroup
metadata:
name: businessautomation-operator
namespace: rhpam-user1
spec:
targetNamespaces:
So the output should be:
apiVersion: operators.coreos.com/v1
kind: OperatorGroup
metadata:
name: businessautomation-operator
namespace: rhpam-user1
spec:
targetNamespaces:
- rhpam-user1
This command worked for changing the namespace:
yq eval '.metadata.namespace = "rhpam-user1"' -i ./file.yaml
When I run the following I am getting an error:
yq eval '.spec.targetNamespaces[+] = "rhpam-user1"' -i ./file.yaml
Error: '' expects 2 args but there is 1
I can't seem to get the new yq command structure right...
If the targetNamespaces is intended to be used as an array type, you need to enclose the target string in [..], with += as below (as tested on version 4.11.2)
yq e '.spec.targetNamespaces += [ "rhpam-user1" ]' yaml
See mikefarah/yq - Relative append
I am not sure how I missed updating my question with the solution I found, but here it is...
Because there is only one array element I was able to get it working with this:
yq eval ".spec.targetNamespaces[0] = \"rhpam-user1\"" -i ./file.yaml
I did not test the solution suggested by #Inian.
Related
I am trying to generate yaml output with kubectl --dry-run.
When I run this
$ kubectl create configmap my-config-map -o yaml --dry-run=client | kubectl annotate -f- --dry-run=client -o yaml --local this-is-my-target-directory='{{ template "This.is.long.path.for.some.value" . }}'
I get the following output where annotations are splitting into 2 lines.
apiVersion: v1
kind: ConfigMap
metadata:
annotations:
this-is-my-target-directory: '{{ template "This.is.long.path.for.some.value" .
}}'
creationTimestamp: null
name: my-config-map
my desired output is that annotations should be in one line. like below
apiVersion: v1
kind: ConfigMap
metadata:
annotations:
this-is-my-target-directory: '{{ template "This.is.long.path.for.some.value" . }}'
creationTimestamp: null
name: my-config-map
If I reduce the size of the string then it becomes 1 line. I could not find anywhere in the documentation anything about line length. Can anyone guide me on how to fix this?
I think you can't find anything because this is a totally valid yaml. So I guess you can use it as it's without the need to put the curly brackets at the same line.
I have a file with three configmaps in it, like the one below.
apiVersion: v1
data:
TEST: "one"
kind: ConfigMap
metadata:
name: test-config-one
---
apiVersion: v1
data:
TEST: "two"
kind: ConfigMap
metadata:
name: test-config-two
---
apiVersion: v1
data:
TEST: "three"
kind: ConfigMap
metadata:
name: test-config-three
I'm trying to apply only test-config-three to the cluster. I know I can break that out into its own file and run kubectl apply -f test-config-three.yaml, but is there a way to do that without having to create a new file?
I was hoping to be able to do something like:
cat file.yml | yq <get only test-config-three> | kubectl apply -f -
But yq doesn't seem to support finding a single resource in a file. I also looked at tools like kubesplit but they tend to output all resources to separate files.
Is there a way to isolate and output a single resource from a yaml file containing multiple resources without creating a new file?
Update
Thanks to #Inian's answer below, I was able to get this full command working.
cat file.yml | yq e 'select(.data.TEST == "three")' - | kubectl apply -f -
There are two versions of yq implemented, one in Python and one in Go as I've highlighted in my answer at How can I parse a YAML file from a Linux shell script?
Using the Python version - kislyuk/yq
yq -y 'select(.data.TEST == "three")' yaml
Go version - mikefarah/yq
yq e 'select(.data.TEST == "three")' yaml
If you have python you can try following
export MYFILE=file.yml
export DOC_NUMBER=2
python3 -c "import yaml; print(yaml.dump(list(yaml.safe_load_all(open('"$MYFILE"')))["$DOC_NUMBER"]))" | kubectl apply -f -
yaml.safe_load_all(open('"$MYFILE"')) loads all the documents in the yaml file into a list. Then you are selecting "$DOC_NUMBER" document with ["$DOC_NUMBER"].
yaml.dump would dump the loaded object back into yaml format which is then printed.
I have a text file with the below text on multiple places:
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: harbor-exporter
labels:
release: mgmt
I want to have:
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: harbor-exporter
namespace: test
labels:
release: mgmt
Because this text is present on multiple places in the file, I do need to search for:
metadata:\n name:harbor-exporter and append the line namespace: test. Next to that, the metadata: should be present at the beginning of the line. What is the best way of achieving this with sed?
I have tried:
cat harbor-exporter.yaml | sed ':a;N;$!ba;s|metadata:\n name: harbor-exporter|NewString|g'
But I do need the word metadata: at the start of a line. So I put a ^ in front of metadata: and that failing.
Command which is failing:
cat harbor-exporter.yaml | sed ':a;N;$!ba;s|^metadata:\n name: harbor-exporter|NewString|g'
Thanks
This might work for you (GNU sed):
sed -E ':a;/metadata:/{n;/name: harbor-export/!ba;p;s/\S.*/namespace: test/}' file
Match on a line containing metadata, print it and fetch the next line. If that line does not contain name: harbor-export, start again. Otherwise, print that line and replace it with namespace:test.
Tested on GNU sed, not sure if this requires changes on other implementations: If metadata: cannot occur in consecutive lines, the logic can be simplified:
sed '/^metadata:$/{n; s/^ name: harbor-exporter$/&\n namespace: test/}'
/^metadata:$/ if whole line content is metadata:
n fetch next line (pattern space is replace with this new line)
s/^ name: harbor-exporter$/&\n namespace: test/ add required line if the next line so fetched has the content name: harbor-exporter
Also, you might be better off with a tool that understands YAML such as https://github.com/TomWright/dasel
I am using ansible version 2.7 for kubernetes deployment.
For sending logs to datadog on kubernetes one of the way is to configure annotations like below,
template:
metadata:
annotations:
ad.datadoghq.com/nginx.logs: '[{"source":"nginx","service":"webapp"}]'
this works fine and I could see logs in DataDog.
However I would like to achieve above configuration via ansible deployment on kubernetes for which I have used below code
template:
metadata:
annotations:
ad.datadoghq.com/xxx.logs: "{{ lookup('template', './datadog.json.j2')}}"
and datadog.json.j2 looks like below
'[{{ '{' }}"source":"{{ sourcea }}"{{ ',' }} "service":"{{ serviceb }}"{{ '}' }}]' **--> sourcea and serviceb are defined as vars**
However the resulting config on deployment is below
template:
metadata:
annotations:
ad.datadoghq.com/yps.logs: |
'[{"source":"test", "service":"test"}]'
and this config does not allow datadog agent to parse logs failing with below error
[ AGENT ] 2019-xx-xx xx10:50 UTC | ERROR | (kubelet.go:97 in parseKubeletPodlist) | Can't parse template for pod xxx-5645f7c66c-s9zj4: could not extract logs config: in logs: invalid character '\'' looking for beginning of value
if I use ansible code as below (using replace)
template:
metadata:
annotations:
ad.datadoghq.com/xxx.logs: "{{ lookup('template', './datadog.json.j2', convert_data=False) | string | replace('\n','')}}"
it generates deployment config as below
template:
metadata:
annotations:
ad.datadoghq.com/yps.logs: '''[{"source":"test", "service":"test"}]'''
creationTimestamp: null
labels:
Which also fails,
to configure the working config with ansible, I have to either remove leading pipe (|) or three quotes coming when using replace).
I would like to have jinja variables substitution in place so that I could configure deployment with desired source and service at deployment time.
kindly suggest
By introducing space in datadog.json.j2 template definition .i.e.
[{"source":"{{ sourcea }}"{{ ',' }} "service":"{{ serviceb }}"}] (space at start)
and running deployment again I got the working config as below
template:
metadata:
annotations:
ad.datadoghq.com/yps.logs: ' [{"source":"test", "service":"test"}]'
However I am not able to understand the behaviour if anyone could help me understand it
The problem is that the YAML being produced is broken. The | character starts an indented scalar ("string" more or less), but the next line includes single-quotes - so the quotes end up being inside the annotation value.
The correct YAML should look like:
template:
metadata:
annotations:
ad.datadoghq.com/yps.logs: |
[{"source":"test", "service":"test"}]
This looks like a bug in how Ansible is generating the output YAML, and your fix must have worked around the bug.
Creating a new ConfigMap from a file:
kubernetes create configmap foo --from-file=foo
This is how the ConfigMap looks internally:
kubernetes get configmaps foo -o yaml
apiVersion: v1
data:
foo: |
VAR1=value1
VAR2=value2
Then, I use this ConfigMap to create a set of environment variables in the container:
apiVersion: v1
kind: Pod
metadata:
labels:
name: app
name: app
spec:
containers:
- name: app-server
image: app:latest
ports:
- containerPort: 3000
envFrom:
- configMapRef:
name: foo
command: ["/bin/bash", "-c", "printenv"]
When the container command runs, I see the following output for printenv:
foo=VAR1=value1
VAR2=value2
So, an echo $foo command in the pod returns:
VAR1=value1 VAR2=value2
According to the documentation for ConfigMap with --from-file, this is expected behaviour.
What would be a creative way (and the proper place) to somehow get the values of this file available to the pod as individual env variables VAR1, VAR2, VAR3, etc. ?
This is not possible with the current version (1.6.x) of Kubernetes. As written in the offical documentation for kubectl create configmap:
--from-file: Key file can be specified using its file path, in which case file basename will be used as configmap key, or optionally with a key and file path, in which case the given key will be used. Specifying a directory will iterate each named file in the directory whose basename is a valid configmap key.
When you want to create a configmap which is used like this, as input for the envFrom container configuration you could create it with the --from-literal option like this:
kubectl create configmap foo --from-literal=var1=value1 --from-literal=var2=value2
To still keep the file, you could transform your file into somethings which then runs this command like this:
eval "kubectl create configmap foo $(cat foo.txt | sed -e 's/^/--from-literal=/' | tr "\n" ' ')"
Along with that maybe checking the outstanding proposals like the --flatten flag proposal on Github are worth your time.
Also keep an eye on the variable naming. iirc VAR1 and VAR2 are not valid property names - they have to be lower case which might cause some issues when passing them on.