I am trying to replace project ParameterKey:Project with ParameterValue:test in the below Cloudformation parameters file
[{
"ParameterKey": "Project",
"ParameterValue": "<changeMe>"
},
{
"ParameterKey": "DockerInstanceType",
"ParameterValue": "m3.medium"
}]
I am trying to execute below jq command
cat config.json |
jq "map(if .ParameterKey == "Project"
then . + {\"ParameterValue\":\"test\"}
else .
end)" > populated_config.json
I am getting the below error
jq: error: Project/0 is not defined at <top-level>, line 1:
map(if .ParameterKey == Project
jq: 1 compile error
You're prematurely closing the string passed to jq by not escaping the quotes for "Project" in the equality.
You can simplify by enclosing the expression with single quotes, and no escaping is necessary:
$ cat config.json | jq 'map(if .ParameterKey == "Project" then . + {"ParameterValue":"test"} else . end)'
[
{
"ParameterKey": "Project",
"ParameterValue": "test"
},
{
"ParameterKey": "DockerInstanceType",
"ParameterValue": "m3.medium"
}
]
Related
I have been using jq for quite a while, but some behaviour today has surprised me:
I expected this:
echo '{"Name":"NAME","Tags":[{"Key":"KEY","Value":"HASH"}]}' \
| jq '{"name": .Name, "hash": (.Tags[]|select(.Key=="hash")|.Value)}'
to produce this:
{
"name": "NAME",
"hash": null
}
but instead, it filters out the entire object from the output completely.
As a sanity check changing the filter to a value that does match as follows::
echo '{"Name":"NAME","Tags":[{"Key":"KEY","Value":"HASH"}]}' \
| jq '{"name": .Name, "hash": (.Tags[]|select(.Key=="KEY")|.Value)}'
produces the expected output:
{
"name": "NAME",
"hash": "HASH"
}
The output of the first case, select(.Key=="hash"), is surprising to me since the select filter is meant to be acting on the stream produced by .Tags[] and not the entire input stream.
How do I express what I want to express, which is if there is no matching Tag, the "hash" property of the output object should be set to null?
I am using jq 1.6 on OSX
If a part of a filter produces empty, it will suck in everything that is compund to it.
Solution: Don't let it produce empty. One way would be to give the select filter an alternative: select(.Key=="hash") // null
jq '{"name": .Name, "hash": (.Tags[] | select(.Key=="hash") // null | .Value)}'
{
"name": "NAME",
"hash": null
}
Demo
How do I just get 1 output from "labels"?
tried doing -o=jsonpath='{.metadata.labels[0]}' in hopes of getting the first string but that threw an error.
"metadata": {
"labels": {
"beta.kubernetes.io/arch": "amd64",
"beta.kubernetes.io/os": "linux",
"kubernetes.io/arch": "amd64",
"kubernetes.io/hostname": "143.110.156.190",
"kubernetes.io/os": "linux",
"node-role.kubernetes.io/controlplane": "true",
"node-role.kubernetes.io/etcd": "true",
"node-role.kubernetes.io/worker": "true"
},
metadata.labels is not an array, so don't think '{.metadata.labels[0]}' will work.
One of the option is perhaps you can try is to use shell to fetch the first value as following:
kubectl get ingress -o json | jq '.items[0].metadata.labels' | head -2 |tr -d , |cat - <(echo "}") | jq
Kubectl uses JSONPath expressions to filter on specific fields in the JSON object and format the output:
kubectl get ingress -o=jsonpath='{.items[0].metadata.labels}'
For reference use the following link:
https://kubernetes.io/docs/reference/kubectl/jsonpath/
Sometimes, I want to explore all deployments/daemonsets which mount specific configmap/secret.
Is there any way can achieve this by kubectl?
You need to have jq to do such a complex queries.
Here you go:
kubectl get -o json deploy,daemonset | jq '[.items[] | . as $parent | .spec.template.spec.volumes[]? | select(.configMap != null) | {kind:$parent.kind, name:$parent.metadata.name, configMap:.configMap.name}]'
The jq command de-constructed:
[ // output as array
.items[] // iterate over root key 'items'
|
. as $parent // store the current entry as $parent to refer to it later
|
.spec.template.spec.volumes[]? // iterate over its volumes (the ? to prevent error if there is no volume
|
select(.configMap != null) // select only those volumes with configMap key
|
{kind:$parent.kind, name:$parent.metadata.name, configMap:.configMap.name} // now construct the output using $parent's kind and name and the configMap's name
]
Example output:
[
{
"kind": "Deployment",
"name": "telemetry-agent",
"configMap": "telemetry-config-map"
},
{
"kind": "DaemonSet",
"name": "fluent-bit",
"configMap": "fluent-bit"
},
{
"kind": "DaemonSet",
"name": "telegraf",
"configMap": "telegraf"
}
]
N.B. If you want to find specific configMap, just replace the select() clause .configMap != null to .configMap.name == "specific-configmap". Also, feel free to add --all-namespaces to the kubectl get command if you want to query from all namespaces
I have a Ansible group_vars directory with the following file within it:
$ cat inventory/group_vars/env1
...
...
ldap_config: !vault |
$ANSIBLE_VAULT;1.1;AES256
31636161623166323039356163363432336566356165633232643932623133643764343134613064
6563346430393264643432636434356334313065653537300a353431376264333463333238383833
31633664303532356635303336383361386165613431346565373239643431303235323132633331
3561343765383538340a373436653232326632316133623935333739323165303532353830386532
39616232633436333238396139323631633966333635393431373565643339313031393031313836
61306163333539616264353163353535366537356662333833653634393963663838303230386362
31396431636630393439306663313762313531633130326633383164393938363165333866626438
...
...
This Ansible encrypted string has a Kubernetes secret encapsulated within it. A base64 blob that looks something like this:
IyMKIyBIb3N0IERhdGFiYXNlCiMKIyBsb2NhbGhvc3QgaXMgdXNlZCB0byBjb25maWd1cmUgdGhlIGxvb3BiYWNrIGludGVyZmFjZQojIHdoZW4gdGhlIHN5c3RlbSBpcyBib290aW5nLiAgRG8gbm90IGNoYW5nZSB0aGlzIGVudHJ5LgojIwoxMjcuMC4wLjEJbG9jYWxob3N0CjI1NS4yNTUuMjU1LjI1NQlicm9hZGNhc3Rob3N0Cjo6MSAgICAgICAgICAgICBsb2NhbGhvc3QKIyBBZGRlZCBieSBEb2NrZXIgRGVza3RvcAojIFRvIGFsbG93IHRoZSBzYW1lIGt1YmUgY29udGV4dCB0byB3b3JrIG9uIHRoZSBob3N0IGFuZCB0aGUgY29udGFpbmVyOgoxMjcuMC4wLjEga3ViZXJuZXRlcy5kb2NrZXIuaW50ZXJuYWwKIyBFbmQgb2Ygc2VjdGlvbgo=
How can I decrypt this in a single CLI?
We can use an Ansible adhoc command to retrieve the variable of interest, ldap_config. To start we're going to use this adhoc to retrieve the Ansible encrypted vault string:
$ ansible -i "localhost," all \
-m debug \
-a 'msg="{{ ldap_config }}"' \
--vault-password-file=~/.vault_pass.txt \
-e#inventory/group_vars/env1
localhost | SUCCESS => {
"msg": "ABCD......."
Make note that we're:
using the debug module and having it print the variable, msg={{ ldap_config }}
giving ansible the path to the secret to decrypt encrypted strings
using the notation -e#< ...path to file...> to pass the file with the encrypted vault variables
Now we can use Jinja2 filters to do the rest of the parsing:
$ ansible -i "localhost," all \
-m debug \
-a 'msg="{{ ldap_config | b64decode | from_yaml }}"' \
--vault-password-file=~/.vault_pass.txt \
-e#inventory/group_vars/env1
localhost | SUCCESS => {
"msg": {
"apiVersion": "v1",
"bindDN": "uid=readonly,cn=users,cn=accounts,dc=mydom,dc=com",
"bindPassword": "my secret password to ldap",
"ca": "",
"insecure": true,
"kind": "LDAPSyncConfig",
"rfc2307": {
"groupMembershipAttributes": [
"member"
],
"groupNameAttributes": [
"cn"
],
"groupUIDAttribute": "dn",
"groupsQuery": {
"baseDN": "cn=groups,cn=accounts,dc=mydom,dc=com",
"derefAliases": "never",
"filter": "(objectclass=groupOfNames)",
"scope": "sub"
},
"tolerateMemberNotFoundErrors": false,
"tolerateMemberOutOfScopeErrors": false,
"userNameAttributes": [
"uid"
],
"userUIDAttribute": "dn",
"usersQuery": {
"baseDN": "cn=users,cn=accounts,dc=mydom,dc=com",
"derefAliases": "never",
"scope": "sub"
}
},
"url": "ldap://192.168.1.10:389"
}
}
NOTE: The above section -a 'msg="{{ ldap_config | b64decode | from_yaml }}" is what's doing the heavy lifting in terms of converting from Base64 to YAML.
References
How to run Ansible without hosts file
https://docs.ansible.com/ansible/latest/user_guide/playbooks_filters.html#filters-for-formatting-data
Base64 Decode String in jinja
How to decrypt string with ansible-vault 2.3.0
If you need a one liner that works with any yaml file (not only in inventory) containing inlined vault vars, and if you are ready to install a pip package for that, there is a solution using yq, a yaml processor built on top of jq
prerequesite: Install yq
pip install yq
Usage
You can get your result with the following command:
yq -r .ldapconfig inventory/group_vars/env1 | ansible_vault decrypt
If you need to type your vault pass interactively, don't forget to add the relevant option
yq -r .ldapconfig inventory/group_vars/env1 | ansible_vault --ask-vault-pass decrypt
Note: the -r option to yq is mandatory to get a raw result without the quotation marks around the value.
I am using awscli and trying to get the value of IpAddress from the output of my query.
I tried to use jq but I get a compile error.
This is the case:
output="$(aws efs describe-mount-targets --file-system-id fs-089b5e31)"
echo $output
{ "MountTargets": [ { "MountTargetId": "fsmt-bb29e666", "IpAddress": "172.20.33.255", "OwnerId": "668225551666", "SubnetId": "subnet-0b61377039d31e666", "NetworkInterfaceId": "eni-045f6ea1376662bdf", "FileSystemId": "fs-089b5e66", "LifeCycleState": "available" } ] }
And this is the command I am using to get the IpAddress:
echo array | jq '.[]MountTarget[]s.IpAddress'
The error I get is this:
parse error: Invalid numeric literal at line 2, column 0
ubuntu#ip-10-10-16-245:~/infra-devops/kops/vector$ echo array | jq '.[]MountTarget[]s.IpAddress'
jq: error: syntax error, unexpected IDENT, expecting $end (Unix shell quoting issues?) at <top-level>, line 1:
.[]MountTarget[]s.IpAddress
jq: 1 compile error
Is my query is the problem or maybe I better use sed instead?
Your syntax to access array is wrong. To get the IP address, use this:
aws efs describe-mount-targets --file-system-id fs-089b5e31 |
jq '.MountTargets[0].IpAddress'
The MountTargets is an array from which you want the first object.
If you need raw data (without double quotes) use -r option in the jq command.