Unexpected action of jq select filter - select

I have been using jq for quite a while, but some behaviour today has surprised me:
I expected this:
echo '{"Name":"NAME","Tags":[{"Key":"KEY","Value":"HASH"}]}' \
| jq '{"name": .Name, "hash": (.Tags[]|select(.Key=="hash")|.Value)}'
to produce this:
{
"name": "NAME",
"hash": null
}
but instead, it filters out the entire object from the output completely.
As a sanity check changing the filter to a value that does match as follows::
echo '{"Name":"NAME","Tags":[{"Key":"KEY","Value":"HASH"}]}' \
| jq '{"name": .Name, "hash": (.Tags[]|select(.Key=="KEY")|.Value)}'
produces the expected output:
{
"name": "NAME",
"hash": "HASH"
}
The output of the first case, select(.Key=="hash"), is surprising to me since the select filter is meant to be acting on the stream produced by .Tags[] and not the entire input stream.
How do I express what I want to express, which is if there is no matching Tag, the "hash" property of the output object should be set to null?
I am using jq 1.6 on OSX

If a part of a filter produces empty, it will suck in everything that is compund to it.
Solution: Don't let it produce empty. One way would be to give the select filter an alternative: select(.Key=="hash") // null
jq '{"name": .Name, "hash": (.Tags[] | select(.Key=="hash") // null | .Value)}'
{
"name": "NAME",
"hash": null
}
Demo

Related

parsing kubectl json output with jq or jsonpath

I would like to select, and list the crds which are containing the "v1beta1" in the
.spec.versions.*.name
The versions part of the crd object looks similar like this
"versions": [
{
"name": "v1alpha2",
"served": true,
"storage": true,
"subresources": {
"status": {}
},
"name": "v1beta1"
"served": true,
"storage": true,
"subresources": {
"status": {}
}
}
]
I tried some different queries like the following, but no success.
$ kubectl get crd -ojson | jq -r '.items[] | map(select(.spec.versions[] | contains("v1beta1"))).metadata.name'
jq: error (at <stdin>:250345): Cannot index string with string "spec"
Jsonpath solution would be also great. I tried something like this without success.
$ kubectl get crd -ojsonpath="{range .items.*.spec.versions.*}{.name[?(#=='v1beta1')].metadata.name}{'\n'}{end}"
Could someone help me please?
This will show the name using jsonpath: kubectl get crd -o jsonpath='{range .items[?(#.spec.versions[].name=="v1beta1")].metadata}{.name}{"\n"}'
You could base a jq solution on:
.spec.versions[] | select(.name | contains("v1beta1"))
or similar, e.g.
.spec.versions[] | select(.name | startswith("v1beta1"))

Getting just a string from a list

How do I just get 1 output from "labels"?
tried doing -o=jsonpath='{.metadata.labels[0]}' in hopes of getting the first string but that threw an error.
"metadata": {
"labels": {
"beta.kubernetes.io/arch": "amd64",
"beta.kubernetes.io/os": "linux",
"kubernetes.io/arch": "amd64",
"kubernetes.io/hostname": "143.110.156.190",
"kubernetes.io/os": "linux",
"node-role.kubernetes.io/controlplane": "true",
"node-role.kubernetes.io/etcd": "true",
"node-role.kubernetes.io/worker": "true"
},
metadata.labels is not an array, so don't think '{.metadata.labels[0]}' will work.
One of the option is perhaps you can try is to use shell to fetch the first value as following:
kubectl get ingress -o json | jq '.items[0].metadata.labels' | head -2 |tr -d , |cat - <(echo "}") | jq
Kubectl uses JSONPath expressions to filter on specific fields in the JSON object and format the output:
kubectl get ingress -o=jsonpath='{.items[0].metadata.labels}'
For reference use the following link:
https://kubernetes.io/docs/reference/kubectl/jsonpath/

Partition JSON data using jq and then send Rest query

I have a json file like below
[
{
"field": {
"empID": "sapid",
"location": "India",
}
},
{
"field": {
"empID": "sapid",
"location": "India",
}
},
{
"field": {
"empID": "sapid",
"location": "India",
}
}
{
"field": {
"empID": "sapid",
"location": "India",
}
},
{
"field": {
"empID": "sapid",
"location": "India",
}
}
{
"field": {
"empID": "sapid",
"location": "India",
}
}
.... upto 1 million
]
I have to use this json as an input for a rest request For example
curl <REST Server URL with temp.json as input> "Content-Type: application/json" -d #temp.json
My server will not accept 1 million json object at a time.
I am looking for an approach where i have to extract the first 500 objects from the main json and send it in one rest query and then next 500 object in second rest query and so on.
Can you please suggest how can i achieve this by jq?
There's an intrinsic tradeoff here between space and time efficiency. In the following, the focus is on the latter.
Assuming that each call to curl must send a JSON array, a time-efficient solution can be constructed along the following lines:
< array.json jq -c '
def batch($n): length as $l | range(0;length;$n) as $i | .[$i: $i+n];
batch(500)
' | while read -r json
do
echo "$json" | curl -X POST -H "Content-Type: application/json" -d -# ....
done
Here .... signifies additional appropriate curl arguments.
GNU parallel
You might also want to consider using GNU parallel, e.g.:
< array.json jq -c '
def batch($n):
length as $l
| range(0;length;$n) as $i
| .[$i: $i+n];
batch(500)
' | parallel --pipe -N1 curl -X POST -H "Content-Type: application/json" -d #- ....
You have not shared any HW information of the system you are running this on. At the minimum you need to do some sort of multiprocessing to make this faster instead of running (1000000/500) curl requests altogether.
One way, would be to use GNU xargs which has a built-in to run number of parallel instances of a given process using the -P flag and number of lines of input to read from at any time with the L flag.
To start with you can do something like below to instruct curl to run on 500 lines at a time and invoke 20 such invocations in parallel. So at a given tick, approximately (500 *20) lines of input are processed. You can tune the numbers depending on your HW capability both on the host and the server side.
xargs -L 500 -P 20 curl -X POST -H "Content-Type: application/json" http://sample-url -d #- < <(jq -c 'range(0;length;500) as $i | .[$i: $i+500]' json)
Modified jq filter to pack the JSON payload as an array of objects (credit peak's answer). The earlier version jq -c '.[]' json might not work as the individual chunk of lines passed at a time doesn't represent a valid JSON.
Note: Not tested due to performance constraints.
Assuming you have this formatting, splitting can be done by unpacking the array and saving the desired number of objects to separate files, e.g.:
<input.json jq -c '.[]' | split -l500
Which creates xaa with the first 500 objects, xab with the next 500 objects, etc. If you want to repackage the objects in an array, use the -s option to jq, e.g.: jq -s . xaa.
If you want to do this from the shell, you could use jq to split your JSON and pass it to xargs to call curl for each object returned.
jq -c '.[]' temp.json | xargs -I {} curl <REST Server URL with temp.json as input> "Content-Type: application/json" -d '{}'
This will send one curl request for each object. However, if you e.g. want to send the first 500 objects in a single curl request, you can specify a subarray in the jq filter. To send all of your JSON objects you will then somehow need to repeat the command, as afaik jq has no built-in way to split the input into chunks of objects.
jq -c '.[0:500]' temp.json | xargs -I {} curl <REST Server URL with temp.json as input> "Content-Type: application/json" -d '{}'
jq -c '.[500:1000]' temp.json | xargs -I {} curl <REST Server URL with temp.json as input> "Content-Type: application/json" -d '{}'
jq -c '.[1000:1500]' temp.json | xargs -I {} curl <REST Server URL with temp.json as input> "Content-Type: application/json" -d '{}'
[...]

How to show all deployments/daemonsets which mount specific configmap/secret?

Sometimes, I want to explore all deployments/daemonsets which mount specific configmap/secret.
Is there any way can achieve this by kubectl?
You need to have jq to do such a complex queries.
Here you go:
kubectl get -o json deploy,daemonset | jq '[.items[] | . as $parent | .spec.template.spec.volumes[]? | select(.configMap != null) | {kind:$parent.kind, name:$parent.metadata.name, configMap:.configMap.name}]'
The jq command de-constructed:
[ // output as array
.items[] // iterate over root key 'items'
|
. as $parent // store the current entry as $parent to refer to it later
|
.spec.template.spec.volumes[]? // iterate over its volumes (the ? to prevent error if there is no volume
|
select(.configMap != null) // select only those volumes with configMap key
|
{kind:$parent.kind, name:$parent.metadata.name, configMap:.configMap.name} // now construct the output using $parent's kind and name and the configMap's name
]
Example output:
[
{
"kind": "Deployment",
"name": "telemetry-agent",
"configMap": "telemetry-config-map"
},
{
"kind": "DaemonSet",
"name": "fluent-bit",
"configMap": "fluent-bit"
},
{
"kind": "DaemonSet",
"name": "telegraf",
"configMap": "telegraf"
}
]
N.B. If you want to find specific configMap, just replace the select() clause .configMap != null to .configMap.name == "specific-configmap". Also, feel free to add --all-namespaces to the kubectl get command if you want to query from all namespaces

Getting value from json parameter doesn't work (jq: 1 compile error)

I am using awscli and trying to get the value of IpAddress from the output of my query.
I tried to use jq but I get a compile error.
This is the case:
output="$(aws efs describe-mount-targets --file-system-id fs-089b5e31)"
echo $output
{ "MountTargets": [ { "MountTargetId": "fsmt-bb29e666", "IpAddress": "172.20.33.255", "OwnerId": "668225551666", "SubnetId": "subnet-0b61377039d31e666", "NetworkInterfaceId": "eni-045f6ea1376662bdf", "FileSystemId": "fs-089b5e66", "LifeCycleState": "available" } ] }
And this is the command I am using to get the IpAddress:
echo array | jq '.[]MountTarget[]s.IpAddress'
The error I get is this:
parse error: Invalid numeric literal at line 2, column 0
ubuntu#ip-10-10-16-245:~/infra-devops/kops/vector$ echo array | jq '.[]MountTarget[]s.IpAddress'
jq: error: syntax error, unexpected IDENT, expecting $end (Unix shell quoting issues?) at <top-level>, line 1:
.[]MountTarget[]s.IpAddress
jq: 1 compile error
Is my query is the problem or maybe I better use sed instead?
Your syntax to access array is wrong. To get the IP address, use this:
aws efs describe-mount-targets --file-system-id fs-089b5e31 |
jq '.MountTargets[0].IpAddress'
The MountTargets is an array from which you want the first object.
If you need raw data (without double quotes) use -r option in the jq command.