I’m using the command gcloud compute instances list in a script, but I’m worried that the exact output format isn’t static. What should I do?
You should use the --format flag, available for most gcloud commands.
For instance, if you’d like to get the exact same output as the current (as of the time of writing of this answer) format, you can run:
$ gcloud compute instances list --format="table(
name,
zone.basename(),
machineType.basename(),
scheduling.preemptible.yesno(yes=true, no=''),
networkInterfaces[0].networkIP:label=INTERNAL_IP,
networkInterfaces[0].accessConfigs[0].natIP:label=EXTERNAL_IP,
status
)"
The output of this command will not change between releases, even if the default output of the command does (unless the resource being formatted changes; this should be rare).1 Showing the default format for resources in commands is a work in progress.2
You can also specify a format like YAML or JSON for machine-readable output:
$ gcloud compute instances list --format=yaml
$ gcloud compute instances list --format=json
Note that this output contains much more information than is present in the default output for this command; this is the information you have to work with when constructing a custom format.
CSV is another format option. Like table, it requires a projection–a specification for how to print each row.3
$ gcloud compute instances list --format="csv(name,zone,status)"
name,zone,status
example-instance,us-central1-f,RUNNING
...
For more information on the formatting capabilities of gcloud, see the output of gcloud topic formats and gcloud topic projections.
You can see all possible fields by running gcloud compute instances list --format=flattened.
For some commands, like gcloud beta test android locales list, you can pass the --verbosity=info flag and look for INFO: Display format.
This is because CSV data cannot be nested like JSON or YAML, and the data structures being printed may be nested.
Related
I'm trying to make argocd cli output yaml/json to prep it for script ingestion.
According to this PR: https://github.com/argoproj/argo-cd/pull/2551
It should be available but I can't find the option in cli help nor in documentation.
#argocd version:
argocd: v2.1.2+7af9dfb
...
argocd-server: v2.0.3+8d2b13d
Some commands accept the -o json flag to request JSON output.
Look in the commands documentation to find commands which support that flag.
argocd cluster list -o json, for example, will return a JSON list of configured clusters. The documentation looks like this:
Options
-h, --help help for get
-o, --output string
Output format. One of: json|yaml|wide|server (default "yaml")
I use to append datasets in a bucket in gcloud using:
gsutil compose gs://bucket/obj1 [gs://bucket/obj2 ...] gs://bucket/composite
However, today when I tried to append some data the terminal prints the error CommandException: The compose command accepts at most 33 arguments.
I didn't know about this restriction. How can I append more than 33 files in my bucket? Is there another command line tool? I would like to avoid to create a virtual machine for what looks like a rather simple task.
I checked the help using gsutil help compose. But it didn't help much. There is only a warning saying "Note that there is a limit (currently 32) to the number of components that can
be composed in a single operation." but no hint on a workaround.
Could you not do it recursively|batch?
I've not tried this.
Given an arbitrary list of files (FILES)
While there is more than 1 file in FILES:
Take the first n where n<=33 from FILES and gsutil compose into temp file
If that succeeds, replace the n names in FILES with the 1 temp file.
Repeat
The file that remains is everything composed.
Update
The question piqued my curiosity and gave me an opportunity to improve my bash ;-)
A rough-and-ready proof-of-concept bash script that generates batches of gsutil compose commands for arbitrary (limited by the string formatting %04) numbers of files.
GSUTIL="gsutil compose"
BATCH_SIZE="32"
# These may be the same (or no) bucket
SRC="gs://bucket01/"
DST="gs://bucket02/"
# Generate test LST
FILES=()
for N in $(seq -f "%04g" 1 100); do
FILES+=("${SRC}/file-${N}")
done
function squish() {
LST=("$#")
LEN=${#LST[#]}
if [ "${LEN}" -le "1" ]; then
# Empty array; nothing to do
return 1
fi
# Only unique for this configuration; be careful
COMPOSITE=$(printf "${DST}/composite-%04d" ${LEN})
if [ "${LEN}" -le "${BATCH_SIZE}" ]; then
# Batch can be composed with one command
echo "${GSUTIL} ${LST[#]} ${COMPOSITE}"
return 1
fi
# Compose 1st batch of files
# NB Provide start:size
echo "${GSUTIL} ${LST[#]:0:${BATCH_SIZE}} ${COMPOSITE}"
# Remove batch from LST
# NB Provide start (to end is implied)
REM=${LST[#]:${BATCH_SIZE}}
# Prepend composite from above batch to the next run
NXT=(${COMPOSITE} ${REM[#]})
squish "${NXT[#]}"
}
squish "${FILES[#]}"
Running with BATCH_SIZE=3, no buckets and 12 files yields:
gsutil compose file-0001 file-0002 file-0003 composite-0012
gsutil compose composite-0012 file-0004 file-0005 composite-0010
gsutil compose composite-0010 file-0006 file-0007 composite-0008
gsutil compose composite-0008 file-0008 file-0009 composite-0006
gsutil compose composite-0006 file-0010 file-0011 composite-0004
gsutil compose composite-0004 file-0012 composite-0002
NOTE How composite-0012 is created by the first command but then knitted into the subsequent command.
I'll leave it to you to improve throughput by not threading the output from each step into the next, parallelizing the gsutil compose commands across the list chopped into batches and then compose the batches.
The docs say that you may only combine 32 components in a single operation, but there is no limit to the number of components that can make up a composite object.
So, if you have more than 32 objects to concatenate, you may perform multiple compose operations, composing 32 objects at a time until you eventually get all of them composed together.
I'm trying to extract a "sub-map" from a K8s object using kubectl. The result has to be legal JSON syntax so it can be parsed by the Groovy JsonSlurper class.
If I run a command like this:
kubectl get configmap ... -o jsonpath="{.data}"
I get output like this:
map[...
This cannot be parsed by JsonSlurper.
If I instead do this:
kubectl get configmap ... -o json | jq .data
I get something like this:
{
"...": "....",
This should be parseable by JsonSlurper.
You might first say, "well, why don't you just DO that then?". I could, but I'm in a situation where I have to limit what applications I assume are installed in the environment. I'm not certain that "jq" is available on all of our build nodes (they aren't running in a container yet).
Is there some way that I can make kubectl with jsonpath output emits a value in legal JSON syntax?
I'm currently using kubectl v1.14.0, with v1.13.5 on the server.
Is there no way to get a file listing out from a Google Cloud Storage bucket that is sorted by date descending? This is very frustrating. I need to check the status of files that are uploaded and the bucket has thousands of objects.
gsutil ls does not have the standard linux -t option.
Google cloud console also lists it but does not offer sorting options.
I use this as a workaround:
gsutil ls -l gs://[bucket-name]/ | sort -k 2
This outputs full listing including date as the second field, sort -k 2 then sorts by this field.
The only ordering supported by GCS is lexicographic.
As a workaround, if it's possible for you to name your objects with a datestamp, that would give you a way to list objects by date.
I am using oozie to run my map-reduce job. I want to create the output file according to the date. But it takes date as a string and ends up printing instead of taking date as the value :
/user/skataria/geooutput/$(date +"%m%d%Y%H%M%S")
Here is the oozie properties file:
nameNode=hdfs://localhost:8020
jobTracker=localhost:8021
date=(date +"%m%d%Y%H%M%S")
oozie.wf.application.path=${nameNode}/services/advert/sid
inputDir=${nameNode}/user/${user.name}/geoinput/testgeo
outputDir=${nameNode}/user/${user.name}/geooutput/${date}
Somehow i cant have oozie as tag because my reputation is under 1500
It looks like you're trying to use a linux shell command (date +"%m%d%Y%H%M%S") in a java properties file - this isn't going to resolve.
One work around, assuming this is part of a manually submitted Workflow job (as opposed to a Coordinator job) is to provide the date property from the command line using the -D key=value option, and linux shell back quotes to resolve the output of a command inline
oozie job -run -config job.properties -D date=`date +"%m%d%Y%H%M%S"`
You'll need to make sure your version of Oozie support the -D key=value option
Yes I agree the shell option works. But that does not solve my usecase. I want to run my map-reduce job daily and schedule this thru Hue. The output directory needs to be parameterized as an job property to Oozie.
By the way I find that Oozie has Expression language Functions,
Unfortunately the function timestamp() returns the UTC current date and time in W3C format down to the second (YYYY-MM-DDThh:mm:ss.sZ). i.e.: 1997-07-16T19:20:30.45Z and completely unusable for creating a sub-directory name in HDFS
So for now,
I have a workaround. I am using the Workflow EL Function wf:id()
In workflow.xml
<property>
<name>mapreduce.output.fileoutputformat.outputdir</name>
<value>/user/sasubramanian/impressions/output/outpdir/${yyyy_mm_dd}/${wf:id()}</value>
</property>
This creates a output directory with subdirectory as,
/user/foouser/subdir1/output/outpdir/0000006-130321100837625-oozie-oozi-W
NOTE: You must specify this in the workflow.xml. This will not work if you specified it in the job.properties