I would like to parse my JSON log with jq, but my command didn't work anymore since few months.
I use tail from azure-cli command to show live log from my webapp
az webapp log tail --resource-group ${RESOURCE} --name ${appNAME} | sed 's/^[^{]*//g' | sed 's/[^}]*$//g' | jq -r .
I use roarr to parse log, but it's a detail
az webapp log tail --resource-group ${RESOURCE} --name ${appNAME} | sed 's/^[^{]*//g' | sed 's/[^}]*$//g' | roarr --exclude-alien true --use-colors --output-format pretty
Exemple of logs received by azure :
2021-09-08T19:04:34.555601536Z {"context":{"package":"slonik","poolId":"xynV9RGHSBewXN4hftNfaQ-0","logLevel":20,"processId":-140462480,"stats":{"idleConnectionCount":0,"totalConnectionCount":1,"waitingRequestCount":0}},"message":"client is checked out from the pool","sequence":648,"time":1631127874554,"version":"1.0.0"}
2021-09-08T19:04:34.561488913Z {"context":{"package":"slonik","poolId":"xynV9RGHSBewXN4hftNfaQ-0","logLevel":20,"processId":-140462480,"stats":{"idleConnectionCount":0,"totalConnectionCount":1,"waitingRequestCount":0}},"message":"client is checked out from the pool","sequence":649,"time":1631127874560,"version":"1.0.0"}
2021-09-08T19:04:34.567543092Z {"context":{"package":"slonik","poolId":"xynV9RGHSBewXN4hftNfaQ-0","logLevel":20,"processId":-140462480,"stats":{"idleConnectionCount":0,"totalConnectionCount":1,"waitingRequestCount":0}},"message":"client is checked out from the pool","sequence":650,"time":1631127874567,"version":"1.0.0"}
2021-09-08T19:04:34.815734729Z {"context":{"package":"slonik","poolId":"xynV9RGHSBewXN4hftNfaQ-0","logLevel":20,"processId":-140462480,"stats":{"idleConnectionCount":0,"totalConnectionCount":1,"waitingRequestCount":0}},"message":"client is checked out from the pool","sequence":651,"time":1631127874814,"version":"1.0.0"}
My expression clean the time header added by azure, but it's not working anymore
It look like there are no "\n" on log streamed by azure... or it like it's in one block, so impossible to parse JSON ...
Do you have any idea? How do you achieve that?
The /g modifier must not being used here, just:
< azure.log sed 's/[^{]*//;s/[^}]*$//' | jq
Related
I want to check how many active meetings there are on the BBB server at any one time from the command line. I have tried
$ bbb-conf --network
but not getting anywhere. I have also checked the number of active connections to port 80 and 443
$ netstat -anp | grep :443 | grep ESTABLISHED | wc -l
but I'm not sure if I can trust that figure.
I know I can use the isMeetingRunning call from the API but I'm just looking for command line.
Any ideas would be appreciated
The following bash script, which can be run from command line on the same machine as the BigBlueButton server, will process the response to the BBB API getMeetings call.
#!/bin/bash
APICallName="getMeetings"
APIQueryString=""
X=$( bbb-conf --secret | fgrep URL: )
APIEndPoint=${X##* }
Y=$( bbb-conf --secret | fgrep Secret: )
Secret=${Y##* }
S=$APICallName$APIQueryString$Secret
Checksum=$( echo -n $S | sha1sum | cut -f 1 -d ' ' )
if [[ "$APIQueryString" == "" ]]
then
URL="${APIEndPoint}api/$APICallName?checksum=$Checksum"
else
URL="${APIEndPoint}api/$APICallName?$APIQueryString&checksum=$Checksum"
fi
wget -q -O - "$URL" | grep -o '<meetingID>' | wc -w
Tested on a live BBB machine.
Note:
The APICallName and APIQueryString can be modified to provide interface to other BBB API calls. See https://docs.bigbluebutton.org/dev/api.html
The command-line sha1sum will output a different result if a newline is appended to its input. This is the reason echo -n is used instead of echo.
In the last line, the script processes the XML output from the API call in a very naïve way, simply counting the number of occurences of the <meetingID> tag. More elaborate processing would probably require parsing the XML.
How can I delete release older than, for example, the 1st of October?
I mean updated before the 1st of October.
Alternatively, delete all releases with the app version lower than _.
Helm ls output:
|NAME|REVISION|UPDATED |STATUS |CHART |APP VERSION|NAMESPACE|
|myapp| 8 |Fri Sep 27 17:27:20 2019|DEPLOYED|myapp.85|85 |default|
The following command deletes just one.
helm delete relase_name
The following is not a great solution as well
helm delete relase_name1 relase_name2 relase_name3
Note1: I don't want to delete all. There is an explanation of how to do that over here Helm delete all releases and I don't want to do that. However, I assume I need to use bash for this task.
Note2: I have already read documentation it is not that big. There is nothing over there about filtering.
https://helm.sh/docs/helm/#helm-delete
Note3: I have already looked into helm sources, I am not 100% sure but It doesn't look like it is possible https://github.com/helm/helm/tree/master/pkg/helm
Thank you in advance!
The command bellow kind of worked for me, it is far from perfect solution but at least it helped me :
helm ls -d -m 25 --namespace default --short | xargs -L1 helm delete
helm ls - lists all of the releases.
-d orders by date.
-m maximum number of releases to fetch (so I take 25).
--namespace default - Show releases within a specific namespace, this option helped me to filter my app releases.
--short - this option limits output of the command to release names only.
The solution is not perfect and I hope that someone can provide a better solution.
Assuming Helm version 3+
You can use jq and xargs to accomplish this:
Question 1, to delete releases that were last updated before $TIMESTAMP (in seconds):
helm ls -o json | jq -r --argjson timestamp $TIMESTAMP '.[] | select (.updated | sub("\\..*";"Z") | sub("\\s";"T") | fromdate < now - $timestamp).name' | xargs -L1 helm uninstall
sub("\\..*";"Z") | sub("\\s";"T") converts the date format
Helm uses in their output to ISO 8601.
Question 2, to delete releases with an app version older than $VERSION:
helm ls -o json | jq -r --argjson version $VERSION '.[] | select(.app_version | sub("\\.[0-9]$";"") | tonumber | . < $version).name' | xargs -L1 helm uninstall
$VERSION should be major or major.minor only (i.e. 2 or
2.1). don't use a patch number.
Question 3, to delete releases by their initial deploy date, you'll have to parse the contents of the helm history RELEASE command for each release.
I won't solve this one, but it would look something like:
loop over helm ls
get release name
get first entry of helm history for that release
pass to jq and process like the answer for question 1
Relevant Links:
https://stedolan.github.io/jq/manual/#select(boolean_expression)
https://stedolan.github.io/jq/manual/#sub(regex;tostring)sub(regex;string;flags)
https://stedolan.github.io/jq/manual/#Dates
https://helm.sh/docs/helm/helm_history/
This is a sample my script for this:
DT=$(date --date="1 weeks ago" +%s)
LIST=$(helm list -A -d --short -f '-[0-9][0-9][0-9]+')
for nm in $LIST
do
DU=$(helm list -A -d -f $nm | awk '{print $4}' | sed 's/UPDATED//' )
DD=$(date --date="$DU" +%s)
if (( "$DT" >= "$DD" ))
then
NAMESPACE=$(helm list -A -f $nm | awk '{print $2}' | sed 's/NAMESPACE//' )
NAME=$nm
echo ---START---
echo $NAME
echo $NAMESPACE
echo ---END---
echo -e '\n'
fi
done
You have variables that is needed for you into the loop, and can do that you want.
For eg:
In my nexus repo - Temp-releases, under com/abc/temp/trial-platform-rpm, I have various folders like
6.1.7.0.34
6.1.8.1.3
7.0.0.0.568
7.0.1.0.89
7.0.2.0.544
So my script will provide the first 4 digits (For eg: if the branch selected is 7.0.2.0) then from nexus, I need to find latest version for this release (which is 7.0.2.0.544)
INPUT
7.0.2.0
OUTPUT
7.0.2.0.544
Curl command is as below and provide the version in grep :
curl -s "https://nexussite/nexus/service/local/repositories/Core-deloy/content/com/abc/item/item-portal-rpm/maven-metadata.xml" | grep "." | sort | uniq | sed -e "s#\(.\)\(\)\(.\)\(\)\(.\)#\3#g"| grep 7.0.1.0 | tail -n1""
I'm running a task in VSTS which performs some operations on a variable from a previous step and I then need to output the result to be used in future tasks. I have the following in a command line task running on a linux build host but am having no luck when trying to use the result later with $(podName3).
COMMAND="$(echo '$(pods)' | grep -oh -P '[^ ]*' | grep schema)"
##vso[task.setvariable variable=podName3]"$COMMAND"
I have tried several variations on this to no avail and need some direction as this has stumped me for too long now
Seems the syntax is incorrect.
Just try to below format:
COMMAND="$(echo '$pods' | grep -oh -P '[^ ]*' | grep schema)"
echo "##vso[task.setvariable variable=podName3]$COMMAND"
Or add a powershell task and run below commands to set the variable:
$COMMAND="$(echo '$env:pods' | grep -oh -P '[^ ]*' | grep schema)"
Write-Host "##vso[task.setvariable variable=podName3]$COMMAND"
More information please see Define and modify your variables in a script
I created a command line tool & an Azure DevOps task for this: https://marketplace.visualstudio.com/items?itemName=riezebosch.setvar
It just lets you pipe the output of a command into the tool and output it as the magic variable string. It's written in Go and cross compiled so works on all major platforms and all different shells.
Your example:
echo '$pods' | grep -oh -P '[^ ]*' | grep schema | setvar -name podName3
You only need to include the setvar task prior to this script task in order to get the tool on the agent.
Adding a PSA to this post, looks like they changed the way the variables are accessed now - you have to access variables like this: $(variable)
COMMAND=$(echo '$pods' | grep -oh -P '[^ ]*' | grep schema)
echo "##vso[task.setvariable variable=podName3]$(COMMAND)"
I am starting with fish and one of the things I could not find in the extensive documentation was autocomplete feeds.
There is mention of Tab Completions in the tutorial but it addresses existing the existence of the mechanism itself, not its configuration.
I have a bunch of virtual machines I connect to via
machinectl shell <name of machine> /bin/bash
I could make alises for all my machines via
function cm
machinectl $argv shell /bin/bash;
end
but this requires to remember and type the machine name.
How could I use the output of machinectl list | tail -n +2 | head -n -2 | cut -f1 -d' ' as a feed/hint to my cm command so that it shows them when using Tab?
EDIT: I somehow missed this right at the top of the documentation: Tab completion (I found it after reviewing the answers)
This should get you off to a good start:
complete --command cm --no-files \
--arguments '(machinectl list | tail -n +2 | head -n -2 | cut -f1 -d" ")'
Entering that at the command line will activate it for the current session; to make it permanent add the line to a completions file as Kurtis describes (~/.config/fish/completions/cm.fish).
See help complete. You'll find the completions that ship with fish, including completions for ssh, in $__fish_datadir/completions. A completion you write for a private function or command would be placed in ~/.config/fish/completions/$cmd_name.fish