I need to get job duration via curl command.
I can get access to it
curl -s $JENKINS_JOB_URL/146/api/json?tree=duration
{"duration":14182142}
How convert the 14182142 to "3 hr 56 min" ? ( via ksh, or in the curl command )
Thanks
You could always use some python:
python -c"import datetime;print datetime.timedelta(milliseconds=14182142);"
3:56:22.142000
Is awk acceptible?
curl -s $JENKINS_JOB_URL/146/api/json?tree=duration | awk -F : '{ print $2 }' | awk -F } '{ printf "%i hr %i min", ($1 / 3600000), ($1 % 3600000)/60000 }'
If anyone knows of a nice way to get the number out of the string and into awk, I can remove the nasty double-invocation of awk.
Related
What is working :
echo "Oct 12 2021" | awk '{print "date -d\""$1FS$2FS$3"\" +%Y%m%d"}' | bash
how could I use a variable on a bash script ?
mydate="Oct 12 2021"
awk -v dateformat= ''$mydate" '{print "date -d\""$1FS$2FS$3"\" +%Y%m%d"}'
You obviously wouldn't really use awk to call date on a single value but I assume you have something more than that in mind in your real code so this is what you asked for (call GNU date from awk using an awk variable as the date):
mydate="Oct 12 2021"
awk -v date="$mydate" 'BEGIN {
system("date -d\047" date "\047 +\047%Y%m%d\047")
}'
20211012
or if you prefer:
awk -v date="$mydate" 'BEGIN {
cmd = "date -d\047" date "\047 +\047%Y%m%d\047"
print ( (cmd | getline line) > 0 ? line : date"=N/A" )
close(cmd)
}'
20211012
Don't do either of those though. It's very slow to call an external command from awk and you don't need to when all you want to do is change text from one format to another:
$ awk -v date="$mydate" 'BEGIN {
split(date,d);
mth = (index("JanFebMarAprMayJunJulAugSepOctNovDec",d[1])+2)/3
printf "%04d%02d%02d\n", d[3], mth, d[2]
}'
20211012
Also, if you have GNU awk it has built-in time functions so even if what you wanted to do wasn't just shuffling bits of text around you could call the internal mktime() and strftime() instead of the external date:
awk -v date="$mydate" 'BEGIN {
split(date,d)
mth = (index("JanFebMarAprMayJunJulAugSepOctNovDec",d[1])+2)/3
secs = mktime(d[3]" "mth" "d[2]" 12 0 0")
print strftime("%Y%m%d",secs)
}'
20211012
Your original example uses awk to prepare arguments that are then passed to the Linux date command running in bash; the date formatting is performed by the date command.
You don't need awk for this, the date command -d option is very flexible in the format it receives.
mydate="Oct 3 2021"
date -d "$mydate" +%Y%m%d
Will give you a formatted date. You can assign the result to a variable using the $() syntax - run a command and assign result
formattedDate=$(date -d "$mydate" +%Y%m%d)
echo $formattedDate
I have a bash script which echos out an html file like
this ./foo.sh > out.html In the html there are timestamps in the
following format 2019-02-24T17:02:19Z.
I wrote a function to
convert the time stamp to the time delta between the timestamp
and now.
my_uptime(){
START=$(date -d "$1" "+%s")
NOW=$(date "+%s")
STD=$(echo "($NOW-$START)/3600" | bc)
MIN=$(echo "(($NOW-$START)/60)%60" | bc)
echo "Uptime $STD h $MIN min"
}
Now I want to replace the timestamp with the output of my_uptime
directly in the stream. I tried this:
echo "<p>some html</p>
2019-02-24T17:02:19Z
<p>some more html</p>" | sed -r "s/[0-9\-]+T[0-9:]+Z/$(my_uptime \0)/"
This fails because the command substitution doesn't recognize the
back reference and puts in a literal 0. Is there another way to
achieve this? Preferably directly in the stream.
... | sed -r "s/[0-9\-]+T[0-9:]+Z/$(my_uptime \0)/"
This code is attempting to pass the matched value from sed's s/// into the shell function. However, $(...) is expanded before sed even sees it.
Using sed is probably not appropriate here.
Here's a perl replacement that effectively combines your shell function and sed:
... | perl -ple '
if (/([0-9-]+T[0-9:]+Z)/) {
$s = `date -d "$1" +%s`;
$n = time;
$h = int(($n-$s)/3600);
$m = int(($n-$s)/60)%60;
$_ = "Uptime $h h $m min";
}'
You could probably do something similar in awk.
For testing one of my application I need some huge data in parquet format. which I don't have.I have written a shell script which is performing slow.I wanted go with spark.How can I generate random data using spark scala.
Each filed has to be in specific range.Id should be in between (1-10),date(any date from 2010-2018),start and end time should be any.
My shell scipt.
!/bin/bash
if [ $# -eq 2 ]; then
LIMIT=40 # to generate 40KB file
for((i=0;i<$2;i++))
{
FILE_NAME="$1$i.csv"
echo "id,date,start_time,end_time,distance,amount,persons,longitude,latitude" >> "$FILE_NAME"
while [ $(du -k $FILE_NAME | cut -f 1) -le $LIMIT ]
do
start_time=`date -d "$(date +%H:%M:%S) + $(shuf -i 0-24 -n 1) hours $(shuf -i 0-60 -n 1) minutes $(shuf -i 0-60 -n 1) seconds" +'%H:%M:%S'`
echo "`shuf -i 1-10 -n 1`,`date -d "2011-01-01 + $(shuf -i 1-2557 -n 1) days" +'%m-%d-%Y'`,$start_time,`date -d "$start_time + $(shuf -i 1-6 -n 1) hours $(shuf -i 0-60 -n 1) minutes $(shuf -i 0-60 -n 1) seconds" +'%H:%M:%S'`,`shuf -i 1-60 -n 1`,`shuf -i 100-1500 -n 1`,`shuf -i 1-6 -n 1`,`shuf -i 10-99 -n 1`.`shuf -i 100000-999999 -n 1`,`shuf -i 10-99 -n 1`.`shuf -i 100000-999999 -n 1`" >> "$FILE_NAME"
done
}
else
printf "Usage: sh GenerateCSV.sh <filename without extension> <No of files to generate> \nThe files will be generated with .csv extension\n"
fi
I want data to be like this,which should be parquet format.
2,20-10-2010,23:18:10,23:40:40
How can I do it in spark.
You can try this option.
Below are the Unix Timestamp values for the dates mentioned.
val ss = SparkSession.builder().appName("local").master("local[*]").getOrCreate()
ss.sqlContext.sql("SELECT unix_timestamp ('2010-01-01', 'yyyy-MM-dd')") // 1262284200
ss.sqlContext.sql("SELECT unix_timestamp ('2018-12-31', 'yyyy-MM-dd')") // 1546194600
You can code in this below way for generating random number between the above numbers.
val r = new scala.util.Random
val x = 1262284200 + r.nextInt((1546194600-1262284200))
You can code in this below way for generating random date between the dates by using above generated value x
ss.sqlContext.sql(s"SELECT FROM_UNIXTIME($x)")
Often I need to analyze large logs in console.
I use the following command to colorize important keywords:
echo "string1\nerror\nsuccess\nstring2\nfail" | perl -p -e 's/(success)/\e[1;32;10m$&\e[0m/g;' -e 's/(error|fail)/\e[0;31;10m$&\e[0m/g'
It will colorize "success" with green, and error messages with red and keeps others lines unchanged (as they contain some useful info).
But in some cases I need to colorize values after some marker, but not marker itself, i.e. in these lines
Marker1: value1
Marker2: value2
need to highlight only value1 and value2 by known markers.
I'm looking for a way to modify my current oneliner to add this function
Also I tried the following solution, which I like less
#!/bin/bash
default=$(tput op)
red=$(tput setaf 1 || tput AF 1)
green=$(tput setaf 2 || tput AF 2)
sed -u -r "s/(Marker1: )(.+)$/\1${red}\2${default}/
s/(Marker2: )(.+)$/\1${green}\2${default}/" "${#}"`
But it has some problem with buffering, so it's ok for some constant file, but log which is continuosly running is not displayed at all
UPDATE:
Found a solution with help of some perl guru.
echo -e "string1\nerror\nsuccess\nstring2\nfail\nMaker1: value1\nMaker2: value2" | \
perl -p \
-e 's/(success)/\e[32m$&\e[0m/g;' \
-e 's/(error|fail)/\e[31m$&\e[0m/g;' \
-e 's/(Maker1:) (.*)/$1 \e[36m$2\e[0m/m;' \
-e 's/(Maker2:) (.*)/$1 \e[01;34m$2\e[0m/m;'
echo -e "string1\nerror\nsuccess\nstring2\nfail\nMaker1: value1\nMaker2: value2" | \
perl -p \
-e 's/(success)/\e[32m$&\e[0m/g;' \
-e 's/(error|fail)/\e[31m$&\e[0m/g;' \
-e 's/(Maker1:) (.*)/$1 \e[36m$2\e[0m/m;' \
-e 's/(Maker2:) (.*)/$1 \e[01;34m$2\e[0m/m;'
#!/bin/bash
default=$(tput op)
red=$(tput setaf 1 || tput AF 1)
green=$(tput setaf 2 || tput AF 2)
#default='e[0m'
#red='e[0;31;10m'
#green='e[1;32;10m'
# automaticaly use passed argument file if any or stdin if not
sed -u -r \
"/success/ s//${green}&${default}/
/error|fail/ s//${red}&${default}/
/^Marker1:/ {s//\1${red}/;s/$/${default}/;}
/^Marker2:/ {s//\1${green}/;s/$/${default}/;}" \
$( [ ${##} -gt 0 ] && echo ${#} )
For a one line:
remove other line thans sed one
replace newline in sed by ;
use directly the terminal code in place of variable
remove the last line if you pipe or use specific file instead
I'm trying to extract data/urls (in this case - someurl) from a file that contains them within some tag ie.
xyz>someurl>xyz
I don't mind using either awk or sed.
I think the best, easiest, way is with cut:
$ echo "xyz>someurl>xyz" | cut -d'>' -f2
someurl
With awk can be done like:
$ echo "xyz>someurl>xyz" | awk 'BEGIN { FS = ">" } ; { print $2 }'
someurl
And with sed is a little bit more tricky:
$ echo "xyz>someurl>xyz" | sed 's/\(.*\)>\(.*\)>\(.*\)/\2/g'
someurl
we get blocks of something1<something2<something3 and print the 2nd one.
grep was born to extract things:
kent$ echo "xyz>someurl>xyz"|grep -Po '>\K[^>]*(?=>)'
someurl
you could kill a fly with a bomb of course:
kent$ echo "xyz>someurl>xyz"|awk -F\> '$0=$2'
someurl
If your grep supports P option then you can use lookahead and lookbehind regular expression to identify the url.
$ echo "xyz>someurl>xyz" | grep -oP '(?<=xyz>).*(?=>xyz)'
someurl
This is just a sample to get you started not the final answer.