Shorten a sed command - using it to make asterisk call logs - sed

I know that asterisk creates it's own call logs in the form of a csv file. For my purposes I need the call logs formatted as I have depicted below. I use:
ls -l /var/spool/asterisk/monitor as the basis for my call logs, which produces this:
-rw------- 1 asterisk asterisk 112684 2013-02-07 17:24 20130207-172424-+15551235566-IN.wav
-rw------- 1 asterisk asterisk 44 2013-02-07 17:53 20130207-175311-+15554561122-IN.wav
-rw------- 1 asterisk asterisk 2019564 2013-02-07 18:00 20130207-175828-15554561122-OUT.wav
-rw------- 1 asterisk asterisk 44 2013-02-07 22:09 20130207-220805-15554561122-OUT.wav
-rw------- 1 asterisk asterisk 44 2013-02-07 22:12 20130207-221204-15551235566-OUT.wav
-rw------- 1 asterisk asterisk 111084 2013-02-07 22:13 20130207-221255-15551235566-OUT.wav
-rw------- 1 asterisk asterisk 364844 2013-02-07 22:39 20130207-223843-15558271212-OUT.wav
-rw------- 1 asterisk asterisk 4279404 2013-02-07 23:53 20130207-234836-5552785454-OUT.wav
-rw------- 1 asterisk asterisk 44 2013-02-08 00:00 20130208-000026-+15559813232-IN.wav
The part I need help with is my command below. It works and produces the exact results I want; however, it seems bulky to me. Can it be shortened?
variables
YESTER=$(date -d "-24 hours" +"%Y-%m-%d-%H%M")
TODAY=$(date +"%Y-%m-%d-%H%M_UTC")
create call log (command I'd like to change)
ls -l /var/spool/asterisk/monitor/ |grep '\.wav'|awk '{print $8 " " $5/1000000}'|sed -e 's/4\.4e\-05/NOT RECORDED/g' -e 's/\.wav//g' -e 's/-/ /g' -e 's/OUT/OUT - Approx Minutes:/g' -e 's/IN/IN - Approx Minutes:/g' -e 's/\(\.[0-9]\).*$/\1/g' -e 's/^.\{15\}/& UTC -/' -e 's/^.\{13\}/&:/' -e 's/^.\{11\}/&:/' -e 's/^.\{6\}/&-/' -e 's/^.\{4\}/& /' -e 's/+//g' > /var/spool/asterisk/monitor/call_logs/${YESTER}__${TODAY}-call-log.txt
For readability here is the command separated by line (without | ):
ls -l /var/spool/asterisk/monitor/
grep '\.wav'
awk '{print $8 " " $5/1000000}'
sed -e 's/4\.4e\-05/NOT RECORDED/g'
-e 's/\.wav//g'
-e 's/-/ /g'
-e 's/OUT/OUT - Approx Minutes:/g'
-e 's/IN/IN - Approx Minutes:/g'
-e 's/\(\.[0-9]\).*$/\1/g'
-e 's/^.\{15\}/& UTC -/'
-e 's/^.\{13\}/&:/'
-e 's/^.\{11\}/&:/'
-e 's/^.\{6\}/&-/'
-e 's/^.\{4\}/& /'
-e 's/+//g'
> /var/spool/asterisk/monitor/call_logs/${YESTER}__${TODAY}-call-log.txt
Output:
2013 02-07 17:24:24 UTC - 15551235566 IN - Approx Minutes: 0.1
2013 02-07 17:53:11 UTC - 15554561122 IN - Approx Minutes: NOT RECORDED
2013 02-07 17:58:28 UTC - 15554561122 OUT - Approx Minutes: 2.0
2013 02-07 22:08:05 UTC - 15554561122 OUT - Approx Minutes: NOT RECORDED
2013 02-07 22:12:04 UTC - 15551235566 OUT - Approx Minutes: NOT RECORDED
2013 02-07 22:12:55 UTC - 15551235566 OUT - Approx Minutes: 0.1
2013 02-07 22:38:43 UTC - 15558271212 OUT - Approx Minutes: 0.3
2013 02-07 23:48:36 UTC - 5552785454 OUT - Approx Minutes: 4.2
2013 02-08 00:00:26 UTC - 15559813232 IN - Approx Minutes: NOT RECORDED

You can put all formating into AWK, why you use sed?
To got it simple, use
[root#gleb monitor]# ls -l --time-style="+%Y-%m-%d %H:%M -"
-rw-r--r-- 1 asterisk asterisk 5195 2013-01-09 21:42 - 20130109-214242-1357756962.1658.WAV
-rw-r--r-- 1 asterisk asterisk 13450 2013-01-13 22:33 - 20130113-223350-1358105630.4124.WAV
Unfortanly i can't give full script, becuase i have other files. Based on such ls command output you now not need rewrite data, so can use it as column.
You can do full features processing including formating in single awk expression.
http://www.gnu.org/software/gawk/manual/html_node/Printf-Examples.html

$ cat tst.awk
{
mins = $5 / 1000000
mins = ( mins == "4.4e-05" ? "NOT RECORDED" : sprintf("%.1f",mins) )
split($8,fname,/-\+?|\./)
date = fname[1]
time = fname[2]
nrs = fname[3]
dir = fname[4]
printf "%s %s-%s ",substr(date,1,4),substr(date,5,2),substr(date,7,2)
printf "%s:%s:%s UTC - ",substr(time,1,2),substr(time,3,2),substr(time,5,2)
printf "%s %s - Approx Minutes: %s\n",nrs,dir,mins
}
$ awk -f tst.awk file
2013 02-07 17:24:24 UTC - 15551235566 IN - Approx Minutes: 0.1
2013 02-07 17:53:11 UTC - 15554561122 IN - Approx Minutes: NOT RECORDED
2013 02-07 17:58:28 UTC - 15554561122 OUT - Approx Minutes: 2.0
2013 02-07 22:08:05 UTC - 15554561122 OUT - Approx Minutes: NOT RECORDED
2013 02-07 22:12:04 UTC - 15551235566 OUT - Approx Minutes: NOT RECORDED
2013 02-07 22:12:55 UTC - 15551235566 OUT - Approx Minutes: 0.1
2013 02-07 22:38:43 UTC - 15558271212 OUT - Approx Minutes: 0.4
2013 02-07 23:48:36 UTC - 5552785454 OUT - Approx Minutes: 4.3
2013 02-08 00:00:26 UTC - 15559813232 IN - Approx Minutes: NOT RECORDED
So just do:
ls -l /var/spool/asterisk/monitor/*.wav | awk -f tst.awk

Although feasible in your case, you should generally avoid parsing ls. I would prefer to see a solution using find instead. If you have access to GNU awk, then you can simplify your pipeline considerably. Run like:
awk -f script.awk <(find /var/spool/asterisk/monitor/ -maxdepth 1 -type f -name "*.wav" -printf "%p %s\n" | sort -n)
Contents of script.awk:
BEGIN {
t = systime()
y = t - 60 * 60 * 24
t = strftime("%Y-%m-%d-%H%M_UTC", t)
y = strftime("%Y-%m-%d-%H%M", y)
}
{
p = "^..(....)(..)(..)-(..)(..)(..)-\\+?([^-]*)-([^\\.]*).*$"
r = "\\1 \\2-\\3 \\4:\\5:\\6 UTC - \\7 \\8 - Approx Minutes:"
s = ($2 != 44 ? sprintf("%.1f", $2/1000000) : "NOT RECORDED")
print gensub(p, r, "", $1) FS s > y "__" t "-call-log.txt"
}
In my testing, this generates a single log file containing your desired output. Because you are now using find, the method could be easily modified if your filesnames were to begin containing whitespace or newline characters. Please let me know how it goes. Cheers.

Related

Why logrotate doesn't properly postrotate only has 1 day delay

I have in /etc/logrotate.d/mikrotik :
/var/log/mikrotik.log {
rotate 2
daily
compress
dateext
dateyesterday
dateformat .%Y-%m-%d
postrotate
#/usr/sbin/invoke-rc.d syslog-ng reload >/dev/null
rsync -avH /var/log/mikrotik*.gz /backup/logs/mikrotik/
/usr/lib/rsyslog/rsyslog-rotate
endscript
}
The mikrotik.log.YYYY-MM-DD.gz file is created daily
The problem is that rsync in postrotate doesn't copy the last file. For example, on September 25, 2021, there are such files in /var/log:
-rw-r ----- 1 root adm 37837 Sep 24 23:49 mikrotik.log. 2021-09-24.gz
-rw-r ----- 1 root adm 36980 Sep 25 23:55 mikrotik.log. 2021-09-25.gz
and in /backup/logs/mikrotik/ are only:
-rw-r ----- 1 root adm 35495 Sep 23 00:00 mikrotik.log. 2021-09-22.gz
-rw-r ----- 1 root adm 36842 Sep 23 23:58 mikrotik.log. 2021-09-23.gz
-rw-r ----- 1 root adm 37837 Sep 24 23:49 mikrotik.log. 2021-09-24.gz
There is no file mikrotik.log.2021-09-25.gz from Sep 25 23:55 it will not be copied until the next rotation.
How to make a file packed today copied by postrotate ?
Problem solved.
It relied on the order in which the operations were performed.
Lgrotate does a 'postrotate' section before compressing to .gz.
The solution to the problem was to change the name from 'postrotate' to 'lastaction'.

Perl touch -t file error for a future date

I am trying to touch a file(for referencing date) with a future date something like -
Current date - $date
Fri Jan 6 03:59:55 EST 2017
touch -t 201702032359.59 /var/tmp/ME_FILE_END
on checking the timestamp of the file as -
$ ls -lrt /var/tmp/ME_FILE_END
getting an output with only date and not the entire timestamp(hhmm.sec)
-rw-r--r-- 1 abcproc abc 0 Feb 3 2017 /var/tmp/ME_FILE_END
But for a date with is less than or equal to current it gives correct result -
touch -t 201612010000.00 /var/tmp/ME_FILE_START
ls -lrt /var/tmp/ME_FILE_START
-rw-r--r-- 1 abcproc abc 0 Dec 1 00:00 /var/tmp/ME_FILE_START
Can someone please suggest why this discrepancy ?
It's just the way ls displays the date. When far from now, the modification time is not displayed.
If you want details regarding the last access / modification / change time, you should be using stat.
stat /var/tmp/ME_FILE_END
You will see the expected output.
For example:
[10:29:41]dabi#gaia:~$ touch -t 201702032359.59 /var/tmp/ME_FILE_END
[10:29:43]dabi#gaia:~$ ls -ltr /var/tmp/ME_FILE_END
-rw-rw-r-- 1 dabi dabi 0 feb. 3 2017 /var/tmp/ME_FILE_END
[10:29:47]dabi#gaia:~$ stat /var/tmp/ME_FILE_END
File : '/var/tmp/ME_FILE_END'
Size : 0 Blocks : 0 I/O blocks : 4096 empty file
Device : 803h/2051d Inode : 5374373 Links : 1
Access : (0664/-rw-rw-r--) UID : ( 1000/ dabi) GID : ( 1000/ dabi)
Access : 2017-02-03 23:59:59.000000000 +0100
Change : 2017-02-03 23:59:59.000000000 +0100
Change : 2017-01-06 10:29:43.364630503 +0100
Birth : -

Convert Unix `cal` output to latex table code: one-liner solution?

Trying to achieve the following struggled my mind:
Convert Unix cal output to latex table code, using a short and sweet one-liner (or few-liner).
E.g cal -h 02 2012 | $magicline should yield
Mo &Tu &We &Th &Fr \\
& & 1 & 2 & 3 \\
6 & 7 & 8 & 9 &10 \\
13 &14 &15 &16 &17 \\
20 &21 &22 &23 &24 \\
27 &28 & & & \\
The only reasonable solution I could come up with so far was
cal -h | sed -r -e '1d' -e \
's/^(..)?(...)?(...)?(...)?(...)?(...)?(...)?$/\2\t\&\3\t\&\4\t\&\5\t\&\6\t\\\\/'
... and I really tried hard. The nice thing about it being that it's uncomplicated and easy to understand, the bad thing about it that it's "unflexible" (It couldn't cope with a week of 8 days) and a little verbose. I'm looking for alternative solutions to learn from ;-)
EDIT: Found another one that seems acceptable
cal -h | tail -n +2 |
perl -ne 'chomp;
$,="\t&";
$\="\t\\\\\n";
$line=$_;
print map {substr($line,$_*3,3)} (1..5)'
EDIT: Nice one:
cal -h | perl \
-F'(.{1,3})' -ane \
'BEGIN{$,="\t&";$\="\t\\\\\n"}
next if $.==1;
print #F[3,5,7,9,11]'
Tested on OS-X:
cal 02 2012 |grep . |tail +2 |perl -F'/(.{3})/' -ane \
'chomp(#F=grep $_,#F); $m=$#F if !$m; printf "%s"."\t&%s"x$m."\t\\\\\n", #F;'
Where cal output has 3-character columns; {3} could be changed to match your cal output.
Using the GNU version of awk:
My output of cal using an english LANG.
Command:
LANG=en_US cal
Output:
February 2012
Su Mo Tu We Th Fr Sa
1 2 3 4
5 6 7 8 9 10 11
12 13 14 15 16 17 18
19 20 21 22 23 24 25
26 27 28 29
The awk one-line:
LANG=en_US cal | awk '
BEGIN {
FIELDWIDTHS = "3 3 3 3 3 3 3";
OFS = "&";
}
FNR == 1 || $0 ~ /^\s*$/ { next }
{
for (i=2; i<=6; i++) {
printf "%-3s%2s", $i, i < 6 ? OFS : "\\\\";
}
printf "\n";
}'
Result:
Mo &Tu &We &Th &Fr \\
& & 1 & 2 & 3 \\
6 & 7 & 8 & 9 &10 \\
13 &14 &15 &16 &17 \\
20 &21 &22 &23 &24 \\
27 &28 &29 & & \\
cal 02 2012|perl -lnE'$.==1||eof||do{$,="\t&";$\="\t\\\\\n";$l=$_;print map{substr($l,$_*3,3)}(1..5)}'
my new favorite:
cal 02 2012|perl -F'(.{1,3})' -anE'BEGIN{$,="\t&";$\="\t\\\\\n"}$.==1||eof||do{$i//=#F;print#F[map{$_*2-1}(1..$i/2)]}'
This might work for you:
cal | sed '1d;2{h;s/./ /g;x};/^\s*$/b;G;s/\n/ /;s/^...\(.\{15\}\).*/\1/;s/.../ &\t\&/g;s/\&$/\\\\/'
This works for my implementation of cal, which uses four-character columns and has an initial title line showing the month and year
cal | perl -pe "next if $.==1;s/..../$&&/g;s/&$/\\\\/"
It looks as though yours may have eight-character columns and has no title line, in which case
cal | perl -pe "s/.{8}/$&&/g;s/&$/\\\\/"
should do the trick, but be prepared to tweak it.
cal -h 02 2012| cut -c4-17 | sed -r 's/(..)\s/\0\t\&/g' | sed 's/$/\t\\\\/' | head -n-1 | tail -n +2
This will produce:
Mo &Tu &We &Th &Fr \\
& & 1 & 2 & 3 \\
6 & 7 & 8 & 9 &10 \\
13 &14 &15 &16 &17 \\
20 &21 &22 &23 &24 \\
27 &28 &29 & & \\
You can easily replace \t with number of spaces you wish

SED: How to remove every 10 lines in a file (thin or subsample the file)

I have this so far:
sed -n '0,10p' yourfile > newfile
But it is not working, just outputs a blank file :(
Your question is ambiguous, so here is every permutation I can think of:
Print only the first 10 lines
head -n10 yourfile > newfile
Skip the first 10 lines
tail -n+10 yourfile > newfile
Print every 10th line
awk '!(NR%10)' yourfile > newfile
Delete every 10th line
awk 'NR%10' yourfile > newfile
(Since an ambiguous questions can only have an ambiguous answer...)
To print every tenth line (GNU sed):
$ seq 1 100 | sed -n '0~10p'
10
20
30
40
...
100
Alternatively (GNU sed):
$ seq 1 100 | sed '0~10!d'
10
20
30
40
...
100
To delete every tenth line (GNU sed):
$ seq 1 100 | sed '0~10d'
1
...
9
11
...
19
21
...
29
31
...
39
41
...
To print the first ten lines (POSIX):
$ seq 1 100 | sed '11,$d'
1
2
3
4
5
6
7
8
9
10
To delete the first ten lines (POSIX):
$ seq 1 100 | sed '1,10d'
11
12
13
14
...
100
python -c "import sys;sys.stdout.write(''.join(line for i, line in enumerate(open('yourfile')) if i%10 == 0 ))" >newfile
It is longer, but it is a single language - not different syntax and aprameters for each thing one tries to do.
With non-GNU sed, to print every 10th line use
sed '10,${p;n;n;n;n;n;n;n;n;n;}'
(GNU : sed -n '0~10p')
and to delete every 10th line use
sed 'n;n;n;n;n;n;n;n;n;d;'
(GNU : sed -n '0~10d')

How to calculate last business day of the month in Korn Shell?

I've seen this question answered in other languages but not the Korn Shell. I need to prevent a script from being run on the last business day of the month (we can assume M-F are business days, ignore holidays).
This function works in Bash, Korn shell and zsh, but it requires a date command (such as GNU date) that has the -d option:
function lbdm { typeset lbdm ldm dwn m y; (( m = $1 + 1 )); if [[ $m = 13 ]]; then m=1; (( y = $2 + 1 )); else y=$2; fi; ldm=$(date -d "$m/1/$y -1 day"); dwn=$(date -d "$ldm" +%u);if [[ $dwn = 6 || $dwn = 7 ]]; then ((offset = 5 - $dwn)); lbdm=$(date -d "$ldm $offset day"); else lbdm=$ldm; fi; echo $lbdm; }
Run it like this:
$ lbdm 10 2009
Fri Oct 30 00:00:00 CDT 2009
Here is a demo script broken into separate lines and with better variable names and some comments:
for Month in {1..12} # demo a whole year
do
Year=2009
LastBusinessDay=""
(( Month = $Month + 1 )) # use the beginning of the next month to find the end of the one we're interested in
if [[ $Month = 13 ]]
then
Month=1
(( Year++ ))
fi;
# these two calls to date could be combined and then parsed out
# this first call is in "American" order, but could be changed - everything else is localized - I think
LastDayofMonth=$(date -d "$Month/1/$Year -1 day") # get the day before the first of the month
DayofWeek=$(date -d "$LastDayofMonth" +%u) # the math is easier than Sun=0 (%w)
if [[ $DayofWeek = 6 || $DayofWeek = 7 ]] # if it's Sat or Sun
then
(( Offset = 5 - $DayofWeek )) # then make it Fri
LastBusinessDay=$(date -d "$LastDayofMonth $Offset day")
else
LastBusinessDay=$LastDayofMonth
fi
echo "$LastDayofMonth - $DayofWeek - $LastBusinessDay"
done
Output:
Sat Jan 31 00:00:00 CST 2009 - 6 - Fri Jan 30 00:00:00 CST 2009
Sat Feb 28 00:00:00 CST 2009 - 6 - Fri Feb 27 00:00:00 CST 2009
Tue Mar 31 00:00:00 CDT 2009 - 2 - Tue Mar 31 00:00:00 CDT 2009
Thu Apr 30 00:00:00 CDT 2009 - 4 - Thu Apr 30 00:00:00 CDT 2009
Sun May 31 00:00:00 CDT 2009 - 7 - Fri May 29 00:00:00 CDT 2009
Tue Jun 30 00:00:00 CDT 2009 - 2 - Tue Jun 30 00:00:00 CDT 2009
Fri Jul 31 00:00:00 CDT 2009 - 5 - Fri Jul 31 00:00:00 CDT 2009
Mon Aug 31 00:00:00 CDT 2009 - 1 - Mon Aug 31 00:00:00 CDT 2009
Wed Sep 30 00:00:00 CDT 2009 - 3 - Wed Sep 30 00:00:00 CDT 2009
Sat Oct 31 00:00:00 CDT 2009 - 6 - Fri Oct 30 00:00:00 CDT 2009
Mon Nov 30 00:00:00 CST 2009 - 1 - Mon Nov 30 00:00:00 CST 2009
Thu Dec 31 00:00:00 CST 2009 - 4 - Thu Dec 31 00:00:00 CST 2009
Note: I discovered during testing that if you try to use this for dates around World War II that it fails due to wartime time zones like CWT and CPT.
Edit: Here's a version that should run on AIX and other systems that can't use the above. It should work on Bourne, Bash, Korn and zsh.
function lbdN { cal $1 $2 | awk 'NF == 0 {next} FNR > 2 {week = $0} END {num = split(week, days); lbdN = days[num]; if ( num == 1 ) { lbdN -= 2 }; if ( num == 7 ) { lbdN-- }; print lbdN }'; }
You may have to make adjustments if your cal starts weeks on Monday.
Here's how you can use it:
month=12; year=2009 # if these are unset or null, the current month/year will be used
if [[ $(date +%d) == $(lbdN $month $year) ]];
then
echo "Don't do stuff today"
else
echo "It's not the last business day of the month"
fi
making appropriate adjustments for your shell's if...then syntax, of course.
Edit: Bug Fix: The previous version of lbdN failed when February ends on Saturday the 28th because of the way it used tail. The new version fixes that. It uses only cal and awk.
Edit: For completeness, I thought it would be handy to include functions for the first business day of the month.
Requires date with -d:
function fbdm { typeset dwn d; dwn=$(date -d "$1/1/$2" +%u); d=1; if [[ $dwn = 6 || $dwn = 7 ]]; then (( d = 9 - $dwn )); fi; echo $(date -d "$1/$d/$2"); }
For May 2010:
Mon May 3 00:00:00 CDT 2010
Requires cal and awk only:
function fbdN { cal $1 $2 | awk 'FNR == 3 { week = $0 } END { num = split(week, days); fbdN = days[1]; if ( num == 1 ) { fbdN += 2 }; if ( num == 7 ) { fbdN++ }; print fbdN }'; }
For August 2010:
2