Logs extracting, limiting cpulimit - sed

I have the following script for extraction of logs.
Need to make work it with cpulimit, to not overload the server. Can anyone help?
nice -20 zcat /var/detail-20150418.gz | sed '/./{H;$!d;};x;/46.241.178.96/!d' > /tmp/new.txt
nice -20 sed -n '/Acct-Status-Type/,/Called-Station-Id/p' /tmp/new.txt > /tmp/new_2.txt
rm /tmp/new.txt
nice -20 sed '/Acct-Status-Type/{x;p;x;}' /tmp/new_2.txt > /tmp/new_3.txt
rm /tmp/new_2.txt
grep -v '\(Acct-Authentic\|Acct-Input-Octets\|Acct-Input-Gigawords\|Acct-Output-Octets\|Acct-Output-Gigawords\|Acct-Input-Packets\|Acct-Output-Packets\|Acct-Session-Time\)' /tmp/new_3.txt > /tmp/new_4.txt
rm /tmp/new_3.txt
sed 's/^[ \t]*//;s/[ \t]*$//' /tmp/new_4.txt | paste -s -d ',' | sed 's/,,/\n/g' | sed 's/^[,]*//;s/[,]*$//' > /tmp/new_5.txt
rm /tmp/new_4.txt
sed 's/Acct-Status-Type = //' /tmp/new_5.txt | sed 's/User-Name = //' |sed 's/Event-Timestamp = //' | sed 's/Acct-Terminate-Cause = //' | sed 's/Framed-IP-Address = //' | sed 's/Called-Station-Id = //' > /tmp/new_6.txt
rm /tmp/new_5.txt
awk -F"," '{ print $3 "," $1 "," $2 "," $6 "," $5 "," $4}' /tmp/new_6.txt | sed -e 's/"//g' | sed -e 's/,,/,/g' > /tmp/log.txt
rm /tmp/new_6.txt

Getting rid of all that disk IO will help a lot:
nice -20 sh <<'END_SCRIPT' > /tmp/log.txt
zcat /var/detail-${date}.gz |
sed '/./{H;$!d;};x;/46.241.178.96/!d' |
sed -n '/Acct-Status-Type/,/Called-Station-Id/p' |
sed '/Acct-Status-Type/{x;p;x;}' |
grep -Ev '(Acct-Authentic|Acct-Input-Octets|Acct-Input-Gigawords|Acct-Output-Octets|Acct-Output-Gigawords|Acct-Input-Packets|Acct-Output-Packets|Acct-Session-Time)' |
sed 's/^[ \t]*//;s/[ \t]*$//' |
paste -s -d ',' |
sed -e 's/,,/\n/g' \
-e 's/^[,]*//;s/[,]*$//' \
-e 's/Acct-Status-Type = //' \
-e 's/User-Name = //' \
-e 's/Event-Timestamp = //' \
-e 's/Acct-Terminate-Cause = //' \
-e 's/Framed-IP-Address = //' \
-e 's/Called-Station-Id = //' |
awk -F"," '{ print $3 "," $1 "," $2 "," $6 "," $5 "," $4}' |
sed -e 's/"//g' |
sed -e 's/,,/,/g'
END_SCRIPT
All the commands except zcat can be consolidated into a single awk or perl script: I don't have the time to help you with that right now.

Related

sed search and replace \" but not \\"

I am trying to replace all escaped characters \" in a string with "" but not if \" is preceded by a \
So that input such as:
\"\"\"\" would return """"""""
\"\\"\"\" would return ""\\"""""
\" would return ""
\"\" would return """"
\\"\" would return \\"""
\"\\" would return ""\\"
\\\\\\\" would return \\\\\\\"
So far I have
$ echo sed -e 's/\([^\]\)\\"/\1""/;s/^\\"/""/'
but in the case of
$ echo '\"\"\"\"\"' | sed -e 's/\([^\]\)\\"/\1""/;s/^\\"/""/'`
I am getting incorrect results.
Any help would be appreciated.
This might work for you (GNU sed):
sed 's/\\\\"/\n/g;s/\\"/""/g;s/\n/\\\\"/g' file
Replace all occurances of the string you want untouched by something else (\n is a good choice), replace the string you want changed globally, reinstate the first set of strings.
How about this:
#!/bin/bash
function myreplace {
echo "$1" | sed -e "s/[\\]\"/MYDUMMY/g" \
-e 's/\\MYDUMMY/\\\\"/g' \
-e 's/MYDUMMY/""/g'
}
myreplace '\"\"\"\"'
myreplace '\"\\"\"\"'
myreplace '\"'
myreplace '\"\"'
myreplace '\\"\"'
myreplace '\"\\"'
myreplace '\\\\\\\"'
Executing the script above results in:
""""""""
""\\"""""
""
""""
\\"""
""\\"
\\\\\\\"
Using a sed loop will allow not having to pick a unique replacement string for an unknown dataset.
sed -e 's/^\\"/""/;:inner; s/\([^\]\)[\]"/\1""/g; t inner'
$ echo '\"\"\"\"' | sed -e 's/^\\"/""/;:inner; s/\([^\]\)[\]"/\1""/g;t inner'
""""""""
$ echo '\"\\"\"\"' | sed -e 's/^\\"/""/;:inner; s/\([^\]\)[\]"/\1""/g; t inner'
""\\"""""
$ echo '\"' | sed -e 's/^\\"/""/;:inner; s/\([^\]\)[\]"/\1""/g; t inner'
""
$ echo '\"\"' | sed -e 's/^\\"/""/;:inner; s/\([^\]\)[\]"/\1""/g; t inner'
""""
$ echo '\\"\"' | sed -e 's/^\\"/""/;:inner; s/\([^\]\)[\]"/\1""/g; t inner'
\\"""
$ echo '\"\\"' | sed -e 's/^\\"/""/;:inner; s/\([^\]\)[\]"/\1""/g; t inner'
""\\"
$ echo '\\\\\\\"' | sed -e 's/^\\"/""/;:inner; s/\([^\]\)[\]"/\1""/g; t inner'
\\\\\\\"

grep + grep + sed = sed: no input files

Can anybody help me please?
grep " 287 " file.txt | grep "HI" | sed -i 's/HIS/HID/g'
sed: no input files
Tried also xargs
grep " 287 " file.txt | grep HI | xargs sed -i 's/HIS/HID/g'
sed: invalid option -- '6'
This works fine
grep " 287 " file.txt | grep HI
If you want to keep your pipeline:
f=file.txt
tmp=$(mktemp)
grep " 287 " "$f" | grep "HI" | sed 's/HIS/HID/g' > "$tmp" && mv "$tmp" "$f"
Or, simplify:
sed -i -n '/ 287 / {/HI/ s/HIS/HID/p}' file.txt
That will filter out any line that does not contain " 287 " and "HI" -- is that what you want? I suspect you really want this:
sed -i '/ 287 / {/HI/ s/HIS/HID/}' file.txt
For lines that match / 287 /, execute the commands in braces. In there, for lines that match /HI/, search for the first "HIS" and replace with "HID". sed implicitly prints all lines if -n is not specified.
Other commands that do the same thing:
awk '/ 287 / && /HI/ {sub(/HIS/, "HID")} {print}' file.txt > new.txt
perl -i -pe '/ 287 / and /HI/ and s/HIS/HID/' file.txt
awk does not have an "in-place" option (except gawk -i inplace for recent gawk versions)

perl - Extract data using grep and sed

I'm using this code to get all titles from urls with http://something.txt:
#!/usr/bin/perl -w
$output = `cat source.html | grep -o '<a .*href=.*>' | grep -E 'txt' | sed -e 's/<a /\n<a /g' | sed -e 's/<a .*title="//' | cut -f1 -d '"'`;
print("$output");
When i run this on perl i get the error:
sed: -e expression #1, char 6: unterminated `s' command
The error is related with this portion of code:
sed -e 's/<a /\n<a /g'
In backquotes, Perl uses the same rules as in double quotes. Therefore, \n corresponds to a newline; you have to backslash the backslash to pass literal \ to the shell:
`sed -e 's/<a /\\n<a /g'`

Removing matching text from line

I have a example cut down from a log file.
112 172.172.172.1#50912 (ssl.bing.com):
I would like some how to remove the # and numbers after and (): from the url.
Would like the result.
112 172.172.172.1 ssl.bing.com
Here is the sed oneliner I have been working on.
cat newdns.log | sed -e 's/.*query: //' | cut -f 1 -d' ' | sort | uniq -c | sort -k2 > old.log
Thanks
Using sed, you could say:
sed 's/#[0-9]*//;s/(\(.*\)):$/\1/' filename
or, in a single substitution:
sed 's/#[0-9]* *(\(.*\)):$/ \1/' filename
Another sed:
sed -r 's/#[^ ]+|[():]//g'
$ echo '112 172.172.172.1#50912 (ssl.bing.com):' | sed -r 's/#[^ ]+|[():]//g'
112 172.172.172.1 ssl.bing.com

get list of sections from ini-file using shell (sed/awk)

I want to create a var from the section names of an ini file like:
[foo]
; ...
[bar]
; ...
[baz:bar]
;...
now I need a var like
SECTIONS="foo bar baz"
thanks in advance
One line solution could be:
export SECTIONS=`grep "^\[" test.ini |sort -u | xargs | tr '\[' ' ' | tr '\]' ' ' `
SECTIONS=$(crudini --get your.ini | sed 's/:.*//')
I'm now using this construct, don't need to know if a section exists. just read it, if it's empty it does not exist.
INI_FILE=test.ini
function ini_get
{
eval `sed -e 's/[[:space:]]*\=[[:space:]]*/=/g' \
-e 's/;.*$//' \
-e 's/[[:space:]]*$//' \
-e 's/^[[:space:]]*//' \
-e "s/^\(.*\)=\([^\"']*\)$/\1=\"\2\"/" \
< $INI_FILE \
| sed -n -e "/^\[$1\]/,/^\s*\[/{/^[^;].*\=.*/p;}"
echo ${!2}
}
IP=$(ini_get 50001 ip)
PORT=$(ini_get 50001 port)
echo $IP:$PORT