Hi I have lots of logfiles with ^[[1m (as vim displays it) in them. I want to watch a logfile life via
tail -n 1000 -f logfile.log | grep <expression-for-escape-sequence>
and only get lines that have bold in them.
I am not sure which grep options I should use and have tried the following already:
tail -n 1000 -f logfile.log | grep "\033\0133\061\0155"
tail -n 1000 -f logfile.log | grep "\033\01331m"
tail -n 1000 -f logfile.log | grep "\033\[1m"
It does not work though... And yes there are bold lines in the last 1000 lines of logfile.log, testing with
echo -e "\033\01331mTest\033\01330m" | grep ...
same results... ;)
Appreciate any help!
Use single quotes with a dollar sign in front—as in $'...'—to have the shell convert the \033 escape sequence into an ESC character:
tail -n 1000 -f logfile.log | grep $'\033\[1m'
From man bash:
Words of the form $'string' are treated specially. The word expands to string, with backslash-escaped characters replaced as specified by the ANSI C standard.
This works (in a POSIX shell, not necessarily bash):
echo -e "\033\01331mTest\033\01330m" | grep "$(printf "\x1b\\[1m")"
Related
This is mostly by curiosity, I am trying to have the same behavior as:
echo -e "test1:test2:test3"| sed 's/:/\n/g' | grep 1
in a single sed command.
I already tried
echo -e "test1:test2:test3"| sed -e "s/:/\n/g" -n "/1/p"
But I get the following error:
sed: can't read /1/p: No such file or directory
Any idea on how to fix this and combine different types of commands into a single sed call?
Of course this is overly simplified compared to the real usecase, and I know I can get around by using multiple calls, again this is just out of curiosity.
EDIT: I am mostly interested in the sed tool, I already know how to do it using other tools, or even combinations of those.
EDIT2: Here is a more realistic script, closer to what I am trying to achieve:
arch=linux64
base=https://chromedriver.storage.googleapis.com
split="<Contents>"
curl $base \
| sed -e 's/<Contents>/<Contents>\n/g' \
| grep $arch \
| sed -e 's/^<Key>\(.*\)\/chromedriver.*/\1/' \
| sort -V > out
What I would like to simplify is the curl line, turning it into something like:
curl $base \
| sed 's/<Contents>/<Contents>\n/g' -n '/1/p' -e 's/^<Key>\(.*\)\/chromedriver.*/\1/' \
| sort -V > out
Here are some alternatives, awk and sed based:
sed -E "s/(.*:)?([^:]*1[^:]*).*/\2/" <<< "test1:test2:test3"
awk -v RS=":" '/1/' <<< "test1:test2:test3"
# or also
awk 'BEGIN{RS=":"} /1/' <<< "test1:test2:test3"
Or, using your logic, you would need to pipe a second sed command:
sed "s/:/\n/g" <<< "test1:test2:test3" | sed -n "/1/p"
See this online demo. The awk solution looks cleanest.
Details
In sed solution, (.*:)?([^:]*1[^:]*).* pattern matches an optional sequence of any 0+ chars and a :, then captures into Group 2 any 0 or more chars other than :, 1, again 0 or more chars other than :, and then just matches the rest of the line. The replacement just keeps Group 2 contents.
In awk solution, the record separator is set to : and then /1/ regex is used to only return the record having 1 in it.
This might work for you (GNU sed):
sed 's/:/\n/;/^[^\n]*1/P;D' file
Replace each : and if the first line in the pattern space contains 1 print it.
Repeat.
An alternative:
sed -Ez 's/:/\n/g;s/^[^1]*$//mg;s/\n+/\n/;s/^\n//' file
This slurps the whole file into memory and replaces all colons by newlines. All lines that do not contain 1 are removed and surplus newlines deleted.
An alternative to the really ugly sed is: grep -o '\w*2\w*'
$ printf "test1:test2:test3\nbob3:bob2:fred2\n" | grep -o '\w*2\w*'
test2
bob2
fred2
grep -o: only matching
Or: grep -o '[^:]*2[^:]*'
echo -e "test1:test2:test3" | sed -En 's/:/\n/g;/^[^\n]*2[^\n]*(\n|$)/P;//!D'
sed -n doesn't print unless told to
sed -E allows using parens to match (\n|$) which is newline or the end of the pattern space
P prints the pattern buffer up to the first newline.
D trims the pattern buffer up to the first newline
[^\n] is a character class that matches anything except a newline
// is sed shorthand for repeating a match
//! is then matching everything that didn't match previously
So, after you split into newlines, you want to make sure the 2 character is between the start of the pattern buffer ^ and the first newline.
And, if there is not the character you are looking for, you want to D delete up to the first newline.
At that point, it works for one line of input, with one string containing the character you're looking for.
To expand to several matches within a line, you have to ta, conditionally branch back to label :a:
$ printf "test1:test2:test3\nbob3:bob2:fred2\n" | \
sed -En ':a s/:/\n/g;/^[^\n]*2[^\n]*(\n|$)/P;D;ta'
test2
bob2
fred2
This is simply NOT a job for sed. With GNU awk for multi-char RS:
$ echo "test1:test2:test3:test4:test5:test6"| awk -v RS='[:\n]' '/1/'
test1
$ echo "test1:test2:test3:test4:test5:test6"| awk -v RS='[:\n]' 'NR%2'
test1
test3
test5
$ echo "test1:test2:test3:test4:test5:test6"| awk -v RS='[:\n]' '!(NR%2)'
test2
test4
test6
$ echo "foo1:bar1:foo2:bar2:foo3:bar3" | awk -v RS='[:\n]' '/foo/ || /2/'
foo1
foo2
bar2
foo3
With any awk you'd just have to strip the \n from the final record before operating on it:
$ echo "test1:test2:test3:test4:test5:test6"| awk -v RS=':' '{sub(/\n$/,"")} /1/'
test1
I have a log that produce lots of text in the line along side the string that i want to get. basically it contains something like:
bla bla bla packet 12 out of 432 bla bla
I have this big command:
tail -f log.txt |grep --line-buffered "packet" |sed -n 's/.*\(packet [0-9]* out of [0-9]*\).*/\1/p' |while read log; do echo "$(date +%F_%H:%m:%S:%N) $log" ; done
and I want to redirect the output to file.
Why >> file does not work? What am I doing wrong?
The reason for this issue is that the sed command holds the output data in a buffer. You are avoided the data synchronization issue for the grep command (2nd in the pipe chain) by using --line-buffered parameter, using -u parameter for next piped sed command help to fix this issue. The command should be:
tail -f log.txt | grep --line-buffered "packet" | sed -n -u 's/.*\(packet [0-9]* out of [0-9]*\).*/\1/p' | while read log; do echo "$(date +%F_%H:%m:%S:%N) $log" >> outputfile.log ; done
sed -n -u 's/.(packet [0-9] out of [0-9])./\1/p'
Instead of
sed -n 's/.(packet [0-9] out of [0-9])./\1/p'
With this change the redirect to file should work.
I use sed to get the content of file from a desire point but I have a problem.
I can not print $variable value into this sed command
count=$(sed -n '/$variable/,$p' file.log | grep '"KO"' -c)
I try with double quotes and close the single but not working
count=$(sed -n "/$variable/,$p" file.log | grep '"KO"' -c) ERROR unexpected `,'
count=$(sed -n '/'$variable'/,$p' file.log | grep '"KO"' -c) ERROR unterminated address regex
I know that the sed reseach is letteral "$variable" but I can not pass the value...
Thanks in advance.
It's a question of getting the quoting right.
Your first example:
count=$(sed -n '/$variable/,$p' file.log | grep '"KO"' -c)
doesn't expand $variable because it's in single quotes, the second:
count=$(sed -n "/$variable/,$p" file.log | grep '"KO"' -c)
expands $variable but has issues with its contents, as mentioned by choroba. It also has issue with the $p which will be interpreted as a shell variable. Your third example:
count=$(sed -n '/'$variable'/,$p' file.log | grep '"KO"' -c)
comes pretty close to what you need, but still suffers if $variable contains characters that sed treats specially, so these need to be escaped, e.g. the following works:
variable="\[17-09-12 00:01:03\]"
count=$(sed -n '/'$variable'/,$p' file.log
And as brackets are also special to the shell you can escape them automatically with the printf %q directive:
variable="[17-09-12 00:01:03]"
variable=$(printf "%q" "$variable")
count=$(sed -n '/'$variable'/,$p' file.log
[ has a special meaning in sed. I would use something more powerful than sed, i.e. Perl. It can escape the variable for you:
perl -ne '/\Q'"$variable"'\E/ and print'
I try to extract digits with sed:
echo hgdfjg678gfdg kjg45nn | sed 's/.*\([0-9]\+\).*/\1/g'
but result is:
5
How to extract: 678 and 45?
Thanks in advance!
The problem is that the . in .* will match digits as well as non-digits, and it keeps on matching as long as it can -- that is as long as there's one digit left unconsumed that can match the [0-9].
Instead of extracting digits, just delete non-digits:
echo hgdfjg678gfdg kjg45nn | sed 's/[^0-9]//g'
or even
echo hgdfjg678gfdg kjg45nn | tr -d -c 0-9
You may use grep with option -o for this:
$ echo hgdfjg678gfdg kjg45nn | grep -E -o "[0-9]+"
678
45
Or use tr:
$ echo hgdfjg678gfdg kjg45nn | tr -d [a-z]
678 45
.* in sed is greedy. And there are no non-greedy option AFAIK.(You must use [^0-9]* in this case for non-greedy matching. But this works only once, so you will get only 678 without 45.)
If you must use only sed, it would not be easy to get the result.
I recommend to use gnu’s grep
$ echo hgdfjg678gfdg kjg45nn | grep -oP '\d+'
678
45
If you really want to stick to sed, this would be one of many possible answers.
$ echo hgdfjg678gfdg kjg45nn | \
sed -e 's/\([0-9^]\)\([^0-9]\)/\1\n\2/g' | \
sed -n 's/[^0-9]*\([0-9]\+\).*/\1/p’
678
45
more file
param1=" 1,deerfntjefnerjfntrjgntrjnvgrvgrtbvggfrjbntr*rfr4fv*frfftrjgtrignmtignmtyightygjn 2,3,4,5,6,7,8,
rfcmckmfdkckemdio8u548384omxc,mor0ckofcmineucfhcbdjcnedjcnywedpeodl40fcrcmkedmrikmckffmcrffmrfrifmtrifmrifvysdfn drfr4fdr4fmedmifmitfmifrtfrfrfrfnurfnurnfrunfrufnrufnrufnrufnruf"****
need to match the content of param1 as
sed -n "/$param1/p" file
but because the line length (very long line) I cant match the line
what’s the best way to match very long lines?
The problem you are facing is that param1 contains special characters which are being interpreted by sed. The asterisk ('*') is used to mean 'zero or more occurrences of the previous character', so when this character is interpreted by sed there is nothing left to match the literal asterisk you are looking for.
The following is a working bash script that should help:
#!/bin/bash
param1=' 1,deerfntjefnerjfntrjgntrjnvgrvgrtbvggfrjbntr\*rfr4fv\*frfftrjgtrignmtignmtyightygjn 2,3,4,5,6,7,8, rfcmckmfdkckemdio8u548384omxc,mor0ckofcmineucfhcbdjcnedjcnywedpeodl40fcrcmkedmrikmckffmcrffmrfrifmtrifmrifvysdfn'
cat <<EOF | sed "s/${param1}/Bubba/g"
1,deerfntjefnerjfntrjgntrjnvgrvgrtbvggfrjbntr*rfr4fv*frfftrjgtrignmtignmtyightygjn 2,3,4,5,6,7,8, rfcmckmfdkckemdio8u548384omxc,mor0ckofcmineucfhcbdjcnedjcnywedpeodl40fcrcmkedmrikmckffmcrffmrfrifmtrifmrifvysdfn
EOF
Maybe the problem is that your $param1 contains special characters? This works for me:
A="$(perl -e 'print "a" x 10000')"
echo $A | sed -n "/$A/p"
($A contains 10 000 a characters).
echo $A | grep -F $A
and
echo $A | grep -P $A
also works (second requires grep with built-in PCRE support. If you want pattern matching you should use either this or pcregrep. If you don't, use the fixed grep (grep -F)).
echo $A | grep $A
is too slow.