I need to used grep / egrep / sed to extract certain parts out of a SNORT rule string.
given a string that can be in the format:
alert tcp any any -> any any (msg:"Some message";
content:"c1"; content:"GET /blah"; offset:0; depth:9; content:"something else";)
How would I go about extracting just the following:
content:"GET /blah"; offset:0; depth:9;
Given that the following are true:
It must match up until the start of the next content match (if there is one)
A rule may only have this content term, it may have more and they may be in any order
Other modifiers may be applied before, after or in between the offset and depth operators, they must also be extracted as follows:
content:"GET "; offset:5; http_uri; depth:12;
Rules can be "malformed" i.e. instead of having a single semicolon after the content term it may have two or more.
What I have so far which I believe would work in other regex systems is:
(GET|POST).*?(?=content)
The idea behind this being that .*? is an ungreedy match on any character any number of times and a non grabbing (not sure if that's the term) match on the next term "content".
I believe this breaks though if there is no following content term and also doesn't seem to extract anything in grep or egrep.
Not sure what to do, any ideas?
This should do the trick:
grep -Po '\bcontent\s*:\s*"(GET|POST)\b[^"]*"((?!;\s*content\s*:)[^"]|"[^"]*")*;'
Sample input:
alert tcp any any -> any any (msg:"Some message";
content:"c1"; content:"GET /blah"; offset:0; depth:9; content:"something else";)
content:"GET "; offset:5; http_uri; depth:12;
Output:
content:"GET /blah"; offset:0; depth:9;
content:"GET "; offset:5; http_uri; depth:12;
Explanation:
Instead of looking ahead for the next content, I am using a negative lookahead to consume anything other than the word content. This way, end of line also qualifies as the end of the match.
The regex in detail:
\b - word boundary (to prevent matching e.g. othercontent)
content\s*:\s* - literally: content followed by a colon; with optional spaces
" - opening quote
(GET|POST) - either one of these verbs
\b - word boundary (to prevent matching e.g. POSTAL)
[^"]*" - everything upto and including the closing quote
( - begin repeating subpattern
(?!;\s*content\s*:) - negative lookahead, to make sure we stop before any subsequent content
[^"] - any non-quote; spaces, letters, colons, semicolons...
| - or...
"[^"]*" - some attribute string; matching this as a whole to prevent the negative lookahead to pick up something between quotes
)* - end repeating subpattern; zero or more times
; - closing semicolon
Related
Can anyone explain what is being done below?
$name=~m,common/([^/]+)/run.*/([^/]+)/([^/]+)$,;
common, run and / are match themselves.
() captures.
[^/]+ matches 1 or more characters that aren't /.
.* matches 0 or more characters that aren't Line Feeds.[1]
$ is equivalent to (\n?\z).[2]
\n optionally matches a Line Feed.
\z matches the end of the string.
I think it's trying to match a path of one or both of the following forms:
.../common/XXX/runYYY/XXX/XXX
common/XXX/runYYY/XXX/XXX
Where
XXX is a sequence of at least one character that doesn't contain /.
YYY is a sequence of any number of characters (incl zero) that doesn't contain /.
It matches more than that, however.
It matches uncommon/XXX/runYYY/XXX/XXX
It matches common/XXX/runYYY/XXX/XXX/XXX/XXX/XXX/XXX
The parts in bold are captured (available to the caller).
When the s flag isn't used.
When the m flag isn't used.
Is it possible to replace half or x characters of a matched group?
I have had a request for a partial email capturing, so something like example123#abcdef.com becomes ***mple123#***def.com
I can do this if the characters before and after the # are 3 characters long,
([^|#]{0,3})([^]{0,3})
This captures 123#abc.com perfectly and I can substitute for ***#***.com but if it's over 3 characters long on each end, for instance example123#abcdef.com it becomes ******e123#abcdef.com
The other way I can see is capture everything until the # and everything to the . but then this won't be a partial capture. Is this possible?
You can use
sed -E 's/^[^#]{0,3}|(#)[^.]{0,3}/\1***/g'
Details
-E - enables the POSIX ERE syntax that does not require too much escaping here
^[^#]{0,3} - zero to three occurrences of any char other than a # at the start of the string
| - or
(#) - Group 1: a # char
[^.]{0,3} - zero to three occurrences of any char other than a .
\1*** replaces with Group 1 value + ***.
I've got an application that has no useful api implemented, and the only way to get certain information is to parse string output. This is proving to be very painful...
I'm trying to achieve this in bash on SLES12.
Given I have the following strings:
QMNAME(QMTKGW01) STATUS(Running)
QMNAME(QMTKGW01) STATUS(Ended normally)
I want to extract the STATUS value, ie "Ended normally" or "Running".
Note that the line structure can move around, so I can't count on the "STATUS" being the second field.
The closest I have managed to get so far is to extract a single word from inside STATUS like so
echo "QMNAME(QMTKGW01) STATUS(Running)" | sed "s/^.*STATUS(\(\S*\)).*/\1/"
This works for "Running" but not for "Ended normally"
I've tried switching the \S* for [\S\s]* in both "grep -o" and "sed" but it seems to corrupt the entire regex.
This is purely a regex issue, by doing \S you requested to match non-white space characters within (..) but the failing case has a space between which does not comply with the grammar defined. Make it simple by explicitly calling out the characters to match inside (..) as [a-zA-Z ]* i.e. zero or more upper & lower case characters and spaces.
sed 's/^.*STATUS(\([a-zA-Z ]*\)).*/\1/'
Or use character classes [:alnum:] if you want numbers too
sed 's/^.*STATUS(\([[:alnum:] ]*\)).*/\1/'
sed 's/.*STATUS(\([^)]*\)).*/\1/' file
Output:
Running
Ended normally
Extracting a substring matching a given pattern is a job for grep, not sed. We should use sed when we must edit the input string. (A lot of people use sed and even awk just to extract substrings, but that's wasteful in my opinion.)
So, here is a grep solution. We need to make some assumptions (in any solution) about your input - some are easy to relax, others are not. In your example the word STATUS is always capitalized, and it is immediately followed by the opening parenthesis (no space, no colon etc.). These assumptions can be relaxed easily. More importantly, and not easy to work around: there are no nested parentheses. You will want the longest substring of non-closing-parenthesis characters following the opening parenthesis, no mater what they are.
With these assumptions:
$ grep -oP '\bSTATUS\(\K[^)]*(?=\))' << EOF
> QMNAME(QMTKGW01) STATUS(Running)
> QMNAME(QMTKGW01) STATUS(Ended normally)
> EOF
Running
Ended normally
Explanation:
Command options: o to return only the matched substring; P to use Perl extensions (the \K marker and the lookahead). The regexp: we look for a word boundary (\b) - so the word STATUS is a complete word, not part of a longer word like SUBSTATUS; then the word STATUS and opening parenthesis. This is required for a match, but \K instructs that this part of the matched string will not be returned in the output. Then we seek zero or more non-closing-parenthesis characters ([^)]*) and we require that this be followed by a closing parenthesis - but the closing parenthesis is also not included in the returned string. That's a "lookahead" (the (?= ... ) construct).
I am trying to do a regex string to find all cases of force unwrapping in swift. This will search all words with exclamation points in the entire code base. However, the regex that I already have has included implicit declaration of variable which I am trying to exclude.
This is the regex that I already have.
(:\s)?\w+(?<!as)\)*!
And it works fine. It searches for "variableName!", "(variableName)!", "hello.hello!". The exclusion of force casting also works. It avoids cases like "hello as! UIView", But I am trying also to exclude another cases such as "var hello: UIView!" which has an exclamation point. That's the problem I am having. I tried negative lookahead and negative lookbehind and nothing solved this kind of case.
This is the sample regex I am working on
(:\s)?\w+(?<!as)\)*!
And this is the result
testing.(**test)))!**
Details lists capture **groups!**
hello as! hello
**Hello!**
**testing!**
testing**.test!**
Hello != World
var noNetworkBanner**: StatusBarNotificationBanner!** <-- need to exclude
"var noNetworkBanner**: StatusBarNotificationBanner!**" <-- need to exclude
You may use
(?<!:\s)\b\w+(?<!\bas)\b\)*!
I added \b word boundaries to match whole words only, and changed the (:\s)? optional group to a negative lookbehind, (?<!:\s), that disallows a : + space before the word you need to match.
See the regex demo and the regex graph:
Details
(?<!:\s) - a negative lookbehind that fails the match if, immediately to the left of the current location, there is a : and a whitespace
\b - word boundary
\w+ - 1+ word chars
(?<!\bas) - a negative lookbehind that fails the match if, immediately to the left of the current location, there is a whole word as
\b - word boundary
\)* - 0 or more ) chars
! - a ! char.
Here is a piece of code
while($l=~/(\\\s*)$/) {
statements;
}
$l contains a line of text taken form file, in effect this code is for go through lines in file.
Questions:
I don't clearly understand what the condition in while is doing. I think it is trying to match group of \ followed by some number of white spaces at the end of line and loop should stop whenever a line ends with \ and may be some white spaces. I am not sure of it.
I came across statement $a ~= s/^(.*$)/$1/ . What I understand that ^ will force matching at the beginning of string, but in (.*$) would mean match all the characters at the end of string . Dose it mean that the statement is trying to find if any group of character at the end is same as group of character in the beginning of text ?
It is interesting to note that this statement:
while ( $l =~ /(\\\s*)$/ ) {
Is an infinite loop unless $l is altered inside the loop so that the regex no longer matches. As has already been mentioned by others, this is what it matches:
( ... ) a capture group, captures string to $1 (that's the number one, not lower case L)
\\ matches a literal backslash
\s* matches 0 or more whitespace characters.
$ matches end of line with optional newline.
Since you do not have the /g modifier, this regex will not iterate through matches, it will simply check if there is a match, resetting the regex each iteration, thereby causing an endless loop.
The statement
$a ~= s/^(.*$)/$1/
Looks rather pointless. It captures a string of characters up until end of string, then replaces it with itself. The captured text is stored in $1 and is simply replaced. The only marginally useful thing about this regex is that:
It matches up until newline \n, and nothing further, which may be of some use to a parser. A period . matches any character except newline, unless the /s modifier is present on the regex.
It captures the line in $1 for future use. However, a simple /^(.*$)/ would do the same.
1. the while
Usually while (regex) is used with the /g modifier, otherwise, if it matches, you get an infinite loop (unless you exit the loop, like using last).
statements would be executed continuously in an infinite loop.
In your case, adding the g
while($l=~/(\\\s*)$/g)
will have the while make only one loop, due to the $ - making a match unique (whatever matches up to the end of string is unique, as $ marks the end, and there is nothing after...).
2. $a ~= s/^(.*$)/$1/
This is a substitution. If the string ^.*$ matches (and it will, since ^.*$ matches (almost, see comment) anything) it is replaced with... $1 or what's inside the (), ie itself, since the match occurs from 1st char to the end of string
^ means beginning of string
(.*) means all chars
$ end of string
so that will replace $a with itself - probably not what you want.
it matches a literal backslash followed by 0 or more spaces followed by the end of the line.
it executes statements for all the lines in that text file that contain a \, followed by zero or more spaces ( \s* ), at the end of the line ($).
It matches lines that end with a backslash character, ignoring any trailing whitespace characters.
Ending a line with a backslash is used in some languages and data files to indicate that the line is being continued on the next line. So I suspect this is part of a parser that merges these continuation lines.
If you enter a regular expression at RegExr and hover your mouse over the pieces, it displays the meaning of each piece in a tooltip.
(\\\s*)$ this regex means --- a \ followed by zero or more number of white space characters which is followed by end of the line. Since you have your regex in (...), you can extract what you matched using $1, if you need.
http://rubular.com/r/dtHtEPh5DX
EDIT -- based on your update
$a ~= s/^(.$)/$1/ --- this is search and replace. So your regex matches a line which contains exactly one character (since you use . http://www.regular-expressions.info/dot.html), except a new-line character. Since you use (...), the character which matched the regex is extracted and stored in variable a
EDIT -- you changed your regex so here is the updated answer
$a ~= s/^(.*$)/$1/ -- same as above except now it matches zero or more characters (except new-line)