I need to extract certain json data (that have datalist member) from the log file, but only of map value is not 200.
right now I have two sed scripts, one extracts json data from a log file:
sed -n 's/.*\({\"datalist\".*}\).*/\1/p' full.log > new.log
the other one skips data if map field has value 200:
sed -n '/.*\"map\":\"200\".*/!p' new.log > map.log
how to combine these two into one?
UPD: I have accepted answer for now, but I wonder why
sed -n 's/.*\({\"datalist\".*\"map\":\"\(?!200\)\".*}\).*/\1/p' full.log > new.log
doesn't work
This might work for you (GNU sed):
sed -n '/"map":"200"/!s/.*\({"datalist".*}\).*/\1/p' full.log > new.log
Strip out the "map:200" lines with grep before sending to sed:
grep -v "\"map\":\"200\"" full.log | sed -n 's/.*\({\"datalist\".*}\).*/\1/p' > new.log
Related
Is it possible to return multiple next word results using a single sed query
String:
profile string.www.com { app-service none cert string.www.com_2018_2020.crt cert-key-chain { string.www.com _2018_2020_geotrust_rsa_ca_2018 { cert string.www.com _2018_2020.crt chain geotrust_rsa_ca_2018.crt key string.www.com_2018_2020.key } } chain geotrust_rsa_ca_2018.crt defaults-from default-clientssl inherit-certkeychain false key string.www.com_2018_2020.key passphrase none }
So for instance something like:
sed -n 's/^.*\(profile \|defaults-from \| passphrase) \([^ ]*\).*/\1/p'
Would return:
string.www.com
default-clientssl
none
This might work for you (GNU sed):
sed -En '/\n/!s/(profile|defaults-from|passphrase) \S+/\n&\n/g;/^(profile|defaults-from|passphrase)/P;D' file
Surround the required keys and values by newlines and only print those keys and values when they present themselves at the start of the line.
To output only the value:
sed -En '/\n/!s/(profile|defaults-from|passphrase) \S+/\n&\n/g;/^(profile|defaults-from|passphrase)/{s/\S+ //;P};D' file
You can't easily extract multiple matches on the same line with sed as its patterns are always "greedy". You may do it with two steps:
sed 's/\(profile\|defaults-from\|passphrase\) \([^ ]*\)/\nXXX\2XXX\n/g' <<< "$s" | \
sed -n 's/^XXX\(.*\)XXX$/\1/p'
First, wrap the matches with XXX and newlines, and then grab those matches with sed -n 's/^XXX\(.*\)XXX$/\1/p'. See this online demo.
If you want a "sngle-step" solution, you can extract those texts much easier with grep.
GNU grep solution:
grep -oP '(profile|defaults-from|passphrase)\s+\K\S+' file
Or, with pcregrep:
pcregrep -o '(profile|defaults-from|passphrase)\s+\K\S+' file
See this online demo.
This is mostly by curiosity, I am trying to have the same behavior as:
echo -e "test1:test2:test3"| sed 's/:/\n/g' | grep 1
in a single sed command.
I already tried
echo -e "test1:test2:test3"| sed -e "s/:/\n/g" -n "/1/p"
But I get the following error:
sed: can't read /1/p: No such file or directory
Any idea on how to fix this and combine different types of commands into a single sed call?
Of course this is overly simplified compared to the real usecase, and I know I can get around by using multiple calls, again this is just out of curiosity.
EDIT: I am mostly interested in the sed tool, I already know how to do it using other tools, or even combinations of those.
EDIT2: Here is a more realistic script, closer to what I am trying to achieve:
arch=linux64
base=https://chromedriver.storage.googleapis.com
split="<Contents>"
curl $base \
| sed -e 's/<Contents>/<Contents>\n/g' \
| grep $arch \
| sed -e 's/^<Key>\(.*\)\/chromedriver.*/\1/' \
| sort -V > out
What I would like to simplify is the curl line, turning it into something like:
curl $base \
| sed 's/<Contents>/<Contents>\n/g' -n '/1/p' -e 's/^<Key>\(.*\)\/chromedriver.*/\1/' \
| sort -V > out
Here are some alternatives, awk and sed based:
sed -E "s/(.*:)?([^:]*1[^:]*).*/\2/" <<< "test1:test2:test3"
awk -v RS=":" '/1/' <<< "test1:test2:test3"
# or also
awk 'BEGIN{RS=":"} /1/' <<< "test1:test2:test3"
Or, using your logic, you would need to pipe a second sed command:
sed "s/:/\n/g" <<< "test1:test2:test3" | sed -n "/1/p"
See this online demo. The awk solution looks cleanest.
Details
In sed solution, (.*:)?([^:]*1[^:]*).* pattern matches an optional sequence of any 0+ chars and a :, then captures into Group 2 any 0 or more chars other than :, 1, again 0 or more chars other than :, and then just matches the rest of the line. The replacement just keeps Group 2 contents.
In awk solution, the record separator is set to : and then /1/ regex is used to only return the record having 1 in it.
This might work for you (GNU sed):
sed 's/:/\n/;/^[^\n]*1/P;D' file
Replace each : and if the first line in the pattern space contains 1 print it.
Repeat.
An alternative:
sed -Ez 's/:/\n/g;s/^[^1]*$//mg;s/\n+/\n/;s/^\n//' file
This slurps the whole file into memory and replaces all colons by newlines. All lines that do not contain 1 are removed and surplus newlines deleted.
An alternative to the really ugly sed is: grep -o '\w*2\w*'
$ printf "test1:test2:test3\nbob3:bob2:fred2\n" | grep -o '\w*2\w*'
test2
bob2
fred2
grep -o: only matching
Or: grep -o '[^:]*2[^:]*'
echo -e "test1:test2:test3" | sed -En 's/:/\n/g;/^[^\n]*2[^\n]*(\n|$)/P;//!D'
sed -n doesn't print unless told to
sed -E allows using parens to match (\n|$) which is newline or the end of the pattern space
P prints the pattern buffer up to the first newline.
D trims the pattern buffer up to the first newline
[^\n] is a character class that matches anything except a newline
// is sed shorthand for repeating a match
//! is then matching everything that didn't match previously
So, after you split into newlines, you want to make sure the 2 character is between the start of the pattern buffer ^ and the first newline.
And, if there is not the character you are looking for, you want to D delete up to the first newline.
At that point, it works for one line of input, with one string containing the character you're looking for.
To expand to several matches within a line, you have to ta, conditionally branch back to label :a:
$ printf "test1:test2:test3\nbob3:bob2:fred2\n" | \
sed -En ':a s/:/\n/g;/^[^\n]*2[^\n]*(\n|$)/P;D;ta'
test2
bob2
fred2
This is simply NOT a job for sed. With GNU awk for multi-char RS:
$ echo "test1:test2:test3:test4:test5:test6"| awk -v RS='[:\n]' '/1/'
test1
$ echo "test1:test2:test3:test4:test5:test6"| awk -v RS='[:\n]' 'NR%2'
test1
test3
test5
$ echo "test1:test2:test3:test4:test5:test6"| awk -v RS='[:\n]' '!(NR%2)'
test2
test4
test6
$ echo "foo1:bar1:foo2:bar2:foo3:bar3" | awk -v RS='[:\n]' '/foo/ || /2/'
foo1
foo2
bar2
foo3
With any awk you'd just have to strip the \n from the final record before operating on it:
$ echo "test1:test2:test3:test4:test5:test6"| awk -v RS=':' '{sub(/\n$/,"")} /1/'
test1
I'm parsing a csv file with a sed command like this:
sed -n -e 's/abc/&/p' -e 's/xyz/&/p' <input >output
Now if there is both in one line (abc and xyz) I'll have the line twice in the output. I'd should have it just once.
Can I do that with sed?
If you only want to print a line with "abc" or "xyz":
sed -n '/abc\|xyz/p'
Other tools:
grep -F -e abc -e xyz
awk '/abc/ || /xyz/'
I believe you are mis-using the s///p just to print the lines. This is not necessary in sed - you can get them printed using //p.
Both expressions will evaluate, though, so you are still at risk of duplication. Your best bet (and fastest, for large datasets) will be to build the 'or' behavior into the matching regexp:
sed -Ene '/abc|xyz/p' input >output
sed -n -r -e 's/(abc|xyz)/&/p' <input >output
-r flag is for enabling extended regular expressions (is in GNU sed)
sed -n '/abc/{p;b
}
/xyz/p' Input > Output
for non GNU sed (where | is not allowed as OR)
I'm trying to convert all occurrences of a certain letter in the header of a file to lowercase, i can achieve this with 2 sed lines but i would like to use one instead.
What i'm trying is this:
cat file.txt | sed -e 'n 1p' -e 's/U/u/g'
Supposing that the letter i want to replace is the 'u'
I feel like i'm very close to it but for some reasons i get sed to complain about an extra char after the 'n' command, but in this case -n needs a parameter, so there should be no reason to complain.
Any hint?
This might work for you (GNU sed):
sed '1y/U/u/' file
try this (GNU sed):
sed '1s/U/u/g' file
I have the following sed command. I need to execute the below command in single line
cat File | sed -n '
/NetworkName/ {
N
/\n.*ims3/ p
}' | sed -n 1p | awk -F"=" '{print $2}'
I need to execute the above command in single line. can anyone please help.
Assume that the contents of the File is
System.DomainName=shayam
System.Addresses=Fr6
System.Trusted=Yes
System.Infrastructure=No
System.NetworkName=AS
System.DomainName=ims5.com
System.DomainName=Ram
System.Addresses=Fr9
System.Trusted=Yes
System.Infrastructure=No
System.NetworkName=Peer
System.DomainName=ims7.com
System.DomainName=mani
System.Addresses=Hello
System.Trusted=Yes
System.Infrastructure=No
System.NetworkName=Peer
System.DomainName=ims3.com
And after executing the command you will get only peer as the output. Can anyone please help me out?
You can use a single nawk command. And you can lost the useless cat
nawk -F"=" '/NetworkName/{n=$2;getline;if($2~/ims3/){print n} }' file
You can use sed as well as proposed by others, but i prefer less regex and less clutter.
The above save the value of the network name to "n". Then, get the next line and check the 2nd field against "ims3". If matched, then print the value of "n".
Put that code in a separate .sh file, and run it as your single-line command.
cat File | sed -n '/NetworkName/ { N; /\n.*ims3/ p }' | sed -n 1p | awk -F"=" '{print $2}'
Assuming that you want the network name for the domain ims3, this command line works without sed:
grep -B 1 ims3 File | head -n 1 | awk -F"=" '{print $2}'
So, you want the network name where the domain name on the following line includes 'ims3', and not the one where the following line includes 'ims7' (even though the network names in the example are the same).
sed -n '/NetworkName/{N;/ims3/{s/.*NetworkName=\(.*\)\n.*/\1/p;};}' File
This avoids abuse of felines, too (not to mention reducing the number of commands executed).
Tested on MacOS X 10.6.4, but there's no reason to think it won't work elsewhere too.
However, empirical evidence shows that Solaris sed is different from MacOS sed. It can all be done in one sed command, but it needs three lines:
sed -n '/NetworkName/{N
/ims3/{s/.*NetworkName=\(.*\)\n.*/\1/p;}
}' File
Tested on Solaris 10.
You just need to put -e pretty much everywhere you'd break the command at a newline or have a semicolon. You don't need the extra call to sed or awk or cat.
sed -n -e '/NetworkName/ {' -e 'N' -e '/\n.*ims3/ s/[^\n]*=\(.*\).*/\1/P' -e '}' File