My input is like this:
start
content A
end
garbage
start
content B
end
I want to extract the second (or first, or third ...) start .. end block. With
sed -ne '/start/,/end/p'
I can filter out the garbage, but how do I get just "start content B end"?
But anyway, if you want sed - you get sed:)
/^start$/{
x
s/^/a/
/^aaa$/{
x
:loop
p
/^end$/q
n
bloop
}
x
}
The number of a's in the middle match equals to which segment you want to get. You could also have it in regexp repetion like Dennis noted. That approach allows for specifying direct number to the script.
Note: the script should be run with -n sed option.
Get all range
$ awk 'BEGIN{RS="end";FS="start"}{ print $NF}' file
content A
content B
Get 2nd range
$ awk 'BEGIN{RS="end";FS="start"}{c++; if (c==2) print $NF}' file
content B
Ruby(1.9+), get first range
$ ruby -0777 -ne 'puts $_.scan(/start(.*?)end/m)[0]' file
content A
Related
I'm trying to extract the name of the file name that has been generated by a Java program. This Java program spits out multiple lines and I know exactly what the format of the file name is going to be. The information text that the Java program is spitting out is as follows:
ABCASJASLEKJASDFALDSF
Generated file YANNANI-0008876_17.xml.
TDSFALSFJLSDJF;
I'm capturing the output in a variable and then applying a sed operator in the following format:
sed -n 's/.*\(YANNANI.\([[:digit:]]\).\([xml]\)*\)/\1/p'
The result set is:
YANNANI-0008876_17.xml.
However, my problem is that want the extraction of the filename to stop at .xml. The last dot should never be extracted.
Is there a way to do this using sed?
Let's look at what your capture group actually captures:
$ grep 'YANNANI.\([[:digit:]]\).\([xml]\)*' infile
Generated file YANNANI-0008876_17.xml.
That's probably not what you intended:
\([[:digit:]]\) captures just a single digit (and the capture group around it doesn't do anything)
\([xml]\)* is "any of x, m or l, 0 or more times", so it matches the empty string (as above – or the line wouldn't match at all!), x, xx, lll, mxxxxxmmmmlxlxmxlmxlm, xml, ...
There is no way the final period is removed because you don't match anything after the capture groups
What would make sense instead:
Match "digits or underscores, 0 or more": [[:digit:]_]*
Match .xml, literally (escape the period): \.xml
Make sure the rest of the line (just the period, in this case) is matched by adding .* after the capture group
So the regex for the string you'd like to extract becomes
$ grep 'YANNANI.[[:digit:]_]*\.xml' infile
Generated file YANNANI-0008876_17.xml.
and to remove everything else on the line using sed, we surround regex with .*\( ... \).*:
$ sed -n 's/.*\(YANNANI.[[:digit:]_]*\.xml\).*/\1/p' infile
YANNANI-0008876_17.xml
This assumes you really meant . after YANNANI (any character).
You can call sed twice: first in printing and then in replacement mode:
sed -n 's/.*\(YANNANI.\([[:digit:]]\).\([xml]\)*\)/\1/p' | sed 's/\.$//g'
the last sed will remove all the last . at the end of all the lines fetched by your first sed
or you can go for a awk solution as you prefer:
awk '/.*YANNANI.[0-9]+.[0-9]+.xml/{print substr($NF,1,length($NF)-1)}'
this will print the last field (and truncate the last char of it using substr) of all the lines that do match your regex.
Can i use matched group from sed command, for another command, which generates replacement. Something like that:
sed -e 's/\(<regex>\)/$(<command using \1 reference and generating replacement>)/g'
I need it for replacement in first file, according to another file contents (replacement not constant and based on concrete replaced line).
As #EtanReisner mentions, this is possible only with GNU sed -- and still somewhat tricky. Also, it is potentially dangerous, and you should only use it if the input comes from a trustworthy source.
Anyway, the e modifier to the s/// command treats the contents of the pattern space after the substitution was made as a shell command, runs it, and replaces the pattern space with the output of that command, which means that the output will have to be shunted into place manually. A general pattern for this is
sed '/regex/ { h; s//\n/; x; s//\n&\n/; s/.*\n\(.*\)\n.*/command \1/e; x; G; s/\([^\n]*\)\n\([^\n]*\)\n\(.*\)/\1\3\2/ }' filename
Let's go through this from the top:
/regex/ { # When we find what we seek:
h # Make a copy of the current line in
# the hold buffer.
s//\n/ # Put a newline where the match occurs
# (// reattempts the last attempted
# regex, which is the one from the
# start). This serves as a marker
# where the output of the command will
# be inserted.
x # Swap the copy back in; the marked
# line moves to the hold buffer
s//\n&\n/ # put markers around the match this
# time,
s/.*\n\(.*\)\n.*/command \1/e # then use those markers to construct
# the command and run it. The pattern
# space contains the output of the
# command now.
x # swap the marked line back in
G # append the output to it
s/\([^\n]*\)\n\([^\n]*\)\n\(.*\)/\1\3\2/ # split, reassemble all that in
# the right order, using the
# newline marker we put there in
# the beginning as a splitting
# point.
}
regex and command have to be replaced with your regex and command, obviously. You can try this out with
echo 'foo /tmp/ bar' | sed '/\/\S*/ { h; s//\n/; x; s//\n&\n/; s/.*\n\(.*\)\n.*/ls \1/e; x; G; s/\([^\n]*\)\n\([^\n]*\)\n\(.*\)/\1\3\2/ }'
This will run ls /tmp/ and put the listing between foo and bar.
You might find it simpler and clearer to use awk. e.g. to multiply some number in the middle of the input by 3:
$ echo 'abc 12 def' |
awk 'match($0,/[0-9]+/) {print substr($0,1,RSTART-1) substr($0,RSTART,RLENGTH)*3 substr($0,RSTART+RLENGTH)}'
abc 36 def
With GNU awk you can use the 3rd arg to match() to save the regexp matching segments:
$ echo 'abc 12 def' |
awk 'match($0,/(.* )([0-9]+)( .*)/,a){print a[1] a[2]*3 a[3]}'
abc 36 def
or to pass it to a shell command (probably not a good idea, but can be done):
$ echo 'abc 12 def' |
awk 'match($0,/(.* )([0-9]+)( .*)/,a){system("echo \"" a[2] "\"")}'
12
I'd like to be able to replace a string between 2 known patterns. The catch is that I want to replace it by a string of the same length that is composed only of 'x'.
Let's say I have a file containing:
Hello.StringToBeReplaced.SecondString
Hello.ShortString.SecondString
I'd like the output to be like this:
Hello.xxxxxxxxxxxxxxxxxx.SecondString
Hello.xxxxxxxxxxx.SecondString
Using sed loops
You can use sed, though the thinking required is not wholly obvious:
sed ':a;s/^\(Hello\.x*\)[^x]\(.*\.SecondString\)/\1x\2/;t a'
This is for GNU sed; BSD (Mac OS X) sed and other versions may be fussier and require:
sed -e ':a' -e 's/^\(Hello\.x*\)[^x]\(.*\.SecondString\)/\1x\2/' -e 't a'
The logic is identical in both:
Create a label a
Substitute the lead string and a sequence of x's (capture 1), followed by a non-x, and arbitrary other data plus the second string (capture 2), and replace it with the contents of capture 1, an x and the content of capture 2.
If the s/// command made a change, go back to the label a.
It stops substituting when there are no non-x's between the two marker strings.
Two tweaks to the regex allow the code to recognize two copies of the pattern on a single line. Lose the ^ that anchors the match to the beginning of the line, and change .* to [^.]* (so that the regex is not quite so greedy):
$ echo Hello.StringToBeReplaced.SecondString Hello.StringToBeReplaced.SecondString |
> sed ':a;s/\(Hello\.x*\)[^x]\([^.]*\.SecondString\)/\1x\2/;t a'
Hello.xxxxxxxxxxxxxxxxxx.SecondString Hello.xxxxxxxxxxxxxxxxxx.SecondString
$
Using the hold space
hek2mgl suggests an alternative approach in sed using the hold space. This can be implemented using:
$ echo Hello.StringToBeReplaced.SecondString |
> sed 's/^\(Hello\.\)\([^.]\{1,\}\)\(\.SecondString\)/\1#\3##\2/
> h
> s/.*##//
> s/./x/g
> G
> s/\(x*\)\n\([^#]*\)#\([^#]*\)##.*/\2\1\3/
> '
Hello.xxxxxxxxxxxxxxxxxx.SecondString
$
This script is not as robust as the looping version but works OK as written when each line matches the lead-middle-tail pattern. It first splits the line into three sections: the first marker, the bit to be mangled, and the second marker. It reorganizes that so that the two markers are separated by #, followed by ## and the bit to be mangled. h copies the result to the hold space. Remove everything up to and including the ##; replace each character in the bit to be mangled by x, then copy the material in the hold space after the x's in the pattern space, with a newline separating them. Finally, recognize and capture the x's, the lead marker, and the tail marker, ignoring the newline, the # and ## plus trailing material, and reassemble as lead marker, x's, and tail marker.
To make it robust, you'd recognize the pattern and then group the commands shown inside { and } to group them so they're only executed when the pattern is recognized:
sed '/^\(Hello\.\)\([^.]\{1,\}\)\(\.SecondString\)/{
s/^\(Hello\.\)\([^.]\{1,\}\)\(\.SecondString\)/\1#\3##\2/
h
s/.*##//
s/./x/g
G
s/\(x*\)\n\([^#]*\)#\([^#]*\)##.*/\2\1\3/
}'
Adjust to suit your needs...
Adjusting to suit your needs
[I tried one of your solutions and it worked fine.]
However when I try to replace the 'hello' by my real string (which is
'1.2.840.') and my second string (which is simply a dot '.'), things stop
working. I guess all these dots confuse the sed command.
What I try to achieve is transform this '1.2.840.10008.' to
'1.2.840.xxxxx.'
And this pattern happens several times in my file with variable number
of characters to be replaced between the '1.2.840.' and the next dot '.'
There are times when it is important to get your question close enough to the real scenario — this may be one such. Dot is a metacharacter in
sed regular expressions (and in most other dialects of regular expression — shell globbing being the noticeable exception). If the 'bit to be mangled' is always digits, then we can tighten up the regular expressions, though actually (when I look at the code ahead) the tightening really isn't imposing much in the way of a restriction.
Pretty much any solution using regular expressions is a balancing act that has to pit convenience and abbreviation against reliability and precision.
Revised code plus data
cat <<EOF |
transform this '1.2.840.10008.' to '1.2.840.xxxxx.'
OK, and hence 1.2.840.21. and 1.2.840.20992. should lose the 21 and 20992.
EOF
sed ':a;s/\(1\.2\.840\.x*\)[^x.]\([^.]*\.\)/\1x\2/;t a'
Example output:
transform this '1.2.840.xxxxx.' to '1.2.840.xxxxx.'
OK, and hence 1.2.840.xx. and 1.2.840.xxxxx. should lose the 21 and 20992.
The changes in the script are:
sed ':a;s/\(1\.2\.840\.x*\)[^x.]\([^.]*\.\)/\1x\2/;t a'
Add 1\.2\.840\. as the start pattern.
Revise the 'character to replace' expression to 'not x or .'.
Use just \. as the tail pattern.
You could replace the [^x.] with [0-9] if you're sure you only want digits matched, in which case you won't have to worry about spaces as discussed below.
You may decide you don't want spaces to be matched so that a casual comment like:
The net prefix is 1.2.840. And there are other prefixes too.
does not end up as:
The net prefix is 1.2.840.xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx.
In which case, you probably need to use:
sed ':a;s/\(1\.2\.840\.x*\)[^x. ]\([^ .]*\.\)/\1x\2/;t a'
And so the changes continue until you've got something precise enough to do what you want without doing anything you don't want on your current data set. Writing bullet-proof regular expressions requires a precise specification of what you want matched, and can be quite hard.
I'd choose perl:
perl -pe 's/(?<=Hello\.)(.*?)(?=\.SecondString)/ "x" x length($1) /e' file
This awk should do:
awk -F. '{for (i=1;i<=length($2);i++) a=a"x";$2=a;a=""}1' OFS="." file
Hello.xxxxxxxxxxxxxxxxxx.SecondString
Hello.xxxxxxxxxxx.SecondString
Bash Works Too
While the perl, sed and awk solutions are probably the better choice, a Bash solution is not that difficult (just longer). Bash has good character-by-character handling abilities as well:
#!/bin/bash
rep=0 # replace flag
skip=0 # delay reset flag
while read -r line; do # read each line
for ((i=0; i<${#line}; i++)); do # for each character in the line
# if '.' and replace on, turn off and set skip
[ ${line:i:1} == '.' -a $rep -eq 1 ] && { rep=0; skip=1; }
# print char or "x" depending on replace flag
[ $rep -eq 0 ] && printf "%c" ${line:i:1} || printf "x"
# if '.' and replace off
if [ ${line:i:1} == '.' -a $rep -eq 0 ]; then
# if skip, turn skip off, else set replace on
[ $skip -eq 1 ] && skip=0 || rep=1
fi
done
printf "\n"
done
exit 0
Input
$ cat dat/replacefile.txt
Hello.StringToBeReplaced.SecondString
Hello.ShortString.SecondString
Output
$ bash replacedot.sh < dat/replacefile.txt
Hello.xxxxxxxxxxxxxxxxxx.SecondString
Hello.xxxxxxxxxxx.SecondString
For the sake of your sanity, just use awk:
$ awk 'BEGIN{FS=OFS="."} {gsub(/./,"x",$2)} 1' file
Hello.xxxxxxxxxxxxxxxxxx.SecondString
Hello.xxxxxxxxxxx.SecondString
I have a huge file that contains lines that follow this format:
New-England-Center-For-Children-L0000392290
Southboro-Housing-Authority-L0000392464
Crew-Star-Inc-L0000391998
Saxony-Ii-Barber-Shop-L0000392491
Test-L0000392334
What I'm trying to do is narrow it down to just this:
New-England-Center-For-Children
Southboro-Housing-Authority
Crew-Star-Inc
Test
Can anyone help with this?
Using GNU awk:
awk -F\- 'NF--' OFS=\- file
New-England-Center-For-Children
Southboro-Housing-Authority
Crew-Star-Inc
Saxony-Ii-Barber-Shop
Test
Set the input and output field separator to -.
NF contains number of fields. Reduce it by 1 to remove the last field.
Using sed:
sed 's/\(.*\)-.*/\1/' file
New-England-Center-For-Children
Southboro-Housing-Authority
Crew-Star-Inc
Saxony-Ii-Barber-Shop
Test
Simple greedy regex to match up to the last hyphen.
In replacement use the captured group and discard the rest.
Version 1 of the Question
The first version of the input was in the form of HTML and parts had to be removed both before and after the desired text:
$ sed -r 's|.*[A-Z]/([a-zA-Z-]+)-L0.*|\1|' input
Special-Restaurant
Eliot-Cleaning
Kennedy-Plumbing
Version 2 of the Question
In the revised question, it is only necessary to remove the text that starts with -L00:
$ sed 's|-L00.*||' input2
New-England-Center-For-Children
Southboro-Housing-Authority
Crew-Star-Inc
Saxony-Ii-Barber-Shop
Test
Both of these commands use a single "substitute" command. The command has the form s|old|new|.
The perl code for this would be: perl -nle'print $1 if(m{-.*?/(.*?-.*?)-})
We can break the Regex down to matching the following:
- for that's between the city and state
.*? match the smallest set of character(s) that makes the Regex work, i.e. the State
/ matches the slash between the State and the data you want
( starts the capture of the data you are interested in
.*?-.*? will match the data you care about
) will close out the capture
- will match the dash before the L####### to give the regex something to match after your data. This will prevent the minimal Regex from matching 0 characters.
Then the print statement will print out what was captured (your data).
awk likes these things:
$ awk -F[/-] -v OFS="-" '{print $(NF-3), $(NF-2)}' file
Special-Restaurant
Eliot-Cleaning
Kennedy-Plumbing
This sets / and - as possible field separators. Based on them, it prints the last_field-3 and last_field-2 separated by the delimiter -. Note that $NF stands for last parameter, hence $(NF-1) is the penultimate, etc.
This sed is also helpful:
$ sed -r 's#.*/(\w*-\w*)-\w*\.\w*</loc>$#\1#' file
Special-Restaurant
Eliot-Cleaning
Kennedy-Plumbing
It selects the block word-word after a slash / and followed with word.word</loc> + end_of_line. Then, it prints back this block.
Update
Based on your new input, this can make it:
$ sed -r 's/(.*)-L\w*$/\1/' file
New-England-Center-For-Children
Southboro-Housing-Authority
Crew-Star-Inc
Saxony-Ii-Barber-Shop
Test
It selects everything up to the block -L + something + end of line, and prints it back.
You can use also another trick:
rev file | cut -d- -f2- | rev
As what you want is every slice of - separated fields, let's get all of them but last one. How? By reversing the line, getting all of them from the 2nd one and then reversing back.
Here's how I'd do it with Perl:
perl -nle 'm{example[.]com/bp/(.*?)/(.*?)-L\d+[.]htm} && print $2' filename
Note: the original question was matching input lines like this:
<loc>http://www.example.com/bp/Lowell-MA/Special-Restaurant-L0000423916.htm</loc>
<loc>http://www.example.com/bp/Houston-TX/Eliot-Cleaning-L0000422797.htm</loc>
<loc>http://www.example.com/bp/New-Orleans-LA/Kennedy-Plumbing-L0000423121.htm</loc>
The -n option tells Perl to loop over every line of the file (but not print them out).
The -l option adds a newline onto the end of every print
The -e 'perl-code' option executes perl-code for each line of input
The pattern:
/regex/ && print
Will only print if the regex matches. If the regex contains capture parentheses you can refer to the first captured section as $1, the second as $2 etc.
If your regex contains slashes, it may be cleaner to use a different regex delimiter ('m' stands for 'match'):
m{regex} && print
If you have a modern Perl, you can use -E to enable modern feature and use say instead of print to print with a newline appended:
perl -nE 'm{example[.]com/bp/(.*?)/(.*?)-L\d+[.]htm} && say $2' filename
This is very concise in Perl
perl -i.bak -lpe's/-[^-]+$//' myfile
Note that this will modify the input file in-place but will keep a backup of the original data in called myfile.bak
My question can be split in 2. First I have a data file (file.dat) that looks like:
Parameter stuff number 1 (1029847) word index 2 (01293487), bla bla
Parameter stuff number 3 (134123) word index 4 (02983457), bla bla
Parameter stuff number 2 (109847) word index 3 (1029473), bla bla
etc...
I want to extract the number in brackets and save it to a variable for example the first one in line one to be 'x1', the second on the same line to be 'y1', for line 2 'x2' and 'y2', and so on... The numbers change randomly line after line, their position (in columns, if you like) stays the same line after line. The number of lines is variable (0 to 'n'). How can I do this? Please.
I have search for answers and I get lost with the many different commands one can use, however those answers attend to particular examples where the word is at the end or in brackets but only one per line, etc. Anyhow, here is what I have done so far (I am newby):
1) I get rid of the characters that are not part of the number in the string
sed -i 's/(//g' file.dat
sed -i 's/),//g' file.dat
2) Out of frustration I decided to output the whole lines to variables (getting closer?)
2.1) Get the number of lines to iterate for:
numlines=$(wc -l < file.dat)
2.2) Loop to numlines (I havent tested this bit yet!)
for i in {1..$numlines}
do
line${!i}=$(sed -n "${numlines}p" file.dat)
done
2.3) I gave up here, any help appreciated.
The second question is similar and merely out of curiosity: imagine a database separated by spaces, or tabs, or comas, any separator; this database has a variable number of lines ('n') and the strings per line may vary too ('k'). How do I extract the value of the 'i'th line on the 'j'th string, and save it to a variable 'x'?
Here is a quick way to store value in bash array variable.
x=("" $(awk -F"[()]" '{printf "%s ",$2}' file))
y=("" $(awk -F"[()]" '{printf "%s ",$4}' file))
echo ${x[2]}
134123
If you are going to use these data for more jobs, I would have done it in awk. Then you can use internal array in awk
awk -F"[()]" '{x[NR]=$2;y[NR]=$4}' file
#!/usr/bin/env bash
x=()
y=()
while read line; do
x+=("$(sed 's/[^(]*(\([0-9]*\)).*/\1/' <<< $line)")
y+=("$(sed 's/[^(]*([^(]*(\([0-9]*\)).*/\1/' <<< $line)")
done < "data"
echo "${x[#]}"
echo "${y[#]}"
x and y are declared as arrays. Then you loop over the input file and invoke a sed command to every line in your input file.
x+=(data) appends the value data to the array x. Instead of writing the value we want to store in the array, we use command substitution, which is done with $(command), instead of appending the literal meaning of $(command) to the array, the command is executed and its return value is stored in the array.
Let's look at the sed commands:
's' is the substitute command, with [^(]* we want to match everything, except (, then we match (. The following characters we want to store in the array, to do that we use \( and \), we can later reference to it again (with \1). The number is matched with [0-9]*. In the end we match the closing bracket ) and everything else with .*. Then we replace everything we matched (the whole line), with \1, which is just what we had between \( and \).
If you are new to sed, this might be highly confusing, since it takes some time to read the sed syntax.
The second sed command is very similar.
How do I extract the value of the 'i'th line on the 'j'th string, and
save it to a variable 'x'?
Try using awk
x=$(awk -v i=$i -v j=$j ' NR==i {print $j; exit}' file.dat)
I want to extract the number in brackets and save it to a variable for
example the first one in line one to be 'x1', the second on the same
line to be 'y1', for line 2 'x2' and 'y2', and so on...
Using awk
x=($(awk -F'[()]' '{print $2}' file.dat))
y=($(awk -F'[()]' '{print $4}' file.dat))
x1 can be accessed as ${x[0]} and y1 as ${y[0]}, likewise for other sequence of variables.