I would like to update a large number of C++ source files with an extra include directive before any existing #includes. For this sort of task, I normally use a small bash script with sed to re-write the file.
How do I get sed to replace just the first occurrence of a string in a file rather than replacing every occurrence?
If I use
sed s/#include/#include "newfile.h"\n#include/
it replaces all #includes.
Alternative suggestions to achieve the same thing are also welcome.
A sed script that will only replace the first occurrence of "Apple" by "Banana"
Example
Input: Output:
Apple Banana
Apple Apple
Orange Orange
Apple Apple
This is the simple script: Editor's note: works with GNU sed only.
sed '0,/Apple/{s/Apple/Banana/}' input_filename
The first two parameters 0 and /Apple/ are the range specifier. The s/Apple/Banana/ is what is executed within that range. So in this case "within the range of the beginning (0) up to the first instance of Apple, replace Apple with Banana. Only the first Apple will be replaced.
Background: In traditional sed the range specifier is also "begin here" and "end here" (inclusive). However the lowest "begin" is the first line (line 1), and if the "end here" is a regex, then it is only attempted to match against on the next line after "begin", so the earliest possible end is line 2. So since range is inclusive, smallest possible range is "2 lines" and smallest starting range is both lines 1 and 2 (i.e. if there's an occurrence on line 1, occurrences on line 2 will also be changed, not desired in this case). GNU sed adds its own extension of allowing specifying start as the "pseudo" line 0 so that the end of the range can be line 1, allowing it a range of "only the first line" if the regex matches the first line.
Or a simplified version (an empty RE like // means to re-use the one specified before it, so this is equivalent):
sed '0,/Apple/{s//Banana/}' input_filename
And the curly braces are optional for the s command, so this is also equivalent:
sed '0,/Apple/s//Banana/' input_filename
All of these work on GNU sed only.
You can also install GNU sed on OS X using homebrew brew install gnu-sed.
# sed script to change "foo" to "bar" only on the first occurrence
1{x;s/^/first/;x;}
1,/foo/{x;/first/s///;x;s/foo/bar/;}
#---end of script---
or, if you prefer: Editor's note: works with GNU sed only.
sed '0,/foo/s//bar/' file
Source
An overview of the many helpful existing answers, complemented with explanations:
The examples here use a simplified use case: replace the word 'foo' with 'bar' in the first matching line only.
Due to use of ANSI C-quoted strings ($'...') to provide the sample input lines, bash, ksh, or zsh is assumed as the shell.
GNU sed only:
Ben Hoffstein's anwswer shows us that GNU provides an extension to the POSIX specification for sed that allows the following 2-address form: 0,/re/ (re represents an arbitrary regular expression here).
0,/re/ allows the regex to match on the very first line also. In other words: such an address will create a range from the 1st line up to and including the line that matches re - whether re occurs on the 1st line or on any subsequent line.
Contrast this with the POSIX-compliant form 1,/re/, which creates a range that matches from the 1st line up to and including the line that matches re on subsequent lines; in other words: this will not detect the first occurrence of an re match if it happens to occur on the 1st line and also prevents the use of shorthand // for reuse of the most recently used regex (see next point).1
If you combine a 0,/re/ address with an s/.../.../ (substitution) call that uses the same regular expression, your command will effectively only perform the substitution on the first line that matches re.
sed provides a convenient shortcut for reusing the most recently applied regular expression: an empty delimiter pair, //.
$ sed '0,/foo/ s//bar/' <<<$'1st foo\nUnrelated\n2nd foo\n3rd foo'
1st bar # only 1st match of 'foo' replaced
Unrelated
2nd foo
3rd foo
A POSIX-features-only sed such as BSD (macOS) sed (will also work with GNU sed):
Since 0,/re/ cannot be used and the form 1,/re/ will not detect re if it happens to occur on the very first line (see above), special handling for the 1st line is required.
MikhailVS's answer mentions the technique, put into a concrete example here:
$ sed -e '1 s/foo/bar/; t' -e '1,// s//bar/' <<<$'1st foo\nUnrelated\n2nd foo\n3rd foo'
1st bar # only 1st match of 'foo' replaced
Unrelated
2nd foo
3rd foo
Note:
The empty regex // shortcut is employed twice here: once for the endpoint of the range, and once in the s call; in both cases, regex foo is implicitly reused, allowing us not to have to duplicate it, which makes both for shorter and more maintainable code.
POSIX sed needs actual newlines after certain functions, such as after the name of a label or even its omission, as is the case with t here; strategically splitting the script into multiple -e options is an alternative to using an actual newlines: end each -e script chunk where a newline would normally need to go.
1 s/foo/bar/ replaces foo on the 1st line only, if found there.
If so, t branches to the end of the script (skips remaining commands on the line). (The t function branches to a label only if the most recent s call performed an actual substitution; in the absence of a label, as is the case here, the end of the script is branched to).
When that happens, range address 1,//, which normally finds the first occurrence starting from line 2, will not match, and the range will not be processed, because the address is evaluated when the current line is already 2.
Conversely, if there's no match on the 1st line, 1,// will be entered, and will find the true first match.
The net effect is the same as with GNU sed's 0,/re/: only the first occurrence is replaced, whether it occurs on the 1st line or any other.
NON-range approaches
potong's answer demonstrates loop techniques that bypass the need for a range; since he uses GNU sed syntax, here are the POSIX-compliant equivalents:
Loop technique 1: On first match, perform the substitution, then enter a loop that simply prints the remaining lines as-is:
$ sed -e '/foo/ {s//bar/; ' -e ':a' -e '$!{n;ba' -e '};}' <<<$'1st foo\nUnrelated\n2nd foo\n3rd foo'
1st bar
Unrelated
2nd foo
3rd foo
Loop technique 2, for smallish files only: read the entire input into memory, then perform a single substitution on it.
$ sed -e ':a' -e '$!{N;ba' -e '}; s/foo/bar/' <<<$'1st foo\nUnrelated\n2nd foo\n3rd foo'
1st bar
Unrelated
2nd foo
3rd foo
1 1.61803 provides examples of what happens with 1,/re/, with and without a subsequent s//:
sed '1,/foo/ s/foo/bar/' <<<$'1foo\n2foo' yields $'1bar\n2bar'; i.e., both lines were updated, because line number 1 matches the 1st line, and regex /foo/ - the end of the range - is then only looked for starting on the next line. Therefore, both lines are selected in this case, and the s/foo/bar/ substitution is performed on both of them.
sed '1,/foo/ s//bar/' <<<$'1foo\n2foo\n3foo' fails: with sed: first RE may not be empty (BSD/macOS) and sed: -e expression #1, char 0: no previous regular expression (GNU), because, at the time the 1st line is being processed (due to line number 1 starting the range), no regex has been applied yet, so // doesn't refer to anything.
With the exception of GNU sed's special 0,/re/ syntax, any range that starts with a line number effectively precludes use of //.
sed '0,/pattern/s/pattern/replacement/' filename
this worked for me.
example
sed '0,/<Menu>/s/<Menu>/<Menu><Menu>Sub menu<\/Menu>/' try.txt > abc.txt
Editor's note: both work with GNU sed only.
You could use awk to do something similar..
awk '/#include/ && !done { print "#include \"newfile.h\""; done=1;}; 1;' file.c
Explanation:
/#include/ && !done
Runs the action statement between {} when the line matches "#include" and we haven't already processed it.
{print "#include \"newfile.h\""; done=1;}
This prints #include "newfile.h", we need to escape the quotes. Then we set the done variable to 1, so we don't add more includes.
1;
This means "print out the line" - an empty action defaults to print $0, which prints out the whole line. A one liner and easier to understand than sed IMO :-)
Quite a comprehensive collection of answers on linuxtopia sed FAQ. It also highlights that some answers people provided won't work with non-GNU version of sed, eg
sed '0,/RE/s//to_that/' file
in non-GNU version will have to be
sed -e '1s/RE/to_that/;t' -e '1,/RE/s//to_that/'
However, this version won't work with gnu sed.
Here's a version that works with both:
-e '/RE/{s//to_that/;:a' -e '$!N;$!ba' -e '}'
ex:
sed -e '/Apple/{s//Banana/;:a' -e '$!N;$!ba' -e '}' filename
With GNU sed's -z option you could process the whole file as if it was only one line. That way a s/…/…/ would only replace the first match in the whole file. Remember: s/…/…/ only replaces the first match in each line, but with the -z option sed treats the whole file as a single line.
sed -z 's/#include/#include "newfile.h"\n#include'
In the general case you have to rewrite your sed expression since the pattern space now holds the whole file instead of just one line. Some examples:
s/text.*// can be rewritten as s/text[^\n]*//. [^\n] matches everything except the newline character. [^\n]* will match all symbols after text until a newline is reached.
s/^text// can be rewritten as s/(^|\n)text//.
s/text$// can be rewritten as s/text(\n|$)//.
#!/bin/sed -f
1,/^#include/ {
/^#include/i\
#include "newfile.h"
}
How this script works: For lines between 1 and the first #include (after line 1), if the line starts with #include, then prepend the specified line.
However, if the first #include is in line 1, then both line 1 and the next subsequent #include will have the line prepended. If you are using GNU sed, it has an extension where 0,/^#include/ (instead of 1,) will do the right thing.
Just add the number of occurrence at the end:
sed s/#include/#include "newfile.h"\n#include/1
A possible solution:
/#include/!{p;d;}
i\
#include "newfile.h"
:a
n
ba
Explanation:
read lines until we find the #include, print these lines then start new cycle
insert the new include line
enter a loop that just reads lines (by default sed will also print these lines), we won't get back to the first part of the script from here
I know this is an old post but I had a solution that I used to use:
grep -E -m 1 -n 'old' file | sed 's/:.*$//' - | sed 's/$/s\/old\/new\//' - | sed -f - file
Basically use grep to print the first occurrence and stop there. Additionally print line number ie 5:line. Pipe that into sed and remove the : and anything after so you are just left with a line number. Pipe that into sed which adds s/.*/replace to the end number, which results in a 1 line script which is piped into the last sed to run as a script on the file.
so if regex = #include and replace = blah and the first occurrence grep finds is on line 5 then the data piped to the last sed would be 5s/.*/blah/.
Works even if first occurrence is on the first line.
i would do this with an awk script:
BEGIN {i=0}
(i==0) && /#include/ {print "#include \"newfile.h\""; i=1}
{print $0}
END {}
then run it with awk:
awk -f awkscript headerfile.h > headerfilenew.h
might be sloppy, I'm new to this.
As an alternative suggestion you may want to look at the ed command.
man 1 ed
teststr='
#include <stdio.h>
#include <stdlib.h>
#include <inttypes.h>
'
# for in-place file editing use "ed -s file" and replace ",p" with "w"
# cf. http://wiki.bash-hackers.org/howto/edit-ed
cat <<-'EOF' | sed -e 's/^ *//' -e 's/ *$//' | ed -s <(echo "$teststr")
H
/# *include/i
#include "newfile.h"
.
,p
q
EOF
I finally got this to work in a Bash script used to insert a unique timestamp in each item in an RSS feed:
sed "1,/====RSSpermalink====/s/====RSSpermalink====/${nowms}/" \
production-feed2.xml.tmp2 > production-feed2.xml.tmp.$counter
It changes the first occurrence only.
${nowms} is the time in milliseconds set by a Perl script, $counter is a counter used for loop control within the script, \ allows the command to be continued on the next line.
The file is read in and stdout is redirected to a work file.
The way I understand it, 1,/====RSSpermalink====/ tells sed when to stop by setting a range limitation, and then s/====RSSpermalink====/${nowms}/ is the familiar sed command to replace the first string with the second.
In my case I put the command in double quotation marks becauase I am using it in a Bash script with variables.
Using FreeBSD ed and avoid ed's "no match" error in case there is no include statement in a file to be processed:
teststr='
#include <stdio.h>
#include <stdlib.h>
#include <inttypes.h>
'
# using FreeBSD ed
# to avoid ed's "no match" error, see
# *emphasized text*http://codesnippets.joyent.com/posts/show/11917
cat <<-'EOF' | sed -e 's/^ *//' -e 's/ *$//' | ed -s <(echo "$teststr")
H
,g/# *include/u\
u\
i\
#include "newfile.h"\
.
,p
q
EOF
This might work for you (GNU sed):
sed -si '/#include/{s//& "newfile.h\n&/;:a;$!{n;ba}}' file1 file2 file....
or if memory is not a problem:
sed -si ':a;$!{N;ba};s/#include/& "newfile.h\n&/' file1 file2 file...
If anyone came here to replace a character for the first occurrence in all lines (like myself), use this:
sed '/old/s/old/new/1' file
-bash-4.2$ cat file
123a456a789a
12a34a56
a12
-bash-4.2$ sed '/a/s/a/b/1' file
123b456a789a
12b34a56
b12
By changing 1 to 2 for example, you can replace all the second a's only instead.
The use case can perhaps be that your occurences are spread throughout your file, but you know your only concern is in the first 10, 20 or 100 lines.
Then simply adressing those lines fixes the issue - even if the wording of the OP regards first only.
sed '1,10s/#include/#include "newfile.h"\n#include/'
The following command removes the first occurrence of a string, within a file. It removes the empty line too. It is presented on an xml file, but it would work with any file.
Useful if you work with xml files and you want to remove a tag. In this example it removes the first occurrence of the "isTag" tag.
Command:
sed -e 0,/'<isTag>false<\/isTag>'/{s/'<isTag>false<\/isTag>'//} -e 's/ *$//' -e '/^$/d' source.txt > output.txt
Source file (source.txt)
<xml>
<testdata>
<canUseUpdate>true</canUseUpdate>
<isTag>false</isTag>
<moduleLocations>
<module>esa_jee6</module>
<isTag>false</isTag>
</moduleLocations>
<node>
<isTag>false</isTag>
</node>
</testdata>
</xml>
Result file (output.txt)
<xml>
<testdata>
<canUseUpdate>true</canUseUpdate>
<moduleLocations>
<module>esa_jee6</module>
<isTag>false</isTag>
</moduleLocations>
<node>
<isTag>false</isTag>
</node>
</testdata>
</xml>
ps: it didn't work for me on Solaris SunOS 5.10 (quite old), but it works on Linux 2.6, sed version 4.1.5
Nothing new but perhaps a little more concrete answer: sed -rn '0,/foo(bar).*/ s%%\1%p'
Example: xwininfo -name unity-launcher produces output like:
xwininfo: Window id: 0x2200003 "unity-launcher"
Absolute upper-left X: -2980
Absolute upper-left Y: -198
Relative upper-left X: 0
Relative upper-left Y: 0
Width: 2880
Height: 98
Depth: 24
Visual: 0x21
Visual Class: TrueColor
Border width: 0
Class: InputOutput
Colormap: 0x20 (installed)
Bit Gravity State: ForgetGravity
Window Gravity State: NorthWestGravity
Backing Store State: NotUseful
Save Under State: no
Map State: IsViewable
Override Redirect State: no
Corners: +-2980+-198 -2980+-198 -2980-1900 +-2980-1900
-geometry 2880x98+-2980+-198
Extracting window ID with xwininfo -name unity-launcher|sed -rn '0,/^xwininfo: Window id: (0x[0-9a-fA-F]+).*/ s%%\1%p' produces:
0x2200003
POSIXly (also valid in sed), Only one regex used, need memory only for one line (as usual):
sed '/\(#include\).*/!b;//{h;s//\1 "newfile.h"/;G};:1;n;b1'
Explained:
sed '
/\(#include\).*/!b # Only one regex used. On lines not matching
# the text `#include` **yet**,
# branch to end, cause the default print. Re-start.
//{ # On first line matching previous regex.
h # hold the line.
s//\1 "newfile.h"/ # append ` "newfile.h"` to the `#include` matched.
G # append a newline.
} # end of replacement.
:1 # Once **one** replacement got done (the first match)
n # Loop continually reading a line each time
b1 # and printing it by default.
' # end of sed script.
A possible solution here might be to tell the compiler to include the header without it being mentioned in the source files. IN GCC there are these options:
-include file
Process file as if "#include "file"" appeared as the first line of
the primary source file. However, the first directory searched for
file is the preprocessor's working directory instead of the
directory containing the main source file. If not found there, it
is searched for in the remainder of the "#include "..."" search
chain as normal.
If multiple -include options are given, the files are included in
the order they appear on the command line.
-imacros file
Exactly like -include, except that any output produced by scanning
file is thrown away. Macros it defines remain defined. This
allows you to acquire all the macros from a header without also
processing its declarations.
All files specified by -imacros are processed before all files
specified by -include.
Microsoft's compiler has the /FI (forced include) option.
This feature can be handy for some common header, like platform configuration. The Linux kernel's Makefile uses -include for this.
I needed a solution that would work both on GNU and BSD, and I also knew that the first line would never be the one I'd need to update:
sed -e "1,/pattern/s/pattern/replacement/"
Trying the // feature to not repeat the pattern did not work for me, hence needing to repeat it.
I will make a suggestion that is not exactly what the original question asks for, but for those who also want to specifically replace perhaps the second occurrence of a match, or any other specifically enumerated regular expression match. Use a python script, and a for loop, call it from a bash script if needed. Here's what it looked like for me, where I was replacing specific lines containing the string --project:
def replace_models(file_path, pixel_model, obj_model):
# find your file --project matches
pattern = re.compile(r'--project.*')
new_file = ""
with open(file_path, 'r') as f:
match = 1
for line in f:
# Remove line ending before we do replacement
line = line.strip()
# replace first --project line match with pixel
if match == 1:
result = re.sub(pattern, "--project='" + pixel_model + "'", line)
# replace second --project line match with object
elif match == 2:
result = re.sub(pattern, "--project='" + obj_model + "'", line)
else:
result = line
# Check that a substitution was actually made
if result is not line:
# Add a backslash to the replaced line
result += " \\"
print("\nReplaced ", line, " with ", result)
# Increment number of matches found
match += 1
# Add the potentially modified line to our new file
new_file = new_file + result + "\n"
# close file / save output
f.close()
fout = open(file_path, "w")
fout.write(new_file)
fout.close()
sed -e 's/pattern/REPLACEMENT/1' <INPUTFILE
I saw a sed command to delete the last 10 rows of data:
sed -e :a -e '$d;N;2,10ba' -e 'P;D'
But I don't understand how it works. Can someone explain it for me?
UPDATE:
Here is my understanding of this command:
The first script indicates that a label “a” is defined.
The second script indicates that it first determines whether the
line currently reading pattern space is the last line. If it is,
execute the "d" command to delete it and restart the next cycle; if
not, skip the "d" command; then execute "N" command: append a new
line from the input file to the pattern space, and then execute
"2,10ba": if the line currently reading the pattern space is a line
in the 2nd to 10th lines, jump to label "a".
The third script indicates that if the line currently read into
pattern space is not a line from line 2 to line 10, first execute "P" command: the first line
in pattern space is printed, and then execute "D" command: the first line in pattern
space is deleted.
My understanding of "$d" is that "d" will be executed when sed reads the last line into the pattern space. But it seems that every time "ba" is executed, "d" will be executed, regardless of Whether the current line read into pattern space is the last line. why?
:a is a label. $ in the address means the last line, d means delete. N stands for append the next line into the pattern space. 2,10 means lines 2 to 10, b means branch (i.e. goto), P prints the first line from the pattern space, D is like d but operates on the pattern space if possible.
In other words, you create a sliding window of the size 10. Each line is stored into it, and once it has 10 lines, lines start to get printed from the top of it. Every time a line is printed, the current line is stored in the sliding window at the bottom. When the last line gets printed, the sliding window is deleted, which removes the last 10 lines.
You can modify the commands to see what's getting deleted (()), stored (<>), and printed by the P ([]):
$ printf '%s\n' {1..20} | \
sed -e ':a ${s/^/(/;s/$/)/;p;d};s/^/</;s/$/>/;N;2,10ba;s/^/[/;s/$/]/;P;D'
[<<<<<<<<<<1>
[<2>
[<3>
[<4>
[<5>
[<6>
[<7>
[<8>
[<9>
[<10>
(11]>
12]>
13]>
14]>
15]>
16]>
17]>
18]>
19]>
20])
a simpler resort, if your data in 'd' file by gnu sed,
sed -Ez 's/(.*\n)(.*\n){10}$/\1/' d
^
pointed 10 is number of last line to remove
just move the brace group to invert, ie. to get only the last 10 lines
sed -Ez 's/.*\n((.*\n){10})$/\1/' d
I have the following content in a file
dhcp_option_domain:
- test.domain
And what I need to do is this:
whenever the value 'dhcp_option_domain:' followed by a newline and then ANY string, replace it with 'dhcp_option_domain:' followed by a newline and a variable.
ie if I set a variable of dhcp_domain="different.com" then then string above would convert to:
dhcp_option_domain:
- different.com
Note that both lines have and need to maintain leading 2 spaces.
I do not want to just do a search and replace on 'test.domain' as I have a few cases to use this and the values could be different each time the sed command is run.
I have tried a few methods such as:
dhcp_domain="something.com"
sed -i 's|dhcp_option_domain:\n.*|dhcp_option_domain:\n - $dhcp_domain|g' filename
however cannot get it to work.
Thanks.
As the manual explains:
sed operates by performing the following cycle on each line of input: first, sed reads one line from the input stream, removes any trailing newline, and places it in the pattern space. Then commands are executed
Your regex (dhcp_option_domain:\n.*) does not match because there is no \n in the pattern space in the first place.
A possible solution:
sed '/dhcp_option_domain:$/{n;c\
- '"$dhcp_domain"'
}'
The /dhcp_option_domain:$/ part is an address. The following command is only executed on lines matching that pattern.
The { } command groups multiple commands into a single block.
The n command prints out the current pattern space and replaces it by the next line of input.
The c\ command replaces the current pattern space by whatever follows in the script. Here it gets a bit tricky. We have:
a literal newline in the sed program (required after c\), then
- (placing those characters in the pattern space literally, then
' (part of shell syntax, terminating the single-quoted part started by sed '...), then
" (starting a double-quoted part), then
$dhcp_domain (which, because it's in a double-quoted part, interpolates the contents of the dhcp_domain shell variable), then
" (terminating the double-quoted part), then
' (starting another single-quoted part), then
a literal newline again (terminating the text after c\), then
} (closing the block started by {).
By default, sed works line by line (using newline character to distinguish newlines)
$ cat ip.txt
foo baz
dhcp_option_domain:
- test.domain
123
dhcp_option_domain:
$ dhcp_domain='something.com'
$ sed '/^ dhcp_option_domain:/{n; s/.*/ - '"$dhcp_domain"'/}' ip.txt
foo baz
dhcp_option_domain:
- something.com
123
dhcp_option_domain:
/^ dhcp_option_domain:/ condition to match
{} to group more than one command to be executed when this condition is satisfied
n get next line
s/.*/ - '"$dhcp_domain"'/ replace it as required - note that shell variables won't be expanded inside single quotes, see sed substitution with bash variables
for details
note that last line in the file didn't trigger the change as there was no further line
tested on GNU sed, syntax might vary for other implementations
From GNU sed manual
n
If auto-print is not disabled, print the pattern space, then,
regardless, replace the pattern space with the next line of input. If
there is no more input then sed exits without processing any more
commands.
This might work for you (GNU sed):
sed '/dhcp_option_domain:$/{p;s// - '"${var}"'/;n;d}' file
Match on dhcp_option_domain:, print it, substitute the new domain name (maintaining indent), print the current line and fetch the next (n) and delete it.
I have the following (gnu) sed script, which is intended to parse another sed script, and output distinct commands on a separate line.
In words, this script should put a newline after each semicolon ;, except semicolons that are inside a matching or substitution command.
Sed script:
#!/bin/sed -rf
# IDEA:
# replace ';' by ';\n' except when it's inside a match expression or subst. expression.
# Ignored patterns:
/^#/b # commented lines
/^$/b # empty lines
# anything in a single line, without semicolon except at the end
/^[^\n;]*;?$/b
# Processed patterns (put on separate lines):
# Any match preceding a semicolon, or the end of the line, or a substitution
s_/^[^/]+/[^;s]*;?_&\n_; t printtopline
s/^\\(.)[^\1]+\1[^;s]*;?/&\n/;t printtopline
# Any substitution (TODO)
# Any other command, separated by semicolon
s/\;/\;\n/; t printtopline;
:printtopline
P;D; # print top line, delete it, start new cycle
For example, I tested it with the following file (actually adapted from an answer of #ctac_ to one of my previous sed questions):
Input file:
#!/bin/sed -f
#/^>/N;
:A;
/\n>/!{s/\n/ /;N;bA}; # join next line if not a sequence label
#h;
#s/\(.*\)\n.*/\1/p;
s/^>//g;P
#x;
#s/.*\n//;
D
bA;
Output
The above script produces the right output, for example, the line /\n>/!{s/\n/ /;N;bA}; # join next line if not a sequence label becomes:
/\n>/!{s/\n/ /;
N;
bA};
# join next line if not a sequence label
Question
However, could you help me understand why this part of the script works:
s/\;/\;\n/; t printtopline;
:printtopline
?
I seems to me that the branching command t printtopline is useless here. I thought whatever the success of the substitution, the next thing to be executed would be :printtopline.
However, if I comment out the t command, or if I replace it with b, the script produces the following output lines:
/\n>/!{s/\n/ /;
N;bA}; # join next line if not a sequence label
From info sed, here is the explanation of t:
't LABEL'
Branch to LABEL only if there has been a successful 's'ubstitution
since the last input line was read or conditional branch was taken.
The LABEL may be omitted, in which case the next cycle is started.
Why isn't the t command immediately followed by its label not behaving like no command at all or the b command?
The crucial part is this:
Branch to label only if there has been a successful substitution since the last input line was read or conditional branch was taken.
I.e. t looks into the past and takes into account the success of all recent substitutions up to the most recent
input, or
conditional branch.
Consider the input line you're asking about. After all the substitutions we have
/\n>/!{s/\n/ /;
N;bA}; # join next line if not a sequence label
in our pattern space when we reach P;D;. The P commands outputs the first line, then D deletes the first line and restarts the main loop. Now we just have
N;bA}; # join next line if not a sequence label
Note that this didn't involve reading any additional lines. No input occurred; D just removed parts of the pattern space.
We process the remaining text (which does nothing because none of the other patterns match) until we reach this part of the code:
s_/^[^/]+/[^;s]*;?_&\n_; t printtopline
The substitution fails (the pattern space doesn't contain /^). But the t command doesn't check the status of just this one s command; it looks at the history of all substitutions since the most recent input or conditional branch taken.
The most recent input occurred when /\n>/!{s/\n/ /;N;bA}; was read.
The most recent conditional branch taken was
s/\;/\;\n/; t printtopline;
:printtopline
in the original version of your code. Since then no other substitution succeeded, so the t command does nothing. The rest of the program continues as expected.
But in the modified version of your code there was no conditional branch at this point (b is an unconditional branch):
s/\;/\;\n/; b printtopline;
:printtopline
That means the t from s_/^[^/]+/[^;s]*;?_&\n_; t printtopline "sees" the s/\;/\;\n/; as having succeeded, so it immediately jumps to the P;D; part. This is what outputs
N;bA}; # join next line if not a sequence label
unmodified.
In summary: t makes a difference here not because of its immediate effect of jumping to a label, but because it serves as a dynamic delimiter for the next t that gets executed. Without t here, the previously executed s command is taken into account for the next t.
Part 1 - how the P;D; sequence works.
Compare this two command's outputs: sed 's/;/;\n/' and sed 's/;/;\n/; P;D;'.
First:
$ sed 's/;/;\n/' <<< 'one;two;three;four'
one;
two;three;four
Second:
$ sed 's/;/;\n/; P;D;' <<< 'one;two;three;four'
one;
two;
three;
four
Why the difference? I will to explain.
The first command substitutes only the first occurrence of the ; character. To substitute all occurrences, the g modifier should be added to the s command: sed 's/;/;\n/g'.
The second command works this way:
sed 's/;/;\n/; - the same as the first command - no difference. Before this command the pattern space is one;two;three;four, after - one\ntwo;three;four.
P; -
from man: "Print up to the first embedded newline of the current pattern space."
That is, it prints up to first newline - one. The pattern space stay unchanged: one\ntwo;three;four
D; -
from man: "If pattern space contains no newline, start a normal new cycle as if the d command was
issued. Otherwise, delete text in the pattern space up to the first newline, and restart
cycle with the resultant pattern space, without reading a new line of input."
In the our case, pattern space has newline - one\ntwo;three;four. The D; removes the one\n part and repeat all commands cycle from the beginning. Now, the pattern space is: two;three;four.
That is, again sed 's/;/;\n/; - pattern space: two\nthree;four, then P; - print two, pattern space unchanged: two\nthree;four, D; - removes two\n, pattern space becomes: three;four. Etc.
Part 2 - what happening with branching.
I looked at the sed source code and found next information:
When the s command is executing and having match, the replaced flag is setting to the true:
/* We found a match, set the 'replaced' flag. */
replaced = true;
The t command is executing, if the replaced flag is true. And it is changing this flag to the false:
case 't':
if (replaced)
{
replaced = false;
So, in the first, s/\;/\;\n/; t printtopline; case, the substitution is successful - therefore, replaced flag is setting to the true. Then, the following t command is running and changing replaced flag back to the false.
In the second case, without t command - s/\;/\;\n/;, substitution is successful, too - therefore, replaced flag is setting to the true.
But now, this flag is stored to the next cycle, initiated by the D command. So, then the first t command appears in the new cycle - s_/^[^/]+/[^;s]*;?_&\n_; t printtopline, it checks the replaced flag, sees, that the flag is true and jumps to the label :printtopline, omitting all other commands before the label.
The pattern space doesn't have newlines, so P;D; sequence just prints pattern space and starts the next cycle with the new line of input.