I came across this command in a project I am working on:
sed -i '/regex/,$d' file
I don't understand how the ,$d part works. If I omit any part of ,$d I get errors. In my tests it looks like it replaces the matching line and anything after it with nothing. Example:
File with contents:
first line
second line regex
third line
fourth line
Comes out as after running that command:
first line
I couldn't find any documentation in the man page that explains this, though I could have easily missed it. The man page is hard for me to parse...
This is example was tested with GNU Sed v 4.2.2.
This is not a replacement command; the sed substitute or replace command looks like s/from/to/.
The general form of a sed script is a sequence of commands - typically a single letter, but some of them take arguments, like the s command above - with an optional address expression before each. You are looking at a d (delete line) command preceded by the address expression /regex/,$
The address range specifies lines from the first regex match through to the end of the file ($ in this context specifies the last line) and the action d deletes the specified lines.
Although many people only ever encounter simple sed scripts which use just the s command, this behavior will be described in any basic introduction to sed, as well as in the man page.
Related
I am currently running the following sed command:
sed 's/P(\(.*\))\\mid(\(.*\))/\\condprob{\1}{\2}/g' myfile.tex
Essentially, I have inherited an oddly formatted tex file, and want to replace everything like this:
P(<foo>)\mid(<bar>)
With this
\condprob{<foo>}{<bar>}
The file I am trying to run sed on contains the following line:
P(\vec{m}_i)\mid(t,h,\alpha) = \prod_{u\in\mathcal{U}} P(\vec{m}_{iu})\mid(t,h,\alpha)
Which I would like to change to this:
\condprob{\vec{m}_i}{t,h,\alpha} = \prod_{u\in\mathcal{U}}\condprob{\vec{m}_{iu}}{t,h,\alpha}
However, sed keeps missing the first \mid and instead gives me this:
\condprob{\vec{m}_i)\mid(t,h,\alpha) = \prod_{u\in\mathcal{U}} P(\vec{m}_{iu}}{t,h,\alpha}
If I add a line break at the = sign it matches everything fine
Can someone please a) help me resolve this, and b) perhaps tell me why it is happening?
Thanks.
Edit: thanks choroba and Sloopjon, you've both answered my why, and Sloopjon's solution is actually exactly what I was needing. choroba: I guess I will have to wait another day to learn perl.
For those that are interested Sloopjon's solution when translated into my problem looks like this (match everything that isn't a closing parenthesis):
sed 's/P(\([^)]*\))\\mid(\([^)]\))/\\condprob{\1}{\2}/g' myfile.tex
It looks like you expect P(\(.*\)) to match only P(\vec{m}_i), but the * quantifier is greedy, so it actually matches P(\vec{m}_i)\mid...P(\vec{m}_{iu}). There are two common fixes for this: use a non-greedy quantifier if your tool supports it, or change the pattern so that it only matches what you expect. For example, if you know that parentheses won't nest in this P() construct, change .* to [^)]*.
Edit: I also suggest that you look for a regex visualizer or debugger when you have a problem like this. For example, pasting your example into debuggex.com makes it clear what's happening.
The problem is the greediness of the * quantifier. It matches as many times as it can, i.e. it doesn't stop at the first ).
You can try Perl, that features "non-greedy" (frugal, lazy) *?:
perl -pe 's/P\((.*?)\)\\mid\((.*?)\)/\\condprob{$1}{$2}/g'
I noticed something a bit odd while fooling around with sed. If you try to remove multiple line intervals (by number) from a file, but any interval specified later in the list is fully contained within an interval earlier in the list, then an additional single line is removed after the specified (larger) interval.
seq 10 > foo.txt
sed '2,7d;3,6d' foo.txt
1
9
10
This behaviour was behind an annoying bug for me, since in my script I generated the interval endpoints on the fly, and in some cases the intervals produced were redundant. I can clean this up, but I can't think of a good reason why sed would behave this way on purpose.
Since this question was highlighted as needing an answer in the Stack Overflow Weekly Newsletter email for 2015-02-24, I'm converting the comments above (which provide the answer) into a formal answer. Unattributed comments here were made by me in essentially equivalent form.
Thank you for a concise, complete question. The result is interesting. I can reproduce it with your script. Intriguingly, sed '3,6d;2,7d' foo.txt (with the delete operations in the reverse order) produces the expected answer with 8 included in the output. That makes it look like it might be a reportable bug in (GNU) sed, especially as BSD sed (on Mac OS X 10.10.2 Yosemite) works correctly with the operations in either order. I tested using 'sed (GNU sed) 4.2.2' from an Ubuntu 14.04 derivative.
More data points for you/them. Both of these include 8 in the output:
sed -e '/2/,/7/d' -e '/3/,/6/d' foo.txt
sed -e '2,7d' -e '/3/,/6/d' foo.txt
By contrast, this does not:
sed -e '/2/,/7/d' -e '3,6d' foo.txt
The latter surprised me (even accepting the basic bug).
Beats me. I thought given some of sed's arcane constructs that you might be missing the batman symbol or something from the middle of your command but sed -e '2,7d' -e '3,6d' foo.txt behaves the same way and swapping the order produces the expected results (GNU sed 4.2.2 on Cygwin). /bin/sed on Solaris always produces the expected result and interestingly so does GNU sed 3.02. Ed Morton
More data: it only seems to happen with sed 4.2.2 if the 2nd range is a subset of the first: sed '2,5d;2,5d' shows the bug, sed '2,5d;1,5d' and sed '2,5d;2,6d' do not. glenn jackman
The GNU sed home page says "Please send bug reports to bug-sed at gnu.org" (except it has an # in place of ' at '). You've got a good reproduction; be explicit about the output you expect vs the output you get (they'll get the point, but it's best to make sure they can't misunderstand). Point out that the reverse ordering of the commands works as expected, and give the various other commands as examples of working or not working. (You could even give this Q&A URL as a cross-reference, but make sure that the bug report is self-contained so that it can be understood even if no-one follows the URL.)
You can also point to BSD sed (and the Solaris version, and the older GNU 3.02 sed) as behaving as expected. With the old version GNU sed working, it means this is arguably a regression. […After a little experimentation…] The breakage occurred in the 4.1 release; the 4.0.9 release is OK. (I also checked 4.1.5 and 4.2.1; both are broken.) That will help the maintainers if they want to find the trouble by looking at what changed.
The OP noted:
Thanks everyone for comments and additional tests. I'll submit a bug report to GNU sed and post their response. santayana
I want to delete newlines after lines containing a keyword e.g. like modifiers private:,public: or protected: to fulfill our coding standard. I need a command line tool (Linux) for this, so please no Notepad++, Emacs, VS, or Vim solutions, if they require user interaction. So in other words I want to do a:
sed -i 's/private:\s*\n\s*\n/private:\n/g'
I've seen this question but was unable to extend it to my needs.
If I understand correctly, you want to remove empty lines which follow a line containing private:, public:, or protected:.
sed ':loop;/private:\|public:\|protected:/{n;/^$/d;Tloop}' inputfile
Explanation:
:loop create a label
/private:\|public:\|protected:/ will search for lines containing the pattern.
n;/^$/d will load the next line (n), check whether it is an empty line (/^$/), and if it is, delete the line (d).
Tloop branch to label loop if there was no match (line was not empty)
I am no sed guru, there might be more elegant ways to do this. There might also be more elegant ways to do this in awk, perl, python, whatever.
perl -0777 -pi -e 's/([ \t\r]*)(private|protected|public):[ \t\r]*\n[ \t\r]*\n/$1$2:\n/g' file
should do the trick, while also take trailing and leading whitespace into account, which I didn't specified in the question as this is not a must requirement.
I trying to use sed in finding a matching pattern in a file then deleting
the next line only.
Ex.
LocationNew York <---- delete USA
LocationLondon <---- deleteUK
I tried sed '/Location/{n; d}' that work on linux but didn't work on solaris.
Thanks.
As I mentioned in my answer about your other sed question, Solaris sed is old-school AND needs more hand-holding (or to put it another way), is more fussy about it's syntax.
All you need is an additional ';' char placed after the `d' char, i.e.
sed '/Location/{n; d;}'
More generally, anything that can be on a new-line inside {...} needs a semi-colon separator when it is rolled up onto a single line. However, you can't roll up the 'a', 'i', 'c' commands onto a single line as you can in Linux.
In Solaris standard sed, the 'a', i', 'c' commands need a trailing '\' with NO spaces or tabs after it, as much data as you like (probably within some K limit) on \n terminated lines (NO \r s), followed by a blank line.
Newer installations of Solaris may also have /usr/xpg4/bin/sed installed. Try
/usr/xpg4/bin/sed '/Location/{n; d}'
If you're lucky, it will support your shortcut syntax. I don't have access to any solaris machines anymore to test this.
Finally, if that doesn't work, there are packages of GNU tools that can be installed that would have a sed that is much more like what you're used to from Linux. Ask your sys-admins if GNU tools are already there, or if they can be installed. I'm not sure what version of gnu sed started to support 'relaxed' syntax, so don't assume that it will be fixed without testing :-)
I hope this helps.
P.S.
Welcome to StackOverflow and let me remind you of three things we usually do here: 1) As you receive help, try to give it too, answering questions in your area of expertise 2) Read the FAQs, http://tinyurl.com/2vycnvr , 3) When you see good Q&A, vote them up by using the gray triangles, http://i.imgur.com/kygEP.png , as the credibility of the system is based on the reputation that users gain by sharing their knowledge. Also remember to accept the answer that better solves your problem, if any, by pressing the checkmark sign , http://i.imgur.com/uqJeW.png
You can append the next line to the current one and then remove everything that is not Location:
$ cat text
Location
New York <---- delete
USA
Location
London <---- delete
UK
$ sed '/Location/{N;s/Location.*$/Location/;}' text
Location
USA
Location
UK
I do not have a Solaris here so I would like to know if this works.
Does this AWK Solution works for you -
[jaypal~/temp]$ cat a.txt
Location
New York <---- delete
USA
Location
London <---- delete
UK
Updated to preserve the empty line -
!NF is used to preserve the blank lines. It means, if the Number of Fields is = 0 then just print the line. NF is an in-built variable which keeps track of number of fields in a record. If we encounter a blank line, we skip the rest of the processing and go to the next line.
!/Location/ will print the lines. This is to preserve the lines which are not followed by Location. Printing is an implicit action in AWK whenever the pattern is true.
The third patter/action is where we print the line when it matches the RegEx /Location/. Apart from printing the line, we do getline twice which effectively deletes your next line and then print it.
[jaypal~/temp]$ awk '!NF{print;next}; !/Location/; /Location/{print;getline;getline;print}' INPUT_FILE
Location
USA
Location
UK
substr($obj_strptime,index($strptime,"sub")+6,0) = <<'ESQ';
shift; # package
....
....
ESQ
What is this ESQ and what is it doing here? Please help me understand these statements.
It marks the end of a here-doc section.
EOF is more traditional than ESQ though.
This construct is known as a here-doc (because you're getting standard input from a document here rather than an external document on the file system somewhere).
It basically reads everything from the next line up to but excluding an end marker line, and uses that as standard input to the program or command that you're running. The end marker line is controlled by the text following the <<.
As an example, in bash (which I'm more familiar with than Perl), the command:
cat <<EOF
hello
goodbye
EOF
will run cat and then send two lines to its standard input (the hello and goodbye lines). Perl also has this feature though the syntax is slightly different (as you would expect, given it's a different language). Still, it's close enough for the explanation to still hold.
Wikipedia has an entry for this which you probably would have found had you known it was called a here-doc, but otherwise it would be rather hard to figure it out.
You can basically use any suitable marker. For example, if one of your input lines was EOF, you couldn't really use that as a marker since the standard input would be terminated prematurely:
cat <<EOF
This section contains the line ...
EOF
but then has more stuff
and this line following is the real ...
EOF
In that case, you could use DONE (or anything else that doesn't appear in the text on its own line).
There are other options such as using quotes around the marker (so the indentation can look better) and the use of single or double quotes to control variable substitution.
If you go to the perlop page and search for <<EOF, it will hopefully all become clear.