France 211 55 Europe
Japan 144 120 Asia
Germany 96 61 Europe
England 94 56 Europe
Taiwan 55 144 Asia
North Korea 44 2134 Asia
The above is my data file.
There are empty lines in it.
There are no spaces or tabs in those empty lines.
I want to remove all empty lines in the data.
I did a search Delete empty lines using SED has given the perfect answer.
Before that, I wrote two sed code myself:
sed -r 's/\n\n+/\n/g' cou.data
sed 's/\n\n\n*/\n/g' cou.data
And I tried awk gsub, not successful either.
awk '{ gsub(/\n\n*/, "\n"); print }' cou.data
But they don't work and nothing changes.
Where did I do wrong about my sed code?
Use the following sed to delete all blank lines.
sed '/./!d' cou.data
Explanation:
/./ matches any character, including a newline.
! negates the selector, i.e. it makes the command apply to lines which do not match the selector, which in this case is the empty line(s).
d deletes the selected line(s).
cou.data is the path to the input file.
Where did you go wrong?
The following excerpt from How sed Works states:
sed operates by performing the following cycle on each line of input: first, sed reads one line from the input stream, removes any trailing newline, and places it in the pattern space. Then commands are executed; each command can have an address associated to it: addresses are a kind of condition code, and a command is only executed if the condition is verified before the command is to be executed.
When the end of the script is reached, unless the -n option is in use, the contents of pattern space are printed out to the output stream, adding back the trailing newline if it was removed.8 Then the next cycle starts for the next input line.
I've intentionally emboldened the parts which are pertinent to why your sed examples are not working. Given your examples:
They seem to disregard that sed reads one line at a time.
The trailing newlines, (\n\n and \n\n\n in your first and second example respectively), which you're trying to match don't actually exist. They've been removed by the time your regexp pattern is executed and then reinstated when the end of the script is reached.
RobC's answer is great if your lines are terminated by newline (linefeed or \n) only, because SED separates lines that way. If your lines are terminated by \r\n (or CRLF) - which you may have your reasons for doing even on a unix system - you will not get a match, because from sed's perspective the line isn't empty - the \r (CR) counts as a character. Instead you can try:
sed '/^\r$/d' filename
Explanation:
^ matches the start of the line
\r matches the carriage return
$ matches the end of the line
d deletes the selected line(s).
filename is the path to the input file.
Related
given text like
_adsp TXT "dkim=all"
VVKMU6SE3C2MF88BG4DJQAECMR9SIIF0 NSEC3 1 1 10 C4F407437E8EA4C5 (
175MCHR31K25LP89OVJI5LCE0JA2N2AP
A MX TXT AAAA RRSIG SPF )
RRSIG NSEC3 7 3 1800 (
20200429171433 20200330161758 11672 example.com
H3l26qmtkuiFZCeSYCCAo5krFE3gjM0I8UeQ9jhj3STy
X6fM0YizCHEuv4VZynOJGJc1XJnHRHI+p7yLlZ+OVseK
UfIkPVP+VOmlerwozEpM+Tnt8evwnMTDbcn0zxf/6YJx
kZeO2AszWkRZ0bctqW7INYo8YuyyuTSxSr8se27fiaPA
4GXQymepGgv/JGqargzHbyhhkDhENmNo7Qwkjl+a0kI4
6qqKcEWCsDvnlYUQiDFzc5oRs2j7TT9uybTfwUDQxV+t
MQFMhzu7LNbRIUuOb16sAEGSdl9mWQ4sZRJ9wuXJWbso
G+3tY0pBbq4ffScz/JKcrJ0qAuBF1F5JcQ== )
$TTL 1800
I want to get rid by the part with the "(not beginning with whitespace) NSEC3 " until the first line not beginning with a whitespace character.
resulting
_adsp TXT "dkim=all"
$TTL 1800
in the example.
I tried sed '/^[^\s].*\sNSEC3\s/,/^[^\s]/d;' filename but that doesn't work as expected, example results in
_adsp TXT "dkim=all"
H3l26qmtkuiFZCeSYCCAo5krFE3gjM0I8UeQ9jhj3STy
X6fM0YizCHEuv4VZynOJGJc1XJnHRHI+p7yLlZ+OVseK
UfIkPVP+VOmlerwozEpM+Tnt8evwnMTDbcn0zxf/6YJx
kZeO2AszWkRZ0bctqW7INYo8YuyyuTSxSr8se27fiaPA
4GXQymepGgv/JGqargzHbyhhkDhENmNo7Qwkjl+a0kI4
6qqKcEWCsDvnlYUQiDFzc5oRs2j7TT9uybTfwUDQxV+t
MQFMhzu7LNbRIUuOb16sAEGSdl9mWQ4sZRJ9wuXJWbso
G+3tY0pBbq4ffScz/JKcrJ0qAuBF1F5JcQ== )
$TTL 1800
so resuming printout way too early?
what do I miss?
thank you
P.S.:
you maybe see what I want to do is removing DNSSEC parts out of an named zone. didn't find any other way to remove RRSIG and NSEC3 entries, yet. If someone has an idea, I would appreciate that too.
[\s] matches a literal \ or s characters. It doesn't match whitespace.
The /^[^\s]/d; (if [\s] would work as you expect) will also include removing the last line with non-leading whitespaces. I think you have to loop manually.
On the example you've given, the following seems to work:
sed -n '/^[^ \t].*\sNSEC3\s/{ :a; n; /^[^ \t]/bb; ba}; :b; p'
This might work for you (GNU sed):
sed -n '/^\S.*NSEC/{:a;n;/^\S/!ba};p' file
Turn off implicit printing by using the -n option.
Throw away lines between one starting with a non-space and containing the string NSEC and any lines not starting with a non-space.
Print all other lines.
Alternative:
sed '/^\S.*NSEC/,/^\S/{/^\S.*NSEC\|^\s/d}' file
Yet another alternative:
sed '/^\S.*NSEC/{:a;N;/\n\S/!ba;s/.*\n//}' file
And another:
sed '/^\S.*NSEC/{:a;N;/\n\S/!s/\n//;ta;D}' file
N.B. The first two solutions will delete lines regardless of a line delimiting the end of the deletions. Whereas the last two solutions will only delete lines if there is a line delimiting the end of the deletions.
Suppose I have a file which contains:
something
line=1
file=2
other
lines
ignore
something
line=2
file=3
other
lines
ignore
Eventually, I want a unique list of the line and file combinations in each section. In the first stage I am trying to get sed to output just those lines combined into one line, like
line=1file=2
line=2file=3
Then I can use sort and uniq.
So I am trying
sed -n -r 's/(line=)(.*?)(\r)(file=)(.*?)(\r)/\1\2\4\5/p' sample.txt
(It isn't necessarily just a number after each)
But it won't match across the lines. I have tried \n and \r\n but it doesn't seem to be the style of new line, since:
sed -n -r 's/(line=)(.*?)(\r)/\1\2/p' sample.txt
Will output the "line=" lines, but I just can't get it to span the new line, and collect the second line as well.
By default, sed will operate only on chunks separated by \n character, so you can never match across multiple lines. Some sed implementations support -z option which will make it to operate on chunks separated by ASCII NUL character instead of newline character (this could work for small files, assuming NUL character won't affect the pattern you want to match)
There are also some sed commands that can be used for multiline processing
sed -n '/line=/{N;s/\n//p}'
N command will add the next line to current chunk being processed (which has to match line= in this case)
s/\n//p then delete the newline character, so that you get the output as single line
If your input has dos style line ending, first convert it to unix style (see Why does my tool output overwrite itself and how do I fix it?) or take care of \r as well
sed -n '/line=/{N;s/\r\n//p}'
Note that these commands were tested on GNU sed, syntax may vary for other implementations
everybody.
I don't understand dollar sign ($) in sed script programming, it is stand for last line of a file or a counter of sed?
I want to reverse order of lines (emulates "tac") of /etc/passwd. like following:
$ cat /etc/passwd | wc -l ----> 52 // line numbers
$ sed '1!G;h;$!d' /etc/passwd | wc -l ----> 52 // working correctly
$ sed '1!G;h;$d' /etc/passwd | wc -l ----> 1326 // no ! followed by $
$ sed '1!G;h;$p' /etc/passwd | wc -l ----> 1430 // instead !d by p
Last two example don't work right, who can tell me what mean does dollar sign stand for?
All the commands "work right." They just do something you don't expect. Let's consider the first version:
sed '1!G;h;$!d
Start with the first two commands:
1!G; h
After these two commands have been executed, the pattern space and the hold space both contain all the lines reads so far but in reverse order.
At this point, if we do nothing, sed would take its default action which is to print the pattern space. So:
After the first line is read, it would print the first line.
After the second line is read, it would print the second line followed by the first line.
After the third line is read, it would print the third line, followed by the second line, followed by the first line.
And so on.
If we are emulating tac, we don't want that. We want it to print only after it has read in the last line. So, that is where the following command comes in:
$!d
$ means the last line. $! means not-the-last-line. $!d means delete if we are not on the last line. Thus, this tells sed to delete the pattern space unless we are on the last line, in which case it will be printed, displaying all lines in reverse order.
With that in mind, consider your second example:
sed '1!G;h;$d'
This prints all the partial tacs except the last one.
Your third example:
sed '1!G;h;$p'
This prints all the partial tacs up through the last one but the last one is printed twice: $p is an explicit print of the pattern space for the last line in addition to the implicit print that would happen anyway.
Both grep and sed handle input line-by-line and, as far as I know, getting either of them to handle multiple lines isn't very straightforward. What I'm looking for is an alternative or alternatives to these two programs that treat newlines as just another character. Is there any tool that fits such a criteria
The tool you want is awk. It is record-oriented, not line-oriented, and you can specify your record-separator by setting the builtin variable RS. In particular, GNU awk lets you set RS to any regular expression, not just a single character.
Here is an example where awk uses one blank line to separate every record. If you show us what data you have, we can help you with it.
cat file
first line
second line
third line
fourth line
fifth line
sixth line
seventh line
eight line
more data
Running awk on this and reconstruct data using blank line as new record.
awk -v RS= '{$1=$1}1' file
first line second line third line
fourth line fifth line sixth line
seventh line eight line
more data
PS RS is not equal to file, is set to RS= blank, equal to RS=""
1) Sed can handle a block lines together, not always line by line.
In sed, normally I use :loop; $!{N; b loop}; to get all the lines available in pattern space delimited by newline.
Sample:
Productivity
Google Search\
Tips
"Web Based Time Tracking,
Web Based Todo list and
Reduce Key Stores etc"
result (remove the content between ")
sed -e ':loop; $!{N; b loop}; s/\"[^\"]*\"//g' thegeekstuff.txt
Productivity
Google Search\
Tips
You should read this URL (Unix Sed Tutorial: 6 Examples for Sed Branching Operation), it will give you detail how it works.
http://www.thegeekstuff.com/2009/12/unix-sed-tutorial-6-examples-for-sed-branching-operation/
2) For grep, check if your grep support -z option, which needn't handle input line by line.
-z, --null-data
Treat the input as a set of lines, each terminated by a zero
byte (the ASCII NUL character) instead of a newline. Like the
-Z or --null option, this option can be used with commands like
sort -z to process arbitrary file names.
Input-
X's Score 1725 and Y's Score 6248 in the match number 576
I want sed to ouput-
1725
6248
My code-
sed 's/Score[[:space:]]\([0-9]+\)/\1/g'
The above code outputs -
1725 and Y's 6248 in the match
You could try the following sed commands
#!/bin/sed f
s/Score\s*/\
/g
s/\n\([0-9]\+\)[^\n]*/\
\1/g
s/^[^\n]*\n//
The first command replaces all "Score"s with newlines, so now all numbers are at the beginning of a line. To insert a newline character, we must write a backslash followed by an actual line break. That's why the command spawns two lines.
The second command will remove everything after the numbers that are on the beginning of a line. It will match a newline character followed by a number (this is how we now that this number was prefixed by a "Score" string). The number will be captured into variable \1. Then it will skip all characters up to the newline character. When writing the replacement, we must restore the newline character and the number that was captured into \1.
Because the first line contains text before the first "Score", we must remove it. That's what the last command does, it matches all characters up to the first newline, starting from the beginning of the contents of the pattern space (ie. our working buffer).
In a single command:
sed -e 's/Score\s*/\
/g;s/\n\([0-9]\+\)[^\n]*/\
\1/g;s/^[^\n]*\n//'
Hope this helps =)
One way using GNU sed because \b that matches a word boundary is an extension.
echo "X's Score 1725 and Y's Score 6248 in the match number 576" | sed -e '
## Surround searched numbers (preceded by "Score") with newline characters.
s/\bScore \([0-9]\+\)\b/\n\1\n/g;
## Delete all numbers not preceded by a newline character.
s/\([^\n0-9]\)[0-9]\+/\1/g;
## Remove all other characters but numbers and newlines.
s/[^0-9\n]\+//g;
## Remove extra newlines.
s/\n\([0-9]\)/\1/g;
s/\n$//
' infile
It yields:
1725
6248
You could AND two egreps:
<infile egrep -o 'Score [0-9]+' | egrep -o '[0-9]+$'