I want to remove lines from file by regex pattern using sed just like in this question Delete lines in a text file that containing a specific string, but only inside a range of lines (not in the whole file). I want to do it starting from some line number till the end of file.
This is how I've done it in combination with tail:
tail -n +731 file|sed '/some_pattern/d' >> file
manually remove edited range in file from previous step
Is there a shorter way to do it with sed only?
Something like sed -i '731,1000/some_pattern/d' file?
You can use this sed,
sed -i.bak '731,1000{/some_pattern/d}' yourfile
Test:
$ cat a
1
2
3
13
23
4
5
$ sed '2,4{/3/d}' a
1
2
23
4
5
You need $ address to match end of file. With GNU sed:
sed -i '731,${/some_pattern/d;}' file
Note that this can be slower than tail -n +number, because sed will start processing at start of file instead of doing lseek() like tail.
(With BSD sed you need sed -i '' ...)
sed is for simple substitutions on individual lines, that is all. For anything even marginally more interesting an awk solution will be clearer, more robust, portable, maintainable, extensible and better in just about ever other desirable attribute of software.
Given this sample input file:
$ cat file
1
2
3
4
1
2
3
4
1
2
3
4
The following script will print every line except a line containing the number 3 that occurs after the 6th line of the input file:
$ awk '!(NR>6 && /3/)' file
1
2
3
4
1
2
4
1
2
4
Want to only do the deletion between lines 6 and 10? No problem:
$ awk '!(NR>6 && NR<10 && /3/)' file
1
2
3
4
1
2
4
1
2
3
4
Want the skipped lines written to a log file? No problem:
awk 'NR>6 && /3/{print > "log";next} {print}' file
Written to stderr?
awk 'NR>6 && /3/{print | "cat>&2";next} {print}' file
Want a count of how many lines you deleted also written to stderr?
awk 'NR>6 && /3/{print | "cat>&2"; cnt++; next} {print} END{print cnt | "cat>&2"}' file
ANYTHING you want to do additionally or differently will be easy and build on what you start with. Try doing any of the above, or just about anything else, with a sed script that satisfies your original requirement.
awk to the rescue!
awk '!(NR>=731 && /pattern/)' input > output
Related
I have a file called error_log for the apache and I want to see the first line and the last line of this file using sed command. Would you please help me how can I do that?
I know how to do that with head and tail commands, but I'm curious if it's possible in sed command too.
I have read the man sed and have googled a lot but nothing is found unfortunately.
This might work for you (GNU sed):
sed '1b;$b;d' file
All sed commands can be prefixed by either an address or a regexp. An address is either a line number or the $ which represents the last line. If neither an address or a regexp is present, the following command applies to all other lines.
The normal sed cycle, presents each line of input (less its newline) in the pattern space. The sed commands are then applied and the final act of the cycle is to re-attach the newline and print the result.
The b command controls command flow; if by itself it jumps out of the following sed commands to the final act of the cycle i.e. where the newline is re-attached and the result printed.
The d command deletes the pattern space and since there is nothing to be printed no further processing is executed (including re-attaching the newline and printing the result).
Thus the solution above prints the first line and the last and deletes the rest.
Sed has some command line options, one of which turns of the implicit printing of the result of the pattern space -n. The p command prints the current state of the pattern space. Thus the dual of the above solution is:
sed -n '1p;$p' file
N.B. If the input file is only one line the first solution will only print one line whereas the second solution will print the same line twice. Also if more than one file is input both solutions will print the first line of the first file and last line of the last file unless the -i option is in place, in which case each file will be amended. The -s option replicates this without amending each file but streams the results to stdout as if each file is treated separately.
This will work:
sed -n '1p ; $p' error_log
1p will print the first line and $p will print the last line.
As a suggestion, take a look at info sed, not only man sed. You can find the some examples about your question at the paragraph 2.1.
First line:
sed '2,$d' error_log
Last line:
sed '$!d' error_log
Based on your new requirement to output nothing if the input file is just 1 line (see How to find only the first and last line of a file using sed):
awk 'NR==1{first=$0} {last=$0} END{if (NR>1) print first ORS last}'
Original answer:
This is one of those things that you can, at face value, do easily enough in sed:
$ seq 3 7
3
4
5
6
7
$ seq 3 7 | sed -n '1p; $p'
3
7
but then how to handle edge cases like one line of input is non-obvious, e.g. is this REALLY the correct output:
$ printf 'foo\n' | sed -n '1p; $p'
foo
foo
or is the correct output just:
foo
and if the latter, how do you tweak that sed command to produce that output? #potong suggested a GNU sed command:
$ printf 'foo\n' | sed '1b;$b;d'
foo
which works but may be GNU-only (idk) and more importantly doesn't look much like the command we started with so the tiniest change in requirements meant a complete rewrite using different constructs.
Now, how about if you want to enhance it to, say, only print the first and last line if the file contained foo? I expect that'd be another challenging exercise with sed and probably involve non-portable constructs too.
It's just all pointless to learn how to do this with sed when you can use a different tool like awk and do whatever you like in a simple, consistent, portable syntax:
$ seq 3 7 |
awk 'NR==1{first=$0} {last=$0} END{print first ORS last}'
3
7
$ printf 'foo\n' |
awk 'NR==1{first=$0} {last=$0} END{print first ORS last}'
foo
foo
$ printf 'foo\n' |
awk 'NR==1{first=$0} {last=$0} END{print first (NR>1 ? ORS last : "")}'
foo
$ printf '3\nfoo\n7\n' |
awk 'NR==1{first=$0} /foo/{f=1} {last=$0} END{if (f) print first (NR>1 ? ORS last : "")}'
3
7
$ printf '3\nbar\n7\n' |
awk 'NR==1{first=$0} /foo/{f=1} {last=$0} END{if (f) print first (NR>1 ? ORS last : "")}'
$
Notice that:
Every command looks like every other command.
A minor change in requirements leads to a minor change in the code, not a complete rewrite.
Once you learn how to do any given thing A, how to do similar things B, C, D, etc. just builds on top of the syntax you already used, you don't have to learn a completely different syntax.
Each of those commands will work using any awk in any shell on every UNIX box.
Now, how about if you want to do that for multiple files such as would be created by the following commands?
$ seq 3 7 > file1
$ seq 12 25 > file2
With awk you can just store the lines in an array for printing in the END:
$ awk 'FNR==1{first[++cnt]=$0} {last[cnt]=$0}
END{for (i=1;i<=cnt;i++) print first[i] ORS last[i]}' file1 file2
3
7
12
25
or with GNU awk you can print them from ENDFILE:
$ awk 'FNR==1{first=$0} {last=$0} ENDFILE{print first ORS last}' file1 file2
3
7
12
25
With sed? An exercise left for the reader.
test.txt contains:
this is a line
another line
one more line
For example, this can remove 1-2 lines, and save the rest into the rest.txt
sed -e '1,2d' test.txt > rest.txt
But the original file remains intact. Then how to get the remaining lines from the file? In this example, I want to remove the first 2 lines, save them into a file 'deleted.txt', and save the 3rd line into 'rest.txt'.
With GNU sed:
seq 1 5 | sed -e '1,2w deleted.txt' -e '1,2d' > rest.txt
w filename: Write the current pattern space to filename.
awk 'NR<=2{print $0 > "deleted.txt"}NR>2{print $0 > "rest.txt"}' test.txt
For lines(here NR) <= 2 redirect the output to deleted.
For lines(here NR) > 2 redirect the output to rest.txt.
I want to delete some specific rows in a file. For example I have a file containing these rows:
+1 1:-51.000000
+1 1:-47.000000
+1 1:-53.000000
+1 1:-48.000000
+1 1:-49.000000
+1 1:-42.000000
I want to delete the last 3 rows. How to do that in bash file?
nl=`wc -l fileName | awk 's=$1-3{print s}'`; head -n $nl fileName > file_withoutlast3rows.txt
wc -l calculates the total number of lines in the file. Using awk, you can print the total number of lines -3. Then use head with -n option to read only that many lines. You can also put that many lines into a new file using >.
You can also use awk to do this :
awk -v nl=$(wc -l <fileName) 'NR<(nl-2)' fileName
Simply count the number of lines, calculate the first index, and use sed -i to delete it.
COUNT=$(wc -l < file)
INDEX=$((COUNT > 3 ? COUNT - 2 : 1))
sed -i "$INDEX,\$d" file
Or just use bash's readarray:
readarray LINES < file
LAST=$((${#LINES[#]} > 3 ? ${#LINES[#]} - 3 : 0))
printf "%s" "${LINES[#]:0:LAST}" > file
This might work for you (GNU sed):
sed '1N;$!N;$d;P;D' file
for say 5 lines:
sed ':a;$!N;s/.*/&/m5;Ta;$d;P;D' file
Using only awk with no external commands like wc.
Read the file twice
awk 'NR==FNR {i++;next} (i-FNR)>=3' file file
So I have a txt file where I need to extract every third number and print it to separate file using Terminal. The txt file is just a long list of numbers, tab delimited:
18 25 0 18 24 5 18 23 5 18 22 8.2 ...
I know there is a way to do this using sed or awk, but so far I've only been able to extract every third line by using:
awk 'NR%3==1' testRain.txt > rainOnly.txt
So here's the answer (or rather, the answer I utilized!):
xargs -n1 < input.txt | awk '!(NR%3)' > output.txt
This gives you an output.txt that has every third number of the original file as a separate line.
A quick pipe line to extract every 3rd number:
$ xargs -n1 < file | sed '3~3!d'
0
5
5
8.2
If you don't want each number on a newline throw the result back through xargs:
$ xargs -n1 < file | sed '3~3!d' | xargs
0 5 5 8.2
Use redirection to store the output in a new file:
$ xargs -n1 < file | sed '3~3!d' | xargs > new_file
With awk using a simple for loop you could do:
$ awk '{for(i=3;i<=NF;i+=3)print $i}' file
0
5
5
8.2
or (adds a trailing tab):
$ awk '{for(i=3;i<=NF;i+=3)printf "%s\t",$i;print ""}' file
0 5 5 8.2
Or by setting the value of RS (adds trailing newline):
$ awk '!(NR%3)' RS='\t' file
0
5
5
8.2
$ awk '!(NR%3)' RS='\t' ORS='\t' file
0 5 5 8.2
You can print every third character by substituting the next two with nothing, globally. When the count straddles a newline, using Perl might be the simplest solution:
perl -p000 -e 's/(.)../$1/gs'
If you want the first, fourth etc character from every line, a line-oriented tool like sed suffices:
sed 's/\(.\)../\1/g'
Using grep -P
grep -oP '([^\t]+\t){2}\K[^\t\n]+' file
0
5
5
8.2
This might work for you (GNU sed):
sed -r 's/(\S+\s){3}/\1/g;s/\s$//' file
#user2718946
Your solution was close, but here you are without xarg.
awk 'NR%3==1' RS=" " file
18
18
18
18
Different start:
awk 'NR%3==0' RS=" " file
0
5
5
8.2
Is fairly easy to strip the first and last character from a string using awk/sed?
Say I have this string
( 1 2 3 4 5 6 7 )
I would like to strip parentheses from it.
How should I do this?
sed way
$ echo '( 1 2 3 4 5 6 7 )' | sed 's/^.\(.*\).$/\1/'
1 2 3 4 5 6 7
awk way
$ echo '( 1 2 3 4 5 6 7 )' | awk '{print substr($0, 2, length($0) - 2)}'
1 2 3 4 5 6 7
POSIX sh way
$ var='( 1 2 3 4 5 6 7 )'; var="${var#?}"; var="${var%?}"; echo "$var"
1 2 3 4 5 6 7
bash way
$ var='( 1 2 3 4 5 6 7 )'; echo "${var:1: -1}"
1 2 3 4 5 6 7
If you use bash then use the bash way.
If not, prefer the posix-sh way. It is faster than loading sed or awk.
Other than that, you may also be doing other text processing, that you can combine with this, so depending on the rest of the script you may benefit using sed or awk in the end.
why doesn't this work? sed '..' s_res.temp > s_res.temp ?
This does not work, as the redirection > will truncate the file before it is read.
To solve this you have some choices:
what you really want to do is edit the file. sed is a stream editor not a file editor.
ed though, is a file editor (the standard one too!). So, use ed:
$ printf '%s\n' "%s/^.\(.*\).$/\1/" "." "wq" | ed s_res.temp
use a temporary file, and then mv it to replace the old one.
$ sed 's/^.\(.*\).$/\1/' s_res.temp > s_res.temp.temp
$ mv s_res.temp.temp s_res.temp
use -i option of sed. This only works with GNU-sed, as -i is not POSIX and GNU-only:
$ sed -i 's/^.\(.*\).$/\1/' s_res.temp
abuse the shell (not recommended really):
$ (rm test; sed 's/XXX/printf/' > test) < test
On Mac OS X (latest version 10.12 - Sierra) bash is stuck to version 3.2.57 which is quite old. One can always install bash using brew and get version 4.x which includes the substitutions needed for the above to work.
There is a collection of bash versions and respective changes, compiled on the bash-hackers wiki
To remove the first and last characters from a given string, I like this sed:
sed -e 's/^.//' -e 's/.$//'
# ^^ ^^
# first char last char
See an example:
sed -e 's/^.//' -e 's/.$//' <<< "(1 2 3 4 5 6 7)"
1 2 3 4 5 6 7
And also a perl way:
perl -pe 's/^.|.$//g'
If I want to remove the First (1) character and the last two (2) characters using sed.
Input "t2.large",
Output t2.large
sed -e 's/^.//' -e 's/..$//'
`