I want to delete part of text:
0
1
test1
a
b
random letter
test2
e
f
g
I want to get:
0
1
test2
e
f
g
I've tried use sed:
sed '/test1/,/test2/d'
But it will remove test2 too
How can I delete text and save test2, if I don't exactly know what text before test2
I need to use awk or sed
give this a try:
sed '/test1/,/test2/{/test2/!d}'
test with your example:
kent$ echo "0
1
test1
a
b
random letter
test2
e
f
g"|sed '/test1/,/test2/{/test2/!d}'
0
1
test2
e
f
g
awk 'BEGIN{p=1}/test1/{p=0}/test2/{p=1}p' your_file
Tested Below:
> cat temp
0
1
test1
a
b
random letter
test2
e
f
g
>
> awk 'BEGIN{p=1}/test1/{p=0}/test2/{p=1}p' temp
0
1
test2
e
f
g
>
If you want to search for whole word in awk:
search like below:
/\<WORD\>/
Alternatively you can go perl as well:
perl -lne 'BEGIN{$p=1}if(/\btest1\b/){$p=0}if(/\btest2\b/){$p=1}print if $p' your_file
Related
I am able to delete lines with certain patterns and shorter sed '/^.\{,20\}$/d' -i FILE or longer sed '/^.\{25\}..*/d' -i FILE than certain length separately, but how do I unite pattern and length in sed?
Lines containing A should be between 20 and 25 characters
Lines containing B should be between 10 and 15 characters
Lines containing C should be between 3 and 8 characters
All other lines should be deleted from the file
1234567890 A 1234567890
12345 A 12345
1 A 1
1234567890 B 1234567890
12345 B 12345
1 B 1
1234567890 C 1234567890
12345 C 12345
1 C 1
So that the output should look like this
1234567890 A 1234567890
12345 B 12345
1 C 1
This is how you can do it with sed:
$ sed -ne '/A/ s/^\(.\{20,25\}\)$/\1/p; /B/ s/^\(.\{10,15\}\)$/\1/p; /C/ s/^\(.\{3,8\}\)$/\1/p;' file
1234567890 A 1234567890
12345 B 12345
1 C 1
How does it work:
-ne - suppress printing pattern
/A/ - look for pattern A
^\(.\{20,25\}\)$ - line with 20-25 characters
/\1/p - print pattern space
Use awk and you can simply write the conditions as a boolean expression, you're not stuck trying to make a condition out of a regexp:
$ awk '(/A/ && /^.{20,25}$/) || (/B/ && /^.{10,15}$/) || (/C/ && /^.{3,8}$/)' file
1234567890 A 1234567890
12345 B 12345
1 C 1
Here's an awk solution
awk '/.*A.*/ && length($0) > 19 && length($0) < 26 \
|| /.*B.*/ && length($0) > 9 && length($0) < 16 \
|| /.*C.*/ && length($0) > 2 && length($0) < 9' test1.dat
edit
And here's a more efficient version, where we only get the length($0) one time
awk '{len=length($0)}
/.*A.*/ && len > 19 && len < 26 \
|| /.*B.*/ && len > 9 && len < 16 \
|| /.*C.*/ && len > 2 && len < 9' test1.dat
output
1234567890 A 1234567890
12345 B 12345
1 C 1
I have incremented/decremented your boundary numbers by one to eliminate the need to test with <= and >= (Which are slightly more expensive tests. On a very large file it might cost you a 30 secs (just a guess!)).
(don't let any whitespace characters creep in after the \ at the end of those continued lines).
(Also, you can remove that \ chars and fold this up onto one-line if you need that.)
This can be enhanced to accept variable values, and I include a short example here, finishing it out to your needs can be seen as an opportunity for learning ;-)
awk -v lim1=10 -v lim2=26 '/.*A.*/ && length($0) > lim1 && length($0) < lim2 ...
IHTH
I have two files of the type:
File1.txt
1 117458 rs184574713 rs184574713
1 119773 rs224578963 rs224500000
1 120000 rs224578874 rs224500045
1 120056 rs60094200 rs60094200
2 120056 rs60094536 rs60094536
File2.txt
10 120200 120400 A 0 189,183,107
1 0 119600 C 0 233,150,122
1 119600 119800 D 0 205,92,92
1 119800 120400 F 0 192,192,192
2 120400 122000 B 0 128,128,128
2 126800 133200 A 0 192,192,192
I want to add the information contained in the second file to the first file. The first column in both files needs to match, while the second column in File1.txt should fall in the interval that is indicated by columns 2 and 3 in File2.txt. So that the output should look like this:
1 117458 rs184574713 rs184574713 C 0 233,150,122
1 119773 rs224578963 rs224500000 D 0 205,92,92
1 120000 rs224578874 rs224500045 F 0 192,192,192
1 120056 rs60094200 rs60094200 F 0 192,192,192
2 120440 rs60094536 rs60094536 B 0 128,128,128
Please help me with awk/perl.. or any other script.
This is how you would do it in bash (with a little help from awk):
join -1 1 -2 1 \
<(sort -n -k1,1 test1) <(sort -n -k1,1 test2) | \
awk '$2 >= $5 && $2 <= $6 {print $1, $2, $3, $4, $7, $8, $9}'
Here is a brief explanation.
First, we use join to join lines based on the common key (the
first field).
But join expects both input files to be already sort (hence
sort).
At least, we employ awk to apply the required condition, and to
project the fields we want.
Try this: (Considering the fact that there is a typo in your output for last entry. 120056 is not between 120400 122000.
$ awk '
NR==FNR {
a[$1,$2,$3]=$4 FS $5 FS $6;
next
}
{
for(x in a) {
split(x,tmp,SUBSEP);
if($1==tmp[1] && $2>=tmp[2] && $2<=tmp[3])
print $0 FS a[x]
}
}' file2 file1
1 117458 rs184574713 rs184574713 C 0 233,150,122
1 119773 rs224578963 rs224500000 D 0 205,92,92
1 120000 rs224578874 rs224500045 F 0 192,192,192
1 120056 rs60094200 rs60094200 F 0 192,192,192
You read through the first file creating an array indexed at column 1,2 and 3 having values of column 4,5 and 6.
For the second file, you look up in your array. For every key, you split the key and check for your condition of first column matching and second column to be in range.
If the condition is true you print the entire line from file 1 followed by the value of array.
I have a large file consisting data in 2 columns
100 5
100 10
100 10
101 2
101 4
102 10
102 2
I want to sum the values in 2nd column with matching values in column 1. For this example, the output I'm expecting is
100 25
101 6
102 12
I'm trying to work on this using bash script preferably. Can someone explain me how can I do this
Using awk:
awk '{a[$1]+=$2}END{for(i in a){print i, a[i]}}' inputfile
For your input, it'd produce:
100 25
101 6
102 12
In a perl oneliner
perl -lane "$s{$F[0]} += $F[1]; END { print qq{$_ $s{$_}} for keys %s}" file.txt
You can use an associative array. The first column is the index and the second becomes what you add to it.
#!/bin/bash
declare -A columns=()
while read -r -a line ; do
columns[${line[0]}]=$((${columns[${line[0]}]} + ${line[1]}))
done < "${1}"
for idx in ${!columns[#]} ; do
echo "${idx} ${columns[${idx}]}"
done
Using awk and maintain the order:
awk '!($1 in a){a[$1]=$2; b[++i]=$1;next} {a[$1]+=$2} END{for (k=1; k<=i; k++) print b[k], a[b[k]]}' file
100 25
101 6
102 12
Python is my choice:
d = {}
for line in f.readlines():
key,value = line.split()
if d[key] == None:
d[key] = 0
d[key] += value
print d
Why would you want a bash script?
Another sed question! I have nucleotide data in pairs
1 Affx-14150122 0 75891 00 CT TT CT TT CT
split by spaces and I need to put a space into every pair, eg
1 Affx-14150122 0 75891 0 0 C T T T C T T T C T
I've tried sed 's/[A-Z][A-Z]/ &/g' and sed 's/[A-Z][A-Z]/& /g'
And both A-Z replaced with .. and it never splits the pair as I'd like it to (it puts spaces before or after or splits every other pair or similar!).
I assume that this will work for you, however it's not perfect!
echo "1 Affx-14150122 0 75891 00 CT TT CT TT CT" | \
sed 's/\(\s[A-Z]\)\([A-Z]\)/\1 \2/g'
gives
1 Affx-14150122 0 75891 00 C T T T C T T T C T
sed 's/\(\s[A-Z]\)\([A-Z]\)/\1 \2/g' matches whitespace (\s) upper case character ([A-Z]), puts that in a group (\(...\)), and then matches upper case character and stores that in second group. Then this match is substituted by first group (\1) space second group (\2).
NOTE:
This fails when you have sequences that are longer than 2 characters.
An solution using awk which modifies only pairs of characters and might be more robust depending on your input data:
echo "1 Affx-14150122 0 75891 00 CT TT CT TT CT" | \
awk '
{
for(i=1;i<=NF;i++) {
if($i ~ /^[A-Z][A-Z]$/){
$i=substr($i,1,1)" "substr($i,2,1)
}
}
}
1'
gives
1 Affx-14150122 0 75891 00 C T T T C T T T C T1
This might work for you (GNU sed):
echo '1 Affx-14150122 0 75891 00 CT TT CT TT CT' |
sed ':a;s/\(\s\S\)\(\S\(\s\|$\)\)/\1 \2/g;ta'
1 Affx-14150122 0 75891 0 0 C T T T C T T T C T
This second method works but might provide false positives:
echo '1 Affx-14150122 0 75891 00 CT TT CT TT CT' | sed 's/\<\(.\)\(.\)\>/\1 \2/g'
1 Affx-14150122 0 75891 0 0 C T T T C T T T C T
This is actually easier in python than in awk:
echo caca | python -c 'import sys;\
for line in sys.stdin: print (" ".join(line))'
c a c a
Q1: Sed specify the whole line and if the line is nothing but the string then delete
I have a file that contains several of the following numbers:
1 1
3 1
12 1
1 12
25 24
23 24
I want to delete numbers that are the same in each line. For that I have either been using:
sed '/1 1/d' < old.file > new.file
OR
sed -n '/1 1/!p' < old.file > new.file
Here is the main problem. If I search for pattern '1 1' that means I get rid of '1 12' as well. So for I want the pattern to specify the whole line and if it does, to delete it.
Q2: Automation of question 1
I am also trying to automate this problem. The range of numbers in the first column and the second column could be from 1 to 25.
So far this is what I got:
for ((i=1;i<26;i++)); do
sed "/'$i' '$i'/d" < oldfile > newfile; mv newfile oldfile;
done
This does nothing to the oldfile in the end. :(
This would be more readable with awk:
awk '$1 == $2 {next} {print}' oldfile > newfile
Update based on comment:
If the requirement is to remove lines where the two values are within 1 of each other:
awk '{d = $1-$2; if (-1 <= d && d <= 1) next; else print}' oldfile
Unfortunately, awk does not have abs() (at least nawk and gawk don't)
Just put the first number in a group (\([0-9]*\)) and then look for it with a backreference (\1). Since the line to delete should contain only the group, repeated, use the ^ to mark the beginning of line and the $ to mark the end of line. For example, for the following file:
$ cat input
1 1
3 1
12 1
1 12
12 12
12 13
13 13
25 24
23 24
...the result is:
$ sed '/^\([0-9]*\) \1$/d' input
3 1
12 1
1 12
12 13
25 24
23 24
You can also do it with grep:
grep -E -v "([0-9])*\s\1" testfile
Look for multiple digits in a row and remember them, followed by a single whitespace, followed by whatever digits you remembered.