I have problem with my sed script. I process this data using the above script.
The problem is that, the lines from script:
/^$/d
s/ $/ajes/g
doesn't work. Of course it will work if I process data again.
This doesn't work:
sed -f script.sed -i data.file
but this will work (double procesing):
sed -f script.sed -i data.file
sed -f script.sed -i data.file
Why is that? Where is problem? Why I can't process it once with all results?
The real problem is that you have just one line. And sed reads it once and tries execute each command one by one, while the whole line is in the patterns space. When you substitute something to \n it is still just one string and it is in the pattern space. It wasn't divided in multiple strings and reread one by one.
Obviously /^$/ doesn't match to your \n\n and s/ $/ajes/g doesn't match to your any space before \n. That's the reason why that doesn't work the way you expect.
You can solve the problem with modifying "non working" commands to
s/\n\+/\n/g # equals to /^$/d
s/ \n/ajes\n/g # equals to s/ $/ajes/g
You have many substitutions there. Let's say you have substitutions
Rule A
Rule B
Rule C.
So, the problem here is if Rule A initially does not match and then Rule B or C matches resulting in a pattern that would match again with A after replacement, then Rule A would not come into effect for that line. So, you need to analyze the order in which the substitutions would be processed and hopefully you'll find the mistake.
Hope this helps you in the proper direction.
Related
In short:
how to convert from fasta to "phylip"-like format (without the sequence and residu counts at the top of the file) using sed ?
A fasta format is like this:
>sequence1
AATCG
GG-AT
>sequence2
AGTCG
GGGAT
The number of lines of a sequence may vary.
I want to convert it to this:
sequence1 AATCG GG-AT
sequence2 AGTCG GGGAT
My question seems simple, but I am lacking a real understanding of the advanced commands in sed, the multiline commands and the commands using the hold buffer.
Here is the implementation idea I had:
fill the pattern space with sequence, and only print it when a new sequence label is encountered. To do this, I would:
Search lines matching ^>. If found:
print the previous pattern space
append line to pattern space
if ^> not found:
append line to pattern space
I read this great manual,
but I am still unsure about a few things, mostly the difference between the capitalized and little letters:
when you use P instead of p:
does it print the first line of the pattern space (in file order)?
I am confused by the use of "up to the next newline".
do I have to use a loop to read lines until the next sequence name, or are the multiline commands sufficient?
do I have to use the hold space in this example?
I know python, perl and awk and I think they would be more "human-friendly" tools to achieve this, but I want to learn some advanced sed.
Nothing I tried worked now, but here are some pieces:
This script uses the line numbers, not trying to do pattern matching.
It shoes what I want to do, and now I need to automate it using match addresses:
#!/bin/sed -nf
1h
2,3H
4{x; s/\n/ /g; p}
5H
6{H;x; s/\n/ /g; p}
sed -nf fa2phy.sed my.fasta returns the expected output.
With sed
sed '/>/N;:A;/\n>/!{s/\n/ /;N;bA};h;s/\(.*\)\n.*/\1/p;x;s/.*\n//;bA' infile
Following simple awk could help you in same.
Solution 1st:
awk '/^>/{sub(/>/,"");if(val){print val, val2};val=$0;val2="";next} {val2=val2?val2 FS $0:$0} END{print val, val2}' Input_file
Solution 2nd:
awk -v RS=">" -v FS="\n" '{for(i=1;i<=NF;i++){printf("%s%s",$i,i==NF?"\n":" ")}}' Input_file
Solution 3rd:
awk -v RS=">" '{gsub(/\n/," ");} NF' Input_file
Alright, I believe I managed to answer my own question.
Here is the script I made: fa2phy.sed:
#!/bin/sed -nf
:readseq
${H;b out} # if last line, append to hold, and goto 'out'
1{h;n;b readseq} # if first, overwrite hold, and start again at 'readseq'
/^>/!{H; n; b readseq} # if not a sequence label, append to hold, read next line, start again at 'readseq'. Else, it continues to 'out'
:out
x; # exchange hold content with pattern content
s/^>//; # substitute the starting '>'
s/\n/ /g; # substitute each newline with 2 spaces
p; # print pattern buffer
Although it works, if someone has a shorter or clearer solution, enlighten me! :)
I'm trying to come up with a sed script to take all lines containing a pattern and move them to the end of the output. This is an exercise in learning hold vs pattern space and I'm struggling to come up with it (though I feel close).
I'm here:
$ echo -e "hi\nfoo1\nbar\nsomething\nfoo2\nyo" | sed -E '/foo/H; //d; $G'
hi
bar
something
yo
foo1
foo2
But I want the output to be:
hi
bar
something
yo
foo1
foo2
I understand why this is happening. It is because the first time we find foo the hold space is empty so the H appends \n to the blank hold space and then the first foo, which I suppose is fine. But then the $G does it again, namely another append which appends \n plus what is in the hold space to the pattern space.
I tried a final delete command with /^$/d but that didn't remove the blank line (I think this is because this pattern is being matched not against the last line, but against the, now, multiline pattern space which has a \n\n in it.
I'm sure the sed gurus have a fix for me.
This might work for you (GNU sed):
sed '/foo/H;//!p;$!d;x;//s/.//p;d' file
If the line contains the required string append it to the hold space (HS) otherwise print it as normal. If it is not the last line delete it otherwise swap the HS for the pattern space (PS). If the required string(s) is now in the PS (what was the HS); since all such patterns were appended, the first character will be a newline, delete the first character and print. Delete whatever is left.
An alternative, using the -n flag:
sed -n '/foo/H;//!p;$!b;x;//s/.//p' file
N.B. When the d or b (without a parameter) command is performed no further sed commands are, a new line is read into the PS and the sed script begins with the first command i.e. the sed commands do not resume following the previous d command.
Why? Stuff like this is absolutely trivial in awk, awk is available everywhere that sed is, and the resulting awk script will be simpler, more portable, faster and better in almost every other way than a sed script to do the same task. All that hold space stuff was necessary in sed before the mid-1970s when awk was invented but there's absolutely no use for it now other than as a mental exercise.
$ echo -e "hi\nfoo1\nbar\nsomething\nfoo2\nyo" |
awk '/foo/{buf = buf $0 RS;next} {print} END{printf "%s",buf}'
hi
bar
something
yo
foo1
foo2
The above will work as-is in every awk on every UNIX installation and I bet you can figure out how it works very easily.
This feels like a hack and I think it should be possible to handle this situation more gracefully. The following works on GNU sed:
echo -e "hi\nfoo1\nbar\nsomething\nfoo2\nyo" | sed -r '/foo/{H;d;}; $G; s/\n\n/\n/g'
However, on OSX/BSD sed, results in this odd output:
hi
bar
something
yonfoo1
foo2
Note the 2 consecutive newlines was replaced with the literal character n
The OSX/BSD vs GNU sed is explained in this article. And the following works (in GNU SED as well):
echo -e "hi\nfoo1\nbar\nsomething\nfoo2\nyo" | sed '/foo/{H;d;}; $G; s/\n\n/\'$'\n''/'
TL;DR; in BSD sed, it does not accept escaped characters in the RHS of the replacement expression and so you either have to put a true LF/newline in there at the command line, or do the above where you split the sed script string where you need the newline on the RHS and put a dollar sign in front of '\n' so the shell interprets it as a line feed.
I'm having issues matching strings even if they start with any number of white spaces. It's been very little time since I started using regular expressions, so I need some help
Here is an example. I have a file (file.txt) that contains two lines
#String1='Test One'
String1='Test Two'
Im trying to change the value for the second line, without affecting line 1 so I used this
sed -i "s|String1=.*$|String1='Test Three'|g"
This changes the values for both lines. How can I make sed change only the value of the second string?
Thank you
With gnu sed, you match spaces using \s, while other sed implementations usually work with the [[:space:]] character class. So, pick one of these:
sed 's/^\s*AWord/AnotherWord/'
sed 's/^[[:space:]]*AWord/AnotherWord/'
Since you're using -i, I assume GNU sed. Either way, you probably shouldn't retype your word, as that introduces the chance of a typo. I'd go with:
sed -i "s/^\(\s*String1=\).*/\1'New Value'/" file
Move the \s* outside of the parens if you don't want to preserve the leading whitespace.
There are a couple of solutions you could use to go about your problem
If you want to ignore lines that begin with a comment character such as '#' you could use something like this:
sed -i "/^\s*#/! s|String1=.*$|String1='Test Three'|g" file.txt
which will only operate on lines that do not match the regular expression /.../! that begins ^ with optional whiltespace\s* followed by an octothorp #
The other option is to include the characters before 'String' as part of the substitution. Doing it this way means you'll need to capture \(...\) the group to include it in the output with \1
sed -i "s|^\(\s*\)String1=.*$|\1String1='Test Four'|g" file.txt
With GNU sed, try:
sed -i "s|^\s*String1=.*$|String1='Test Three'|" file
or
sed -i "/^\s*String1=/s/=.*/='Test Three'/" file
Using awk you could do:
awk '/String1/ && f++ {$2="Test Three"}1' FS=\' OFS=\' file
#String1='Test One'
String1='Test Three'
It will ignore first hits of string1 since f is not true.
This should be extremely simple, but for the life of me I just can't get gnu-sed to do it this afternoon.
The file in question has lines that look like this:
PART NUMBER PART NUMBER QUANTITY WEIGHT -999 -4,999 -9,999
w/ UL APPROVAL
MIN-3
I need to prepend every line like the "MIN-3" line with a ">" character, and the only thing specifically differentiating those lines from the others are two things:
The first character is a space " ".
The lines do not contain a comma.
I've tried mostly things like any of the following:
/^ +[^,]+$/ s/^/>/
/^ +[\w\-]+$/ s/^/>/
/^ +(\w|\-)+$/ s/^/>/
I will admit, I am somewhat new to sed. :)
Edit: Answers that use perl, or awk could also be appreciated, though my initial target is sed.
try this:
sed '/^ [^,]*$/s/^/>/'
the output is, only the line with MIN-3 with leading >
sed default uses basic regex. so the + should be \+ in your script. I think that could be the problem killing your time. You could add -r however, to let sed use extended-regex.
According to your description this should do:
sed 's/^\([ ][^,]*\)$/> \1/' input
which matches the complete line if the line starts with a space and then contains anything but a comma until the end.
Here is a simple answer:
sed 's/^ [^,]*$/>&/'
Is there a way to substitute only within the match space using sed?
I.e. given the following line, is there a way to substitute only the "." chars that are contained within the matching single quotes and protect the "." chars that are not enclosed by single quotes?
Input:
'ECJ-4YF1H10.6Z' ! 'CAP' ! '10.0uF' ! 'TOL' ; MGCDC1008.S1 MGCDC1009.A2
Desired result:
'ECJ-4YF1H10-6Z' ! 'CAP' ! '10_0uF' ! 'TOL' ; MGCDC1008.S1 MGCDC1009.A2
Or is this just a job to which perl or awk might be better suited?
Thanks for your help,
Mark
Give the following a try which uses the divide-and-conquer technique:
sed "s/\('[^']*'\)/\n&\n/g;s/\(\n'[^.]*\)\.\([^']*Z'\)/\1-\2/g;s/\(\n'[^.]*\)\.\([^']*uF'\)/\1_\2/g;s/\n//g" inputfile
Explanation:
s/\('[^']*'\)/\n&\n/g - Add newlines before and after each pair of single quotes with their contents
s/\(\n'[^.]*\)\.\([^']*Z'\)/\1-\2/g - Using a newline and the single quotes to key on, replace the dot with a dash for strings that end in "Z"
s/\(\n'[^.]*\)\.\([^']*uF'\)/\1_\2/g - Using a newline and the single quotes to key on, replace the dot with a dash for strings that end in "uF"
s/\n//g - Remove the newlines added in the first step
You can restrict the command to acting only on certain lines:
sed "/foo/{s/\('[^']*'\)/\n&\n/g;s/\(\n'[^.]*\)\.\([^']*Z'\)/\1-\2/g;s/\(\n'[^.]*\)\.\([^']*uF'\)/\1_\2/g;s/\n//g}" inputfile
where you would substitute some regex in place of "foo".
Some versions of sed like to be spoon fed (instead of semicolons between commands, use -e):
sed -e "/foo/{s/\('[^']*'\)/\n&\n/g" -e "s/\(\n'[^.]*\)\.\([^']*Z'\)/\1-\2/g" -e "s/\(\n'[^.]*\)\.\([^']*uF'\)/\1_\2/g" -e "s/\n//g}" inputfile
$ cat phoo1234567_sedFix.sed
#! /bin/sed -f
/'[0-9][0-9]\.[0-9][a-zA-Z][a-zA-Z]'/s/'\([0-9][0-9]\)\.\([0-9][a-zA-Z][a-zA-Z]\)'/\1_\2/
This answers your specific question. If the pattern you need to fix isn't always like the example you provided, they you'll need multiple copies of this line, with reg-expressions modified to match your new change targets.
Note that the cmd is in 2 parts, "/'[0-9][0-9].[0-9][a-zA-Z][a-zA-Z]'/" says, must match lines with this pattern, while the trailing "s/'([0-9][0-9]).([0-9][a-zA-Z][a-zA-Z])'/\1_\2/", is the part that does the substitution. You can add a 'g' after the final '/' to make this substitution happen on all instances of this pattern in each line.
The \(\) pairs in match pattern get converted into the numbered buffers on the substitution side of the command (i.e. \1 \2). This is what gives sed power that awk doesn't have.
If your going to do much of this kind of work, I highly recommend O'Rielly's Sed And Awk book. The time spent going thru how sed works will be paid back many times.
I hope this helps.
P.S. as you appear to be a new user, if you get an answer that helps you please remember to mark it as accepted, or give it a + (or -) as a useful answer.
this is a job most suitable for awk or any language that supports breaking/splitting strings.
IMO, using sed for this task, which is regex based , while doable, is difficult to read and debug, hence not the most appropriate tool for the job. No offense to sed fanatics.
awk '{
for(i=1;i<=NF;i++) {
if ($i ~ /\047/ ){
gsub(".","_",$i)
}
}
}1' file
The above says for each field (field seperator by default is white space), check to see if there is a single quote, and if there is , substitute the "." to "_". This method is simple and doesn't need complicated regex.