I have the following sed command:
sed -i -E "s/\{\{(.*)\}\}/$(echo "$\1")/g" test.conf
In test.conf, I have this:
this is a {{TEST}}
and this is an {{ANSWER}} here.
And I have the follow environment variables set:
export TEST=1234
export ANSWER=5678
When I run the sed command, I end up with this result:
this is a $TEST
and this is an $ANSWER here.
I want 1234 and 5678 there respectively. Is there a reason the echo command is interpreting things literally?
Backreferences are used internally by a single sed command. The echo has no idea about sed backreferences and would have been invoked by the shell before the sed command has even run so the $(echo "$\1") is outputing $\1 so
sed -i -E "s/\{\{(.*)\}\}/$(echo "$\1")/g" test.conf
is really:
sed -i -E "s/\{\{(.*)\}\}/$\1/g" test.conf
hence the output you are seeing.
Anyway, sed is for simple subsitutions on individual lines, for anything else you should be using awk:
$ export TEST=1234 ANSWER=5678
$ awk 'match($0,/(.*)\{\{(.*)\}\}(.*)/,a){$0=a[1] ENVIRON[a[2]] a[3]} 1' file
this is a 1234
and this is an 5678 here.
The above uses GNU awk for the 3rd arg to match(), with other awks it'd be:
$ awk 'match($0,/\{\{(.*)\}\}/){$0=substr($0,1,RSTART-1) ENVIRON[substr($0,RSTART+2,RLENGTH-4)] substr($0,RSTART+RLENGTH)} 1' file
this is a 1234
and this is an 5678 here.
If anyone suggests running eval or similar on the sed output - don't do it (google eval is evil and friends), just use the awk command above for simple string operations.
You can use perl which conveniently has a %ENV hash variable, see perldoc for more info
perl -pe 's/\{\{(.*)\}\}/$ENV{$1}/' test.conf
Related
I'm looking for an equivalent of perl -pe. Ideally, it would be replace with sed if it's possible. Any help is highly appreciated.
The code is:
perl -pe 's/^\[([^\]]+)\].*$/$1/g'
$ echo '[foo] 123' | perl -pe 's/^\[([^\]]+)\].*$/$1/g'
foo
$ echo '[foo] 123' | sed -E 's/^\[([^]]+)\].*$/\1/'
foo
sed by default accepts code from command line, so -e isn't needed (though it can be used)
printing the pattern space is default, so -p isn't needed and sed -n is similar to perl -n
-E is used here to be as close as possible to Perl regex. sed supports BRE and ERE (not as feature rich as Perl) and even that differs from implementation to implementation.
with BRE, the command for this example would be: sed 's/^\[\([^]]*\)\].*$/\1/'
\ isn't special inside character class unless it is an escape sequence like \t, \x27 etc
backreferences use \N format (and limited to maximum 9)
Also note that g flag isn't needed in either case, as you are using line anchors
I'm basically trying to modify tomcat server.xml connector tag and add a address attribute to it.
I want to find the below string in server.xml
I'm doing the below with sed,
export currlistener=\<Connector\ port\=\"18443\"
export newlistener=\<Connector\ port\=\"18443\"\ address\=\"127.0.0.1\"\
echo $currlistener
echo $newlistener
sed -i -e 's/'$currlistener'/'$newlistener'/g' server.xml
But I get the error
sed: -e expression #1, char 12: unterminated `s' command
I guess sed is interpreting the special characters and erroring out.
How would I do the same using awk?
Regards,
Anand.
Using sed
The problem was that the shell variables were unquoted. Try:
sed -i -e "s/$currlistener/$newlistener/g" server.xml
Using awk
The sed solution requires that you trust the source of your shell variables. For a case like this, awk is safer. Using a modern GNU awk:
awk -i inplace -v a="$currlistener" -v b="$newlistener" '{gsub(a, b)} 1' server.xml
Or, using other awk:
awk -v a="$currlistener" -v b="$newlistener" '{gsub(a, b)} 1' server.xml >tmp && mv tmp server.sml
Simplifying the variable assignments
Separately, the shell variables can be defined without requiring so many escapes:
currlistener='<Connector port="18443"'
newlistener='<Connector port="18443" address="127.0.0.1"'
It is only necessary to export them if they are to be used in a child process.
I'm parsing a csv file with a sed command like this:
sed -n -e 's/abc/&/p' -e 's/xyz/&/p' <input >output
Now if there is both in one line (abc and xyz) I'll have the line twice in the output. I'd should have it just once.
Can I do that with sed?
If you only want to print a line with "abc" or "xyz":
sed -n '/abc\|xyz/p'
Other tools:
grep -F -e abc -e xyz
awk '/abc/ || /xyz/'
I believe you are mis-using the s///p just to print the lines. This is not necessary in sed - you can get them printed using //p.
Both expressions will evaluate, though, so you are still at risk of duplication. Your best bet (and fastest, for large datasets) will be to build the 'or' behavior into the matching regexp:
sed -Ene '/abc|xyz/p' input >output
sed -n -r -e 's/(abc|xyz)/&/p' <input >output
-r flag is for enabling extended regular expressions (is in GNU sed)
sed -n '/abc/{p;b
}
/xyz/p' Input > Output
for non GNU sed (where | is not allowed as OR)
I have values in a file like this ' value-to-remove '(without the ' characters). I want to use sed to run through the file and replace the values including the space before and after. I am running this via a bash script.
How can I do this?
The sed command I'm using at the moment replaces the values but leaves behind the two spaces.
sed -i 's/ '$value' / /g' test.conf
In script I have
sed -i -e 's/\s'$DOMAIN'-'$SITE'\s/\s/g' gitosis.conf
echoed as
sed -i -e s/\sffff.com-eeee\s/\s/g test.conf
Not working though.
IMHO your sed does not know '\s', so use [ \t], and use double quotes, otherwise your variables will not expand. e.g.:
sed -i -e "s/[ \t]'$DOMAIN'-'$SITE'[ \t]/ /g" gitosis.conf
Let me know if this is what you need
echo 'Some values to remove value-to-remove and more' | sed -e 's/\svalue-to-remove\s/CHANGED/g'
output: Some values to removeCHANGEDand more
I need to replace several URLs in a text file with some content dependent on the URL itself. Let's say for simplicity it's the first line of the document at the URL.
What I'm trying is this:
sed "s/^URL=\(.*\)/TITLE=$(curl -s \1 | head -n 1)/" file.txt
This doesn't work, since \1 is not set. However, the shell is getting called. Can I somehow push the sed match variables to that subprocess?
The accept answer is just plain wrong. Proof:
Make an executable script foo.sh:
#! /bin/bash
echo $* 1>&2
Now run it:
$ echo foo | sed -e "s/\\(foo\\)/$(./foo.sh \\1)/"
\1
$
The $(...) is expanded before sed is run.
So you are trying to call an external command from inside the replacement pattern of a sed substitution. I dont' think it can be done, the $... inside a pattern just allows you to use an already existent (constant) shell variable.
I'd go with Perl, see the /e option in the search-replace operator (s/.../.../e).
UPDATE: I was wrong, sed plays nicely with the shell, and it allows you do to that. But, then, the backlash in \1 should be escaped. Try instead:
sed "s/^URL=\(.*\)/TITLE=$(curl -s \\1 | head -n 1)/" file.txt
Try this:
sed "s/^URL=\(.*\)/\1/" file.txt | while read url; do sed "s#URL=\($url\)#TITLE=$(curl -s $url | head -n 1)#" file.txt; done
If there are duplicate URLs in the original file, then there will be n^2 of them in the output. The # as a delimiter depends on the URLs not including that character.
Late reply, but making sure people don't get thrown off by the answers here -- this can be done in gnu sed using the e command. The following, for example, decrements a number at the beginning of a line:
echo "444 foo" | sed "s/\([0-9]*\)\(.*\)/expr \1 - 1 | tr -d '\n'; echo \"\2\";/e"
will produce:
443 foo