This question already has answers here:
sed substitution with Bash variables
(6 answers)
Closed 24 days ago.
The sed command works as expected at the command prompt, but does not work in a shell script.
new_db_name=`echo "$new_db_name" | sed 's/$replace_string/$replace_with/'`
Why is that, and how can I fix it?
Use double quotes for the sed expression.
new_db_name=$(echo "$new_db_name" | sed "s/$replace_string/$replace_with/")
If you use bash, this should work:
new_db_name=${new_db_name/$replace_string/$replace_with}
This worked for me in using env arguments.
export a=foo
export b=bar
echo a/b | sed 's/a/'$b'/'
bar/b
Guys: I used the following to pass bash variables to a function in a bash script using sed. I.e., I passed bash variables to a sed command.
#!/bin/bash
function solveOffendingKey(){
echo "We will delete the offending key in file: $2, in line: $1"
sleep 5
eval "sed -i '$1d' $2"
}
line='4'
file=~/ivan/known_hosts
solveOffendingKey $number $file
Kind regards!
depending on how your variables are initialized, you are better off using brackets:
new_db_name=`echo "$new_db_name" | sed "s/${replace_string}`/${replace_with}/"
Maybe I'm missing something but new_db_name=echo "$new_db_name" doesn't make sense here. $new_db_name is empty so you're echoing a null result, and then the output of the sed command. To capture stdout as a variable, backticks aren't recommended anymore. Capture output surrounded by $().
new_db_name=$(sed "s/${replace_string}/${replace_with}/")
Take the following example:
replace_string="replace_me"
replace_with=$(cat replace_file.txt | grep "replacement_line:" | awk FS" '{print $1}')
Where replace_file.txt could look something like:
old_string: something_old
I like cats
replacement_line: "shiny_new_db"
Just having the variable in the sed expresion $replace_with won't work. bash doesn't have enough context to escape the variable expression. ${replace_with} tells bash to explicitly use the contents of the command issued by the variable.
Related
Playing with makefile and bash shell scripting where I ended-up having a variable containing:
--global-option=build_ext --global-option=--b2-args=libtorrent-python-pic=on --global-option=--b2-args=address-model=32
I need to convert it so double-quotes gets appended at the right place such as:
--global-option=build_ext --global-option=--b2-args="libtorrent-python-pic=on" --global-option=--b2-args="address-model=32"
I tried the following without success:
echo $myvar | sed -e 's/ /\n/' | sed -z '{s/=/="/2;t;s/$/"/}'
--global-option=build_ext
--global-option="--b2-args=libtorrent-python-pic=on
EDIT: Note that this time it's --b2-args= but this could be a conjunction of --anything=, and the reason why I was focussing on the second instance of = to change for =" and if true append = at the end of word.
Since your question doesn't discuss anything about prepending --global-option= if it's missing as in the final --b2-args... string on the provided input line, I think your input was really supposed to be:
$ cat file
--global-option=build_ext --global-option=--b2-args=libtorrent-python-pic=on --global-option=--b2-args=address-model=32
in which case using any sed in any shell on every Unix box:
$ sed 's/\([^ =]*=[^= ]*=\)\([^ ]*\)/\1"\2"/g' file
--global-option=build_ext --global-option=--b2-args="libtorrent-python-pic=on" --global-option=--b2-args="address-model=32"
We have a process which can use a file containing sed commands to alter piped input.
I need to replace a placeholder in the input with a variable value, e.g. in a single -e type of command I can run;
$ echo "Today is XX" | sed -e "s/XX/$(date +%F)/"
Today is 2012-10-11
However I can only specify the sed aspects in a file (and then point the process at the file), E.g. a file called replacements.sed might contain;
s/XX/Thursday/
So obviously;
$ echo "Today is XX" | sed -f replacements.sed
Today is Thursday
If I want to use an environment variable or shell value, though, I can't find a way to make it expand, e.g. if replacements.txt contains;
s/XX/$(date +%F)/
Then;
$ echo "Today is XX" | sed -f replacements.sed
Today is $(date +%F)
Including double quotes in the text of the file just prints the double quotes.
Does anyone know a way to be able to use variables in a sed file?
This might work for you (GNU sed):
cat <<\! > replacements.sed
/XX/{s//'"$(date +%F)"'/;s/.*/echo '&'/e}
!
echo "Today is XX" | sed -f replacements.sed
If you don't have GNU sed, try:
cat <<\! > replacements.sed
/XX/{
s//'"$(date +%F)"'/
s/.*/echo '&'/
}
!
echo "Today is XX" | sed -f replacements.sed | sh
AFAIK, it's not possible. Your best bet will be :
INPUT FILE
aaa
bbb
ccc
SH SCRIPT
#!/bin/sh
STRING="${1//\//\\/}" # using parameter expansion to prevent / collisions
shift
sed "
s/aaa/$STRING/
" "$#"
COMMAND LINE
./sed.sh "fo/obar" <file path>
OUTPUT
fo/obar
bbb
ccc
As others have said, you can't use variables in a sed script, but you might be able to "fake" it using extra leading input that gets added to your hold buffer. For example:
[ghoti#pc ~/tmp]$ cat scr.sed
1{;h;d;};/^--$/g
[ghoti#pc ~/tmp]$ sed -f scr.sed <(date '+%Y-%m-%d'; printf 'foo\n--\nbar\n')
foo
2012-10-10
bar
[ghoti#pc ~/tmp]$
In this example, I'm using process redirection to get input into sed. The "important" data is generated by printf. You could cat a file instead, or run some other program. The "variable" is produced by the date command, and becomes the first line of input to the script.
The sed script takes the first line, puts it in sed's hold buffer, then deletes the line. Then for any subsequent line, if it matches a double dash (our "macro replacement"), it substitutes the contents of the hold buffer. And prints, because that's sed's default action.
Hold buffers (g, G, h, H and x commands) represent "advanced" sed programming. But once you understand how they work, they open up new dimensions of sed fu.
Note: This solution only helps you replace entire lines. Replacing substrings within lines may be possible using the hold buffer, but I can't imagine a way to do it.
(Another note: I'm doing this in FreeBSD, which uses a different sed from what you'll find in Linux. This may work in GNU sed, or it may not; I haven't tested.)
I am in agreement with sputnick. I don't believe that sed would be able to complete that task.
However, you could generate that file on the fly.
You could change the date to a fixed string, like
__DAYOFWEEK__.
Create a temp file, use sed to replace __DAYOFWEEK__ with $(date +%Y).
Then parse your file with sed -f $TEMPFILE.
sed is great, but it might be time to use something like perl that can generate the date on the fly.
To add a newline in the replacement expression using a sed file, what finally worked for me is escaping a literal newline. Example: to append a newline after the string NewLineHere, then this worked for me:
#! /usr/bin/sed -f
s/NewLineHere/NewLineHere\
/g
Not sure it matters but I am on Solaris unix, so not GNU sed for sure.
I have a csh script, my goal is to read an ini config file with perl module Config::Simple. I want to execute my perl command and assign the result to one variable.
perl -MConfig::Simple -e '$cfg = new Config::Simple("../config.ini"); $myvar = $cfg->param("myvar");'
What is the syntax ?
Receive the script's return value into a variable? I con't know the csh syntax, but in bash that is:
myvar=`perl ....`;
But if you wanted to set several variables, not sure.
For setting several variables you could have the perl script print csh syntax that the shell would evaluate.
I don't know csh but in bash it should be done like
#!/bin/sh
eval `perl -E 'say "FOO=123"; say "BAR=456"'`
echo "FOO is $FOO"
Command substitution in csh looks like this:
#!/bin/csh
set VAR=`perl -E 'say q(hello world)'`
echo ${VAR}
And, as an aside, I hope your using a descendent of csh like tcsh. The original csh implementation is a brain-dead mangled shell. This classic paper describes why.
I had a similar requirement, where I was using python instead of perl.
my python script was producing 3 output strings and the values were needed to be set in csh as variable.
Here's a solution.. hope it will help!
Inside Python:
I was using three variables : x , y and z I had only
one print statement in python which printed: x,y,z
Inside CSH:
set py_opt = `./my_python_script`
set csh_x = `echo $py_opt | sed 's/,/ /g' | awk '{print $1}'`
set csh_y = `echo $py_opt | sed 's/,/ /g' | awk '{print $2}'`
set csh_z = `echo $py_opt | sed 's/,/ /g' | awk '{print $3}'`
I need to replace several URLs in a text file with some content dependent on the URL itself. Let's say for simplicity it's the first line of the document at the URL.
What I'm trying is this:
sed "s/^URL=\(.*\)/TITLE=$(curl -s \1 | head -n 1)/" file.txt
This doesn't work, since \1 is not set. However, the shell is getting called. Can I somehow push the sed match variables to that subprocess?
The accept answer is just plain wrong. Proof:
Make an executable script foo.sh:
#! /bin/bash
echo $* 1>&2
Now run it:
$ echo foo | sed -e "s/\\(foo\\)/$(./foo.sh \\1)/"
\1
$
The $(...) is expanded before sed is run.
So you are trying to call an external command from inside the replacement pattern of a sed substitution. I dont' think it can be done, the $... inside a pattern just allows you to use an already existent (constant) shell variable.
I'd go with Perl, see the /e option in the search-replace operator (s/.../.../e).
UPDATE: I was wrong, sed plays nicely with the shell, and it allows you do to that. But, then, the backlash in \1 should be escaped. Try instead:
sed "s/^URL=\(.*\)/TITLE=$(curl -s \\1 | head -n 1)/" file.txt
Try this:
sed "s/^URL=\(.*\)/\1/" file.txt | while read url; do sed "s#URL=\($url\)#TITLE=$(curl -s $url | head -n 1)#" file.txt; done
If there are duplicate URLs in the original file, then there will be n^2 of them in the output. The # as a delimiter depends on the URLs not including that character.
Late reply, but making sure people don't get thrown off by the answers here -- this can be done in gnu sed using the e command. The following, for example, decrements a number at the beginning of a line:
echo "444 foo" | sed "s/\([0-9]*\)\(.*\)/expr \1 - 1 | tr -d '\n'; echo \"\2\";/e"
will produce:
443 foo
I'd like to use Sed to expand variables inside a file.
Suppose I exported a variable VARIABLE=something, and have a "test" file with the following:
I'd like to expand this: "${VARIABLE}"
I've been trying commands like the following, but to no avail:
cat test | sed -e "s/\(\${[A-Z]*}\)/`eval "echo '\1'"`/" > outputfile
The result is the "outputfile" with the variable still not expanded:
I'd like to expand this: "${VARIABLE}"
Still, running eval "echo '${VARIABLE}' in bash console results in the value "something" being echoed. Also, I tested and that pattern is trully being matched.
The desired output would be
I'd like to expand this: "something"
Can anyone shed a light on this?
Consider your trial version:
cat test | sed -e "s/\(\${[A-Z]*}\)/`eval "echo '\1'"`/" > outputfile
The reason this doesn't work is because it requires prescience on the part of the shell. The sed script is generated before any pattern is matched by sed, so the shell cannot do that job for you.
I've done this a couple of ways in the past. Normally, I've had a list of known variables and their values, and I've done the substitution from that list:
for var in PATH VARIABLE USERNAME
do
echo 's%${'"$var"'}%'$(eval echo "\$$var")'%g'
done > sed.script
cat test | sed -f sed.script > outputfile
If you want to map variables arbitrarily, then you either need to deal with the whole environment (instead of the fixed list of variable names, use the output from env, appropriately edited), or use Perl or Python instead.
Note that if the value of an environment variable contains a slash in your version, you'd run into problems using the slash as the field separator in the s/// notation. I used the '%' since relatively few environment variables use that - but there are some found on some machines that do contain '%' characters and so a complete solution is trickier. You also need to worry about backslashes in the value. You probably have to use something like '$(eval echo "\$$var" | sed 's/[\%]/\\&/g')' to escape the backslashes and percent symbols in the value of the environment variable. Final wrinkle: some versions of sed have (or had) a limited capacity for the script size - older versions of HP-UX had a limit of about 100. I'm not sure whether that is still an issue, but it was as recently as 5 years ago.
The simple-minded adaptation of the original script reads:
env |
sed 's/=.*//' |
while read var
do
echo 's%${'"$var"'}%'$(eval echo "\$$var" | sed 's/[\%]/\\&/g')'%g'
done > sed.script
cat test | sed -f sed.script > outputfile
However, a better solution uses the fact that you already have the values in the output from env, so we can write:
env |
sed 's/[\%]/\\&/g;s/\([^=]*\)=\(.*\)/s%${\1}%\2%/' > sed.script
cat test | sed -f sed.script > outputfile
This is altogether safer because the shell never evaluates anything that should not be evaluated - you have to be so careful with shell metacharacters in variable values. This version can only possibly run into any trouble if some output from env is malformed, I think.
Beware - writing sed scripts with sed is an esoteric occupation, but one that illustrates the power of good tools.
All these examples are remiss in not cleaning up the temporary file(s).
Maybe you can get by without using sed:
$ echo $VARIABLE
something
$ cat test
I'd like to expand this: ${VARIABLE}
$ eval "echo \"`cat test`\"" > outputfile
$ cat outputfile
I'd like to expand this: something
Let shell variable interpolation do the work.