I already know that sed uses own approach to deal with single quote but I think it still possible to use it in my automation script.
I had to replace value of fingerprint in Saltstack config file.
Current value:
#master_finger: ''
Target value
master_finger: 'some:value'
My current command which doesn't work:
$ sed -i 's/#master_finger: ''/master_finger: 'some:value'/g' /etc/salt/minion
returns:
master_finger: some:value''
How can I solve this?
just use the double quotes to enclose the script.
$ echo "#master_finger: ''" | sed "s/#master_finger: ''/master_finger: 'some:value'/"
master_finger: 'some:value'
It's not sed that's making handling of 's difficult, it's the shell because the shell does not allow 's within any '-quoted string, including scripts.
You could save the sed script in a file and run it with -f or use a here document:
$ sed -f- file <<'EOF'
s/#master_finger: ''/master_finger: 'some:value'/g
EOF
master_finger: 'some:value'
To see the difference between the above and #karakfas suggestion:
$ sed -f- file <<'EOF'
s/#master_finger: ''/master_finger: '$(date)'/g
EOF
master_finger: '$(date)'
$ sed "s/#master_finger: ''/master_finger: '$(date)'/" file
master_finger: 'Sun Feb 14 06:50:43 CST 2021'
and imagine if date was replace by rm -rf * or something worse.
Also consider:
$ sed 's/#master_finger: '\'\''/master_finger: '\''$(date)'\''/' file
master_finger: '$(date)'
Related
I want to replace a placeholder on a file <<string>> in the example to the contents of a var that has several special characters.
file.txt
My string: <<string>>
script.sh
STRING="something-else;device=name.of.device;key=abcd1234/wtp="
sed -i "s/<<string>>/${STRING}/g" file.txt
I get this error:
sed: -e expression #1, char 165: unknown option to `s'
I already use this sed command for other vars that do not have special characters. Any way to escape the var ${STRING} entirely?
You can't do this job easily and robustly with sed, see Is it possible to escape regex metacharacters reliably with sed. Instead just use a tool like awk that understands literal strings:
$ string='~`!##$%^&*()-_+={[}]|\:;"'\''<,>.?/\1'
$ echo "$string"
~`!##$%^&*()-_+={[}]|\:;"'<,>.?/\1
$ string="$string" awk -i inplace 'match($0,/(.*)(<<string>>)(.*)/,a){ $0=a[1] ENVIRON["string"] a[3] } 1' file.txt
$ cat file.txt
My string: ~`!##$%^&*()-_+={[}]|\:;"'<,>.?/\1
That above will work for any characters (or backreference substrings like \1) that string might contain because it's simply using a literal string operation (concatenation) for the replacement.
It's using GNU awk for -i inplace just the same as your original script used GNU sed for -i.
Don't use all upper case for non-exported variable names by the way to avoid clashes with exported and built-in variables and not obfuscate your code by making it look like you're using exported variables, see Correct Bash and shell script variable capitalization.
Note that if you have multiple <<whatever>> placeholders you can easily parameterize the above, e.g.:
$ foo='Now is the Winter'
$ bar='Of our discontent'
$ cat file.txt
My foo string: <<foo>>
My bar string: <<bar>>
$ foo="$foo" bar="$bar" awk -i inplace 'match($0,/(.*)<<(\w+)>>(.*)/,a) && (a[2] in ENVIRON){ $0=a[1] ENVIRON[a[2]] a[3] } 1' file.txt
$ cat file.txt
My foo string: Now is the Winter
My bar string: Of our discontent
If you don't want to set foo and bar on the awk command line you can export them before it, or read them from a config file or a here-doc or ... - lots of options.
Since STRING contains a /, you should use a other delimiter, for example, you can use ^ like so:
sed 's^<<string>>^'"$STRING"'^g' file.txt
The quoting logic (''""'') is explained nicely on this SO answer.
Example on my locale machine:
$
$ cat file.txt
My string: <<string>>
$
$
$ STRING="something-else;device=name.of.device;key=abcd1234/wtp="
$
$
$ sed -i 's^<<string>>^'"$STRING"'^g' file.txt
$
$ cat file.txt
My string: something-else;device=name.of.device;key=abcd1234/wtp=
$
$
I have the following sed command:
sed -i -E "s/\{\{(.*)\}\}/$(echo "$\1")/g" test.conf
In test.conf, I have this:
this is a {{TEST}}
and this is an {{ANSWER}} here.
And I have the follow environment variables set:
export TEST=1234
export ANSWER=5678
When I run the sed command, I end up with this result:
this is a $TEST
and this is an $ANSWER here.
I want 1234 and 5678 there respectively. Is there a reason the echo command is interpreting things literally?
Backreferences are used internally by a single sed command. The echo has no idea about sed backreferences and would have been invoked by the shell before the sed command has even run so the $(echo "$\1") is outputing $\1 so
sed -i -E "s/\{\{(.*)\}\}/$(echo "$\1")/g" test.conf
is really:
sed -i -E "s/\{\{(.*)\}\}/$\1/g" test.conf
hence the output you are seeing.
Anyway, sed is for simple subsitutions on individual lines, for anything else you should be using awk:
$ export TEST=1234 ANSWER=5678
$ awk 'match($0,/(.*)\{\{(.*)\}\}(.*)/,a){$0=a[1] ENVIRON[a[2]] a[3]} 1' file
this is a 1234
and this is an 5678 here.
The above uses GNU awk for the 3rd arg to match(), with other awks it'd be:
$ awk 'match($0,/\{\{(.*)\}\}/){$0=substr($0,1,RSTART-1) ENVIRON[substr($0,RSTART+2,RLENGTH-4)] substr($0,RSTART+RLENGTH)} 1' file
this is a 1234
and this is an 5678 here.
If anyone suggests running eval or similar on the sed output - don't do it (google eval is evil and friends), just use the awk command above for simple string operations.
You can use perl which conveniently has a %ENV hash variable, see perldoc for more info
perl -pe 's/\{\{(.*)\}\}/$ENV{$1}/' test.conf
I'm requesting help with a very simple script...
#!/usr/bin/sed -f
sed '/11,yahoo/d'
sed '/2506,stackover flow/d'
sed '/2536,reddit/d'
Just need it to remove three matches that account for 18408 in my file, data.csv
% sed -f remove.sed < data.csv
sed: 3: remove.sed: unterminated substitute pattern
Doing these same lines individually is no problem at all, so what am I doing wrong with this?
Using freeBSD 10.1 and its implementation of sed, if that matters.
This, being a sed script, should not have "sed" at each line.
Either change it to:
#!/usr/bin/sed -f
/11,yahoo/d
/2506,stackover flow/d
/2536,reddit/d
Or to
#!/bin/sh
sed -e /11,yahoo/d \
-e /2506,stackover flow/d \
-e /2536,reddit/d
I'm using Sed on a Mac. I am trying to do a simple string replace on a file that is not in the directory. I do:
sed -i 's/old/new/' /Users/A/file
and it says invalid command code A.
What do I need to do?
The -i option in OSX/BSD sed is a little different than the GNU/Linux version in that it requires a backup extension to be given, even if it's an empty string (which means that no backup will be made). The "invalid command code" error message occurs because s/old/new is taken as the backup extension and /Users/A/file is taken as the script (where A is seen as an invalid command name). So it needs to be something like:
sed -i '' 's/old/new/' /Users/A/file
if you have perl:
perl -p -i -e 's/old/new/' /Users/A/file
The -i in sed is not a standard across OS's (not a POSIX standard). This should work every time:
cp /Users/A/file /Users/A/file.sed
cat /Users/A/file.sed | sed 's/old/new' > /Users/A/file
I have a variables that needs to be modified so that minutes %M is added. I know the sed command and it is working as expected.
# cat mysed.txt
myfirstfile="comany$mydb`date +'%d-%b-%Y-%H'`.sql"
# sed -i 's/\%H/\%H-\%M/' mysed.txt
# cat mysed.txt
myfirstfile="company$mydb`date +'%d-%b-%Y-%H-%M'`.sql"
But if I run the same sed command again, it will add %M again as follows.
# sed -i 's/\%H/\%H-\%M/' mysed.txt
# cat mysed.txt
myfirstfile="company$mydb`date +'%d-%b-%Y-%H-%M-%M'`.sql"
I need a sed command that should add the minutes %M only once even if sed command is executed twice (by mistake)
# sed -i "s/%H'/%H-%M'/" mysed.txt
This should work. This way it will only do the replacement if there is a quote mark next to the %H.
Test if it is present before substituting:
sed '/%H-%M/! s/%H/%H-%M/' mysed.txt
Btw. % does not need to be escaped.
I believe below command will work fine.Please replace test.dat file with your file name.
cat test.dat|sed "s/\%H\'/\%H\-\%M\'/" > test.dat
Without cat command
sed -i "s/\%H\'/\%H\-\%M\'/" test.dat > test.dat
Cheers!