sed - Multiple replacements in single command - sed

EDIT
My use case does require substitution of a URL pattern.
thanks #glenn jackman for response but there may be a problem escaping the // characters in the URL. That was why I was using | for the delimiter.
sh-4.4$ EC2_ID="http://test.com"
sh-4.4$ sed -E '
s|\\$\{instance-hostname\}|\ ${EC2_HOSTNAME}|
s|\\$\{instance-id\}|\ ${EC2_ID}|
' test.txt
AWS EC2 Instance DNS: ${instance-hostname}
AWS EC2 Instance ID: ${instance-id}
sh-4.4$ sed -E '
s/\$\{instance-hostname\}/'"${EC2_HOSTNAME}"'/
s/\$\{instance-id\}/'"${EC2_ID}"'/
' test.txt
sed: -e expression #1, char 65: unknown option to `s'
============================================
I would appreciate assistance with my sed script.
sh-4.4$ EC2_HOSTNAME="A"
sh-4.4$ EC2_ID="B"
sh-4.4$ sed -r "s|\\$\{instance-hostname\}|\ ${EC2_HOSTNAME}|" -r "s|\\$\{instance-id\}|\ ${EC2_ID}|" test.txt
sed: can't read s|\$\{instance-id\}|\ B|: No such file or directory
AWS EC2 Instance DNS: A
AWS EC2 Instance ID: ${instance-id}
sh-4.4$ cat test.txt
AWS EC2 Instance DNS: ${instance-hostname}
AWS EC2 Instance ID: ${instance-id}
sh-4.4$
Substitution works if only one replacement instruction is provided - on either replacement token.
sh-4.4$ sed -r "s|\\$\{instance-hostname\}|\ ${EC2_HOSTNAME}|" test.txt
AWS EC2 Instance DNS: A
AWS EC2 Instance ID: ${instance-id}
sh-4.4$ sed -r "s|\\$\{instance-id\}|\ ${EC2_ID}|" test.txt
AWS EC2 Instance DNS: ${instance-hostname}
AWS EC2 Instance ID: B
sh-4.4$
FYI
sh-4.4$ cat /etc/os-release
NAME="Red Hat Enterprise Linux"
VERSION="8.5 (Ootpa)"
ID="rhel"
ID_LIKE="fedora"
VERSION_ID="8.5"
sed version
sh-4.4$ sed --version
sed (GNU sed) 4.5
Copyright (C) 2018 Free Software Foundation, Inc.
Thanks

The sed body can contain newlines, so this is a readable way to write it:
sed -E '
s/\$\{instance-hostname\}/'"${EC2_HOSTNAME}"'/
s/\$\{instance-id\}/'"${EC2_ID}"'/
' test.txt
AWS EC2 Instance DNS: A
AWS EC2 Instance ID: B
If the shell variables contain the / character, sed will throw errors. There are (ugly) workarounds if this is an issue.
perl allows a more dynamic approach, assuming the "instance_*" and "EC2_*" variable names align
export EC2_HOSTNAME EC2_ID
perl -pe 's<\$\{instance-(\w+)\}>< $key = "EC2_" . uc($1); $ENV{$key} >e' test.txt

You really don't want to use sed for this. If the original author of test.txt had used reasonable names, this would be trivial. Fortunately, your current input is easily fixable:
$ cat test.txt
AWS EC2 Instance DNS: ${instance-hostname}
AWS EC2 Instance ID: ${instance-id}
$ tr - _ < test.txt | { instance_hostname=A instance_id=B envsubst; }
AWS EC2 Instance DNS: A
AWS EC2 Instance ID: B

Related

Airflow bash operator: Problem using sed in hdfs

I have a airflow task that I am trying to use sed command for replacing LF with CRLF:
hdfs dfs -cat /test/file.txt | sed 's/$/\r/g' | hdfs dfs -put -f - /test/file.txt
I get following error:
error: sed: -e expression #1, char 4: unterminated `s' command
I think it is due to \r which it is conflicting with. How do I solve this problem?
I found the reason, the \ is a special character in Python.
To solved it I just added an extra \ is it becomes sed 's/$/\\r/g' , another option is to use prefixing.

Replace value in single quotes using sed

I already know that sed uses own approach to deal with single quote but I think it still possible to use it in my automation script.
I had to replace value of fingerprint in Saltstack config file.
Current value:
#master_finger: ''
Target value
master_finger: 'some:value'
My current command which doesn't work:
$ sed -i 's/#master_finger: ''/master_finger: 'some:value'/g' /etc/salt/minion
returns:
master_finger: some:value''
How can I solve this?
just use the double quotes to enclose the script.
$ echo "#master_finger: ''" | sed "s/#master_finger: ''/master_finger: 'some:value'/"
master_finger: 'some:value'
It's not sed that's making handling of 's difficult, it's the shell because the shell does not allow 's within any '-quoted string, including scripts.
You could save the sed script in a file and run it with -f or use a here document:
$ sed -f- file <<'EOF'
s/#master_finger: ''/master_finger: 'some:value'/g
EOF
master_finger: 'some:value'
To see the difference between the above and #karakfas suggestion:
$ sed -f- file <<'EOF'
s/#master_finger: ''/master_finger: '$(date)'/g
EOF
master_finger: '$(date)'
$ sed "s/#master_finger: ''/master_finger: '$(date)'/" file
master_finger: 'Sun Feb 14 06:50:43 CST 2021'
and imagine if date was replace by rm -rf * or something worse.
Also consider:
$ sed 's/#master_finger: '\'\''/master_finger: '\''$(date)'\''/' file
master_finger: '$(date)'

Using sed with command-substitution text in the replacement part is not working

I have a CFN with cfn-init to deploy a Apache web server with specified virtual hosts. In the template, I use a AWS::CloudFormation::Init configset to replace local IPs with the Instance's private IP.
config:
packages:...
files:...
services:...
commands:
replacePrivateIP:
cwd: "/etc/httpd/conf"
command: !Sub |
sed -i 's#127.0.0.1#$(curl -s http://169.254.169.254/latest/meta-data/local-ipv4)#g' httpd.conf
The sed command works fine outside the CFN template. But in the CFN-init process, it simply replace "127.0.0.1" with the whole $(curl -s http://...) string.
How can I feed the instance private IP correctly into the httpd.conf file through cfn-init?
The command-substitution syntax $(..) does not work when wrapped in single quotes which is as expected in bash or most other shells as they preserve the literal value present inside. For your substitution to happen, put it inside double-quotes as
sed -i 's#127.0.0.1#'"$(curl -s http://169.254.169.254/latest/meta-data/local-ipv4)"'#g' httpd.conf
Compare the outputs of echo '$(date)' and echo "$(date)" for a simple example of your case.

using sed to replace string with special characters

I'm basically trying to modify tomcat server.xml connector tag and add a address attribute to it.
I want to find the below string in server.xml
I'm doing the below with sed,
export currlistener=\<Connector\ port\=\"18443\"
export newlistener=\<Connector\ port\=\"18443\"\ address\=\"127.0.0.1\"\
echo $currlistener
echo $newlistener
sed -i -e 's/'$currlistener'/'$newlistener'/g' server.xml
But I get the error
sed: -e expression #1, char 12: unterminated `s' command
I guess sed is interpreting the special characters and erroring out.
How would I do the same using awk?
Regards,
Anand.
Using sed
The problem was that the shell variables were unquoted. Try:
sed -i -e "s/$currlistener/$newlistener/g" server.xml
Using awk
The sed solution requires that you trust the source of your shell variables. For a case like this, awk is safer. Using a modern GNU awk:
awk -i inplace -v a="$currlistener" -v b="$newlistener" '{gsub(a, b)} 1' server.xml
Or, using other awk:
awk -v a="$currlistener" -v b="$newlistener" '{gsub(a, b)} 1' server.xml >tmp && mv tmp server.sml
Simplifying the variable assignments
Separately, the shell variables can be defined without requiring so many escapes:
currlistener='<Connector port="18443"'
newlistener='<Connector port="18443" address="127.0.0.1"'
It is only necessary to export them if they are to be used in a child process.

passing variable to sed command

i need to pass a variable value to sed command. here is the description, what i am doing.
i am assigning a value to variable :
export ALSIZE=14420
$ echo $ALSIZE
14420
now i am using that value in sed command to read file from $ALSIZE line to end of the file. and i got the error
$ sed -n '$ALSIZE,$p' /db1/u04/oradata/GG11/ggserr.log
Unrecognized command: $ALSIZE,$p ====== >>
used the variable value in "" (double quotes) still got error.
$ sed -n '"$ALSIZE",$p' /db1/u04/oradata/GG11/ggserr.log
Unrecognized command: "$ALSIZE",$p ===== >>>
i am getting response back
$ sed -n '14420,$p' /db1/u04/oradata/GG11/ggserr.log
2013-12-26 06:36:17 INFO OGG-01026 Oracle GoldenGate Capture for Oracle, dpsbprd.prm: Rolling over remote file ./dirdat/siebel/r1003911.
2013-12-26 06:43:31 INFO OGG-01026 Oracle GoldenGate Capture for Oracle, dpsbprd.prm: Rolling over remote file ./dirdat/siebel/r1003912.
2013-12-26 11:07:47 INFO OGG-01026 Oracle GoldenGate Capture for Oracle, dpsbprd.prm: Rolling over remote file ./dirdat/siebel/r1003913.
what is the mistake i am doing. could you please advice ?
All of these would work:
sed -n $ALSIZE',$p' /db1/u04/oradata/GG11/ggserr.log
sed -n "$ALSIZE"',$p' /db1/u04/oradata/GG11/ggserr.log
sed -n "$ALSIZE,\$p" /db1/u04/oradata/GG11/ggserr.log
The key is understanding how quoting works:
You can quote part of a string or all of it: hello, hel'lo', 'hello', 'hel''lo' are all the same to the shell
Shell variables within double quotes are expanded to their value: "$ALSIZE" is expanded to "14420", which doesn't need to be quoted, so you could write simply $ALSIZE without quoting
Shell variables within single quotes are not expanded, so '$ALSIZE' will be used a the literal text "$ALSIZE" which is not what you want
In the "$ALSIZE,\$p" example the second $ has to be escaped to prevent the shell from expanding the $p variable, which probably has no value
The single quotes means that the '$' in $ALSIZE is expanded as a literal "\$ALSIZE". What happens if you rewrite it to:
sed -n $ALSIZE',$p' /db1/u04/oradata/GG11/ggserr.log
(leaving the single quotes until after the $ALSIZE variable)?
use double quotes:
kent$ s=5
kent$ seq 10 |sed -n "$s,$ p"
5
6
7
8
9
10