Using sed to replace to a var with special characters - sed

I want to replace a placeholder on a file <<string>> in the example to the contents of a var that has several special characters.
file.txt
My string: <<string>>
script.sh
STRING="something-else;device=name.of.device;key=abcd1234/wtp="
sed -i "s/<<string>>/${STRING}/g" file.txt
I get this error:
sed: -e expression #1, char 165: unknown option to `s'
I already use this sed command for other vars that do not have special characters. Any way to escape the var ${STRING} entirely?

You can't do this job easily and robustly with sed, see Is it possible to escape regex metacharacters reliably with sed. Instead just use a tool like awk that understands literal strings:
$ string='~`!##$%^&*()-_+={[}]|\:;"'\''<,>.?/\1'
$ echo "$string"
~`!##$%^&*()-_+={[}]|\:;"'<,>.?/\1
$ string="$string" awk -i inplace 'match($0,/(.*)(<<string>>)(.*)/,a){ $0=a[1] ENVIRON["string"] a[3] } 1' file.txt
$ cat file.txt
My string: ~`!##$%^&*()-_+={[}]|\:;"'<,>.?/\1
That above will work for any characters (or backreference substrings like \1) that string might contain because it's simply using a literal string operation (concatenation) for the replacement.
It's using GNU awk for -i inplace just the same as your original script used GNU sed for -i.
Don't use all upper case for non-exported variable names by the way to avoid clashes with exported and built-in variables and not obfuscate your code by making it look like you're using exported variables, see Correct Bash and shell script variable capitalization.
Note that if you have multiple <<whatever>> placeholders you can easily parameterize the above, e.g.:
$ foo='Now is the Winter'
$ bar='Of our discontent'
$ cat file.txt
My foo string: <<foo>>
My bar string: <<bar>>
$ foo="$foo" bar="$bar" awk -i inplace 'match($0,/(.*)<<(\w+)>>(.*)/,a) && (a[2] in ENVIRON){ $0=a[1] ENVIRON[a[2]] a[3] } 1' file.txt
$ cat file.txt
My foo string: Now is the Winter
My bar string: Of our discontent
If you don't want to set foo and bar on the awk command line you can export them before it, or read them from a config file or a here-doc or ... - lots of options.

Since STRING contains a /, you should use a other delimiter, for example, you can use ^ like so:
sed 's^<<string>>^'"$STRING"'^g' file.txt
The quoting logic (''""'') is explained nicely on this SO answer.
Example on my locale machine:
$
$ cat file.txt
My string: <<string>>
$
$
$ STRING="something-else;device=name.of.device;key=abcd1234/wtp="
$
$
$ sed -i 's^<<string>>^'"$STRING"'^g' file.txt
$
$ cat file.txt
My string: something-else;device=name.of.device;key=abcd1234/wtp=
$
$

Related

sed equivalent of perl -pe

I'm looking for an equivalent of perl -pe. Ideally, it would be replace with sed if it's possible. Any help is highly appreciated.
The code is:
perl -pe 's/^\[([^\]]+)\].*$/$1/g'
$ echo '[foo] 123' | perl -pe 's/^\[([^\]]+)\].*$/$1/g'
foo
$ echo '[foo] 123' | sed -E 's/^\[([^]]+)\].*$/\1/'
foo
sed by default accepts code from command line, so -e isn't needed (though it can be used)
printing the pattern space is default, so -p isn't needed and sed -n is similar to perl -n
-E is used here to be as close as possible to Perl regex. sed supports BRE and ERE (not as feature rich as Perl) and even that differs from implementation to implementation.
with BRE, the command for this example would be: sed 's/^\[\([^]]*\)\].*$/\1/'
\ isn't special inside character class unless it is an escape sequence like \t, \x27 etc
backreferences use \N format (and limited to maximum 9)
Also note that g flag isn't needed in either case, as you are using line anchors

Use sed to replace complex line using environment variable

I have a file: foo
The file has a line:
JVMDATA="$(${TIMEOUT} sudo /usr/lib/jvm/jdk1.8.0_92/bin/java -jar ${JVMINSPECTOR} ${PID} 2>&1)"
I would like to replace /usr/lib/jvm/jdk1.8.0_92/bin/java with:
JAVA_HOME/bin/java
How do I do this?
One way to do it could be this:
sed -e 's|/usr/lib/jvm/jdk1.8.0_92/bin/java|JAVA_HOME/bin/java|' original.txt > new.txt
cp original.txt original.txt.sav
mv new.txt original.txt
This uses the sed pattern substitution s/old/new/.
Normally it is a pain to do with slashes because you need to escape them.
On my system sed tolerates vertical-bar pattern delimiters so that can make it more readable.
I suspect you're going to want to change it to $JAVA_HOME... (note the leading dollar sign).
Just for completeness, here's a version using the more common slash pattern delimiter:
sed -e 's/\/usr\/lib\/jvm\/jdk1.8.0_92\/bin\/java/JAVA_HOME\/bin\/java/' original.txt > new.txt

sed to copy part of line to end

I'm trying to copy part of a line to append to the end:
ftp://ftp.ncbi.nlm.nih.gov/genomes/all/GCA/900/169/985/GCA_900169985.1_IonXpress_024_genomic.fna.gz
becomes:
ftp://ftp.ncbi.nlm.nih.gov/genomes/all/GCA/900/169/985/GCA_900169985.1/GCA_900169985_IonXpress_024_genomic.fna.gz
I have tried:
sed 's/\(.*(GCA_\)\(.*\))/\1\2\2)'
$ f1=$'ftp://ftp.ncbi.nlm.nih.gov/genomes/all/GCA/900/169/985/GCA_900169985.1_IonXpress_024_genomic.fna.gz'
$ echo "$f1"
ftp://ftp.ncbi.nlm.nih.gov/genomes/all/GCA/900/169/985/GCA_900169985.1_IonXpress_024_genomic.fna.gz
$ sed -E 's/(.*)(GCA_.[^.]*)(.[^_]*)(.*)/\1\2\3\/\2\4/' <<<"$f1"
ftp://ftp.ncbi.nlm.nih.gov/genomes/all/GCA/900/169/985/GCA_900169985.1/GCA_900169985_IonXpress_024_genomic.fna.gz
sed -E (or -r in some systems) enables extended regex support in sed , so you don't need to escape the group parenthesis ( ).
The format (GCA_.[^.]*) equals to "get from GCA_ all chars up and excluding the first found dot" :
$ sed -E 's/(.*)(GCA_.[^.]*)(.[^_]*)(.*)/\2/' <<<"$f1"
GCA_900169985
Similarly (.[^_]*) means get all chars up to first found _ (excluding _ char). This is the regex way to perform a non greedy/lazy capture (in perl regex this would have been written something like as .*_?)
$ sed -E 's/(.*)(GCA_.[^.]*)(.[^_]*)(.*)/\3/' <<<"$f1"
.1
Short sed approach:
s="ftp://ftp.ncbi.nlm.nih.gov/genomes/all/GCA/900/169/985/GCA_900169985.1_IonXpress_024_genomic.fna.gz"
sed -E 's/(GCA_[^._]+)\.([^_]+)/\1.\2\/\1/' <<< "$s"
The output:
ftp://ftp.ncbi.nlm.nih.gov/genomes/all/GCA/900/169/985/GCA_900169985.1/GCA_900169985_IonXpress_024_genomic.fna.gz

search a string which contains "/" and replace using sed

How to search a pattern and remove the line using sed which contains special characters like "ranasnfs2:/SA_kits/prod"
I tried using a variable to hold the complete string and then recall the variable in sed command but it is not working.
echo $a
ranasnfs2:/SA_kits/prod
sed -i '/"$a"/d' test.txt
cat test.txt | grep -i SA
/SA_kits -rw,suid,soft,retry=4 ranasnfs2:/SA_kits/prod
You need to escape the slash character.
Use this for deleting lines which contain a /:
sed '/\//d' file

How to search in sed for any name matching

How to find structures matching a pattern
struct struct_name {
....
....
};
I'm using
sed -n -e '/struct{/,/}/p'
how to search for any struct_name
To extract all struct definitions (POSIX-compliant command):
sed -n '/struct [^ {]\{1,\} {/,/}/p' file
More robust with respect to whitespace variations (POSIX-compliant):
sed -n '/struct[[:blank:]]\{1,\}[^ {]\{1,\}[[:blank:]]*{/,/}/p' file
Alternative, using an extended regular expression (works with both GNU and BSD/macOS sed):
sed -E -n '/struct[[:blank:]]+[^ {]+[[:blank:]]*\{/,/\}/p' file
awk alternative (awk only uses extended regexes):
awk '/struct[[:blank:]]+[^ {]+[[:blank:]]*\{/,/\}/' file
The awk solution has the added advantage that a given struct definition will also be extracted correctly if it is all on a single line: awk looks for the end of a range on the same input line as the start of the range, whereas sed does not.
To extract a specific struct definition by name:
sed doesn't support variables, so your best bet is to splice in a shell variable that the shell expands up front.
name='struct_name' # define name to search for as shell var.
sed -n '/struct '"$name"' {/,/}/p' file # splice shell var. into sed script
Note that I've deliberately not used sed -n "/struct $name {/,/}/p" - a single, double-quoted string expanded by the shell as a whole - so as to make it clear which part of the sed script is expanded by the shell up front.
This works in this simple case, but is tricky business in general, because you must ensure that the expanded variable value contains no regex/sed metacharacters that break the command.
Here's an awk alternative that uses awk variables and literal substring matching to bypass the problem of potentially having to escape the variable value:
awk -v name='struct_name' 'index($0, "struct " name " {"),/}/' file
This solution has the added advantage that the struct definition will also be extracted correctly if it is all on a single line: awk looks for the end of a range on the same input line as the start of the range, whereas sed does not.
This will search a text file for struct_name. You can use the -E switch to use a regular expression.
grep -no struct_name test.txt
The -n switch causes the line number to be included, the -o means only the matching element of the line will be displayed.