I've been looking for a way to find-and-replace strings in all fortran files in my current directory. Most answers on here are along the lines of using:
sed -i 's/INCLUDE \'atm/params.inc\'/USE params/g' *.f
or
perl -pi -w -e 's/INCLUDE \'atm/params.inc\'/USE params/g' *.f
However, when I use either of these the bash line continuation > pops up on the next line as if it's expecting input or another argument. I haven't seen anyone else encounter this and I am not sure what to do with it. Are my commands incorrect; am I missing something?
There are two problems with the original:
You weren't protecting your / in the literal data from being parsed by sed rather than treated as data. One very readable and explicit way to do this is with [/].
You were trying to use \' to put a literal ' in a single-quoted string. That doesn't work. The common idiom is '"'"', which, character-by-character, does the following:
' - exits the original single-quoted context
" - opens a double-quoted context
' - adds a literal single-quote (protected by the surrounding double quotes)
" - ends that double-quoted context
' - resumes the outer single-quoted context.
Thus, consider:
# note: this works with GNU sed, not MacOS sed or others
sed -i 's/INCLUDE '"'"'atm[/]params.inc'"'"'/USE params/g' *.f
Related
The following command fails.
sed 's/user=\'mysql\'/user=`whoami`/g' input_file
An example input_file contains the following line
user='mysql'
The corresponding expected output is
user=`whoami`
(Yes, I literally want whoami between backticks, I don't want it to expand my userid.)
This should be what you need:
Using double quotes to enclose the sed command,
so that you are free to use single quotes in it;
escape backticks to avoid the expansion.
sed "s/user='mysql'/user=\`whoami\`/g" yourfile
I've intentionally omitted the -i option for the simple reason that it is not part of the issue.
To clarify the relation between single quotes and escaping, compare the following two commands
echo 'I didn\'t know'
echo 'I didn'\''t know'
The former will wait for further input as there's an open ', whereas the latter will work fine, as you are concatenating a single quoted string ('I didn'), an escaped single quote (\'), and another single quoted string ('t know').
I have one file named `config_3_setConfigPW.ldif? containing the following line:
{pass}
on terminal, I used following commands
SLAPPASSWD=Pwd&0011
sed -i "s#{pass}#$SLAPPASSWD#" config_3_setConfigPW.ldif
It should replace {pass} to Pwd&0011 but it generates Pwd{pass}0011.
The reason is that the SLAPPASSWD shell variable is expanded before sed sees it. So sed sees:
sed -i "s#{pass}#Pwd&0011#" config_3_setConfigPW.ldif
When an "&" is on the right hand side of a pattern it means "copy the matched input", and in your case the matched input is "{pass}".
The real problem is that you would have to escape all the special characters that might arise in SLAPPASSWD, to prevent sed doing this. For example, if you had character "#" in the password, sed would think it was the end of the substitute command, and give a syntax error.
Because of this, I wouldn't use sed for this. You could try gawk or perl?
eg, this will print out the modified file in awk (though it still assumes that SLAPPASSWD contains no " character
awk -F \{pass\} ' { print $1"'${SLAPPASSWD}'"$2 } ' config_3_setConfigPW.ldif
That's because$SLAPPASSWD contains the character sequences & which is a metacharacter used by sed and evaluates to the matched text in the s command. Meaning:
sed 's/{pass}/match: &/' <<< '{pass}'
would give you:
match: {pass}
A time ago I've asked this question: "Is it possible to escape regex metacharacters reliably with sed". Answers there show how to reliably escape the password before using it as the replacement part:
pwd="Pwd&0011"
pwdEscaped="$(sed 's/[&/\]/\\&/g' <<< "$pwd")"
# Now you can safely pass $pwd to sed
sed -i "s/{pass}/$pwdEscaped/" config_3_setConfigPW.ldif
Bear in mind that sed NEVER operates on strings. The thing sed searches for is a regexp and the thing it replaces it with is string-like but has some metacharacters you need to be aware of, e.g. & or \<number>, and all of it needs to avoid using the sed delimiters, / typically.
If you want to operate on strings you need to use awk:
awk -v old="{pass}" -v new="$SLAPPASSWD" 's=index($0,old){ $0 = substr($0,1,s-1) new substr($0,s+length(old))} 1' file
Even the above would need tweaked if old or new contained escape characters.
I am using sed to replace 14 different abbreviations like CA_23456, CB_scaffold34532,... with 'proper' names in a file and it works putting it all on one line.
acc=$1
sed -e 's/CA_[A-Z]*[a-z]*[0-9]*/Hesperocyparis_arizonica/;s/CB_[A-Z]*[a-z]*[0-9]*/Hesperocyparis_bakeri/;s/CM_[A-Z]*[a-z]*[0-9]*/Hesperocyparis_macrocarpa/;s/CS_[A-Z]*[a-z]*[0-9]*/Cupressus_sempervirens/;s/CT_[A-Z]*[a-z]*[0-9]*/Cupressus_torulosa/;s/JD_[A-Z]*[a-z]*[0-9]*/Juniperus_drupacea/;s/JF_[A-Z]*[a-z]*[0-9]*/Juniperus_flaccida/;s/JI_[A-Z]*[a-z]*[0-9]*/Juniperus_indica/;s/JP_[A-Z]*[a-z]*[0-9]*/Juniperus_phoenicea/;s/JX_[A-Z]*[a-z]*[0-9]*/Juniperus_procera/;s/JS_[A-Z]*[a-z]*[0-9]*/Juniperus_scopulorum/;s/MD_[A-Z]*[a-z]*[0-9]*/Microbiota_decussata/;s/XN_[A-Z]*[a-z]*[0-9]*/Xanthocyparis_nootkatensis/;s/XV_[A-Z]*[a-z]*[0-9]*/Xanthocyparis_vietnamensis/' ${acc}.nex > ${acc}_replaced.nex
To make it more readable I'd like to have the command split over multiple lines using '\' (not all the replacements are shown for brevity)
acc=$1
sed -e 's/CA_[A-Z]*[a-z]*[0-9]*/Hesperocyparis_arizonica/;\
s/CB_[A-Z]*[a-z]*[0-9]*/Hesperocyparis_bakeri/;\
s/CM_[A-Z]*[a-z]*[0-9]*/Hesperocyparis_macrocarpa/'\
${acc}.nex > ${acc}_replaced.nex
However, I get an error message: sed: -e expression #1, char 168: unterminated address regex. I have looked at the answers to similar problems on various webforums and tried various things (using 's/.../.../' on every line, leaving ';' out,....) but I can't get it to work. What am I doing wrong?
Drop the \ that escapes the newlines. (They are not actually doing it!, they are interpreted as wrong syntax by sed). However I would suggest to put it into a file and run it like this:
sed -f script.sed input
where script.sed looks like this:
s/CA_[A-Z]*[a-z]*[0-9]*/Hesperocyparis_arizonica/
s/CB_[A-Z]*[a-z]*[0-9]*/Hesperocyparis_bakeri/
s/CM_[A-Z]*[a-z]*[0-9]*/Hesperocyparis_macrocarpa/
Remove the backslashes from the sed code.
Inside singly-quoted shell strings, backslashes are not needed to escape newlines and are not removed because they are not parsed as escape characters. This has the effect that sed sees them as part of its code, and it then expects to find an address regex with a different delimiter than / before the command ends at the next newline (similar to \,/home/, !d). This address regex does not appear (nor an associated command), and so sed complains about invalid code.
Apart from that: The semicolons in the sed code are no longer necessary when you terminate commands with newlines, and anything involving shell variables should be quoted to avoid splitting in case of whitespace.
In sum:
sed -e 's/CA_[A-Z]*[a-z]*[0-9]*/Hesperocyparis_arizonica/
s/CB_[A-Z]*[a-z]*[0-9]*/Hesperocyparis_bakeri/
s/CM_[A-Z]*[a-z]*[0-9]*/Hesperocyparis_macrocarpa/' \
"${acc}.nex" > "${acc}_replaced.nex"
This is the command working for me:
sed "50,99999{/^\s*printf/d}" the_file
So this command delete all the lines between 50 and 99999 which have "printf" in it and there is only whitespace before printf at the line.
Now my questions are:
how to replace 99999 with some meta symbol to indicate the real line number
I tried sed "50,${/^\s*PUTS/d}" the_file, but it is not right.
how to replace "printf" with an environment variable? I tried
set pattern printf
sed "50,99999{/^\s*$pattern/d}" the_file
but it is not right.
Assuming a Bourne-like shell such as bash:
Simply define shell variables and splice them into your sed command string:
endLine=99999
pattern='printf'
sed '50,'"$endLine"'{ /^\s*'"$pattern"'/d; }' the_file
Note that the static parts of the sed command strings are single-quoted, as that protects them from interpretation by the shell (which means you needn't quote $ and `, for instance).
You can put everything into a single double-quoted string so as to be able to embed variable references directly, but distinguishing between what the shell will interpret up front and what sed will interpret can get confusing quickly.
That said, using a double-quoted string for the case at hand is simple:
sed "50,$endLine { /^\s*$pattern/d; }" the_file
sed "50,${/^\s*PUTS/d}" the_file
this line won't work, because you used double quotes, and you need escape the dollar: \$ or use single quote: '50,${/.../d}' file
sed "50,99999{/^\s*$pattern/d}" file
this line should work.
EDIT
wait, I just noticed that you set env var via set... this is not correct if you were with Bash. you should use export PAT="PUT" in your script.
check #Jonathan and #tripleee 's comments
I am using the Unix sed command on a string that can contain all types of characters (&, |, !, /, ?, etc).
Is there a complex delimiter (with two characters?) that can fix the error:
sed: -e expression #1, char 22: unknown option to `s'
The characters in the input file are of no concern - sed parses them fine. There may be an issue, however, if you have most of the common characters in your pattern - or if your pattern may not be known beforehand.
At least on GNU sed, you can use a non-printable character that is highly improbable to exist in your pattern as a delimiter. For example, if your shell is Bash:
$ echo '|||' | sed s$'\001''|'$'\001''/'$'\001''g'
In this example, Bash replaces $'\001' with the character that has the octal value 001 - in ASCII it's the SOH character (start of heading).
Since such characters are control/non-printable characters, it's doubtful that they will exist in the pattern. Unless, that is, you are doing something weird like modifying binary files - or Unicode files without the proper locale settings.
Another way to do this is to use Shell Parameter Substitution.
${parameter/pattern/replace} # substitute replace for pattern once
or
${parameter//pattern/replace} # substitute replace for pattern everywhere
Here is a quite complex example that is difficult with sed:
$ parameter="Common sed delimiters: [sed-del]"
$ pattern="\[sed-del\]"
$ replace="[/_%:\\#]"
$ echo "${parameter//$pattern/replace}"
result is:
Common sed delimiters: [/_%:\#]
However: This only work with bash parameters and not files where sed excel.
There is no such option for multi-character expression delimiters in sed, but I doubt
you need that. The delimiter character should not occur in the pattern, but if it appears in the string being processed, it's not a problem. And unless you're doing something extremely weird, there will always be some character that doesn't appear in your search pattern that can serve as a delimiter.
You need the nested delimiter facility that Perl offers. That allows to use stuff like matching, substituting, and transliterating without worrying about the delimiter being included in your contents. Since perl is a superset of sed, you should be able to use it for whatever you’re used sed for.
Consider this:
$ perl -nle 'print if /something/' inputs
Now if your something contains a slash, you have a problem. The way to fix this is to change delimiter, preferably to a bracketing one. So for example, you could having anything you like in the $WHATEVER shell variable (provided the backets are balanced), which gets interpolated by the shell before Perl is even called here:
$ perl -nle "print if m($WHATEVER)" /usr/share/dict/words
That works even if you have correctly nested parens in $WHATEVER. The four bracketing pairs which correctly nest like this in Perl are < >, ( ), [ ], and { }. They allow arbitrary contents that include the delimiter if that delimiter is balanced.
If it is not balanced, then do not use a delimiter at all. If the pattern is in a Perl variable, you don’t need to use the match operator provided you use the =~ operator, so:
$whatever = "some arbitrary string ( / # [ etc";
if ($line =~ $whatever) { ... }
With the help of Jim Lewis, I finally did a test before using sed :
if [ `echo $1 | grep '|'` ]; then
grep ".*$1.*:" $DB_FILE | sed "s#^.*$1*.*\(:\)## "
else
grep ".*$1.*:" $DB_FILE | sed "s|^.*$1*.*\(:\)|| "
fi
Thanks for help
Wow. I totally did not know that you could use any character as a delimiter.
At least half the time I use the sed and BREs its on paths, code snippets, junk characters, things like that. I end up with a bunch of horribly unreadable escapes which I'm not even sure won't die on some combination I didn't think of. But if you can exclude just some character class (or just one character even)
echo '#01Y $#1+!' | sed -e 'sa$#1+ashita' -e 'su#01YuHolyug'
> > > Holy shit!
That's so much easier.
Escaping the delimiter inline for BASH to parse is cumbersome and difficult to read (although the delimiter does need escaping for sed's benefit when it's first used, per-expression).
To pull together thkala's answer and user4401178's comment:
DELIM=$(echo -en "\001");
sed -n "\\${DELIM}${STARTING_SEARCH_TERM}${DELIM},\\${DELIM}${ENDING_SEARCH_TERM}${DELIM}p" "${FILE}"
This example returns all results starting from ${STARTING_SEARCH_TERM} until ${ENDING_SEARCH_TERM} that don't match the SOH (start of heading) character with ASCII code 001.
There's no universal separator, but it can be escaped by a backslash for sed to not treat it like separator (at least unless you choose a backslash character as separator).
Depending on the actual application, it might be handy to just escape those characters in both pattern and replacement.
If you're in a bash environment, you can use bash substitution to escape sed separator, like this:
safe_replace () {
sed "s/${1//\//\\\/}/${2//\//\\\/}/g"
}
It's pretty self-explanatory, except for the bizarre part.
Explanation to that:
${1//\//\\\/}
${ - bash expansion starts
1 - first positional argument - the pattern
// - bash pattern substitution pattern separator "replace-all" variant
\/ - literal slash
/ - bash pattern substitution replacement separator
\\ - literal backslash
\/ - literal slash
} - bash expansion ends
example use:
$ input="ka/pus/ta"
$ pattern="/pus/"
$ replacement="/re/"
$ safe_replace "$pattern" "$replacement" <<< "$input"
ka/re/ta