echo "eh]" | sed -r 's/[\]a]/lol/g' returns eh] instead of ehlol. no errors are raised so I assume \] is some kind of sed magic. Im tired of constant sed problems, but I have to use it. so how to escape ] in sed substition expression?
You can simply use echo "eh]" | sed -r 's/[]a]/lol/g':
without the backslash
and ] as first character inside the brackets.
] and \ have some specialties when used inside brackets.
Let me cite from the documentation:
A leading ^ reverses the meaning of list, so that it matches any single character not in list. To include ] in the list, make it the first character (after the ^ if needed), to include - in the list, make it the first or last; to include ^ put it after the first character.
The characters $, *, ., [, and \ are normally not special within list. For example, [\*] matches either ‘\’ or ‘*’, because the \ is not special here. However, strings like [.ch.], [=a=], and [:space:] are special within list and represent collating symbols, equivalence classes, and character classes, respectively, and [ is therefore special within list when it is followed by ., =, or :. Also, when not in POSIXLY_CORRECT mode, special escapes like \n and \t are recognized within list. See Escapes.
Related
I want to replace specific strings in php files automatically using sed. Some work, and some do not. I already investigated this is not an issue with the replacement string but with the string that is to be replaced. I already tried to escape [ and ] with no success. It seems to be the whitespace within the () - not whitespaces in general. The first whitespaces (around the = ) do not have any problems. Please can someone point me to the problem:
sed -e "1,\$s/$adm = substr($path . rawurlencode($upload['name']) , 16);/$adm = rawurlencode($upload['name']); # fix 23/g" -i administration/identify.php
I already tried to shorten the string which should be replaced and the result was if I cut it directly behind $path it works, with the following whitespace it does not. Escaping whitespace has no effect...
what must be escaped for sed
The following characters have special meaning in sed and have to be escaped with \ for the regex to be taken literally:
\
[
the character used in separating s command parts, ie. / here
.
*
& only replacement string
Newline character is handled specially as the end of the string, but can be replaced for \n.
So first escape all special characters in input and then pass it to sed:
rgx="$adm = substr($path . rawurlencode($upload['name']) , 16);"
rgx_escaped=$(sed 's/[\\\[\.\*\/&]/\\&/g' <<<"$rgx")
sed "s/$rgx_escaped/ etc."
See Escape a string for a sed replace pattern for a generic escaping solution.
You may use
sed -i 's/\$adm = substr(\$path \. rawurlencode(\$upload\['"'"'name'"'"']) , 16);/$adm = rawurlencode($upload['"'"'name'"'"']); # fix 23/g' administration/identify.php
Note:
the sed command is basically wrapped in single quotes, the variable expansion won't occur inside single quotes
In the POSIX BRE syntax, ( matches a literal (, you do not need to escape ) either, but you need to escape [ and . that must match themselves
The single quotes require additional quoting with concatenation.
I'm trying to replace some text in a file using sed but I'm having troubles.
sed -ir 's/(\$hello = )true/\1false/' /path/to/my/file.txt gives the error sed: -e expression #1, char 27: invalid reference \1 on 's' command's RHS.
I want to replace $hello = true with $hello = false, so in order to avoid typing $hello = twice I wanted to use capture groups - which isn't working.
What am I doing wrong?
You don't have to escape parentheses in extended regex mode, if it was your intent with the r into -ir, but actually if you want both options -i and -r then you have to keep them apart or use -ri instead of -ir because the latter interprets the part after -i as an optional backup suffix.
From sed manual
Because -i takes an optional argument, it should
not be followed by other short options:
sed -Ei '...' FILE
Same as -E -i with no backup suffix - FILE will be edited in-place without creating a backup.
sed -iE '...' FILE
This is equivalent to --in-place=E, creating FILEE as backup
of FILE
You must escape the parenthesis with backslashes \(...\), to be used as grouping.
See THE SED FAQ, section "3.1.2. Escape characters on the right side of "s///"" has an example:
3.1.2. Escape characters on the right side of "s///"
The right-hand side (the replacement part) in "s/find/replace/" is
almost always a string literal, with no interpolation of these
metacharacters:
. ^ $ [ ] { } ( ) ? + * |
Three things are interpolated: ampersand (&), backreferences, and
options for special seds. An ampersand on the RHS is replaced by
the entire expression matched on the LHS. There is never any
reason to use grouping like this:
s/\(some-complex-regex\)/one two \1 three/
And later in section "F. GNU sed v2.05 and higher versions":
F. GNU sed v2.05 and higher versions
...
Undocumented -r switch:
Beginning with version 3.02, GNU sed has an undocumented -r switch
(undocumented till version 4.0), activating Extended Regular
Expressions in the following manner:
? - 0 or 1 occurrence of previous character
+ - 1 or more occurrences of previous character
| - matches the string on either side, e.g., foo|bar
(...) - enable grouping without backslash
{...} - enable interval expression without backslash
When the -r switch (mnemonic: "regular expression") is used, prefix
these symbols with a backslash to disable the special meaning.
For documentation of regular expression syntax used in (GNU) sed, see Overview of basic regular expression syntax
5.3 Overview of basic regular expression syntax
...
\(regexp\)
Groups the inner regexp as a whole, this is used to:
Apply postfix operators, like (abcd)*: this will search for zero or more whole sequences of ‘abcd’, while abcd* would search for ‘abc’ followed by zero or more occurrences of ‘d’. Note that support for (abcd)* is required by POSIX 1003.1-2001, but many non-GNU implementations do not support it and hence it is not universally portable.
Use back references (see below).
I wanted to change out the home directory that I has in some files.
I figured that I would have to escape the curly brackets - like this :
sed -i 's/\$\{HOME\}/\/casper\/home/g' /var/tmp/casper.txt
Everything that I experience in sed tells me that I would have to escape the brackets, but I did not. The only thing that I needed to escape is the Dollar sign.
What kind of regex engine does sed use and where is a list of the special characters that need to be escaped and what not does not need to be escaped.
sed -i 's/\${HOME}/\/casper\/home/g' /var/tmp/casper.txt
I am not an expert, just an user of regexps, so I cannot give you strict technical answer, but according to info sed, if invoked without -r (--regexp-extended), than it uses basic regular expressions. If -r is put, then extended regular expressions are used. There is an explanation in info's Appendix:
The only difference between basic and extended regular expressions is in
the behavior of a few characters: '?', '+', parentheses, braces ('{}'),
and '|'. While basic regular expressions require these to be escaped if
you want them to behave as special characters, when using extended
regular expressions you must escape them if you want them _to match a
literal character_.
By the way, if you need to put a slash inside sed's regex, it is very useful to use different symbol as a separator. It doesn't need to be a slash, you can choose the symbol arbibtrary, for example it could be #. So instead this:
sed -i 's/\${HOME}/\/casper\/home/g' /var/tmp/casper.txt
you can make this:
sed -i 's#\${HOME}#/casper/home#g' /var/tmp/casper.txt
I don't understand why the following sed command contains an # symbol:
sed 's#session\s*required\s*pam_loginuid.so#session optional pam_loginuid.so#g' -i /etc/pam.d/sshd
I've looked at /etc/pam.d/sshd for the before/after effects of this command:
BEFORE:
...
# Set the loginuid process attribute.
session required pam_loginuid.so
...
AFTER:
...
# Set the loginuid process attribute.
session optional pam_loginuid.so
....
Is the # symbol possibly part of regex or sed syntax?
Could not find any doco on this.
Note: The above sed command is actually part of a Dockerfile RUN command in tutorial:
https://docs.docker.com/examples/running_ssh_service/
These are alternate delimiters for the regular expressions and replacement string. Handy when your regex or replacement string includes '/'.
From the sed manual
The syntax of the s (as in substitute) command is ‘s/regexp/replacement/flags’. The / characters may be uniformly replaced by any other single character within any given s command. The / character (or whatever other character is used in its stead) can appear in the regexp or replacement only if it is preceded by a \ character.
From the POSIX specification:
[2addr]s/BRE/replacement/flags
Substitute the replacement string for instances of the BRE in the pattern space. Any character other than <backslash> or <newline> can be used instead of a to delimit the BRE and the replacement. Within the BRE and the replacement, the BRE delimiter itself can be used as a literal character if it is preceded by a <backslash>.
as other says, it is another delimiter than traditionnal / in the s///action. This is usually used when / is found/part of the pattern like searching (or replacing by) a unix path that need to escape the /
s/\/my\/path/\/Your\/path/
# same as
s#my/path#/Your/path#
You often use a character that is not alpha numeric (but you can). The only (logical) constraint is to avoid a special character (aka special meaning like ^$[]{}()+\*.) for regex that make it difficult to read (but functionnal) and without the feature of this character in the pattern
echo "b(a)l" | sed 's(.)()('
I have a file of string records where one of the fields - delimited by "," - can contain one or more "-" inside it.
The goal is to delete the field value if it contains more than two "-".
i am trying to recoup my past knowledge of sed/awk but can't make much headway
==========
info,whitepaper,Data-Centers,yes-the-6-top-problems-in-your-data-center-lane
info,whitepaper,Data-Centers,the-evolution-center
info,whitepaper,Data-Centers,the-evolution-of-lan-technology-lanner
==========
expected outcome:
info,whitepaper,Data-Centers
info,whitepaper,Data-Centers,the-evolution-center
info,whitepaper,Data-Centers
thanks
Try
sed -r 's/(^|,)([^,-]+-){3,}[^,]+(,|$)/\3/g'
or if you're into slashes
sed 's/\(^\|,\)\([^,-]\+-\)\{3,\}[^,]\+\(,\|$\)/\3/g'
Explanation:
I'm using the most basic sed command: substitution. The syntax is: s/pattern/replacement/flags.
Here pattern is (^|,)([^,-]+-){3,}[^,]+(,|$), replacement is \3, flags is g.
The g flag means global replacement (all matching parts are replaced, not only the first in line).
In pattern:
brackets () create a group. Somewhat like in math. They also allow to refer to a group with a number later.
^ and $ mean beginning and end of the string.
| means "or", so (^|,) means "comma or beginning of the string".
square brackets [] mean a character class, ^ inside means negation. So [^,-] means "anything but comma or hyphen". Not that usually the hyphen has a special meaning in character classes: [a-z] means all lowercase letters. But here it's just a hyphen because it's not in the middle.
+ after an expression means "match it 1 or more times" (like * means match it 0 or more times).
{N} means "match it exactly N times. {N,M} is "from N to M times". {3,} means "three times or more". + is equivalent to {1,}.
So this is it. The replacement is just \3. This refers to the third group in (), in this case (,|$). This will be the only thing left after the substitution.
P.S. the -r option just changes what characters need to be escaped: without it all of ()-{}| are treated as regular chars unless you escape them with \. Conversely, to match literal ( with -r option you'll need to escape it.
P.P.S. Here's a reference for sed. man sed is your friend as well.
Let me know if you have further questions.
You could try perl instead of sed or awk:
perl -F, -lane 'print join ",", grep { !/-.*-.*-/ } #F' < file.txt
This might work for you:
sed 's/,\{,1\}[^,-]*\(-[^,]*\)\{3,\}//g file
sed 's/\(^\|,\)\([^,]*-\)\{3\}[^,]*\(,\|$\)//g'
This should work in more cases:
sed 's/,$/\n/g;s/\(^\|,\|\n\)\([^,\n]*-\)\{3\}[^,\n]*\(,\|\n\|$\)/\3/g;s/,$//;s/\n/,/g'