I would like to extract substring with sed as below:
#!/bin/bash
txt="[audio.sys.offload.pstimeout.secs]: [3]"
echo $txt|sed -r -e 's/\[[a-zA-Z0-9_.]+\].*/\1/'
expected output is:
audio.sys.offload.pstimeout.secs
Error message:
sed: -e expression #1, char 26: invalid reference \1 on `s' command's RHS
#!/bin/bash
txt="[audio.sys.offload.pstimeout.secs]: [3]"
echo $txt | sed -r -e 's/^\[(.*)\]:.*/\1/'
we're grabbing all the characters from the 1st [ until the last ]: and putting them in a capture group.
Would you like the regex to remain mostly like yours?
by the way - with lazy matching (which isn't supported by sed),
the regex could be cleaner, simply ^\[(.*?\])
I have a sample.txt file with contents as:
test sometest
test2 againsometext
My shell script uses sed to find the line which has "test" and replaces the whole line with "test google.com::100-ws sample.com::SAMPLE". sed fails with "unterminated `s' command".
#!/bin/sh
TEST1=test
TEST2=google.com::100-ws sample.com::GOLD-WS
echo $TEST1
echo $TEST2
if [[ ! -e sample.txt ]]; then
touch sample.txt
echo $TEST1 $TEST2 >> sample.txt
else
grep -q $TEST1 sample.txt
if [ $? -eq 0 ];then
sed -i 's/^$TEST1 .*$/$TEST1 '$TEST2'/' sample.txt
else
echo $TEST1 $TEST2 >> sample.txt
fi
fi
I have this same script in other places where the replacement text does not have any spaces and it works fine.
Can someone suggest some ideas to try?
TIA
Assuming bash is available in your environment, it will be just
enough to say:
#!/bin/bash
test1="test"
test2="google.com::100-ws sample.com::GOLD-W"
sed -i "s/^$test1.*/$test1 $test2/" sample.txt
You need to quote strings which contain blank character(s) to prevent word-splitting.
It is not a good practice to use uppercase alphabets for normal variable names.
You need to use not single quotes but double quotes to enable variable expansion as
a command to sed.
That space between test and msg is important. Otherwise you'll obliterate every line that begins with test. Also you can backrefence \1 your captured group \(test \) https://regular-expressions.mobi/refcapture.html
#! /bin/bash
replace="google.com::100-ws sample.com::GOLD-W"
sed -i "s/^\(test \).*/\1$replace/" sample.txt
I 've got a file named file.conf containing:
this is the configuration text and this is the WORD to change.
Running:
sed -i 's/WORD/"ONE TWO"/g' file.conf
I will have file.conf modified:
this is the configuration text and this is the "ONE TWO" to change.
now if I make a script, using read:
read -p 'word to change' TEXT -> "ONE TWO"
echo $TEXT -> "ONE TWO"
sed -i 's/WORD/'$TEXT'/g' file.conf
it does not work with error message:
sed: -e expression #1, char 11: unterminated `s' command
file.conf is not modified in this case.
but it works if I read $TEXT with only one word without spaces: "ONE" for instance.
Thanx folks.
Double quote variable like this:
sed -i 's/WORD/'"$TEXT"'/g' file.conf
Even safer:
sed -i 's/WORD/'"${TEXT}"'/g' file.conf
I want to replace one string with another but I can't. The code is:
updatedb
MCRYPTINI=$(locate mcrypt.ini | grep 'apache2')
MCRYPTSO=$(locate mcrypt.so | grep "/mcrypt.so")
OLD="extension=mcrypt.so"
NEW="extension=$MCRYPTSO"
echo $MCRYPTINI
echo $MCRYPTSO
echo $OLD
echo $NEW
echo "'s/$OLD/$NEW' $MCRYPTINI"
sed -i 's/$OLD/$NEW' $MCRYPTINI
And the result is:
sudo sh testScript.sh
/etc/php5/apache2/conf.d/20-mcrypt.ini
/usr/lib/php5/20121212/mcrypt.so
extension=mcrypt.so
extension=/usr/lib/php5/20121212/mcrypt.so
's/extension=mcrypt.so/extension=/usr/lib/php5/20121212/mcrypt.so' /etc/php5/apache2/conf.d/20-mcrypt.ini
sed: -e expression #1, char 11: unterminated `s' command
For the response I don't need to use 'sed', but it's looks easy and good.
I use sh not bash because I want the code can use in all the systems, so I prefer answers that follow that principle
UPDATE
sed -i "s/$OLD/$NEW/" $MCRYPTINI
error:
sed: -e expression #1, char 14: unknown option to `s'
Add a slash and double quotes:
sed -i "s/$OLD/$NEW/" file
The solution could be:
sed -i "s/$OLD/$NEW/" $MCRYPTINI
but $NEW is a path, so I need to change "/" by other character, for example "+"
sed -i "s+$OLD+$NEW+" $MCRYPTINI
I have a simple shell script that removes trailing whitespace from a file. Is there any way to make this script more compact (without creating a temporary file)?
sed 's/[ \t]*$//' $1 > $1__.tmp
cat $1__.tmp > $1
rm $1__.tmp
You can use the in place option -i of sed for Linux and Unix:
sed -i 's/[ \t]*$//' "$1"
Be aware the expression will delete trailing t's on OSX (you can use gsed to avoid this problem). It may delete them on BSD too.
If you don't have gsed, here is the correct (but hard-to-read) sed syntax on OSX:
sed -i '' -E 's/[ '$'\t'']+$//' "$1"
Three single-quoted strings ultimately become concatenated into a single argument/expression. There is no concatenation operator in bash, you just place strings one after the other with no space in between.
The $'\t' resolves as a literal tab-character in bash (using ANSI-C quoting), so the tab is correctly concatenated into the expression.
At least on Mountain Lion, Viktor's answer will also remove the character 't' when it is at the end of a line. The following fixes that issue:
sed -i '' -e's/[[:space:]]*$//' "$1"
Thanks to codaddict for suggesting the -i option.
The following command solves the problem on Snow Leopard
sed -i '' -e's/[ \t]*$//' "$1"
It is best to also quote $1:
sed -i.bak 's/[[:blank:]]*$//' "$1"
var1="\t\t Test String trimming "
echo $var1
Var2=$(echo "${var1}" | sed 's/^[[:space:]]*//;s/[[:space:]]*$//')
echo $Var2
I have a script in my .bashrc that works under OSX and Linux (bash only !)
function trim_trailing_space() {
if [[ $# -eq 0 ]]; then
echo "$FUNCNAME will trim (in place) trailing spaces in the given file (remove unwanted spaces at end of lines)"
echo "Usage :"
echo "$FUNCNAME file"
return
fi
local file=$1
unamestr=$(uname)
if [[ $unamestr == 'Darwin' ]]; then
#specific case for Mac OSX
sed -E -i '' 's/[[:space:]]*$//' $file
else
sed -i 's/[[:space:]]*$//' $file
fi
}
to which I add:
SRC_FILES_EXTENSIONS="js|ts|cpp|c|h|hpp|php|py|sh|cs|sql|json|ini|xml|conf"
function find_source_files() {
if [[ $# -eq 0 ]]; then
echo "$FUNCNAME will list sources files (having extensions $SRC_FILES_EXTENSIONS)"
echo "Usage :"
echo "$FUNCNAME folder"
return
fi
local folder=$1
unamestr=$(uname)
if [[ $unamestr == 'Darwin' ]]; then
#specific case for Mac OSX
find -E $folder -iregex '.*\.('$SRC_FILES_EXTENSIONS')'
else
#Rhahhh, lovely
local extensions_escaped=$(echo $SRC_FILES_EXTENSIONS | sed s/\|/\\\\\|/g)
#echo "extensions_escaped:$extensions_escaped"
find $folder -iregex '.*\.\('$extensions_escaped'\)$'
fi
}
function trim_trailing_space_all_source_files() {
for f in $(find_source_files .); do trim_trailing_space $f;done
}
For those who look for efficiency (many files to process, or huge files), using the + repetition operator instead of * makes the command more than twice faster.
With GNU sed:
sed -Ei 's/[ \t]+$//' "$1"
sed -i 's/[ \t]\+$//' "$1" # The same without extended regex
I also quickly benchmarked something else: using [ \t] instead of [[:space:]] also significantly speeds up the process (GNU sed v4.4):
sed -Ei 's/[ \t]+$//' "$1"
real 0m0,335s
user 0m0,133s
sys 0m0,193s
sed -Ei 's/[[:space:]]+$//' "$1"
real 0m0,838s
user 0m0,630s
sys 0m0,207s
sed -Ei 's/[ \t]*$//' "$1"
real 0m0,882s
user 0m0,657s
sys 0m0,227s
sed -Ei 's/[[:space:]]*$//' "$1"
real 0m1,711s
user 0m1,423s
sys 0m0,283s
Just for fun:
#!/bin/bash
FILE=$1
if [[ -z $FILE ]]; then
echo "You must pass a filename -- exiting" >&2
exit 1
fi
if [[ ! -f $FILE ]]; then
echo "There is not file '$FILE' here -- exiting" >&2
exit 1
fi
BEFORE=`wc -c "$FILE" | cut --delimiter=' ' --fields=1`
# >>>>>>>>>>
sed -i.bak -e's/[ \t]*$//' "$FILE"
# <<<<<<<<<<
AFTER=`wc -c "$FILE" | cut --delimiter=' ' --fields=1`
if [[ $? != 0 ]]; then
echo "Some error occurred" >&2
else
echo "Filtered '$FILE' from $BEFORE characters to $AFTER characters"
fi
In the specific case of sed, the -i option that others have already mentioned is far and away the simplest and sanest one.
In the more general case, sponge, from the moreutils collection, does exactly what you want: it lets you replace a file with the result of processing it, in a way specifically designed to keep the processing step from tripping over itself by overwriting the very file it's working on. To quote the sponge man page:
sponge reads standard input and writes it out to the specified file. Unlike a shell redirect, sponge soaks up all its input before writing the output file. This allows constructing pipelines that read from and write to the same file.
https://joeyh.name/code/moreutils/
To remove trailing whitespace for all files in the current directory, I use
ls | xargs sed -i 's/[ \t]*$//'
These answers confused me. Both of these sed commands worked for me on a Java source file:
sed 's/\s\+$/ filename
sed 's/[[:space:]]\+$// filename
for test purposes, I used:
$ echo " abc " | sed 's/\s\+$/-xx/'
abc-xx
$ echo -e " abc \t\t " | sed 's/\s\+$/-xx/'
abc-xx
Replacing all trailing whitespace with "-xx".
#Viktor wishes to avoid a temporay file, personally I would only use the -i => in-place with a back-up suffix. At least until I know the command works.
Sorry, I just found the existing responses a little oblique. sed is straightforward tool. It is easier to approach it in a straightforward way 90% of the time. Or perhaps I missed something, happy to corrected there.
To only strip whitespaces (in my case spaces and tabs) from lines with at least one non-whitespace character (this way empty indented lines are not touched):
sed -i -r 's/([^ \t]+)[ \t]+$/\1/' "$file"