In a bash script, files with spaces show up as "File\ with\ spaces.txt" and I want to substitute those slashed-spaces with either _ or +.
How can I tell sed to do that? I had no success using;
$1=~/File\ with\ spaces.txt
ext=$1
web=$(echo "$ext" | sed 's/\ /+/')
I'm open to suggestions if there's a better way than through sed.
[EDIT]: Foo Bah's solution works well, but it substitutes only the first space because the text following it is treated as arguments rather than part of the $1. Any way around this?
sed 's/\\\\ /+/';
\\\\ evaluates to a \\ at the shell level, and then into a literal \ within sed.
Sed recognises \ as space just fine:
bee#i20 ~ $ echo file\ 123 | sed 's/\ /+/'
file+123
Your bash script syntax is all wrong, though.
Not sure what you were trying to do with the script, but here is an example of replacing spaces with +:
ext=~/File\ with\ spaces.txt
web=`echo "$ext" | sed 's/\ /+/g'`
echo $web
Upd:
Oh, and you need the g flag to replace all occurences of space, not only the first one. Fixed above.
you want to escape the slash:
web=$(echo "$ext" | sed 's/\\ /_/g')
single quotes are your friend
the following should be used with single quoted args for $1 and $2
#!/bin/bash
ESCAPE='\\'
if [ $# -ne 2 ];then
echo "$0 <TO_ESCAPE> <IN_STRING>"
echo args should be in single quotes!!
exit 1
fi
TO_ESCAPE="${1}"
IN_STRING="${2}"
if [ ${TO_ESCAPE} = '\' ];then
TO_ESCAPE='\\'
fi
echo "${IN_STRING}" | sed "s/${TO_ESCAPE}/${ESCAPE}${TO_ESCAPE}/g"
Related
How do you escape line beginning and line end in bracket expressions in sed?
For example, let's say I want to replace both comma, line beginning, and line end in each line with pipe:
echo "a,b,c" | sed 's/,/|/g'
# a|b|c
echo "a,b,c" | sed 's/^/|/g'
# |a,b,c
echo "a,b,c" | sed 's/$/|/g'
# a,b,c|
echo "a,b,c" | sed 's/[,^$]/|/g'
# a|b|c
I would expect the last command to produce |a|b|c|. I also tried escaping the line beginning and line end via backslash, with no change.
With GNU sed with extended regular expressions, you can do:
$ echo "a,b,c" | /opt/gnu/bin/sed -E 's/^|,|$/|/g'
|a|b|c|
$
The -E option enables the extended regular expressions, as does -r, but -E is also used by other sed variants for the same purpose, unlike -r.
However, for reasons which elude me, the BSD (macOS) variant of sed produces:
$ echo "a,b,c" | sed -E 's/^|,|$/|/g'
|a|b|c
$
I can't think why.
If this variability is unacceptable, go with the three-substitution solution:
$ echo "a,b,c" | sed -e "s/^/|/" -e "s/$/|/" -e "s/,/|/g"
|a|b|c|
$
which should work with any variant of sed. However, note that echo "" | sed …3 subs… produces || whereas the -E variant produces |. I'm not sure if there's an easy fix for that.
You tried this, but it didn't do what you wanted:
$ echo "a,b,c" | sed 's/[,^$]/|/g'
a|b|c
$
This is what should be expected. Inside character classes, most special characters lose their special-ness. There is nothing special about $ (or , but it isn't a metacharacter anyway) in a character class; ^ is only special at the start of the class and it negates the character class. That means that what follows shows the correct, expected behaviour from this permutation of the contents of your character class:
$ echo "a,b\$\$b,c" | sed 's/[^,$]/|/g'
|,|$$|,|
$
It mapped all the non-comma, non-dollar characters to pipes. I should be using single quotes around the echo; then the backslashes wouldn't be necessary. I just followed the question's code quietly.
Following sed may help you in same.
echo "a,b,c" | sed 's/^/|/;s/,/|/g;s/$/|/'
Output will be as follows.
|a|b|c|
I can chain multiple sed substitutions and a awk operation to achieve this, but is there a single sed substitution that can do it?
Also is there any other tool that is more suitable for this parsing task?
You could try:
sed -r 's!rec:id=(.*?)&name=(.*?)&age=(.*?)!\1 \2 \3!' input_file
If you don't know the rec:id etc in advance but you know there's three, you could try:
sed -r 's![^=]+=(.*?)&[^=]+=(.*?)&[^=]+=(.*?)!\1 \2 \3!' input_file
If you don't know how many &name=value pairs you're after in advance but want to output all the values, you could try something like:
grep -P -o '(?<==)([^&]*)(?=&|$)' | xargs
where the -P means 'perl regex', the regex says "find the string followed by an & (or end of string) and preceded by and equals sign", the -o means to print just the matches (ie the 1, zz, and 21) each on their own line, and the | xargs moves these from their own line to one line and space separated (ie 1\nzz\n21 to 1 zz 21).
This might work for you:
echo "rec:id=1&name=zz&age=21" | sed 's/[^=]*=\([^&]*\)/\1 /g'
1 zz 21
However this leaves an extra space at the end, to solve this use:
echo "rec:id=1&name=zz&age=21"|sed 's/[^=]*=\([^&]*\)/\1 /g:;s/ $//'
1 zz 21
How about parsing the values directly into variables?
inbound="rec:id=1&name=zz&age=21"
eval $(echo $inbound | cut -c5- | tr \& "\n")
echo "Name:$name, ID:$id, Age:$age"
Or even better, though slightly more arcane:
inbound="rec:id=1&name=zz&age=21"
IFS=\& eval $(cut -c5- <<< $inbound)
echo "Name:$name, ID:$id, Age:$age"
How do I remove the first and the last quotes?
echo "\"test\"" | sed 's/"//' | sed 's/"$//'
The above is working as expected, But I guess there must be a better way.
You can combine the sed calls into one:
echo "\"test\"" | sed 's/"//;s/"$//'
The command you posted will remove the first quote even if it's not at the beginning of the line. If you want to make sure that it's only done if it is at the beginning, then you can anchor it like this:
echo "\"test\"" | sed 's/^"//;s/"$//'
Some versions of sed don't like multiple commands separated by semicolons. For them you can do this (it also works in the ones that accept semicolons):
echo "\"test\"" | sed -e 's/^"//' -e 's/"$//'
Maybe you prefer something like this:
echo '"test"' | sed 's/^"\(.*\)"$/\1/'
if you are sure there are no other quotes besides the first and last, just use /g modifier
$ echo "\"test\"" | sed 's/"//g'
test
If you have Ruby(1.9+)
$ echo $s
blah"te"st"test
$ echo $s | ruby -e 's=gets.split("\"");print "#{s[0]}#{s[1..-2].join("\"")+s[-1]}"'
blahte"sttest
Note the 2nd example the first and last quotes which may not be exactly at the first and last positions.
example with more quotes
$ s='bl"ah"te"st"tes"t'
$ echo $s | ruby -e 's=gets.split("\"");print "#{s[0]}#{s[1..-2].join("\"")+s[-1]}"'
blah"te"st"test
I have a simple shell script that removes trailing whitespace from a file. Is there any way to make this script more compact (without creating a temporary file)?
sed 's/[ \t]*$//' $1 > $1__.tmp
cat $1__.tmp > $1
rm $1__.tmp
You can use the in place option -i of sed for Linux and Unix:
sed -i 's/[ \t]*$//' "$1"
Be aware the expression will delete trailing t's on OSX (you can use gsed to avoid this problem). It may delete them on BSD too.
If you don't have gsed, here is the correct (but hard-to-read) sed syntax on OSX:
sed -i '' -E 's/[ '$'\t'']+$//' "$1"
Three single-quoted strings ultimately become concatenated into a single argument/expression. There is no concatenation operator in bash, you just place strings one after the other with no space in between.
The $'\t' resolves as a literal tab-character in bash (using ANSI-C quoting), so the tab is correctly concatenated into the expression.
At least on Mountain Lion, Viktor's answer will also remove the character 't' when it is at the end of a line. The following fixes that issue:
sed -i '' -e's/[[:space:]]*$//' "$1"
Thanks to codaddict for suggesting the -i option.
The following command solves the problem on Snow Leopard
sed -i '' -e's/[ \t]*$//' "$1"
It is best to also quote $1:
sed -i.bak 's/[[:blank:]]*$//' "$1"
var1="\t\t Test String trimming "
echo $var1
Var2=$(echo "${var1}" | sed 's/^[[:space:]]*//;s/[[:space:]]*$//')
echo $Var2
I have a script in my .bashrc that works under OSX and Linux (bash only !)
function trim_trailing_space() {
if [[ $# -eq 0 ]]; then
echo "$FUNCNAME will trim (in place) trailing spaces in the given file (remove unwanted spaces at end of lines)"
echo "Usage :"
echo "$FUNCNAME file"
return
fi
local file=$1
unamestr=$(uname)
if [[ $unamestr == 'Darwin' ]]; then
#specific case for Mac OSX
sed -E -i '' 's/[[:space:]]*$//' $file
else
sed -i 's/[[:space:]]*$//' $file
fi
}
to which I add:
SRC_FILES_EXTENSIONS="js|ts|cpp|c|h|hpp|php|py|sh|cs|sql|json|ini|xml|conf"
function find_source_files() {
if [[ $# -eq 0 ]]; then
echo "$FUNCNAME will list sources files (having extensions $SRC_FILES_EXTENSIONS)"
echo "Usage :"
echo "$FUNCNAME folder"
return
fi
local folder=$1
unamestr=$(uname)
if [[ $unamestr == 'Darwin' ]]; then
#specific case for Mac OSX
find -E $folder -iregex '.*\.('$SRC_FILES_EXTENSIONS')'
else
#Rhahhh, lovely
local extensions_escaped=$(echo $SRC_FILES_EXTENSIONS | sed s/\|/\\\\\|/g)
#echo "extensions_escaped:$extensions_escaped"
find $folder -iregex '.*\.\('$extensions_escaped'\)$'
fi
}
function trim_trailing_space_all_source_files() {
for f in $(find_source_files .); do trim_trailing_space $f;done
}
For those who look for efficiency (many files to process, or huge files), using the + repetition operator instead of * makes the command more than twice faster.
With GNU sed:
sed -Ei 's/[ \t]+$//' "$1"
sed -i 's/[ \t]\+$//' "$1" # The same without extended regex
I also quickly benchmarked something else: using [ \t] instead of [[:space:]] also significantly speeds up the process (GNU sed v4.4):
sed -Ei 's/[ \t]+$//' "$1"
real 0m0,335s
user 0m0,133s
sys 0m0,193s
sed -Ei 's/[[:space:]]+$//' "$1"
real 0m0,838s
user 0m0,630s
sys 0m0,207s
sed -Ei 's/[ \t]*$//' "$1"
real 0m0,882s
user 0m0,657s
sys 0m0,227s
sed -Ei 's/[[:space:]]*$//' "$1"
real 0m1,711s
user 0m1,423s
sys 0m0,283s
Just for fun:
#!/bin/bash
FILE=$1
if [[ -z $FILE ]]; then
echo "You must pass a filename -- exiting" >&2
exit 1
fi
if [[ ! -f $FILE ]]; then
echo "There is not file '$FILE' here -- exiting" >&2
exit 1
fi
BEFORE=`wc -c "$FILE" | cut --delimiter=' ' --fields=1`
# >>>>>>>>>>
sed -i.bak -e's/[ \t]*$//' "$FILE"
# <<<<<<<<<<
AFTER=`wc -c "$FILE" | cut --delimiter=' ' --fields=1`
if [[ $? != 0 ]]; then
echo "Some error occurred" >&2
else
echo "Filtered '$FILE' from $BEFORE characters to $AFTER characters"
fi
In the specific case of sed, the -i option that others have already mentioned is far and away the simplest and sanest one.
In the more general case, sponge, from the moreutils collection, does exactly what you want: it lets you replace a file with the result of processing it, in a way specifically designed to keep the processing step from tripping over itself by overwriting the very file it's working on. To quote the sponge man page:
sponge reads standard input and writes it out to the specified file. Unlike a shell redirect, sponge soaks up all its input before writing the output file. This allows constructing pipelines that read from and write to the same file.
https://joeyh.name/code/moreutils/
To remove trailing whitespace for all files in the current directory, I use
ls | xargs sed -i 's/[ \t]*$//'
These answers confused me. Both of these sed commands worked for me on a Java source file:
sed 's/\s\+$/ filename
sed 's/[[:space:]]\+$// filename
for test purposes, I used:
$ echo " abc " | sed 's/\s\+$/-xx/'
abc-xx
$ echo -e " abc \t\t " | sed 's/\s\+$/-xx/'
abc-xx
Replacing all trailing whitespace with "-xx".
#Viktor wishes to avoid a temporay file, personally I would only use the -i => in-place with a back-up suffix. At least until I know the command works.
Sorry, I just found the existing responses a little oblique. sed is straightforward tool. It is easier to approach it in a straightforward way 90% of the time. Or perhaps I missed something, happy to corrected there.
To only strip whitespaces (in my case spaces and tabs) from lines with at least one non-whitespace character (this way empty indented lines are not touched):
sed -i -r 's/([^ \t]+)[ \t]+$/\1/' "$file"
I have the following: param="/var/tmp/test"
I need to replace the word test with another word such as new_test
need a smart way to replace the last word after "/" with sed
echo 'param="/var/tmp/test"' | sed 's/\/[^\/]*"/\/REPLACEMENT"/'
param="/var/tmp/REPLACEMENT"
echo '/var/tmp/test' | sed 's/\/[^\/]*$/\/REPLACEMENT/'
/var/tmp/REPLACEMENT
Extracting bits and pieces with sed is a bit messy (as Jim Lewis says, use basename and dirname if you can) but at least you don't need a plethora of backslashes to do it if you are going the sed route since you can use the fact that the delimiter character is selectable (I like to use ! when / is too awkward, but it's arbitrary):
$ echo 'param="/var/tmp/test"' | sed ' s!/[^/"]*"!/new_test"! '
param="/var/tmp/new_test"
We can also extract just the part that was substituted, though this is easier with two substitutions in the sed control script:
$ echo 'param="/var/tmp/test"' | sed ' s!.*/!! ; s/"$// '
test
You don't need sed for this...basename and dirname are a better choice for assembling or disassembling pathnames. All those escape characters give me a headache....
param="/var/tmp/test"
param_repl=`dirname $param`/newtest
It's not clear whether param is part of the string that you need processed or it's the variable that holds the string. Assuming the latter, you can do this using only Bash (you don't say which shell you're using):
shopt -s extglob
param="/var/tmp/test"
param="${param/%\/*([^\/])//new_test}"
If param= is part of the string:
shopt -s extglob
string='param="/var/tmp/test"'
string="${string/%\/*([^\/])\"//new}"
This might work for you:
echo 'param="/var/tmp/test"' | sed -r 's#(/(([^/]*/)*))[^"]*#\1newtest#'
param="/var/tmp/newtest"