How it is possible to make a dry run with sed?
I have this command:
find ./ -type f | xargs sed -i 's/string1/string2/g'
But before I really substitute in all the files, i want to check what it WOULD substitute. Copying the whole directory structure to check is no option!
Remove the -i and pipe it to less to paginate though the results. Alternatively, you can redirect the whole thing to one large file by removing the -i and appending > dryrun.out
I should note that this script of yours will fail miserably with files that contain spaces in their name or other nefarious characters like newlines or whatnot. A better way to do it would be:
while IFS= read -r -d $'\0' file; do
sed -i 's/string1/string2/g' "$file"
done < <(find ./ -type f -print0)
I would prefer to use the p-option:
find ./ -type f | xargs sed 's/string1/string2/gp'
Could be combined with the --quiet parameter for less verbose output:
find ./ -type f | xargs sed --quiet 's/string1/string2/gp'
From man sed:
p:
Print the current pattern space.
--quiet:
suppress automatic printing of pattern space
I know this is a very old thread and the OP doesn't really need this answer, but I came here looking for a dry run mode myself, so thought of adding the below piece of advice for anyone coming here in future. What I wanted to do was to avoid stomping the backup file unless there is something really changing. If you blindly run sed using the -i option with backup suffix, existing backup file gets overwritten, even when there is nothing substituted.
The way I ended up doing is to pipe sed output to diff and see if anything changed and then rerun sed with in-place update option, something like this:
if ! sed -e 's/string1/string2/g' $fpath | diff -q $fpath - > /dev/null 2>&1; then
sed -i.bak -e 's/string1/string2/g' $fpath
fi
As per OP's question, if the requirement is to just see what would change, then instead of running the in-pace sed, you could do the diff again with some informative messages:
if ! sed -e 's/string1/string2/g' $fpath | diff -q $fpath - > /dev/null 2>&1; then
echo "File $fpath will change with the below diff:"
sed -e 's/string1/string2/g' $fpath | diff $fpath -
fi
You could also capture the output in a variable to avoid doing it twice:
diff=$(sed -e 's/string1/string2/g' $fpath | diff $fpath -)
if [[ $? -ne 0 ]]; then
echo "File $fpath will change with the below diff:"
echo "$diff"
fi
Related
I need a help in the shell scripting processing the file. The script should read each file in the path and replace the string in each row.
It should read each line and replace the 7th column with XXXX mentioned in the sample output. Any help in appreciated.
Input file data
"2013-04-30"|"X"|"0000628"|"15000231"|"1999-12-05"|"ST"|"2455525445552000"|"1111-11-11"|75.00|"XXE11111"|"224425"
"2013-04-30"|"Y"|"0000928"|"95000232"|"1999-12-05"|"VT"|"2455525445552000"|"1111-11-11"|95.00|"VVE11111"|"224425"
output file
"2013-04-30"|"X"|"0000628"|"15000231"|"1999-12-05"|"ST"|"24555XXXXXXXXXX"|"1111-11-11"|75.00|"XXE11111"|"224425"
"2013-04-30"|"Y"|"0000928"|"95000232"|"1999-12-05"|"VT"|"24555XXXXXXXXXX"|"1111-11-11"|95.00|"VVE11111"|"224425"
Script I used to run but it is not editing the input file
FILES=/home/auto/*.txt
for f in $FILES
do
echo "Processing $f file..."
cat $f | awk 'BEGIN {FS="|"; OFS="|"} {$7=substr($7, 1, 6)"XXXXXXXXXX\"";print}'
done
but I can't edit the exiting file in the directory. I need to use the sed -i option but it's not working.
I tried using the script in below server but I am getting the following error.
SunOS 5.10 Generic January 2005
echo "hello"
FILES=/export/home/*.txt
for f in $FILES
do
echo "Processing $f file..."
sed -i -r 's/"([^"]{6})[^"]*"/"\1XXXXXXXXXX"/6' "$f"
done
I get
sed: illegal option -- i
Using GNU sed with -i optoin
sed -i -r 's/"([^"]{5})[^"]*"/"\1XXXXXXXXXX"/5' file
"2013-04-30"|"X"|"0000628"|"15000231"|"1999-12-05"|"ST"|"24555XXXXXXXXXX"|"1111-11-11"|75.00|"XXE11111"|"224425"
"2013-04-30"|"Y"|"0000928"|"95000232"|"1999-12-05"|"VT"|"24555XXXXXXXXXX"|"1111-11-11"|95.00|"VVE11111"|"224425"
if your awk is gnu awk 4.1.0, there is in-place option, read man/info page.
otherwise, you could do:
awk '..code..' inputfile > tmpfile && mv tmpfile inputfile
note, the cat is not necessary, could (should) be removed.
A little ugly but you can try something like this with sed
sed -i 's/\(\([^|]*|\)\{6\}\)\(.\{6\}\).\{11\}\(.*\)/\1\3XXXXXXXXXXX\4/' file
So with your existing script, it will be -
FILES=/home/auto/*.txt
for f in $FILES
do
echo "Processing $f file..."
sed -i 's/\(\([^|]*|\)\{6\}\)\(.\{6\}\).\{11\}\(.*\)/\1\3XXXXXXXXXXX\4/' "$f"
done
The following command is correctly changing the contents of 2 files.
sed -i 's/abc/xyz/g' xaa1 xab1
But what I need to do is to change several such files dynamically and I do not know the file names. I want to write a command that will read all the files from current directory starting with xa* and sed should change the file contents.
I'm surprised nobody has mentioned the -exec argument to find, which is intended for this type of use-case, although it will start a process for each matching file name:
find . -type f -name 'xa*' -exec sed -i 's/asd/dsg/g' {} \;
Alternatively, one could use xargs, which will invoke fewer processes:
find . -type f -name 'xa*' | xargs sed -i 's/asd/dsg/g'
Or more simply use the + exec variant instead of ; in find to allow find to provide more than one file per subprocess call:
find . -type f -name 'xa*' -exec sed -i 's/asd/dsg/g' {} +
Better yet:
for i in xa*; do
sed -i 's/asd/dfg/g' $i
done
because nobody knows how many files are there, and it's easy to break command line limits.
Here's what happens when there are too many files:
# grep -c aaa *
-bash: /bin/grep: Argument list too long
# for i in *; do grep -c aaa $i; done
0
... (output skipped)
#
You could use grep and sed together. This allows you to search subdirectories recursively.
Linux: grep -r -l <old> * | xargs sed -i 's/<old>/<new>/g'
OS X: grep -r -l <old> * | xargs sed -i '' 's/<old>/<new>/g'
For grep:
-r recursively searches subdirectories
-l prints file names that contain matches
For sed:
-i extension (Note: An argument needs to be provided on OS X)
Those commands won't work in the default sed that comes with Mac OS X.
From man 1 sed:
-i extension
Edit files in-place, saving backups with the specified
extension. If a zero-length extension is given, no backup
will be saved. It is not recommended to give a zero-length
extension when in-place editing files, as you risk corruption
or partial content in situations where disk space is exhausted, etc.
Tried
sed -i '.bak' 's/old/new/g' logfile*
and
for i in logfile*; do sed -i '.bak' 's/old/new/g' $i; done
Both work fine.
#PaulR posted this as a comment, but people should view it as an answer (and this answer works best for my needs):
sed -i 's/abc/xyz/g' xa*
This will work for a moderate amount of files, probably on the order of tens, but probably not on the order of millions.
Another more versatile way is to use find:
sed -i 's/asd/dsg/g' $(find . -type f -name 'xa*')
I'm using find for similar task. It is quite simple: you have to pass it as an argument for sed like this:
sed -i 's/EXPRESSION/REPLACEMENT/g' `find -name "FILE.REGEX"`
This way you don't have to write complex loops, and it is simple to see, which files you are going to change, just run find before you run sed.
u can make
'xxxx' text u search and will replace it with 'yyyy'
grep -Rn '**xxxx**' /path | awk -F: '{print $1}' | xargs sed -i 's/**xxxx**/**yyyy**/'
There's some good answers above. I thought I'd throw in one more that is succinct and parallelizable, using GNU parallel, which I often prefer to xargs:
parallel sed -i 's/abc/xyz/g' {} ::: xa*
Combine this with the -j N option to run N jobs in parallel.
If you are able to run a script, here is what I did for a similar situation:
Using a dictionary/hashMap (associative array) and variables for the sed command, we can loop through the array to replace several strings. Including a wildcard in the name_pattern will allow to replace in-place in files with a pattern (this could be something like name_pattern='File*.txt' ) in a specific directory (source_dir).
All the changes are written in the logfile in the destin_dir
#!/bin/bash
source_dir=source_path
destin_dir=destin_path
logfile='sedOutput.txt'
name_pattern='File.txt'
echo "--Begin $(date)--" | tee -a $destin_dir/$logfile
echo "Source_DIR=$source_dir destin_DIR=$destin_dir "
declare -A pairs=(
['WHAT1']='FOR1'
['OTHER_string_to replace']='string replaced'
)
for i in "${!pairs[#]}"; do
j=${pairs[$i]}
echo "[$i]=$j"
replace_what=$i
replace_for=$j
echo " "
echo "Replace: $replace_what for: $replace_for"
find $source_dir -name $name_pattern | xargs sed -i "s/$replace_what/$replace_for/g"
find $source_dir -name $name_pattern | xargs -I{} grep -n "$replace_for" {} /dev/null | tee -a $destin_dir/$logfile
done
echo " "
echo "----End $(date)---" | tee -a $destin_dir/$logfile
First, the pairs array is declared, each pair is a replacement string, then WHAT1 will be replaced for FOR1 and OTHER_string_to replace will be replaced for string replaced in the file File.txt. In the loop the array is read, the first member of the pair is retrieved as replace_what=$i and the second as replace_for=$j. The find command searches in the directory the filename (that may contain a wildcard) and the sed -i command replaces in the same file(s) what was previously defined. Finally I added a grep redirected to the logfile to log the changes made in the file(s).
This worked for me in GNU Bash 4.3 sed 4.2.2 and based upon VasyaNovikov's answer for Loop over tuples in bash.
The Silver Searcher Solution
I'm adding another option for those people who don't know about the amazing tool called The Silver Searcher (command line tool is ag).
Note: You can use grep and other tools to do the same thing here, but The Silver Searcher is fantastic :)
TLDR
ag -l 'abc' | xargs sed -i 's/abc/xyz/g'
Install The Silver Searcher
sudo apt install silversearcher-ag # Debian / Ubuntu
sudo pacman -S the_silver_searcher # Arch / EndeavourOS
sudo yum install epel-release the_silver_searcher # RHEL / CentOS
Demo Files
Paste the following into your terminal to create some demonstration files:
mkdir /tmp/food
cd /tmp/food
content="Everybody loves to abc this food!"
echo "$content" > ./milk
echo "$content" > ./bread
mkdir ./fastfood
echo "$content" > ./fastfood/pizza
echo "$content" > ./fastfood/burger
mkdir ./fruit
echo "$content" > ./fruit/apple
echo "$content" > ./fruit/apricot
Using 'ag'
The following ag command will recursively find all the files that contain the string 'abc'. It ignores the .git directory, .gitignore files, and other ignore files:
$ ag 'abc'
milk
1:Everybody loves to abc this food!
bread
1:Everybody loves to abc this food!
fastfood/burger
1:Everybody loves to abc this food!
fastfood/pizza
1:Everybody loves to abc this food!
fruit/apple
1:Everybody loves to abc this food!
fruit/apricot
1:Everybody loves to abc this food!
To just list the files that contain the string 'abc', use the -l switch:
$ ag -l 'abc'
bread
fastfood/burger
fastfood/pizza
fruit/apricot
milk
fruit/apple
Changing Multiple Files
Finally, using xargs and sed, we can replace the 'abc' string with another string:
ag -l 'abc' | xargs sed -i 's/abc/eat/g'
In the above command, ag is listing all the files that contain the string 'abc'. The xargs command is splitting the file names and piping them individually into the sed command.
I have a bunch of image files that were incorrectly named 'something#x2.png' and they need to be 'something#2x.png'. They're spread across multiple directories like so:
/images
something#x2.png
/icons
icon#x2.png
/backgrounds
background#x2.png
How can I use grep + sed to find/replace as needed?
Ruby(1.9+)
$ ruby -e 'Dir["**/*#x2.png"].each{|x| File.rename( x, x.sub(/#x2/,"#2x") ) }'
Look at qmv and rename
find -iname '*.png' -print0 | xargs -0 qmv -d
will launch your default editor and allow you to interactively edit the names
rename s/#x2/#2x/ *.png
Slashes look linuxy/unixoid to me. Do you have find and rename?
find -name "*#x2*" -execdir rename 's/#x2/#2x/' {} +
rename is worth installing, comes in some perl-package.
With bash 2.x/3.x
#!/bin/bash
while IFS= read -r -d $'\0' file; do
echo mv "$file" "${file/#x2/#2x}"
done < <(find images/ -type f -name "something*#x2*.png" -print0)
With bash 4.x
#!/bin/bash
shopt -s globstar
for file in images/**; do
[[ "$file" == something*#x2*.png ]] && echo mv "$file" "${file/#x2/#2x}"
done
Note:
In each case I left in an echo so you can do a dry-run, remove the echo if the output is sufficient
I have a simple shell script that removes trailing whitespace from a file. Is there any way to make this script more compact (without creating a temporary file)?
sed 's/[ \t]*$//' $1 > $1__.tmp
cat $1__.tmp > $1
rm $1__.tmp
You can use the in place option -i of sed for Linux and Unix:
sed -i 's/[ \t]*$//' "$1"
Be aware the expression will delete trailing t's on OSX (you can use gsed to avoid this problem). It may delete them on BSD too.
If you don't have gsed, here is the correct (but hard-to-read) sed syntax on OSX:
sed -i '' -E 's/[ '$'\t'']+$//' "$1"
Three single-quoted strings ultimately become concatenated into a single argument/expression. There is no concatenation operator in bash, you just place strings one after the other with no space in between.
The $'\t' resolves as a literal tab-character in bash (using ANSI-C quoting), so the tab is correctly concatenated into the expression.
At least on Mountain Lion, Viktor's answer will also remove the character 't' when it is at the end of a line. The following fixes that issue:
sed -i '' -e's/[[:space:]]*$//' "$1"
Thanks to codaddict for suggesting the -i option.
The following command solves the problem on Snow Leopard
sed -i '' -e's/[ \t]*$//' "$1"
It is best to also quote $1:
sed -i.bak 's/[[:blank:]]*$//' "$1"
var1="\t\t Test String trimming "
echo $var1
Var2=$(echo "${var1}" | sed 's/^[[:space:]]*//;s/[[:space:]]*$//')
echo $Var2
I have a script in my .bashrc that works under OSX and Linux (bash only !)
function trim_trailing_space() {
if [[ $# -eq 0 ]]; then
echo "$FUNCNAME will trim (in place) trailing spaces in the given file (remove unwanted spaces at end of lines)"
echo "Usage :"
echo "$FUNCNAME file"
return
fi
local file=$1
unamestr=$(uname)
if [[ $unamestr == 'Darwin' ]]; then
#specific case for Mac OSX
sed -E -i '' 's/[[:space:]]*$//' $file
else
sed -i 's/[[:space:]]*$//' $file
fi
}
to which I add:
SRC_FILES_EXTENSIONS="js|ts|cpp|c|h|hpp|php|py|sh|cs|sql|json|ini|xml|conf"
function find_source_files() {
if [[ $# -eq 0 ]]; then
echo "$FUNCNAME will list sources files (having extensions $SRC_FILES_EXTENSIONS)"
echo "Usage :"
echo "$FUNCNAME folder"
return
fi
local folder=$1
unamestr=$(uname)
if [[ $unamestr == 'Darwin' ]]; then
#specific case for Mac OSX
find -E $folder -iregex '.*\.('$SRC_FILES_EXTENSIONS')'
else
#Rhahhh, lovely
local extensions_escaped=$(echo $SRC_FILES_EXTENSIONS | sed s/\|/\\\\\|/g)
#echo "extensions_escaped:$extensions_escaped"
find $folder -iregex '.*\.\('$extensions_escaped'\)$'
fi
}
function trim_trailing_space_all_source_files() {
for f in $(find_source_files .); do trim_trailing_space $f;done
}
For those who look for efficiency (many files to process, or huge files), using the + repetition operator instead of * makes the command more than twice faster.
With GNU sed:
sed -Ei 's/[ \t]+$//' "$1"
sed -i 's/[ \t]\+$//' "$1" # The same without extended regex
I also quickly benchmarked something else: using [ \t] instead of [[:space:]] also significantly speeds up the process (GNU sed v4.4):
sed -Ei 's/[ \t]+$//' "$1"
real 0m0,335s
user 0m0,133s
sys 0m0,193s
sed -Ei 's/[[:space:]]+$//' "$1"
real 0m0,838s
user 0m0,630s
sys 0m0,207s
sed -Ei 's/[ \t]*$//' "$1"
real 0m0,882s
user 0m0,657s
sys 0m0,227s
sed -Ei 's/[[:space:]]*$//' "$1"
real 0m1,711s
user 0m1,423s
sys 0m0,283s
Just for fun:
#!/bin/bash
FILE=$1
if [[ -z $FILE ]]; then
echo "You must pass a filename -- exiting" >&2
exit 1
fi
if [[ ! -f $FILE ]]; then
echo "There is not file '$FILE' here -- exiting" >&2
exit 1
fi
BEFORE=`wc -c "$FILE" | cut --delimiter=' ' --fields=1`
# >>>>>>>>>>
sed -i.bak -e's/[ \t]*$//' "$FILE"
# <<<<<<<<<<
AFTER=`wc -c "$FILE" | cut --delimiter=' ' --fields=1`
if [[ $? != 0 ]]; then
echo "Some error occurred" >&2
else
echo "Filtered '$FILE' from $BEFORE characters to $AFTER characters"
fi
In the specific case of sed, the -i option that others have already mentioned is far and away the simplest and sanest one.
In the more general case, sponge, from the moreutils collection, does exactly what you want: it lets you replace a file with the result of processing it, in a way specifically designed to keep the processing step from tripping over itself by overwriting the very file it's working on. To quote the sponge man page:
sponge reads standard input and writes it out to the specified file. Unlike a shell redirect, sponge soaks up all its input before writing the output file. This allows constructing pipelines that read from and write to the same file.
https://joeyh.name/code/moreutils/
To remove trailing whitespace for all files in the current directory, I use
ls | xargs sed -i 's/[ \t]*$//'
These answers confused me. Both of these sed commands worked for me on a Java source file:
sed 's/\s\+$/ filename
sed 's/[[:space:]]\+$// filename
for test purposes, I used:
$ echo " abc " | sed 's/\s\+$/-xx/'
abc-xx
$ echo -e " abc \t\t " | sed 's/\s\+$/-xx/'
abc-xx
Replacing all trailing whitespace with "-xx".
#Viktor wishes to avoid a temporay file, personally I would only use the -i => in-place with a back-up suffix. At least until I know the command works.
Sorry, I just found the existing responses a little oblique. sed is straightforward tool. It is easier to approach it in a straightforward way 90% of the time. Or perhaps I missed something, happy to corrected there.
To only strip whitespaces (in my case spaces and tabs) from lines with at least one non-whitespace character (this way empty indented lines are not touched):
sed -i -r 's/([^ \t]+)[ \t]+$/\1/' "$file"
This question already has answers here:
How can I do a recursive find/replace of a string with awk or sed?
(37 answers)
Closed 1 year ago.
Using just grep and sed, how do I replace all occurrences of:
a.example.com
with
b.example.com
within a text file under the /home/user/ directory tree recursively finding and replacing all occurrences in all files in sub-directories as well.
Try this:
find /home/user/ -type f | xargs sed -i 's/a\.example\.com/b.example.com/g'
In case you want to ignore dot directories
find . \( ! -regex '.*/\..*' \) -type f | xargs sed -i 's/a\.example\.com/b.example.com/g'
Edit: escaped dots in search expression
Try this:
grep -rl 'SearchString' ./ | xargs sed -i 's/REPLACESTRING/WITHTHIS/g'
grep -rl will recursively search for the SEARCHSTRING in the directories ./ and will replace the strings using sed.
Ex:
Replacing a name TOM with JERRY using search string as SWATKATS in directory CARTOONNETWORK
grep -rl 'SWATKATS' CARTOONNETWORK/ | xargs sed -i 's/TOM/JERRY/g'
This will replace TOM with JERRY in all the files and subdirectories under CARTOONNETWORK wherever it finds the string SWATKATS.
On macOS, none of the answers worked for me. I discovered that was due to differences in how sed works on macOS and other BSD systems compared to GNU.
In particular BSD sed takes the -i option but requires a suffix for the backup (but an empty suffix is permitted)
grep version from this answer.
grep -rl 'foo' ./ | LC_ALL=C xargs sed -i '' 's/foo/bar/g'
find version from this answer.
find . \( ! -regex '.*/\..*' \) -type f | LC_ALL=C xargs sed -i '' 's/foo/bar/g'
Don't omit the Regex to ignore . folders if you're in a Git repo. I realized that the hard way!
That LC_ALL=C option is to avoid getting sed: RE error: illegal byte sequence if sed finds a byte sequence that is not a valid UTF-8 character. That's another difference between BSD and GNU. Depending on the kind of files you are dealing with, you may not need it.
For some reason that is not clear to me, the grep version found more occurrences than the find one, which is why I recommend to use grep.
I know this is a really old question, but...
#vehomzzz's answer uses find and xargs when the questions says explicitly grep and sed only.
#EmployedRussian and #BrooksMoses tried to say it was a dup of awk and sed, but it's not - again, the question explicitly says grep and sed only.
So here is my solution, assuming you are using Bash as your shell:
OLDIFS=$IFS
IFS=$'\n'
for f in `grep -rl a.example.com .` # Use -irl instead of -rl for case insensitive search
do
sed -i 's/a\.example\.com/b.example.com/g' $f # Use /gi instead of /g for case insensitive search
done
IFS=$OLDIFS
If you are using a different shell, such as Unix SHell, let me know and I will try to find a syntax adjustment.
P.S.: Here's a one-liner:
OLDIFS=$IFS;IFS=$'\n';for f in `grep -rl a.example.com .`;do sed -i 's/a\.example\.com/b.example.com/g' $f;done;IFS=$OLDIFS
Sources:
Bash: Iterating over lines in a variable
grep(1) - Linux man page
Official Grep Manual
sed(1) - Linux man page
Official sed Manual
For me works the next command:
find /path/to/dir -name "file.txt" | xargs sed -i 's/string_to_replace/new_string/g'
if string contains slash 'path/to/dir' it can be replace with another character to separate, like '#' instead '/'.
For example: 's#string/to/replace#new/string#g'
We can try using the more powerful ripgrep as
rg "BYE_BYE_TEXT" ./ --files-with-matches | xargs sed -i "s/BYE_BYE_TEXT/WELCOME_TEXT/g"
Because ripgrep is good at finding and sed is great at replacing.
it is much simpler than that.
for i in `find *` ; do sed -i -- 's/search string/target string/g' $i; done
find i => will iterate over all the files in the folder and in subfolders.
sed -i => will replace in the files the relevant string if exists.
Try this command:
/home/user/ directory - find ./ -type f \
-exec sed -i -e 's/a.example.com/b.example.com/g' {} \;
The command below will search all the files recursively whose name matches the search pattern and will replace the string:
find /path/to/searchdir/ -name "serachpatter" -type f | xargs sed -i 's/stringone/StrIngTwo/g'
Also if you want to limit the depth of recursion you can put the limits as well:
find /path/to/searchdir/ -name "serachpatter" -type f -maxdepth 4 -mindepth 2 | xargs sed -i 's/stringone/StrIngTwo/g'