Perhaps I'm stetching bash a bit far, but I have this variable containing a list of urls.
#!/bin/bash
/* returns
/path/page/one.php
/path/subseciton/
/path/to/this/section/
/path/to/yet/aother_section/about.php
etc
*/
list_of_urls = $(pull_urls.sh)
Then I have a for loop running, pulling text content from the old server and the new server. On each of those I'm running various diff commands to see what has changed.
for i in $urls
do
echo $i
storage_area=./working/$i/
mkdir -p $storage_area
xidel http://oldserver/$i -e '//div[#id="maincontent"]//p' > $storage_area/old.txt
xidel http://newserver/$i -e '//div[#id="content"]//p' > $storage_area/new.txt
diff $storage_area/old.txt $storage_area/new.txt > $storage_area/diff.diff
wdiff $storage_area/old.txt $storage_area/new.txt > $storage_area/wdiff.wdiff
done
My problem is that I need to remove a trailing slash. Is it advisable to do so in the following way?
// inside the loop, before xidel calls
i=$(echo $i | sed -e 's/\/$//g')
how about in this way:
kent$ i=foo/
kent$ i=${i%/}
kent$ echo $i
foo
if you prefer doing it with sed, you could consider to use another separator other than /, since you want to use slash in your pattern:
kent$ i=/path/to/this/section/
kent$ i=$(sed 's#/$##'<<<"$i")
kent$ echo $i
/path/to/this/section
Related
I have a simple .csv file with lines that holds 't' values. Here is the example:
2ABC;t;t;t;tortuga;fault;t;t;bored
I want to replace them to '1' using sed.
If I make sed "s/;t;/;1;/g" I get the next result:
2ABC;1;t;1;tortuga;fault;1;t;bored
As you can see, consecutive ';t;' have been replaced through one. Yes, I can replace all ';t;' by sed -e "s/;t;/;1;/g" -e "s/;t;/;1;/g" but this is boring.
How can I make the replacement by one sed command?
If there is something to replace, branch to replace again.
sed ': again; /;t;/{ s//;1;/; b again }'
Overall, parsing cvs with sed is crude. Consider awk.
awk -F';' -v OFS=';' '{ for(i=1;i<=NF;++i) if ($i=="t") $i=1 } 1'
Lookarounds is helpful in such cases:
$ s='t;2ABC;t;t;t;tortuga;fault;t;t;bored;t'
$ echo "$s" | perl -lpe 's/(?<![^;])t(?![^;])/1/g'
1;2ABC;1;1;1;tortuga;fault;1;1;bored;1
echo '2ABC;t;t;t;tortuga;fault;t;t;bored' |
— gawk-specific solution
gawk -be '(ORS = RT)^!(NF = NF)' FS='^t$' OFS=1 RS=';'
— cross-awk-solution
{m,g,n}awk 'gsub(FS, OFS, $!(NF = NF))^_' FS=';t;' OFS=';1;' RS=
2ABC;1;1;1;tortuga;fault;1;1;bored
I have a CSV. I want to edit the 35th field of the CSV and write the change back to the 35th field. This is what I am doing on bash:
awk -F "," '{print $35}' test.csv | sed -i 's/^0/+91/g'
so, I am pulling the 35th entry using awk and then replacing the "0" in the starting position in the string with "+91". This one works perfet and I get desired output on the console.
Now I want this new entry to get written in the file. I am thinking of sed's "in -place" replacement feature but this fetuare needs and input file. In above command, I cannot provide input file because my primary command is awk and sed is taking the input from awk.
Thanks.
You should choose one of the two tools. As for sed, it can be done as follows:
sed -ri 's/^(([^,]*,){34})0([^,]*)/\1+91\3/' test.csv
Not sure about awk, but #shellter's comment might help with that.
The in-place feature of sed is misnamed, as it does not edit the file in place. Instead, it creates a new file with the same name. eg:
$ echo foo > foo
$ ln -f foo bar
$ ls -i foo bar # These are the same file
797325 bar 797325 foo
$ echo new-text > foo # Changes bar
$ cat bar
new-text
$ printf '/new/s//newer\nw\nq\n' | ed foo # Edit foo "in-place"; changes bar
9
newer-text
11
$ cat bar
newer-text
$ ls -i foo bar # Still the same file
797325 bar 797325 foo
$ sed -i s/new/newer/ foo # Does not edit in-place; creates a new file
$ ls -i foo bar
797325 bar 792722 foo
Since sed is not actually editing the file in place, but writing a new file and then renaming it to the old file, you might as well do the same.
awk ... test.csv | sed ... > test.csv.1 && mv test.csv.1 test.csv
There is the misperception that using sed -i somehow avoids the creation of the temporary file. It does not. It just hides the fact from you. Sometimes abstraction is a good thing, but other times it is unnecessary obfuscation. In the case of sed -i, it is the latter. The shell is really good at file manipulation. Use it as intended. If you do need to edit a file in place, don't use the streaming version of ed; just use ed
So, it turned out there are numerous ways to do it. I got it working with sed as below:
sed -i 's/0\([0-9]\{10\}\)/\+91\1/g' test.csv
But this is little tricky as it will edit any entry which matches the criteria. however in my case, It is working fine.
Similar implementation of above logic in perl:
perl -p -i -e 's/\b0(\d{10})\b/\+91$1/g;' test.csv
Again, same caveat as mentioned above.
More precise way of doing it as shown by Lev Levitsky because it will operate specifically on the 35th field
sed -ri 's/^(([^,]*,){34})0([^,]*)/\1+91\3/g' test.csv
For more complex situations, I will have to consider using any of the csv modules of perl.
Thanks everyone for your time and input. I surely know more about sed/awk after reading your replies.
This might work for you:
sed -i 's/[^,]*/+91/35' test.csv
EDIT:
To replace the leading zero in the 35th field:
sed 'h;s/[^,]*/\n&/35;/\n0/!{x;b};s//+91/' test.csv
or more simply:
|sed 's/^\(\([^,]*,\)\{34\}\)0/\1+91/' test.csv
If you have moreutils installed, you can simply use the sponge tool:
awk -F "," '{print $35}' test.csv | sed -i 's/^0/+91/g' | sponge test.csv
sponge soaks up the input, closes the input pipe (stdin) and, only then, opens and writes to the test.csv file.
As of 2015, moreutils is available in package repositories of several major Linux distributions, such as Arch Linux, Debian and Ubuntu.
Another perl solution to edit the 35th field in-place:
perl -i -F, -lane '$F[34] =~ s/^0/+91/; print join ",",#F' test.csv
These command-line options are used:
-i edit the file in-place
-n loop around every line of the input file
-l removes newlines before processing, and adds them back in afterwards
-a autosplit mode – split input lines into the #F array. Defaults to splitting on whitespace.
-e execute the perl code
-F autosplit modifier, in this case splits on ,
#F is the array of words in each line, indexed starting with 0
$F[34] is the 35 element of the array
s/^0/+91/ does the substitution
use sed to replace every occurrence of /dir with $dir (replace / with $) in every script in a directory.
sed "s#/dir#$dir#g"
The $ keeps being interpreted as a function or variable call.
Is there a way around this?
thanks
Read your shell's friendly manual:
man sh
In the shell, "double quotes" around text allow variable interpretation inside, while 'single quotes' do not, a convention adopted by later languages such as Perl and PHP (but not e.g. JavaScript).
sed 's#/dir#$dir#g' *
To perform the replacement within the scripts do something like
find * -maxdepth 0 -type f | while read f; do mv $f $f.old && sed 's#/dir#$dir#' $f.old > $f; done
or just
perl -pi.old -e 's#/dir#\$dir#' * # Perl also interpolates variables in s commands
You can simply escape it with a backslash:
sed "s#/dir#\$dir#g"
shell approach
for file in file*
do
if [ -f "$file ];then
while read -r line
case "$line" in
*/dir* ) line=${line///dir/\$dir}
esac
echo $line > temp
done < "file"
mv temp $file
fi
done
I have a file that contains this kind of paths:
C:\bad\foo.c
C:\good\foo.c
C:\good\bar\foo.c
C:\good\bar\[variable subdir count]\foo.c
And I would like to get the following file:
C:\bad\foo.c
C:/good/foo.c
C:/good/bar/foo.c
C:/good/bar/[variable subdir count]/foo.c
Note that the non matching path should not be modified.
I know how to do this with sed for a fixed number of subdir, but a variable number is giving me trouble. Actually, I would have to use many s/x/y/ expressions (as many as the max depth... not very elegant).
May be with awk, but this kind of magic is beyond my skills.
FYI, I need this trick to correct some gcov binary files on a cygwin platform.
I am dealing with binary files; therefore, I might have the following kind of data:
bindata\bindata%bindataC:\good\foo.c
which should be translated as:
bindata\bindata%bindataC:/good/foo.c
The first \ must not be translated, despite that it is on the same line.
However, I have just checked my .gcno files while editing this text and it looks like all the paths are flanked with zeros, so most of the answers below should fit.
sed -e '/^C:\\good/ s/\\/\//g' input_file.txt
I would recommend you look into the cygpath utility, which converts path names from one format to another. For instance on my machine:
$ cygpath `pwd`
/home/jericson
$ cygpath -w `pwd`
D:\root\home\jericson
$ cygpath -m `pwd`
D:/root/home/jericson
Here's a Perl implementation of what you asked for:
$ echo 'C:\bad\foo.c
C:\good\foo.c
C:\good\bar\foo.c
C:\good\bar\[variable subdir count]\foo.c' | perl -pe 's|\\|/|g if /good/'
C:\bad\foo.c
C:/good/foo.c
C:/good/bar/foo.c
C:/good/bar/[variable subdir count]/foo.c
It works directly with the string, so it will work anywhere. You could combine it with cygpath, but it only works on machines that have that path:
perl -pe '$_ = `cygpath -m $_` if /good/'
(Since I don't have C:\good on my machine, I get output like C:goodfoo.c. If you use a real path on your machine, it ought to work correctly.)
You want to substitute '/' for all '\' but only on the lines that match the good directory path. Both sed and awk will let you do this by having a LHS (matching) expression that only picks the lines with the right path.
A trivial sed script to do this would look like:
/[Cc]:\\good/ s/\\/\//g
For a file:
c:\bad\foo
c:\bad\foo\bar
c:\good\foo
c:\good\foo\bar
You will get the output below:
c:\bad\foo
c:\bad\foo\bar
c:/good/foo
c:/good/foo/bar
Here's how I would do it in awk:
# fixpaths.awk
/C:\\good/ {
gsub(/\\/,"/",$1);
print $1 >> outfile;
}
Then run it using the command:
awk -f fixpaths.awk paths.txt; mv outfile paths.txt
Or with some help from good ol' Bash:
#!/bin/bash
cat file | while read LINE
do
if <bad_condition>
then
echo "$LINE" >> newfile
else
echo "$LINE" | sed -e "s/\\/\//g" >> newfile
fi
done
try this
sed -re '/\\good\\/ s/\\/\//g' temp.txt
or this
awk -F"\\" '{if($2=="good"){OFS="\/"; $1=$1;} print $0}' temp.txt
I need to do the following for hundreds of files:
Append the name of the file (which may contain spaces) to the end of each line in the file.
It seems to me there should be some way to do this:
sed -e 's/$/FILENAME/' *
where FILENAME represents the name of the current file. Is there a sed variable representing the current filename? Or does anyone have a different solution using bash, awk, etc.?
I'm sure there are other ways to do it, I'd use perl:
perl -p -i -e 's/$/$ARGV/;' *
Some versions of sed support the "--in-place" argument so you can condense Tyler's solution to
for i in * ; do
sed -e "s/\$/$i/" --in-place "$i"
done
You could do it with a bash script
for i in *
do
sed -e "s/\$/$i/" "$i"
done
One-liner version:
for i in * ; do sed -e "s/\$/$i/" "$i" ; done
Edit: If you want to replace the contents of the file with the new, name-appended lines, do this:
TFILE=`mktemp`
for i in *
do
sed -e "s/\$/$i/" "$i" > $TFILE
cp -f $TFILE "$i"
done
rm -f $TFILE
awk '{print $0,FILENAME}' > tmpfile
In BASH, I'd do something to the effect of:
for f in *; do echo $f >> $f; done
More or less how Tyler suggested, just with some modifications to allow for spaces in the name. I was hoping for a one-liner though...
(
OLDIFS=$IFS
IFS=$'\n'
for f in *
do
IFS=OLDIFS
sed -e "s/\$/$f/" $f > tmpfile
mv tmpfile $f
IFS=$'\n'
done
)
This might work for you:
printf "%s\n" * | sed 's/.*/sed -i "s|$| &|" &/' | bash