Have been looking at SED documention but need a little pointer in the right direction
I have 200 files I want to modify in a batch.
Source is html file.
Need to create a new file for the changes.
Want to delete the first part of each file up to the first tag (This is 20 or so lines but can vary slightly).
Then insert the contents of a source file (the same for all files) into the new target file starting at line 1, for 30 or so lines. The number of lines to insert does not match the number that are deleted though.
Hope you can help.
Paul
This can certainly be done with sed(1), but I would probably use the vanilla editor ed(1).
$ cat > bigfix.sh
for i in "$#"; do
ed "$i" << \eof
1,/<tag>/-1d
0r otherfile.html
w
q
eof
done
$ sh bigfix.sh file*.html
This shell script takes arguments and runs ed(1) on each arg. It deletes lines starting from the first and ending on the line right before the one with <tag>. It then puts otherfile.html at the top and writes out the result.
For an individual file:
sed -e '1,/tag/{/tag/r insertfile' -e ';d}' inputfile > outputfile
For many files:
find . -name 'criterion*.ext' -type f -exec sh -c 'sed -e "1,/tag/{/tag/r insertfile" -e ';d}" "{}" > "{}.new"' \;
Edit:
Fixed the find command to use sh because of the redirection. Note the change in quoting from the previous version.
Related
I have nearly 300 files in 60 folders .
As per the C++ coding guidelines, I need to replace below lines from *.cpp and *.cl files (wants to remove extra space between if and for statement) -
for (* .....)
with
for(* .....)
and also
if (* .....)
with
if(* .....)
Can any one suggest me the grep command to do search and replace for all files.
Edited:
I tried with below commands:
sed -i 's/for (/for(/g' *.cpp
But got error like below:
sed: can't read *.cpp: No such file or directory
I think you need sed command (stream editor, see man sed on your mashine). It is more suitable for file editing.
sed -i -E 's/(for|if)[ ]+(\(.*\))/\1\2/g'
Let me explain:
-i stands for inline, that means that all changes will be done and saved in the file
-E is needed to use extended regular expression inside with sed
s/(for|if)[ ]+(\(.*\))/\1\2/g
s stands for substitute
/ is a separator, which separates different parts of command. Between first / and second / there is pattern that you need to find (and then replace). After second / and third / there that we want to have after substitution.
g in the end stands for global, that means to make changes in the whole file.
How to apply to every file that you need?
This question is already exist, so in the end you need to run in directory where are your files stored following command
find ./ -type f -exec sed -i -E 's/(for|if)[ ]+(\(.*\))/\1\2/g' {} \;
I hope, this will help:)
I have created the file "brol.txt", with following content:
for (correct
for(wrong
if (correct
if(wrong
I have launched following grep command:
grep -E "for \(|if \(" brol.txt
With following result:
for (correct
if (correct
Explanation:
grep -E means extended grep (allows to search for expression1 OR expression2,
separated by a pipe character)
\( means the search for a round bracket. The backslash is an escape character.
I'm trying to add the filename of a text file into the first line of a the same text file. for example if the file name is called test1.txt, then the first line when you open the file should be test1.
below is what I've done so for, the only problem i have is that the word "$file" is being written to the file not the file name. any help is appreciated.
for file in *.txt; do
sed -i '1 i\$file' $file;
awk 'sub("$", "\r")' "$file" > "$file"1;
mv "$file"1 "$file";
done
Without concise, testable sample input and expected output it's an untested guess but it SOUNDS like all you need is:
awk -i inplace -v ORS='\r\n' 'FNR==1{print FILENAME}1' *
No shell loop or multiple commands required.
The above uses GNU awk for inplace editing and I'm assuming the sub() in your code was intended to add a \r at the end of every line.
I've just started learning more about sed and awk and put this into a file called insert.sed and sourced it and passed it a file name:
sed -i '1s/^./'$1'\'$'\n/g' $1
In trying it, it seems to work okay:
rent$ cat x.txt
<<< Who are you?
rent$ source insert.sed x.txt
rent$ cat x.txt
x.txt
<< Who are you?
It is cutting off the first character of the first line so I'd have to fix that otherwise it does add the file name to first line.
I'm sure there's better ways of doing it.
If you want test1 on first line, with gnu sed
sed -i '1{x;s/.*/fich=$(ps -p $PPID -o args=);fich=${fich##*\\} };echo ${fich%%.*}/e;G}' test1.txt
OS: Ubuntu 14.04
I have 12 large json files (2-4 gb each) that I want to perform different operations on. I want to remove the first line, find "}," and replace it with "}" and remove all "]".
I am using sed to do the operations and my command is:
sed -i.bak -e '1d' -e 's/},/}/g' -e '/]/d' file.json
When i run the command on a small file (12,7kb) it works fine. file.json contains the content with the changes and file.json.bak contains the original content.
But when i run the command on my larger files the original file is emptied, e.g. file.json is empty and file.json.bak contains the original content. The run time is also what I consider to be "to fast", about 2-3 seconds.
What am I doing wrong here?
Are you sure your input file contains newlines as recognized by the platform you are running your commands on? If it doesn't then deleting one line would delete the whole file. What does wc -l < file tell you?
If it's not that then you probably don't have enough file space to duplicate the file so sed is doing something internally like
mv file backup && sed '...' backup > file
but doesn't have space to create the new file after moving the original to backup. Check your available file space and if you don't have enough and can't get more then you'll need to do something like:
while [ -s oldfile ]
do
copy first N bytes of oldfile into tmpfile &&
remove first N bytes from oldfile using real inplace editing &&
sed 'script' tmpfile >> newfile &&
rm -f tmpfile
done
mv newfile oldfile
See https://stackoverflow.com/a/17331179/1745001 for how to remove the first N bytes inplace from a file. Pick the largest value for N that does fit in your available space.
I have a CSV. I want to edit the 35th field of the CSV and write the change back to the 35th field. This is what I am doing on bash:
awk -F "," '{print $35}' test.csv | sed -i 's/^0/+91/g'
so, I am pulling the 35th entry using awk and then replacing the "0" in the starting position in the string with "+91". This one works perfet and I get desired output on the console.
Now I want this new entry to get written in the file. I am thinking of sed's "in -place" replacement feature but this fetuare needs and input file. In above command, I cannot provide input file because my primary command is awk and sed is taking the input from awk.
Thanks.
You should choose one of the two tools. As for sed, it can be done as follows:
sed -ri 's/^(([^,]*,){34})0([^,]*)/\1+91\3/' test.csv
Not sure about awk, but #shellter's comment might help with that.
The in-place feature of sed is misnamed, as it does not edit the file in place. Instead, it creates a new file with the same name. eg:
$ echo foo > foo
$ ln -f foo bar
$ ls -i foo bar # These are the same file
797325 bar 797325 foo
$ echo new-text > foo # Changes bar
$ cat bar
new-text
$ printf '/new/s//newer\nw\nq\n' | ed foo # Edit foo "in-place"; changes bar
9
newer-text
11
$ cat bar
newer-text
$ ls -i foo bar # Still the same file
797325 bar 797325 foo
$ sed -i s/new/newer/ foo # Does not edit in-place; creates a new file
$ ls -i foo bar
797325 bar 792722 foo
Since sed is not actually editing the file in place, but writing a new file and then renaming it to the old file, you might as well do the same.
awk ... test.csv | sed ... > test.csv.1 && mv test.csv.1 test.csv
There is the misperception that using sed -i somehow avoids the creation of the temporary file. It does not. It just hides the fact from you. Sometimes abstraction is a good thing, but other times it is unnecessary obfuscation. In the case of sed -i, it is the latter. The shell is really good at file manipulation. Use it as intended. If you do need to edit a file in place, don't use the streaming version of ed; just use ed
So, it turned out there are numerous ways to do it. I got it working with sed as below:
sed -i 's/0\([0-9]\{10\}\)/\+91\1/g' test.csv
But this is little tricky as it will edit any entry which matches the criteria. however in my case, It is working fine.
Similar implementation of above logic in perl:
perl -p -i -e 's/\b0(\d{10})\b/\+91$1/g;' test.csv
Again, same caveat as mentioned above.
More precise way of doing it as shown by Lev Levitsky because it will operate specifically on the 35th field
sed -ri 's/^(([^,]*,){34})0([^,]*)/\1+91\3/g' test.csv
For more complex situations, I will have to consider using any of the csv modules of perl.
Thanks everyone for your time and input. I surely know more about sed/awk after reading your replies.
This might work for you:
sed -i 's/[^,]*/+91/35' test.csv
EDIT:
To replace the leading zero in the 35th field:
sed 'h;s/[^,]*/\n&/35;/\n0/!{x;b};s//+91/' test.csv
or more simply:
|sed 's/^\(\([^,]*,\)\{34\}\)0/\1+91/' test.csv
If you have moreutils installed, you can simply use the sponge tool:
awk -F "," '{print $35}' test.csv | sed -i 's/^0/+91/g' | sponge test.csv
sponge soaks up the input, closes the input pipe (stdin) and, only then, opens and writes to the test.csv file.
As of 2015, moreutils is available in package repositories of several major Linux distributions, such as Arch Linux, Debian and Ubuntu.
Another perl solution to edit the 35th field in-place:
perl -i -F, -lane '$F[34] =~ s/^0/+91/; print join ",",#F' test.csv
These command-line options are used:
-i edit the file in-place
-n loop around every line of the input file
-l removes newlines before processing, and adds them back in afterwards
-a autosplit mode – split input lines into the #F array. Defaults to splitting on whitespace.
-e execute the perl code
-F autosplit modifier, in this case splits on ,
#F is the array of words in each line, indexed starting with 0
$F[34] is the 35 element of the array
s/^0/+91/ does the substitution
I'm attempting to extract data from log files and organise it systematically. I have about 9 log files which are ~100mb each in size.
What I'm trying to do is: Extract multiple chunks from each log file, and for each chunk extracted, I would like to create a new file and save this extracted data to it. Each chunk has a clear start and end point.
Basically, I have made some progress and am able to extract the data I need, however, I've hit a wall in trying to figure out how to create a new file for each matched chunk.
I'm unable to use a programming language like Python or Perl, due to the constraints of my environment. So please excuse the messy command.
My command thus far:
find Logs\ 13Sept/Log_00000000*.log -type f -exec \
sed -n '/LRE Starting chunk/,/LRE Ending chunk/p' {} \; | \
grep -v -A1 -B1 "Starting chunk" > Logs\ 13Sept/Chunks/test.txt
The LRE Starting chunk and LRE Ending chunk are my boundaries. Right now my command works, but it saves all matched chunks to one file (whose size is becoming exessive).
How do I go about creating a new file for each match and add the matched content to it? keeping in mind that each file could hold multiple chunks and is not limited to one chunk per file.
Probably need something more programmable than sed: I'm assuming awk is available.
awk '
/LRE Ending chunk/ {printing = 0}
printing {print > "chunk" n ".txt"}
/LRE Starting chunk/ {printing = 1; n++}
' *.log
Try something like this:
find Logs\ 13Sept/Log_00000000*.log -type f -print | while read file; do \
sed -n '/LRE Starting chunk/,/LRE Ending chunk/p' "$file" | \
grep -v -A1 -B1 "Starting chunk" > "Logs 13Sept/Chunks/$file.chunk.txt";
done
This loops over the find results and executes for each file and then create one $file.chunk.txt for each of the files.
Something like this perhaps?
find Logs\ 13Sept/Log_00000000*.log -type f -exec \
sed -n '/LRE Starting chunk/,/LRE Ending chunk/{;/LRE .*ing chunk/d;w\
'"{}.chunk"';}' {} \;
This uses sed's w command to write to a file named (inputfile).chunk. If that is not acceptable, perhaps you can use sh -c '...' to pass in a small shell script to wrap the sed command with. (Or is a shell script also prohibited for some reason?)
Perhaps you could use csplit to do the splitting, then truncate the output files at the chunk end.