replace a string from first line on multiple files - perl

I got 10,000 text files which I have to make changes.
First line on every file contains a url.
By mistake for few files url missking 'com'
eg:
1) http://www.supersonic./psychology
2) http://www.supersonic./social
3) http://www.supersonic.com/science
my task is to check and add 'com' if it is missing
eg:
1) http://www.supersonic.com/psychology
2) http://www.supersonic.com/social
3) http://www.supersonic.com/science
all urls are of same domain(supersonic.com)
can you suggest me any fast and easy approach ?
Tried this : replacing supersonic./ with supersonic.com
sed -e '1s/supersonic.//supersonic.com/' *
no change in the output.

Use -i to change the files instead of just outputting the changed lines.
Use a different delimiter than / if you want to use / in the regex (or use \/ in the regex).
Use \. to match a dot literally, . matches anything.
sed -i~ -e '1s=supersonic\./=supersonic.com/=' *
Some versions of sed don't support -i.

You are very close with your code, but you need to account for the trailing / char after the . char.
Assuming you are using a modern sed with the -i (inplace-edit) option you can do
sed -i '1s#supersonic\./#supersonic.com/#' *
Note that rather than have to escape / inside of the s/srchpat\/withSlash/replaceStr/', you can use another char after the the s command as the delimiter, here I use s#...#...#. If your search pattern had a # char, then you would have to use a different char.
Some older versions of sed need to you to escape the alternate delimiter at the first use, so
sed 's\#srchStr#ReplStr#' file
for those cases.
If you're using a sed that doesn't support the -i options, then
you'll need to loop on your file, and manage the tmp files, i.e.
for f in *.html ; do
sed '1s#supersonic\./#supersonic.com/#' "$f" > /tmp/"$f".fix \
&& /bin/mv /tmp/"$f".fix "$f"
done
Warning
But as you're talking about 10,000+files, you'll want to do some testing before using either of these solutions. Copy a good random set of those files to /tmp/mySedTest/ dir and run one of these solutions there to make sure there are no surprises.
And you're likely to blow out the cmd-line MAX_SIZE with 10,000+ files, so read about find and xargs. There are many posts here about [sed] find xargs. Check them out if needed.
IHTH

Related

Why is my sed multiline find-and-replace not working as expected?

I have a simple sed command that I am using to replace everything between (and including) //thistest.com-- and --thistest.com with nothing (remove the block all together):
sudo sed -i "s#//thistest\.com--.*--thistest\.com##g" my.file
The contents of my.file are:
//thistest.com--
zone "awebsite.com" {
type master;
file "some.stuff.com.hosts";
};
//--thistest.com
As I am using # as my delimiter for the regex, I don't need to escape the / characters. I am also properly (I think) escaping the . in .com. So I don't see exactly what is failing.
Why isn't the entire block being replaced?
You have two problems:
Sed doesn't do multiline pattern matches—at least, not the way you're expecting it to. However, you can use multiline addresses as an alternative.
Depending on your version of sed, you may need to escape alternate delimiters, especially if you aren't using them solely as part of a substitution expression.
So, the following will work with your posted corpus in both GNU and BSD flavors:
sed '\#^//thistest\.com--#, \#^//--thistest\.com# d' /tmp/corpus
Note that in this version, we tell sed to match all lines between (and including) the two patterns. The opening delimiter of each address pattern is properly escaped. The command has also been changed to d for delete instead of s for substitute, and some whitespace was added for readability.
I've also chosen to anchor the address patterns to the start of each line. You may or may not find that helpful with this specific corpus, but it's generally wise to do so when you can, and doesn't seem to hurt your use case.
# separation by line with 1 s//
sed -n -e 'H;${x;s#^\(.\)\(.*\)\1//thistest.com--.*\1//--thistest.com#\2#;p}' YourFile
# separation by line with address pattern
sed -e '\#//thistest.com--#,\#//--thistest.com# d' YourFile
# separation only by char (could be CR, CR/LF, ";" or "oneline") with s//
sed -n -e '1h;1!H;${x;s#//thistest.com--.*\1//--thistest.com##;p}' YourFile
Note:
assuming there is only 1 section thistest per file (if not, it remove anything between the first opening until the last closing section) for the use of s//
does not suite for huge file (load entire file into memory) with s//
sed using addresses pattern cannot select section on the same line, it search 1st pattern to start, and a following line to stop but very efficient on big file and/or multisection

Insert specific lines from file before first occurrence of pattern using Sed

I want to insert a range of lines from a file, say something like 210,221r before the first occurrence of a pattern in a bunch of other files.
As I am clearly not a GNU sed expert, I cannot figure how to do this.
I tried
sed '0,/pattern/{210,221r file
}' bunch_of_files
But apparently file is read from line 210 to EOF.
Try this:
sed -r 's/(FIND_ME)/PUT_BEFORE\1/' test.text
-r enables extendend regular expressions
the string you are looking for ("FIND_ME") is inside parentheses, which creates a capture group
\1 puts the captured text into the replacement.
About your second question: You can read the replacement from a file like this*:
sed -r 's/(FIND_ME)/`cat REPLACEMENT.TXT`\1/' test.text
If replace special characters inside REPLACEMENT.TXT beforehand with sed you are golden.
*= this depends on your terminal emulator. It works in bash.
In https://stackoverflow.com/a/11246712/4328188 CodeGnome gave some "sed black magic" :
In order to insert text before a pattern, you need to swap the pattern space into the hold space before reading in the file. For example:
sed '/pattern/ {
h
r file
g
N
}' in
However, to read specific lines from file, one may have to use a two-calls solution similar to dummy's answer. I'd enjoy knowing of a one-call solution if it is possible though.

sed and perl not replacing a letter in a file

I have a file 1.htm. I want to replace a letter ṣ (s with dot below). I tried with both sed and perl and it does not replace.
sed -i 's/ṣ/s/g' "1.htm"
perl -i -pe 's/ṣ/s/g' "1.htm"
can anyone suggest what to do
1.html (not replacing ṣ)
Also i have found another strange thing. Sed (same command as above) replaces in one file but not the other I am putting the links
replacable.html
unreplacable.html same as 1.html
Why is it happening so. sed is able to replace ṣ in one file but not the other.
You have combined characters in the html file. That is, the "ṣ" is really a "s" followed by a " ̣" (a COMBINING DOT BELOW). One possibility to fix the oneliner is:
perl -C -i -pe 's/s\x{0323}/s/g' "1.htm"
That is, turn utf8 mode for stdout/stdin on (-C) and explicitely write the two characters in the left side of the s///.
Another possibility is to normalize all the combining characters using Unicode::Normalize, e.g.:
perl -C -MUnicode::Normalize -Mutf8 -i -pe '$_=NFKC($_); s/ṣ/s/g' "1.htm"
But this would also normalize all the other characters in the input file, which may or may not be OK for you.
This might work for you (GNU sed):
sed 's/\o341\o271\o243/s/g' file
To find seds octal interpretation of a character use:
echo 'ṣ'| sed l
This returns (for me):
\341\271\243$
ṣ
Then use \onnn (or combinations of) to find the correct pattern in the lefthandside (LFH) of the substitute command.
N.B. \onnn may also be used in the RHS of the substitute command.

Remove a hyphen from a specific line in a file

I have a data file that needs to have several uniq identifiers stripped of hyphens.
So I have:
(Special_Section "data-values")
and I want to have it replaced with:
(Special_Section "datavalues")
I wanted to use a simple sed find/replace, but the data and values are different each time. Preferably, I'd run this in-place since the file has a lot of other information I want to keep in tact.
Does sed or awk have a way to remove the hyphen from the matched portion only?
Currently I can match with: sed -i 's/Special_Section "[a-zA-Z0-9]*-[a-zA-Z0-9]*"/&/g *myfiles*
But I would like to then run s/-// on & if it's possible.
You seems to be using GNU sed, so something like this might work:
sed -ri '
s/(Special_Section [^-]*)-([^)]*)/\1\2/g
' <your_filename_glob>
does this work?
sed -i '/(Special_Section ".*-.*")/{s/-//}' yourFile
Close - scan for the lines and then substitute on those that match:
sed -i '/Special_Section "[a-zA-Z0-9]*-[a-zA-Z0-9]*"/s/\( "[a-zA-Z0-9]*\)-\([a-zA-Z0-9]*\)"/\1\2/' *myfiles*
You can split that over several lines to avoid the scroll bar in SO:
sed -i '/Special_Section "[a-zA-Z0-9]*-[a-zA-Z0-9]*"/{
s/\( "[a-zA-Z0-9]*\)-\([a-zA-Z0-9]*\)"/\1\2/
}' *myfiles*
And on further thoughts, you can also do:
sed -i 's/\(Special_Section "[a-zA-Z0-9]*\)-\([a-zA-Z0-9]*"\)/\1\2/' *myfiles*
This is more compact. You can add the g qualifier if you need it. Both solutions use the special \(...\) notation to capture parts of the regular expression.

Deleting multiline text from multiple files

I have a bunch of java files from which I want to remove the javadoc lines with the license [am changing it on my code].
The pattern I am looking for is
^\* \* ProjectName .* USA\.$
but matched across lines
Is there a way sed [or a commonly used editor in Windows/Linux] can do a search/replace for a multiline pattern?
Here's the appropriate reference point in my favorite sed tutorial.
Probably someone is still looking for such solution from time to time. Here is one.
Use awk to find the lines to be removed. Then use diff to remove the lines and let sed clean up.
awk "/^\* \* ProjectName /,/ USA\.$/" input.txt \
| diff - input.txt \
| sed -n -e"s/^> //p" \
>output.txt
A warning note: if the first pattern exist while the second does not, you will loose all text below the first pattern - so check that first.
Yes. Are you using sed, awk, perl, or something else to solve this problem?
Most regular expression tools allow you to specify multi-line patterns. Just be careful with regular expressions that are too greedy, or they'll match the code between comments if it exists.
Here's an example:
/\*(?:.|[\r\n])*?\*/
perl -0777ne 'print m!/\*(?:.|[\r\n])*?\*/!g;' <file>
Prints out all the comments run
together. The (?: notation must be
used for non-capturing parenthesis. /
does not have to be escaped because !
delimits the expression. -0777 is used
to enable slurp mode and -n enables
automatic reading.
(From: http://ostermiller.org/findcomment.html )