I'm trying to do an inline replacement using sed in an EC2 Amazon instance running their standard Ubuntu flavor but I can't seem to figure out the right syntax.
Basically I need to edit the java.security file in a couple locations so I'm using find + sed to do it and it works perfectly as expected but then, when the tests are completed and I do the cleanup, I want to return it to the expected default values I want and that's where I'm running intro trouble.
This is how my stock file looks like:
:/etc/java-8-oracle/security# cat java.security | grep jdk.tls.disabledAlgorithms
# jdk.tls.disabledAlgorithms=MD5, SSLv3, DSA, RSA keySize < 2048
jdk.tls.disabledAlgorithms=SSLv3, RC4, MD5withRSA, DH keySize < 768
And I want to replace that second line with another value (not append).
If I do this:
find / -name java.security -type f -exec sed -i 's/jdk.tls.disabledAlgorithms=*/jdk.tls.disabledAlgorithms=SSLv3, RC4, MD5withRSA, DH keySize < 768/g' {} +
It will just append it so I'll end up with an increasingly longer and useless string every time it runs.
but if I do this:
find / -name java.security -type f -exec sed -i '/jdk.tls.disabledAlgorithms=*/c\jdk.tls.disabledAlgorithms=SSLv3, RC4, MD5withRSA, DH keySize < 768' {} +
It replaces both the string I want and the commented example which leaves me with 2 identical and uncommented lines.
I'm using find because I need to edit 3 files in different locations at the same time. I'm narrowing it down to / now just for testing and trial/error purposes.
Anyone knows where I'm screwing up or what I'm missing?.
Thanks!.
You've already gotten a working solution in the comment: refine your regex to only match entries starting with jdk.tls.disabledAlgorithms. I've still opted to post an answer since the question you've asked (or at least its title) was slightly different and I felt compelled to elaborate on that in case you were still curious and also someone else landed her searching for an answer how to skip a specific line. So here we go:
In sed commands like s can be preceded by an address or address range which they apply to, so while this:
$ sed -i 's#\(jdk.tls.disabledAlgorithms=\).*#\1NEWVALUES < 768#g' FILE
Would replace each occurrence of jdk.tls.disabledAlgorithms with NEWVALUES.
$ sed -i '/^#/!s#\(jdk.tls.disabledAlgorithms=\).*#\1NEWLIST < 768#g' FILE
Would not do that for any lines whose first character is #. /^#/ matches such lines and ! is a logical not. Only execute the command for lines not matching.
I've also used back reference (to make the line shorter and reduce risk of typos, also during maintenance). And I used # instead of / since I try to go for more visually distinctive separator especially if I have other (back)slashes around. If you wanted to be consistent with that across the board, you could replace the initial /^#/ with \#^##.
Generally though it is indeed advisable to write your regular expressions to match accurately exactly what you want to match.
Related
I have a csv file and in a number of fields there are floating point numbers where the negative at the end of the number, I want these to be altered so as to have the minus/negative symbol at the front.
ie
23.4954-,23.12-
0.23-,16.5453
2495.1-,12,134-
I would like those to read
-23.4954,-23.12
-0.23,16.5453
-2495.1,-12,134
Out of 20 columns and a few thousand rows, there are probably about 80 instances per file, but it is a real pain to just go and replace them
I was hoping to use sed on the files to alter them if possible.
Any help is really appreciated.
Use the following approach:
sed -ri 's/([0-9.]+)-/-\1/g' filepath
-r (--regexp-extended) - allows using extended regular expressions rather than basic regular expressions
-i - option which allows to change file in place
You could use this sed command:
sed -E 's/([0-9.]+)-/-\1/g' file
This moves the - sign before a number composed of 0-9. characters.
I got 10,000 text files which I have to make changes.
First line on every file contains a url.
By mistake for few files url missking 'com'
eg:
1) http://www.supersonic./psychology
2) http://www.supersonic./social
3) http://www.supersonic.com/science
my task is to check and add 'com' if it is missing
eg:
1) http://www.supersonic.com/psychology
2) http://www.supersonic.com/social
3) http://www.supersonic.com/science
all urls are of same domain(supersonic.com)
can you suggest me any fast and easy approach ?
Tried this : replacing supersonic./ with supersonic.com
sed -e '1s/supersonic.//supersonic.com/' *
no change in the output.
Use -i to change the files instead of just outputting the changed lines.
Use a different delimiter than / if you want to use / in the regex (or use \/ in the regex).
Use \. to match a dot literally, . matches anything.
sed -i~ -e '1s=supersonic\./=supersonic.com/=' *
Some versions of sed don't support -i.
You are very close with your code, but you need to account for the trailing / char after the . char.
Assuming you are using a modern sed with the -i (inplace-edit) option you can do
sed -i '1s#supersonic\./#supersonic.com/#' *
Note that rather than have to escape / inside of the s/srchpat\/withSlash/replaceStr/', you can use another char after the the s command as the delimiter, here I use s#...#...#. If your search pattern had a # char, then you would have to use a different char.
Some older versions of sed need to you to escape the alternate delimiter at the first use, so
sed 's\#srchStr#ReplStr#' file
for those cases.
If you're using a sed that doesn't support the -i options, then
you'll need to loop on your file, and manage the tmp files, i.e.
for f in *.html ; do
sed '1s#supersonic\./#supersonic.com/#' "$f" > /tmp/"$f".fix \
&& /bin/mv /tmp/"$f".fix "$f"
done
Warning
But as you're talking about 10,000+files, you'll want to do some testing before using either of these solutions. Copy a good random set of those files to /tmp/mySedTest/ dir and run one of these solutions there to make sure there are no surprises.
And you're likely to blow out the cmd-line MAX_SIZE with 10,000+ files, so read about find and xargs. There are many posts here about [sed] find xargs. Check them out if needed.
IHTH
How can I use the bash sed command to change this string:
<Directory /var/www/>
Options Indexes FollowSymLinks
AllowOverride None
Require all granted
</Directory>
into the following string? (only changing the 3rd line of string)
<Directory /var/www/>
Options Indexes FollowSymLinks
AllowOverride All
Require all granted
</Directory>
NOTE 1: I don't just want to target the string 'AllowOverride None' because there are other occurrences in the file that should not be changed. I need to target the entire string starting with <Directory /var/www>
NOTE 2: I also need to overwrite the file. So, take that into account in your answer. And provide different versions for GNU/non-GNU versions of sed just in case.
Since the patterns contain slashes, use \% (for any character %) to mark the search patterns. Then use:
sed -e '\%^<Directory /var/www/>%,\%^</Directory>% s/AllowOverride None/AllowOverride All/'
The search patterns inside \%…% limit the search to lines between the matching patterns, and the { s/…/…/; } looks for the desired pattern within the range and makes the appropriate replacement.
If you don't want to restrict it to a single directory section but to all directory sections, adjust the start pattern appropriately. For example, this will match any <Directory> section:
sed -e '\%^<Directory [^>]*>%,\%^</Directory>% s/AllowOverride None/AllowOverride All/'
You can make it more selective depending on your requirements.
The simple version, relying on the AllowOverride line coming within two lines after <Directory...> and using a GNU sed extension, is this:
sed '/^<Directory/,+2 { s/AllowOverride None/AllowOverride All/g; }'
UPDATE: Here is the version not relying on any GNU extension (I tried it first, but made a typo and was surprised that it didn't work, that's why a posted the other version first):
sed '/^<Directory/,/^<\/Directory>/ { s/AllowOverride None/AllowOverride All/; }'
I realize this is not what you asked by maybe its worth not using sed?
How about a python solution? It walks directory passed as first parameter to script and replaces exactly <Directory element as you wrote it while only changing None to All and writes changes back to the file. It will also work with different indentation levels while preserving original indentation. Works on both python2 and python3.
After all i am assuming if you have sed you probably have python too.
#!/usr/bin/env python
import re
r = re.compile(r'(<Directory /var/www/>\s+Options Indexes FollowSymLinks\s+AllowOverride )None(\s+Require all granted\s+</Directory>)', re.MULTILINE)
for root, dirs, files in os.walk(sys.argv[1]):
for file_name in files:
if file_name.endswith('.conf'):
file_path = os.path.join(root, file_name)
with open(file_path) as fp:
data = r.sub(r'\1All\2', fp.read())
with open(file_path, 'w+') as fp:
fp.write(data)
Using Gnu Sed:
sed -zie 's!\(<Directory /var/www/>[^<]*AllowOverride\) None!\1 All!' ex1.txt
Option -z is for Null separated records: all the file is one record,
so just make a simple substitution.
[^<]* (multiline) regular expression respects Directory boundaries, and allows flexible format and order.
Your question is a good illustration of the mantra, don't use sed. Really, you shouldn't use any regex engine for context-free language like XML. But you can get close, maybe close enough, with awk.
#! /usr/bin/awk -f
/<Directory \/var\/www\/>/ {
line = NR
}
/ AllowOverride None/ && line + 2 == NR {
gsub( /None/, "All" )
}
{ print }
That way you don't have any fancy, nonstandard regex to read, and your code says exactly what it means: If you find "AllowOverride" 2 lines after the "Directory" line, replace it. The above regexes are both very simple (and Posix compliant) and should work with any version of awk.
Your answer is already given by this user just check here.
Some Reference
In the simplest calling of sed, it has one line of text in the pattern space, ie. 1 line of \n delimited text from the input. The single line in the pattern space has no \n... That's why your regex is not finding anything.
You can read multiple lines into the pattern-space and manipulate things surprisingly well, but with a more than normal effort.. Sed has a set of commands which allow this type of thing... Here is a link to a Command Summary for sed. It is the best one I've found, and got me rolling.
However forget the "one-liner" idea once you start using sed's micro-commands. It is useful to lay it out like a structured program until you get the feel of it... It is surprisingly simple, and equally unusual. You could think of it as the "assembler language" of text editing.
Summary: Use sed for simple things, and maybe a bit more, but in general, when it gets beyond working with a single line, most people prefer something else...
I'll let someone else suggest something else.. I'm really not sure what the best choice would be (I'd use sed, but that's because I don't know perl well enough.)
sed '/^a test$/{
$!{ N # append the next line when not on the last line
s/^a test\nPlease do not$/not a test\nBe/
# now test for a successful substitution, otherwise
#+ unpaired "a test" lines would be mis-handled
t sub-yes # branch_on_substitute (goto label :sub-yes)
:sub-not # a label (not essential; here to self document)
# if no substituion, print only the first line
P # pattern_first_line_print
D # pattern_ltrunc(line+nl)_top/cycle
:sub-yes # a label (the goto target of the 't' branch)
# fall through to final auto-pattern_print (2 lines)
}
}' alpha.txt
Here it is the same script, condensed into what is obviously harder to read and work with, but some would dubiously call a one-liner
sed '/^a test$/{$!{N;s/^a test\nPlease do not$/not a test\nBe/;ty;P;D;:y}}' alpha.txt
Here is my command "cheat-sheet"
: # label
= # line_number
a # append_text_to_stdout_after_flush
b # branch_unconditional
c # range_change
d # pattern_delete_top/cycle
D # pattern_ltrunc(line+nl)_top/cycle
g # pattern=hold
G # pattern+=nl+hold
h # hold=pattern
H # hold+=nl+pattern
i # insert_text_to_stdout_now
l # pattern_list
n # pattern_flush=nextline_continue
N # pattern+=nl+nextline
p # pattern_print
P # pattern_first_line_print
q # flush_quit
r # append_file_to_stdout_after_flush
s # substitute
t # branch_on_substitute
w # append_pattern_to_file_now
x # swap_pattern_and_hold
y # transform_chars
Please bear with me as I'm new to the forums and tried to do my research before posting this. What I'm trying to do is to use sed to look through multiple lines of a file and any line that contains the words 'CPU Usage" I want it to comment out that line and also 19 lines immediately after that.
Example file.txt
This is some random text CPU USAGE more random text
Line2
Line3
Line4
Line5
etc.
I want sed to find the string of text CPU usage and comment out the line and the 19 lines following
#This is some random text CPU USAGE more random text
#Line2
#Line3
#Line4
#Line5
#etc.
This is what I've been trying but obviously it is not working since I'm posting on here asking for help
sed '/\/(CPU Usage)s/^/#/+18 > File_name
sed: -e expression #1, char 17: unknown command: `^'
I'd like to be able to use this on multiple files. Any help you can provide is much appreciated!
GNU sed has a non-standard extension (okay, it has many non-standard extensions, but there's one that's relevant here) of permitting /pattern/,+N to mean from the line matching pattern to that line plus N.
I'm not quite sure what you expected your sed command to do with the \/ part of the pattern, and you're missing a single quote in what you show, but this does the trick:
sed '/CPU Usage/,+19 s/^/#/'
If you want to overwrite the original files, add -i .bak (or just -i if you don't mind losing your originals).
If you don't have GNU sed, now might be a good time to install it.
This can easily be done with awk
awk '/CPU Usage/ {f=20} f && f-- {$0="#"$0}1' file
When CPU Usage is found, set flag f=20
If flag f is true, decrements until 0 and for every time, add # in front of the line and print it.
Think this should work, cant test it, if anyone finds something wrong just let me know :)
awk '/CPU Usage/{t=1}t{x++;$0="#"$0}x==19{t=0;x=0}1' file
I'm trying to extract a list of CentOS domain names only from http://mirrorlist.centos.org/?release=6.4&arch=x86_64&repo=os
Truncating prefix "http://" and "ftp://" to the first "/" character only resulting a list of
yum.phx.singlehop.com
mirror.nyi.net
bay.uchicago.edu
centos.mirror.constant.com
mirror.teklinks.com
centos.mirror.netriplex.com
centos.someimage.com
mirror.sanctuaryhost.com
mirrors.cat.pdx.edu
mirrors.tummy.com
I searched stackoverflow for the sed method but I'm still having trouble.
I tried doing this with sed
curl "http://mirrorlist.centos.org/?release=6.4&arch=x86_64&repo=os" | sed '/:\/\//,/\//p'
but doesn't look like it is doing anything. Can you give me some advice?
Here you go:
curl "http://mirrorlist.centos.org/?release=6.4&arch=x86_64&repo=os" | sed -e 's?.*://??' -e 's?/.*??'
Your sed was completely wrong:
/x/,/y/ is a range. It selects multiple lines, from a line matching /x/ until a line matching /y/
The p command prints the selected range
Since all lines match both the start and end pattern you used, you effectively selected all lines. And, since sed echoes the input by default, the p command results in duplicated lines (all lines printed twice).
In my fix:
I used s??? instead of s/// because this way I didn't need to escape all the / in the patterns, so it's a bit more readable this way
I used two expressions with the -e flag:
s?.*://?? matches everything up until :// and replaces it with nothing
s?/.*?? matches everything from / until the end replaces it with nothing
The two expressions are executed in the given order
In modern versions of sed you can omit -e and separate the two expressions with ;. I stick to using -e because it's more portable.