I'm struggling since a moment now to update some yaml files by adding an element in a new line with sed.
The sed command (or another linux command) must match a string (image: or - image:), add a new element on a new line with the same indentation as previous line.
The thing is that the new line must be exactly just under the string image: and not under - image:.
Example: When the sed matches the string image, it adds the new line with correct indentation (just above image)
kind: DaemonSet
apiVersion: apps/v1
metadata:
name: calico-node
spec:
template:
spec:
containers:
- name: calico-node
image: myimage
imagePullSecret: mysecret
...
Example 2: when the sed matches the string - image, it adds the new line with correct indentation (just above image and not -):
kind: DaemonSet
apiVersion: apps/v1
metadata:
name: calico-node
spec:
template:
spec:
containers:
- image: myimage
imagePullSecret: mysecret
...
What is not wanted is
kind: DaemonSet
apiVersion: apps/v1
metadata:
name: calico-node
spec:
template:
spec:
containers:
- image: myimage
imagePullSecret: mysecret
I already tried the yq command to deal with this but it's a gigantic pain...
I found this code in another thread but it doesn't work when matching the - image string.
sed '/^ *image: .*/ { G; s/^\( *\)image: .*/&\1imagePullSecret: mysecret/; }' sed.yaml
I don't condone the use of sed to parse yaml, but if you really want to do that, you could try something like:
$ nl='
'
$ printf ' image:\n - image:\n' |
sed -e "s/\(^\( *\)image:\)/\1\\$nl\2inserted text/" \
-e "s/\(^\( *\)- image:\)/\1\\$nl\2 inserted text/"
image:
inserted text
- image:
inserted text
I'd suggest:
sed '
/^( *)image: .*/ {p; s//\1imagePullSecret: mysecret/; }
/^( *)- image: .*/ {p; s//\1 imagePullSecret: mysecret/; }
' sed.yaml
The empty pattern in s//replacement/ reuses the most-recently used pattern.
You could use this awk:
awk '/^[[:space:]-]*image/{ split($0,arr,/image.*/)
gsub(/-/," ", arr[1])
rep=arr[1]
print $0}
rep{ printf "%s%s\n", rep, "imagePullSecret: mysecret"
rep=""
next} 1' file
Using GNU awk to utilise gensub:
awk '/image: /{ str=$0 } str { print;str1=gensub(/(^.*)(image:.*$)/,"\\1imagePullSecret: mysecret",str);sub("-"," ",str1);print str1;del str;next }1' file
Explantion:
awk '/image: /{ # Process the image tag
str=$0 # Set a variable str equal to the line
}
str { # Process when we have a str set
print; # Print the current line
str1=gensub(/(^.*)(image:.*$)/,"\\1imagePullSecret: mysecret",str); # Split the line into two sections and set str1 to the first section (get the leading spaces) and then the text we want to add.
sub("-"," ",str1); # Replace any - in the text with a space
print str1; # Print the new line
del str; # Remove the str variable
next # Skip to the next line
}1' file
Yq v4 doesn't yet support maps for addition, but you could convert to JSON, use jq, and then convert back to YAML:
yq --tojson eval infile.yaml \
| jq '.spec.template.spec.containers[0] += {imagePullSecret: "mysecret"}' \
| yq eval -prettyPrint
This might work for you (GNU sed):
sed -E '/(-( ))?image:.*/{p;s//\2\2imagePullSecret: mySecret/}' file
Match on image:.* with an optional - prepended.
Print the current line.
Replace the match by imagePullSecret: mySecret with the optional space following the - prepended twice.
Related
As the question title specifies , i have to replace a block to text in a file with a new block of text
I have searched all over for this thing but every solution i ever found was just too specific to the question. Isn't it possible to create a function which is flexible/reusable ?
To be very specific i need something which has options like
1) File ( where changes are to be done )
2) Exiting block of text
3) New block of text
( 2nd & 3 option could be either as manually pasted text or cat $somefile)
whereby i could change these 3 and use the script for all cases of text block replacement , i am sure it will help many other people too
As for an example , currently i need to replace the below block of text with one at bottom and say the file is $HOME/block.txt . Although i need the solution which is easily reusable/flexible as mentioned above
- name: Set default_volumes variable
set_fact:
default_volumes:
- "/opt/lidarr:/config"
- "/opt/scripts:/scripts"
- "/mnt:/mnt"
- "/mnt/unionfs/Media/Music:/music"
- name: Set default_volumes variable
set_fact:
default_volumes:
- "/opt/lidarr:/config"
- "/opt/scripts:/scripts"
- "/mnt:/mnt"
- "/mnt/unionfs/Media/Music:/music"
- "/mnt/unionfs/downloads/lidarr:/downloads-amd"
PS / while replacement i need the spacing and indentation to be preserved.
Your data is serialized using YAML. You should treat it as such.
Using yq
yq eval '
.[0].set_fact.default_volumes +=
[ "/mnt/unionfs/downloads/lidarr:/downloads-amd" ]
'
yq doesn't natively support in-place editing, but you can use sponge to achieve the same thing.
yq eval '
.[0].set_fact.default_volumes +=
[ "/mnt/unionfs/downloads/lidarr:/downloads-amd" ]
' a.yaml | sponge a.yaml
Using Perl
perl -MYAML -0777ne'
my $d = Load($_);
push #{ $d->[0]{set_fact}{default_volumes} },
"/mnt/unionfs/downloads/lidarr:/downloads-amd";
print Dump($d);
'
As per specifying file to process to Perl one-liner, editing in place would look like this:
perl -i -MYAML -0777ne'
my $d = Load($_);
push #{ $d->[0]{set_fact}{default_volumes} },
"/mnt/unionfs/downloads/lidarr:/downloads-amd";
print Dump($d);
' file.yaml
Using GNU awk for multi-char RS and ARGIND, this will work for any chars in your old or new text including regexp metachars, delimiters, quotes, and backreferences as it's just doing literal string search and replace:
awk -v RS='^$' -v ORS= '
ARGIND==1 { old=$0; next }
ARGIND==2 { new=$0; next }
s=index($0,old) {
$0 = substr($0,1,s-1) new substr($0,s+length(old))
}
1' old new file
or you can do the same using any awk in any shell on every Unix box with:
awk -v ORS= '
{ rec = (FNR>1 ? rec RS : "") $0 }
FILENAME==ARGV[1] { old=rec; next }
FILENAME==ARGV[2] { new=rec; next }
END {
$0 = rec
if ( s=index($0,old) ) {
$0 = substr($0,1,s-1) new substr($0,s+length(old))
}
print
}
' old new file
For example:
$ head old new file
==> old <==
- name: Set default_volumes variable
set_fact:
default_volumes:
- "/opt/lidarr:/config"
- "/opt/scripts:/scripts"
- "/mnt:/mnt"
- "/mnt/unionfs/Media/Music:/music"
==> new <==
- name: Set default_volumes variable
set_fact:
default_volumes:
- "/opt/lidarr:/config"
- "/opt/scripts:/scripts"
- "/mnt:/mnt"
- "/mnt/unionfs/Media/Music:/music"
- "/mnt/unionfs/downloads/lidarr:/downloads-amd"
==> file <==
foo
- name: Set default_volumes variable
set_fact:
default_volumes:
- "/opt/lidarr:/config"
- "/opt/scripts:/scripts"
- "/mnt:/mnt"
- "/mnt/unionfs/Media/Music:/music"
bar
$ awk -v RS='^$' -v ORS= 'ARGIND==1{old=$0; next} ARGIND==2{new=$0; next} s=index($0,old){ $0=substr($0,1,s-1) new substr($0,s+length(old))} 1' old new file
foo
- name: Set default_volumes variable
set_fact:
default_volumes:
- "/opt/lidarr:/config"
- "/opt/scripts:/scripts"
- "/mnt:/mnt"
- "/mnt/unionfs/Media/Music:/music"
- "/mnt/unionfs/downloads/lidarr:/downloads-amd"
bar
For a task like this, you could just use existing commands rather than
reinventing the wheel:
sed '/some text to change/,/with indentation/d; /a bit more/r new_file' your_file
I used two example files:
# original file
some original text to keep
a bit more
some text to remove
- with indentation
rest of original text
is kept
and:
# replacement text
SOME TEXT TO ADD
- WITH DIFFERENT INDENTATION
- ANOTHER LEVEL
Then the command works by first deleting lines between and including two
lines matching patterns:
sed '/some text to change/,/with indentation/d;
Then reading the replacement text from some other file, using a pattern
matching just where the old text used to start:
/a bit more/r new_file' your_file
To yield the result:
some original text to keep
a bit more
SOME TEXT TO ADD
- WITH DIFFERENT INDENTATION
- ANOTHER LEVEL
rest of original text
is kept
Edit
The above is better than my original way:
sed '/a bit more/q' your_file > composite; cat new_file >> composite; sed -n '/rest of original text/,/$/p' your_file >> composite
I have file with this text:
mirrors:
docker.io:
endpoint:
- "http://registry:5000"
registry:5000:
endpoint:
- "http://registry:5000"
localhost:
endpoint:
- "http://registry:5000"
I need to replace it with this text in POSIX shell script (not bash):
mirrors:
docker.io:
endpoint:
- "http://docker.io"
registry:5000:
endpoint:
- "http://registry:5000"
localhost:
endpoint:
- "http://localhost"
Replace should be done dynamically in all places without hard-coded names. I mean we should take sub-string from a first line ("docker.io", "registry:5000", "localhost") and replace with it sub-string "registry:5000" in a third line.
I've figure out regex, that splits it on 5 groups: (^ )([^ ]*)(:[^"]*"http:\/\/)([^"]*)(")
Then I've tried to use sed to print group 2 instead of 4, but this didn't work: sed -n 's/\(^ \)\([^ ]*\)\(:[^"]*"http:\/\/\)\([^"]*\)\("\)/\1\2\3\2\5/p'
Please help!
This might work for you (GNU sed):
sed -E '1N;N;/\n.*endpoint:.*\n/s#((\S+):.*"http://)[^"]*#\1\2#;P;D' file
Open up a three line window into the file.
If the second line contains endpoint:, replace the last piece of text following http:// with the first piece of text before :
Print/Delete the first line of the window and then replenish the three line window by appending the next line.
Repeat until the end of the file.
Awk would be a better candidate for this, passing in the string to change to as a variable str and the section to change (" docker.io" or " localhost" or " registry:5000") and so:
awk -v findstr=" docker.io" -v str="http://docker.io" '
$0 ~ findstr { dockfound=1 # We have found the section passed in findstr and so we set the dockfound marker
}
/endpoint/ && dockfound==1 { # We encounter endpoint after the dockfound marker is set and so we set the found marker
found=1;
print;
next
}
found==1 && dockfound==1 { # We know from the found and the dockfound markers being set that we need to process this line
match($0,/^[[:space:]]+-[[:space:]]"/); # Match the start of the line to the beginning quote
$0=substr($0,RSTART,RLENGTH)str"\""; # Print the matched section followed by the replacement string (str) and the closing quote
found=0; # Reset the markers
dockfound=0
}1' file
One liner:
awk -v findstr=" docker.io" -v str="http://docker.io" '$0 ~ findstr { dockfound=1 } /endpoint/ && dockfound==1 { found=1;print;next } found==1 && dockfound==1 { match($0,/^[[:space:]]+-[[:space:]]"/);$0=substr($0,RSTART,RLENGTH)str"\"";found=0;dockfound=0 }1' file
I'm new to unix programming like sed, perl etc. I've searched and no result found match my case. I need to append substring from top line in the same file. My file content text.txt :
Name: sur.name.custom
Tel: xxx
Address: yyy
Website: www.site.com/id=
Name: sur.name.custom1
Tel: xxx
Address: yyy
Website: www.site.com/id=
I need to append every Name (sur.name.*) to every website on its block.
So Expected ouput:
Name: sur.name.custom
Tel: xxx
Address: yyy
Website: www.site.com/id=sur.name.custom
Name: sur.name.custom1
Tel: xxx
Address: yyy
Website: www.site.com/id=sur.name.custom1
I've tried the following sed command:
sed -n "/^Website:.*id=$/ s/$/sur.name..*/p" ./text.txt;
But sed returned: Website: www.site.com/id=sur.name.* same string I put.
I'm sure sed can append from regex pattern. I need both sed and perl if possible.
Why don't you use awk for this? Assuming names doesn't contain spaces following command should work:
awk '$1=="Name:"{name=$2} $1=="Website:"{print $0 name;next} 1' file
Perl equivalent:
perl -pale'
$F[0] eq "Name:" and $name = $F[1];
$F[0] eq "Website:" and $_ .= $name;
' file
(Line breaks may be removed.)
Here' a sed solution:
sed '/^Name:/{h;s/Name: *//;x;};/^Website:/{G;s/\n//;}' filename
Translation: If the line begins with Name:, save the name to the hold space; if the line starts with Website:, append the (latest) name from the holdspace.
I have a script that output about 50 lines and there are about five lines that match when I am trying to replace only one of them.
Example:
...
metadata
name: A
...
spec:
- labels:
...
name: B
name: C
- labels:
...
name: D
name: E
I am trying to use sed to replace the entire of "name: B" to "name: {firstVar}" and "name: D" to "name: {secondVar}". I know that I can just search for the whole line "name: B" or "name: D" but these aren't always going to be the same and can be changed by others. The only thing that's consistent is their placement. So, I am looking to be able to replace the 2nd and 4th match but anytime I try "s/name:.*/name: {firstVar}/2" or something similar it doesn't work or it will replace all the matches.
Any help would be greatly appreciated.
This might work for you (GNU sed):
second="firstVar" fourth="secondVar"
sed -z 's/\(name:\s*\)[^\n]*/\1'"$second"'/2;s//\1'"$fourth"'/4' file
Set variables in the shell and then use them as replacements in the pattern matching substitutions. The positive integer flag in the substitution command replaces that number pattern in the pattern space. The -z option, slurps the whole of the the file into memory and thus the pattern space contains the whole of the file.
Here is one in awk:
$ awk '
BEGIN {
a[2]="firstVar" # define the pairs
a[4]="secondVar"
}
$1=="name:" { # search for match
sub($1 " " $2,$1 " " ((++i in a)?a[i]:$2),$0) # replace
}1' file
Output:
...
metadata
name: A
...
spec:
- labels:
...
name: firstVar
name: C
- labels:
...
name: secondVar
name: E
I have one data file and one reg template file:
data file contain:
c01218 172.20.13.50
c01203 172.20.13.35
c01204 172.20.13.36
c01220 172.20.13.52
c01230 172.20.13.55
reg template:
[HKEY_USERS\S-1-5-21-2000478354-2111687655-1801674531-230160\Software\SimonTatham\PuTTY\Sessions\name]
"Present"=dword:00000001
"HostName"="172.28.130.0"
I want to create loop which to create new reg files from the template with the name from the first colum and to change "name" located in HKEY_USERS also with the first column and to change the IP address with the second column.
For example:
sed -e "s/name/name1/g" -e "s/172.28.130.0/172.28.130.1/g" 1.reg
Expected view after the command:
#cat c01218.reg
[HKEY_USERS\S-1-5-21-2000478354-2111687655-1801674531-230160\Software\SimonTatham\PuTTY\Sessions\c01218]
"Present"=dword:00000001
"HostName"="172.20.13.50"
sed is an excellent tool for simple substitutions on a single line, for anything else just use awk:
awk '{ printf "[HKEY_USERS\\S-1-5-21-2000478354-2111687655-1801674531-230160\\Software\\SimonTatham\\PuTTY\\Sessions\\name]\n\"Present\"=dword:00000001\n\"HostName\"=\"%s\"\n", $2 > $1 }' data
or if you prefer:
awk -v template="\
[HKEY_USERS\\S-1-5-21-2000478354-2111687655-1801674531-230160\\Software\\SimonTatham\\PuTTY\\Sessions\\name]
\"Present\"=dword:00000001
\"HostName\"=\"%s\"
" '{ printf template, $2 > $1 }' data
Try:
$ while read a b; do sed "s/^\"HostName.*$/\"HostName\"=\"$b\"/" template > $a; done < data
A little messy since " has to be used for the shell to expand the variables in the sed-substitution and all additional " need to be escaped.
output:
$ ls
c01203 c01204 c01218 c01220 c01230 data template
$ cat c*
[HKEY_USERS\S-1-5-21-2000478354-2111687655-1801674531-230160\Software\SimonTatham\PuTTY\Sessions\name]
"Present"=dword:00000001
"HostName"="172.20.13.35"
[HKEY_USERS\S-1-5-21-2000478354-2111687655-1801674531-230160\Software\SimonTatham\PuTTY\Sessions\name]
"Present"=dword:00000001
"HostName"="172.20.13.36"
[HKEY_USERS\S-1-5-21-2000478354-2111687655-1801674531-230160\Software\SimonTatham\PuTTY\Sessions\name]
"Present"=dword:00000001
"HostName"="172.20.13.50"
[HKEY_USERS\S-1-5-21-2000478354-2111687655-1801674531-230160\Software\SimonTatham\PuTTY\Sessions\name]
"Present"=dword:00000001
"HostName"="172.20.13.52"
[HKEY_USERS\S-1-5-21-2000478354-2111687655-1801674531-230160\Software\SimonTatham\PuTTY\Sessions\name]
"Present"=dword:00000001
"HostName"="172.20.13.55"