I have a script that output about 50 lines and there are about five lines that match when I am trying to replace only one of them.
Example:
...
metadata
name: A
...
spec:
- labels:
...
name: B
name: C
- labels:
...
name: D
name: E
I am trying to use sed to replace the entire of "name: B" to "name: {firstVar}" and "name: D" to "name: {secondVar}". I know that I can just search for the whole line "name: B" or "name: D" but these aren't always going to be the same and can be changed by others. The only thing that's consistent is their placement. So, I am looking to be able to replace the 2nd and 4th match but anytime I try "s/name:.*/name: {firstVar}/2" or something similar it doesn't work or it will replace all the matches.
Any help would be greatly appreciated.
This might work for you (GNU sed):
second="firstVar" fourth="secondVar"
sed -z 's/\(name:\s*\)[^\n]*/\1'"$second"'/2;s//\1'"$fourth"'/4' file
Set variables in the shell and then use them as replacements in the pattern matching substitutions. The positive integer flag in the substitution command replaces that number pattern in the pattern space. The -z option, slurps the whole of the the file into memory and thus the pattern space contains the whole of the file.
Here is one in awk:
$ awk '
BEGIN {
a[2]="firstVar" # define the pairs
a[4]="secondVar"
}
$1=="name:" { # search for match
sub($1 " " $2,$1 " " ((++i in a)?a[i]:$2),$0) # replace
}1' file
Output:
...
metadata
name: A
...
spec:
- labels:
...
name: firstVar
name: C
- labels:
...
name: secondVar
name: E
Related
As the question title specifies , i have to replace a block to text in a file with a new block of text
I have searched all over for this thing but every solution i ever found was just too specific to the question. Isn't it possible to create a function which is flexible/reusable ?
To be very specific i need something which has options like
1) File ( where changes are to be done )
2) Exiting block of text
3) New block of text
( 2nd & 3 option could be either as manually pasted text or cat $somefile)
whereby i could change these 3 and use the script for all cases of text block replacement , i am sure it will help many other people too
As for an example , currently i need to replace the below block of text with one at bottom and say the file is $HOME/block.txt . Although i need the solution which is easily reusable/flexible as mentioned above
- name: Set default_volumes variable
set_fact:
default_volumes:
- "/opt/lidarr:/config"
- "/opt/scripts:/scripts"
- "/mnt:/mnt"
- "/mnt/unionfs/Media/Music:/music"
- name: Set default_volumes variable
set_fact:
default_volumes:
- "/opt/lidarr:/config"
- "/opt/scripts:/scripts"
- "/mnt:/mnt"
- "/mnt/unionfs/Media/Music:/music"
- "/mnt/unionfs/downloads/lidarr:/downloads-amd"
PS / while replacement i need the spacing and indentation to be preserved.
Your data is serialized using YAML. You should treat it as such.
Using yq
yq eval '
.[0].set_fact.default_volumes +=
[ "/mnt/unionfs/downloads/lidarr:/downloads-amd" ]
'
yq doesn't natively support in-place editing, but you can use sponge to achieve the same thing.
yq eval '
.[0].set_fact.default_volumes +=
[ "/mnt/unionfs/downloads/lidarr:/downloads-amd" ]
' a.yaml | sponge a.yaml
Using Perl
perl -MYAML -0777ne'
my $d = Load($_);
push #{ $d->[0]{set_fact}{default_volumes} },
"/mnt/unionfs/downloads/lidarr:/downloads-amd";
print Dump($d);
'
As per specifying file to process to Perl one-liner, editing in place would look like this:
perl -i -MYAML -0777ne'
my $d = Load($_);
push #{ $d->[0]{set_fact}{default_volumes} },
"/mnt/unionfs/downloads/lidarr:/downloads-amd";
print Dump($d);
' file.yaml
Using GNU awk for multi-char RS and ARGIND, this will work for any chars in your old or new text including regexp metachars, delimiters, quotes, and backreferences as it's just doing literal string search and replace:
awk -v RS='^$' -v ORS= '
ARGIND==1 { old=$0; next }
ARGIND==2 { new=$0; next }
s=index($0,old) {
$0 = substr($0,1,s-1) new substr($0,s+length(old))
}
1' old new file
or you can do the same using any awk in any shell on every Unix box with:
awk -v ORS= '
{ rec = (FNR>1 ? rec RS : "") $0 }
FILENAME==ARGV[1] { old=rec; next }
FILENAME==ARGV[2] { new=rec; next }
END {
$0 = rec
if ( s=index($0,old) ) {
$0 = substr($0,1,s-1) new substr($0,s+length(old))
}
print
}
' old new file
For example:
$ head old new file
==> old <==
- name: Set default_volumes variable
set_fact:
default_volumes:
- "/opt/lidarr:/config"
- "/opt/scripts:/scripts"
- "/mnt:/mnt"
- "/mnt/unionfs/Media/Music:/music"
==> new <==
- name: Set default_volumes variable
set_fact:
default_volumes:
- "/opt/lidarr:/config"
- "/opt/scripts:/scripts"
- "/mnt:/mnt"
- "/mnt/unionfs/Media/Music:/music"
- "/mnt/unionfs/downloads/lidarr:/downloads-amd"
==> file <==
foo
- name: Set default_volumes variable
set_fact:
default_volumes:
- "/opt/lidarr:/config"
- "/opt/scripts:/scripts"
- "/mnt:/mnt"
- "/mnt/unionfs/Media/Music:/music"
bar
$ awk -v RS='^$' -v ORS= 'ARGIND==1{old=$0; next} ARGIND==2{new=$0; next} s=index($0,old){ $0=substr($0,1,s-1) new substr($0,s+length(old))} 1' old new file
foo
- name: Set default_volumes variable
set_fact:
default_volumes:
- "/opt/lidarr:/config"
- "/opt/scripts:/scripts"
- "/mnt:/mnt"
- "/mnt/unionfs/Media/Music:/music"
- "/mnt/unionfs/downloads/lidarr:/downloads-amd"
bar
For a task like this, you could just use existing commands rather than
reinventing the wheel:
sed '/some text to change/,/with indentation/d; /a bit more/r new_file' your_file
I used two example files:
# original file
some original text to keep
a bit more
some text to remove
- with indentation
rest of original text
is kept
and:
# replacement text
SOME TEXT TO ADD
- WITH DIFFERENT INDENTATION
- ANOTHER LEVEL
Then the command works by first deleting lines between and including two
lines matching patterns:
sed '/some text to change/,/with indentation/d;
Then reading the replacement text from some other file, using a pattern
matching just where the old text used to start:
/a bit more/r new_file' your_file
To yield the result:
some original text to keep
a bit more
SOME TEXT TO ADD
- WITH DIFFERENT INDENTATION
- ANOTHER LEVEL
rest of original text
is kept
Edit
The above is better than my original way:
sed '/a bit more/q' your_file > composite; cat new_file >> composite; sed -n '/rest of original text/,/$/p' your_file >> composite
I have a bunch of . pdf file names. For example:
901201_HKW_RNT_HW21_136_137_DE_442_Freigabe_DE_CLX.pdf
and i am trying to remove everything but this pattern XXX_XXX where X is always a digit.
The result should be:
136_137
So far i did the opposite .. manage to match the pattern by using :
set NoSpacesString to do shell script "echo " & quoted form of insideName & " | sed 's/([0-9][0-9][0-9]_[0-9][0-9][0-9])//'"
My goal is to set NoSpaceString to 136_137
Little bit of help please.
Thank you !
P.S. The rest of the code is in AppleScript if this matters
Fixing sed command...
You can use
sed -n 's/.*\([0-9]\{3\}_[0-9]\{3\}\).*/\1/p'
See the online demo
Details
-n - suppresses the default line output
s/.*\([0-9]\{3\}_[0-9]\{3\}\).*/\1/ - finds the .*\([0-9]\{3\}_[0-9]\{3\}\).* pattern that matches
.* - any zero or more chars
\([0-9]\{3\}_[0-9]\{3\}\) - Group 1 (the \1 in the RHS refers to this group value): three digits, _, three digits
.* - any zero or more chars
p - prints the result of the substitution only.
The regex above is a POSIX BRE compliant pattern. The same can be written in POSIX ERE:
sed -En 's/.*([0-9]{3}_[0-9]{3}).*/\1/p'
Final AppleScript code
set noSpacesString to do shell script "sed -En 's/.*([0-9]{3}_[0-9]{3}).*/\\1/p' <<<" & insideName's quoted form
This might work for you (GNU sed):
sed -E '/\n/{P;D};s/[0-9]{3}_[0-9]{3}/\n&\n/;D' file
This solution will print all occurrences of the pattern on a separate line.
The initial command is dependant on what follows.
The second command replaces the desired pattern prepending and appending newlines either side.
The D command removes up to the first newline, but as the pattern space is not empty, restarts the sed cycle (without append the next line).
Now the initial command comes into play. The front of the line is printed and then deleted along with its appended newline.
Again, the sed cycle is restarted as if the line had never been presented but minus any characters up to and including the first desired pattern.
This flip-flop pattern of control is repeated until nothing is left and then repeated on subsequent lines until the end of the file.
Here is a copy of the debug log for a suitable one line input containing two representations of the desired pattern:
SED PROGRAM:
/\n/ {
P
D
}
s/[0-9]{3}_[0-9]{3}/
&
/
D
INPUT: 'file' line 1
PATTERN: aaa123_456bbb123_456ccc
COMMAND: /\n/ {
COMMAND: }
COMMAND: s/[0-9]{3}_[0-9]{3}/
&
/
MATCHED REGEX REGISTERS
regex[0] = 3-10 '123_456'
PATTERN: aaa\n123_456\nbbb123_456ccc
MATCHED REGEX REGISTERS
regex[0] = 0-3 'aaa'
PATTERN: \n123_456\nbbb123_456ccc
COMMAND: D
PATTERN: 123_456\nbbb123_456ccc
COMMAND: /\n/ {
COMMAND: P
123_456
COMMAND: D
PATTERN: bbb123_456ccc
COMMAND: /\n/ {
COMMAND: }
COMMAND: s/[0-9]{3}_[0-9]{3}/
&
/
MATCHED REGEX REGISTERS
regex[0] = 3-10 '123_456'
PATTERN: bbb\n123_456\nccc
MATCHED REGEX REGISTERS
regex[0] = 0-3 'bbb'
PATTERN: \n123_456\nccc
COMMAND: D
PATTERN: 123_456\nccc
COMMAND: /\n/ {
COMMAND: P
123_456
COMMAND: D
PATTERN: ccc
COMMAND: /\n/ {
COMMAND: }
COMMAND: s/[0-9]{3}_[0-9]{3}/
&
/
PATTERN: ccc
MATCHED REGEX REGISTERS
regex[0] = 0-3 'ccc'
PATTERN:
COMMAND: D
I'm struggling since a moment now to update some yaml files by adding an element in a new line with sed.
The sed command (or another linux command) must match a string (image: or - image:), add a new element on a new line with the same indentation as previous line.
The thing is that the new line must be exactly just under the string image: and not under - image:.
Example: When the sed matches the string image, it adds the new line with correct indentation (just above image)
kind: DaemonSet
apiVersion: apps/v1
metadata:
name: calico-node
spec:
template:
spec:
containers:
- name: calico-node
image: myimage
imagePullSecret: mysecret
...
Example 2: when the sed matches the string - image, it adds the new line with correct indentation (just above image and not -):
kind: DaemonSet
apiVersion: apps/v1
metadata:
name: calico-node
spec:
template:
spec:
containers:
- image: myimage
imagePullSecret: mysecret
...
What is not wanted is
kind: DaemonSet
apiVersion: apps/v1
metadata:
name: calico-node
spec:
template:
spec:
containers:
- image: myimage
imagePullSecret: mysecret
I already tried the yq command to deal with this but it's a gigantic pain...
I found this code in another thread but it doesn't work when matching the - image string.
sed '/^ *image: .*/ { G; s/^\( *\)image: .*/&\1imagePullSecret: mysecret/; }' sed.yaml
I don't condone the use of sed to parse yaml, but if you really want to do that, you could try something like:
$ nl='
'
$ printf ' image:\n - image:\n' |
sed -e "s/\(^\( *\)image:\)/\1\\$nl\2inserted text/" \
-e "s/\(^\( *\)- image:\)/\1\\$nl\2 inserted text/"
image:
inserted text
- image:
inserted text
I'd suggest:
sed '
/^( *)image: .*/ {p; s//\1imagePullSecret: mysecret/; }
/^( *)- image: .*/ {p; s//\1 imagePullSecret: mysecret/; }
' sed.yaml
The empty pattern in s//replacement/ reuses the most-recently used pattern.
You could use this awk:
awk '/^[[:space:]-]*image/{ split($0,arr,/image.*/)
gsub(/-/," ", arr[1])
rep=arr[1]
print $0}
rep{ printf "%s%s\n", rep, "imagePullSecret: mysecret"
rep=""
next} 1' file
Using GNU awk to utilise gensub:
awk '/image: /{ str=$0 } str { print;str1=gensub(/(^.*)(image:.*$)/,"\\1imagePullSecret: mysecret",str);sub("-"," ",str1);print str1;del str;next }1' file
Explantion:
awk '/image: /{ # Process the image tag
str=$0 # Set a variable str equal to the line
}
str { # Process when we have a str set
print; # Print the current line
str1=gensub(/(^.*)(image:.*$)/,"\\1imagePullSecret: mysecret",str); # Split the line into two sections and set str1 to the first section (get the leading spaces) and then the text we want to add.
sub("-"," ",str1); # Replace any - in the text with a space
print str1; # Print the new line
del str; # Remove the str variable
next # Skip to the next line
}1' file
Yq v4 doesn't yet support maps for addition, but you could convert to JSON, use jq, and then convert back to YAML:
yq --tojson eval infile.yaml \
| jq '.spec.template.spec.containers[0] += {imagePullSecret: "mysecret"}' \
| yq eval -prettyPrint
This might work for you (GNU sed):
sed -E '/(-( ))?image:.*/{p;s//\2\2imagePullSecret: mySecret/}' file
Match on image:.* with an optional - prepended.
Print the current line.
Replace the match by imagePullSecret: mySecret with the optional space following the - prepended twice.
I have an input file (customers.txt) that looks like this:
Name, Age, Email,
Hank, 22, hank#mail.com
Nathan, 32, nathan#mail.com
Gloria, 24, gloria#mail.com
I'm trying to have to output to a file (customersnew.txt) to have it look like this:
Name: Hank Age: 22 Email: hank#mail.com
Name: Nathan Age: 32 Email: nathan#mail.com
Name: Gloria Age: 24 Email: gloria#mail.com
So far, I've only been able to get an output like:
Name: Hank, 22, hank#mail.com
Name: Nathan, 32, nathan#mail.com
Name: Gloria, 24, gloria#mail.com
The code that I'm using is
sed -e '1d'\
-e 's/.*/Name: &/g' customers.txt > customersnew.txt
I've also tried separating the data using -e 's/,/\n/g'\ and then -e '2s/.*Age: &/g'. It doesn't work. Any help would be highly appreciated.
Have you considered using awk for this? Like:
$ awk 'BEGIN {FS=", ";OFS="\t"} NR==1 {split($0,hdr);next} {for(i=1;i<=NF;i++)$i=hdr[i]": "$i} 1' file
Name: Hank Age: 22 Email: hank#mail.com
Name: Nathan Age: 32 Email: nathan#mail.com
Name: Gloria Age: 24 Email: gloria#mail.com
This simply saves headers into an array and prepends each field in following records with <header>:.
This might work for you (GNU sed & column):
sed -E '1h;1d;G;s/^/,/;:a;s/,\s*(.*\n)([^,]+),/\2: \1/;ta;P;d' file | column -t
Copy the header to the hold space.
Append the header to each detail line.
Prepend a comma to the start of the line.
Create a substitution loop that replaces the first comma by the first heading in the appended header.
When all the commas have been replaced, print the first line and delete the rest.
To display in neat columns use the column command with the -t option.
Could you please try following.
awk '
BEGIN{
FS=", "
OFS="\t"
}
FNR==1{
for(i=1;i<=NF;i++){
value[i]=$i
}
next
}
{
for(i=1;i<=NF;i++){
$i=value[i] ": " $i
}
}
1
' Input_file
Explanation: Adding explanation for above solution.
awk ' ##Starting awk program from here.
BEGIN{ ##Starting BEGIN section of this program from here.
FS=", " ##Setting field separator as comma space here.
OFS="\t" ##Setting output field separator as TAB here for all lines.
}
FNR==1{ ##Checking here if this is first line then do following.
for(i=1;i<=NF;i++){ ##Starting a for loop to traverse through all elements of fields here.
value[i]=$i ##Creating an array named value with index variable i and value is current field value.
}
next ##next will skip all further statements from here.
}
{
for(i=1;i<=NF;i++){ ##Traversing through all fields of current line here.
$i=value[i] ": " $i ##Setting current field value adding array value with index i colon space then current fiedl value here.
}
}
1 ##1 will print all lines here.
' Input_file ##Mentioning Input_file name here.
I'm new to unix programming like sed, perl etc. I've searched and no result found match my case. I need to append substring from top line in the same file. My file content text.txt :
Name: sur.name.custom
Tel: xxx
Address: yyy
Website: www.site.com/id=
Name: sur.name.custom1
Tel: xxx
Address: yyy
Website: www.site.com/id=
I need to append every Name (sur.name.*) to every website on its block.
So Expected ouput:
Name: sur.name.custom
Tel: xxx
Address: yyy
Website: www.site.com/id=sur.name.custom
Name: sur.name.custom1
Tel: xxx
Address: yyy
Website: www.site.com/id=sur.name.custom1
I've tried the following sed command:
sed -n "/^Website:.*id=$/ s/$/sur.name..*/p" ./text.txt;
But sed returned: Website: www.site.com/id=sur.name.* same string I put.
I'm sure sed can append from regex pattern. I need both sed and perl if possible.
Why don't you use awk for this? Assuming names doesn't contain spaces following command should work:
awk '$1=="Name:"{name=$2} $1=="Website:"{print $0 name;next} 1' file
Perl equivalent:
perl -pale'
$F[0] eq "Name:" and $name = $F[1];
$F[0] eq "Website:" and $_ .= $name;
' file
(Line breaks may be removed.)
Here' a sed solution:
sed '/^Name:/{h;s/Name: *//;x;};/^Website:/{G;s/\n//;}' filename
Translation: If the line begins with Name:, save the name to the hold space; if the line starts with Website:, append the (latest) name from the holdspace.