I am trying to remove any black spaces from the end of lines. I am doing this and I am wondering why it is not working.
sed -i '' 's/\s*$/ endOfLine/g' myFile
I get this
MyLine
endOfLine
MyLine
endOfLine
I expect this
MyLine endOfLine
MyLine endOfLine
Related
I have a file named file, whose content is
noise
noise
X noise STUFF1 noise STUFF2 noise
noise
Y noise STUFF3 noise
noise
and I assert that X and Y are distinct, that each occur once in file, and that X occurs first.
I'm able to issue a sed command to extract the first pieces of information, the like of
$ sed -n '/X/s/\(.*\)\(…\)\(.*\)\(…\)/\2 \4/p' < file
STUFF1 STUFF2
$
and a similar one to extract STUFF3 (¹), but what I'd really like to do is to find the right sed incantation so that
$ sed … < file
STUFF3 STUFF1 STUFF2
$
(and possibly learn, at last! how sed's hold buffer works).
(1) This is not a question on regular expression, I know how to insulate the pieces of text that I need. I need to save the info I've collected and output it at the right time.
Using sed
$ sed -n '/^X/{s/.[^[:upper:]]*\([[:alnum:]]*\)/\1 /g;h};/^Y/{s/.[^[:upper:]]*\([[:alnum:]]*\)/\1 /g;G;s/\n//p}' file
STUFF3 STUFF1 STUFF2
$ cat script.sed
/^X/{ #Match line beginning with X
s/.[^[:upper:]]*\([[:alnum:]]*\)/\1 /g #As you know how to extract what you need, this is just for your sample data to extract needed strings
h #Retain the output of the substitution in the hold buffer
}
/^Y/{ #Match line beginning with Y
s/.[^[:upper:]]*\([[:alnum:]]*\)/\1 /g #Same as above
G #Append the contents of the hold space
s/\n//p #Remov the new line
}
sed -nf script.sed file
STUFF3 STUFF1 STUFF2
sed -n ' # Do not print by default
/X/{
# pattern space holds 'X noise STUFF1 noise STUFF2 noise'
s/.*\(STUFF1).*\(STUFF2\).*/\1 \2/
# pattern space holds 'STUFF1 STUFF2'
# add stuff from pattern space to hold space with __leading newline__
H
# hold space holds '\nSTUFF1 STUFF2'
# use l to inspect
d
}
/Y/{
s/.*\(…\).*/\1/p
H
# hold space holds '\nSTUFF1 STUFF2\nSTUFF3'
d
}
${ # last line?
# switch hold space with pattern space
x
# we have '\nSTUFF1 STUFF2\nSTUFF3' in paterrn space, let's make it nice with spaces
s/\n/ /g
s/ */ /g
s/^ *//g
s/ *$//g
# print it
p
}
'
This might work for you (GNU sed):
sed -En '/^X/h;/^Y/{G;s/\s+/ /g;s/.*/echo "&"|cut -d" " -f3,7,9/ep}' file
Make a copy of the line starting X in the hold space.
Append the copy to a line starting Y.
Replace one or more white spaces by a space globally on the above line(s).
Replace the contents of that line by required columns using the cut command.
I have file with this text:
mirrors:
docker.io:
endpoint:
- "http://registry:5000"
registry:5000:
endpoint:
- "http://registry:5000"
localhost:
endpoint:
- "http://registry:5000"
I need to replace it with this text in POSIX shell script (not bash):
mirrors:
docker.io:
endpoint:
- "http://docker.io"
registry:5000:
endpoint:
- "http://registry:5000"
localhost:
endpoint:
- "http://localhost"
Replace should be done dynamically in all places without hard-coded names. I mean we should take sub-string from a first line ("docker.io", "registry:5000", "localhost") and replace with it sub-string "registry:5000" in a third line.
I've figure out regex, that splits it on 5 groups: (^ )([^ ]*)(:[^"]*"http:\/\/)([^"]*)(")
Then I've tried to use sed to print group 2 instead of 4, but this didn't work: sed -n 's/\(^ \)\([^ ]*\)\(:[^"]*"http:\/\/\)\([^"]*\)\("\)/\1\2\3\2\5/p'
Please help!
This might work for you (GNU sed):
sed -E '1N;N;/\n.*endpoint:.*\n/s#((\S+):.*"http://)[^"]*#\1\2#;P;D' file
Open up a three line window into the file.
If the second line contains endpoint:, replace the last piece of text following http:// with the first piece of text before :
Print/Delete the first line of the window and then replenish the three line window by appending the next line.
Repeat until the end of the file.
Awk would be a better candidate for this, passing in the string to change to as a variable str and the section to change (" docker.io" or " localhost" or " registry:5000") and so:
awk -v findstr=" docker.io" -v str="http://docker.io" '
$0 ~ findstr { dockfound=1 # We have found the section passed in findstr and so we set the dockfound marker
}
/endpoint/ && dockfound==1 { # We encounter endpoint after the dockfound marker is set and so we set the found marker
found=1;
print;
next
}
found==1 && dockfound==1 { # We know from the found and the dockfound markers being set that we need to process this line
match($0,/^[[:space:]]+-[[:space:]]"/); # Match the start of the line to the beginning quote
$0=substr($0,RSTART,RLENGTH)str"\""; # Print the matched section followed by the replacement string (str) and the closing quote
found=0; # Reset the markers
dockfound=0
}1' file
One liner:
awk -v findstr=" docker.io" -v str="http://docker.io" '$0 ~ findstr { dockfound=1 } /endpoint/ && dockfound==1 { found=1;print;next } found==1 && dockfound==1 { match($0,/^[[:space:]]+-[[:space:]]"/);$0=substr($0,RSTART,RLENGTH)str"\"";found=0;dockfound=0 }1' file
My file contains x number of lines, I would like to remove the string before and after the reference string at the beginning and end of each line.
The reference string and string to remove are separated by space.
The file contains :
test.user.passs
test.user.location
global.user
test.user.tel
global.pass
test.user.email string_err
#ttt...> test.user.car ->
test.user.address
è_ 788 test.user.housse
test.user.child
{kl78>&é} global.email
global.foo
test.user.foo
How to remove the string at the start of each line which contain "test" string and also the end of each line separated by space or tab with sed?
The desired result is :
test.user.passs
test.user.location
global.user
test.user.tel
global.pass
test.user.email
test.user.car
test.user.address
test.user.housse
test.user.child
{kl78>&é} global.email
global.foo
test.user.foo
I interpret your question as: find the first word that is "word characters and at least one dots"
Tcl:
echo '
set fh [open [lindex $argv 1] r]
while {[gets $fh line] != -1} {puts [regexp -inline {\w+(?:\.\w+)+} $line]}
' | tclsh - file
sed
sed -r 's/.*\<([[:alpha:]]+(\.[[:alpha:]]+)).*/\1/' file
perl
perl -nE '/(\w+(\.\w+)+)/ and say $1' file
using sed like
sed -r 's/^[^ ]+[ ]+([^ ]+)[ ]+[^ ]*/\1/' file
This might work for you (GNU sed):
sed -r 's/.*(test\S+).*/\1/' file
What is a sed script that will remove the "\n" character but only if it is inside "" characters (delimited string), not the \n that is actually at the end of the (virtual) line?
For example, I want to turn this file
"lalala","lalalslalsa"
"lalalala","lkjasjdf
asdfasfd"
"lalala","dasdf"
(line 2 has an embedded \n ) into this one
"lalala","lalalslalsa"
"lalalala","lkjasjdf \\n asdfasfd"
"lalala","dasdf"
(Line 2 and 3 are now joined, and the real line feed was replaced with the character string \\n (or any other easy to spot character string, I'm not picky))
I don't just want to remove every other newline as a previous question asked, nor do I want to remove ALL newlines, just those that are inside quotes. I'm not wedded to sed, if awk would work, that's fine too.
The file being operated on is too large to fit in memory all at once.
sed is an excellent tool for simple substitutions on a single line but for anything else you should use awk., e.g:
$ cat tst.awk
{
if (/"$/) {
print prev $0
prev = ""
}
else {
prev = prev $0 " \\\\n "
}
}
$ awk -f tst.awk file
"lalala","lalalslalsa"
"lalalala","lkjasjdf \\n asdfasfd"
"lalala","dasdf"
Below was my original answer but after seeing #NeronLeVelu's approach of just testing for a quote at the end of the line I realized I was doing this in a much too complicated way. You could just replace gsub(/"/,"&") % 2 below with /"$/ and it'd work the same but the above code is a simpler implementation of the same functionality and will now handle embedded escaped double quotes as long as they aren't at the end of a line.
$ cat tst.awk
{ $0 = saved $0; saved="" }
gsub(/"/,"&") % 2 { saved = $0 " \\\\n "; next }
{ print }
$ awk -f tst.awk file
"lalala","lalalslalsa"
"lalalala","lkjasjdf \\n asdfasfd"
"lalala","dasdf"
The above only stores 1 output line in memory at a time. It just keeps building up an output line from input lines while the number of double quotes in that output line is an odd number, then prints the output line when it eventually contains an even number of double quotes.
It will fail if you can have double quotes inside your quoted strings escaped as \", not "", but you don't show that in your posted sample input so hopefully you don't have that situation. If you have that situation you need to write/use a real CSV parser.
sed -n ':load
/"$/ !{N
b load
}
:cycle
s/^\(\([^"]*"[^"]*"\)*\)\([^"]*"[^"]*\)\n/\1\3 \\\\n /
t cycle
p' YourFile
load the lines in working buffer until a close line (ending with ") is found or end reach
replace any \n that is after any couple of open/close " followed by a single " with any other caracter that " between from the start of file by the escapped version of new line (in fact replace starting string + \n by starting string and escaped new line)
if any substitution occur, retry another one (:cycle and t cycle)
print the result
continue until end of file
thanks to #Ed Morton for remark about escaped new line
When I want to print an output like this
./myScript (prints some lines)
or
cat myFile
I want the output to show with linebreakers , for example each line will include not more than 100 chars.
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaffffff
vbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbf
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
there is something I can add to the command line to get this result ?
Thanks.
You can use sed if you want the line terminator as ,.
$ cat myfile
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaffffffvbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbfaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
$ sed -r 's/.{50}/&,\n/g' myfile
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa,
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaffffffvbbb,
bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbfaaaaaa,
aaaaaaaaaaaaaaaaaaaaaaaaaaaaa
fold is another utility but won't add a , at the end
$ fold -w50 myfile
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaffffffvbbb
bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbfaaaaaa
aaaaaaaaaaaaaaaaaaaaaaaaaaaaa