I have a script on a centos server and I wrote the script on the server using VIM. The script is to edit a configuration file. When I check the configuration file after it has been edited, there is a ^M at the end of every line that was NOT edited. The lines that were edited are fine.
cat hibernate.properties |
sed -i.bk \
-e 's%\(^hibernate\.connection\.url\=ristor:jdbc:postgresql:\/\/127\.0\.0\.1/\).*%\'1$dbname'%' \
-e 's/\(^hibernate\.connection\.username\=\).*/\'1$dbuser'/' \
-e 's/\(^hibernate\.connection\.password\=\).*/\'1$pws'/' hibernate.properties
This is the code that is being used to edit the configuration file. Why is it putting ^M at the end of every line that is NOT edited?
The ^M being shown are probably windows-style line endings on some lines. Try to run your file through dos2unix before running your script.
For example:
dos2unix hibernate.properties
This is not likely to add \r, it's more like that the file had them already, but was detected as dos fileformat by vim. Your script actually removed it from each line it has touched and vim doesn't consider the file dos anymore and therefore shows carriage returns that are still left in. Once you remove them (%s/<Ctrl-V><Ctrl-M>$// in vim should do), it is not likely to happen again.
Related
I am trying to parse some logs and there is a strange ^# symbol in there. I can remove it in vim by cutting that character and paste/searching for it, but how do I remove it in the bash command line automatically.
This doesn't work
sed 's/^#//'
When faced with an unwanted byte in a text file represented by some other stand-in symbol, a tool like hexdump or od helps. Try this:
Make a copy of the original file.
Remove everything in the copied file, except a line or two that includes the mystery symbol. Save the file.
To see what the byte really is, do:
hexdump -v -e '/1 "%_ad# "' -e '/1 " _%_u\_\n"' file
From which listing find the hex code for the unwanted byte, (let's
say it's 00), and try:
sed 's/\x00//' file
If that works, run the same sed line on the original file.
Quite certainly I miss something basic. My file contains lines like
fooLOCATION=sdfmsvdnv
fooLOCATION=
barLOCATION=sadssf
barLOCATION=
and I want to delete all lines ending with LOCATION=.
sed -i '/LOCATION=$/d' file
does not do, it deletes nothing, and I have tried endless variations, but I don't get it. What inline sed command can do this?
There are two approaches here, either print all non-matching lines with
sed -in '/LOCATION=$/!p' file
or delete all matching names with
sed -i '/LOCATION=$/d' file
The first uses the n command line option to suppress the default action of printing the line. We then test for lines that end in LOCATION= and invert the pattern (only keeping those that don't match). When we get a desirable line, we print it with the p option.
The second looks for lines matching the end of line pattern, and deletes those that do.
Your file contains blank lines, and both of these keep those. If we don't want to keep those, we can change the first option to
sed -in '/^$/!{/LOCATION=$/!p}' file
which first checks if a line is not empty, and only bothers checking if it should be printed if it isn't empty. We can modify the second option to
sed -i '/^$/d;/LOCATION=$/d' file
which deletes blank lines and then checks about deleting the other pattern.
We can modify the options to work with different line ending by specifying the difference in the pattern. The difference between line endings on Unix/Linux (\n) and Windows (\r\n) is the presence of an extra carriage return on Windows. Modifying the four commands above to accept either, we get
sed -in '/LOCATION=\r\{0,1\}$/!p' file
sed -i '/LOCATION=\r\{0,1\}$/d' file
sed -in '/^\r\{0,1\}$/!{/LOCATION=\r\{0,1\}$/!p}' file
sed -i '/^\r\{0,1\}$/d;/LOCATION=\r\{0,1\}$/d' file
Note that in each of these we allow an optional \r before the end of line. We use the curly bracket notation, as sed does not support the question mark optional quantifier in normal mode (using the r option to GNU sed for enabling extended regular expressions, we can replace \{0,1\} with ?).
On a Windows shell, all of the options above require double quotes instead of single quotes.
Your command does work for me:
$ sed -i '/LOCATION=$/d' file
Results, viewed using cat:
$ cat file
fooLOCATION=sdfmsvdnv
barLOCATION=sadssf
Note
If a file has non-Unix line endings such as files from Windows with DOS-formatted line-endings, it can be a reason for failure. A typical remedy is to use dos2unix:
$ dos2unix file
This converter fixes the newline issues, so that file will now have Unix-style line endings. Sed should now properly recognize those line endings, so retry your sed command and it should work.
This might work for you (GNU sed):
sed -i '/LOCATION=\s*$/d' file
This deletes the line if LOCATION= is at the end of the line or if there is any optional white space following the pattern.
I have two commands I'd like to place in a sed script (to add some headings to a text file that's already been created). The two commands are as follows:
sed '/TEST/i\"-TEST-"\' grades.txt
sed '/PROJECT/i\"-PROJECT-"\' grades.txt
The problem I'm running into is creating the actual script. I start by typing sed -f nameofscript.sed at the command line. From there I type in an opening comment using #, hit enter, then type in my commands (without the grades.txt at the end). Every command I type is echo'd back to me. When I'm finished I again type sed -f nameofscript.sed grades.txt. Nothing seems to end the script. so I hit control-d. When I go to review the script (cat nameofscript.sed) there's nothing there. I've been at this for quite a while so any help would be appreciated. Thank you so much.
You use a text editor to create a script file and then sed -f nameofscript.sed to execute that file.
sed -f does not start an interactive editor.
It runs sed over standard input (which is why it echos your lines back at you since it prints by default and you haven't given it any sed script to execute).
If nameofscript.sed didn't exist then sed -f nameofscript.sed would give you an error message about that when you tried to run it.
To be more explicit, when you run sed -f some_empty_file.sed you are telling sed to run an empty script over standard input. There is no standard input so sed waits and listens for input from your (via the keyboard). When you type into that "dead air" you are feeding sed content lines. It then processes those through its script (which is empty) and executes the default print action and spits them back out at you.
I have this line inside a file:
ULNET-PA,client_sgcib,broker_keplersecurities
,KEPLER
I try to get rid of that ^M (carriage return) character so I used:
sed 's/^M//g'
However this does remove everything after ^M:
[root#localhost tmp]# vi test
ULNET-PA,client_sgcib,broker_keplersecurities^M,KEPLER
[root#localhost tmp]# sed 's/^M//g' test
ULNET-PA,client_sgcib,broker_keplersecurities
What I want to obtain is:
[root#localhost tmp]# vi test
ULNET-PA,client_sgcib,broker_keplersecurities,KEPLER
Use tr:
tr -d '^M' < inputfile
(Note that the ^M character can be input using Ctrl+VCtrl+M)
EDIT: As suggested by Glenn Jackman, if you're using bash, you could also say:
tr -d $'\r' < inputfile
still the same line:
sed -i 's/^M//g' file
when you type the command, for ^M you type Ctrl+VCtrl+M
actually if you have already opened the file in vim, you can just in vim do:
:%s/^M//g
same, ^M you type Ctrl-V Ctrl-M
You can simply use dos2unix which is available in most Unix/Linux systems. However I found the following sed command to be better as it removed ^M where dos2unix couldn't:
sed 's/\r//g' < input.txt > output.txt
Hope that helps.
Note: ^M is actually carriage return character which is represented in code as \r
What dos2unix does is most likely equivalent to:
sed 's/\r\n/\n/g' < input.txt > output.txt
It doesn't remove \r when it is not immediately followed by \n and replaces both with just \n. This fails with certain types of files like one I just tested with.
alias dos2unix="sed -i -e 's/'\"\$(printf '\015')\"'//g' "
Usage:
dos2unix file
If Perl is an option:
perl -i -pe 's/\r\n$/\n/g' file
-i makes a .bak version of the input file
\r = carriage return
\n = linefeed
$ = end of line
s/foo/bar/g = globally substitute "foo" with "bar"
In awk:
sub(/\r/,"")
If it is in the end of record, sub(/\r/,"",$NF) should suffice. No need to scan the whole record.
This is the better way to achieve
tr -d '\015' < inputfile_name > outputfile_name
Later rename the file to original file name.
I agree with #twalberg (see accepted answer comments, above), dos2unix on Mac OSX covers this, quoting man dos2unix:
To run in Mac mode use the command-line option "-c mac" or use the
commands "mac2unix" or "unix2mac"
I settled on 'mac2unix', which got rid of my less-cmd-visible '^M' entries, introduced by an Apple 'Messages' transfer of a bash script between 2 Yosemite (OSX 10.10) Macs!
I installed 'dos2unix', trivially, on Mac OSX using the popular Homebrew package installer, I highly recommend it and it's companion command, Cask.
This is clean and simple and it works:
sed -i 's/\r//g' file
where \r of course is the equivalent for ^M.
Simply run the following command:
sed -i -e 's/\r$//' input.file
I verified this as valid in Mac OSX Monterey.
remove any \r :
nawk 'NF+=OFS=_' FS='\r'
gawk 3 ORS= RS='\r'
remove end of line \r :
mawk2 8 RS='\r?\n'
mawk -F'\r$' NF=1
Standard linux patch hard-coded only for unix text files.
PS: I do no want convert ALL to unix and then convert result back.
I've run into this problem before a few times. This is what I've discovered:
The Linux patch command will not recognize a patchfile that has CRLF in the patch 'meta-lines'.
The line-endings of the actual patch content must match the line endings of files being patched.
So this is what I did:
Use dos2unix to convert patch files to LF line-endings only.
Use dos2unix to convert the files being patched to LF line-endings only.
Apply patch.
You can use unix2dos to convert patched files back to CRLF line-endings if you want to maintain that convention.
Use the --binary option. Here is the relevant snippet from the man page:
--binary
Write all files in binary mode, except for standard output and /dev/tty. When reading, disable
the heuristic for transforming CRLF line endings into LF line endings. This option is needed
on POSIX systems when applying patches generated on non-POSIX systems to non-POSIX files. (On
POSIX systems, file reads and writes never transform line endings. On Windows, reads and writes
do transform line endings by default, and patches should be generated by diff --binary when
line endings are significant.)
Combined:
dos2unix patchfile.diff
dos2unix $(grep 'Index:' patchfile.diff | awk '{print $2}')
patch --verbose -p0 -i patchfile.diff
unix2dos $(grep 'Index:' patchfile.diff | awk '{print $2}')
The last line depends on whether you want to keep the CRLFs or not.
M.
PS. This should've been a reply to cscrimge's post. DS.
This is a solution one of our guys came up with in our office, so I'm not taking credit for it but it works for me here.
We have a situation of mixed linux and windows line endings in the same file sometimes, and we also create patch files from windows and apply them on linux.
If you are experience a patch problem after creating your patch file on windows or you have mixed line endings then do this:
dos2unix patch-file
dos2unix $(sed -n 's/^Index: //p' patch-file)
patch -p0 -i patch-file
perl -i.bak -pe's/\R/\n/g' inputfile to convert any line ending to the standard.