I have this line inside a file:
ULNET-PA,client_sgcib,broker_keplersecurities
,KEPLER
I try to get rid of that ^M (carriage return) character so I used:
sed 's/^M//g'
However this does remove everything after ^M:
[root#localhost tmp]# vi test
ULNET-PA,client_sgcib,broker_keplersecurities^M,KEPLER
[root#localhost tmp]# sed 's/^M//g' test
ULNET-PA,client_sgcib,broker_keplersecurities
What I want to obtain is:
[root#localhost tmp]# vi test
ULNET-PA,client_sgcib,broker_keplersecurities,KEPLER
Use tr:
tr -d '^M' < inputfile
(Note that the ^M character can be input using Ctrl+VCtrl+M)
EDIT: As suggested by Glenn Jackman, if you're using bash, you could also say:
tr -d $'\r' < inputfile
still the same line:
sed -i 's/^M//g' file
when you type the command, for ^M you type Ctrl+VCtrl+M
actually if you have already opened the file in vim, you can just in vim do:
:%s/^M//g
same, ^M you type Ctrl-V Ctrl-M
You can simply use dos2unix which is available in most Unix/Linux systems. However I found the following sed command to be better as it removed ^M where dos2unix couldn't:
sed 's/\r//g' < input.txt > output.txt
Hope that helps.
Note: ^M is actually carriage return character which is represented in code as \r
What dos2unix does is most likely equivalent to:
sed 's/\r\n/\n/g' < input.txt > output.txt
It doesn't remove \r when it is not immediately followed by \n and replaces both with just \n. This fails with certain types of files like one I just tested with.
alias dos2unix="sed -i -e 's/'\"\$(printf '\015')\"'//g' "
Usage:
dos2unix file
If Perl is an option:
perl -i -pe 's/\r\n$/\n/g' file
-i makes a .bak version of the input file
\r = carriage return
\n = linefeed
$ = end of line
s/foo/bar/g = globally substitute "foo" with "bar"
In awk:
sub(/\r/,"")
If it is in the end of record, sub(/\r/,"",$NF) should suffice. No need to scan the whole record.
This is the better way to achieve
tr -d '\015' < inputfile_name > outputfile_name
Later rename the file to original file name.
I agree with #twalberg (see accepted answer comments, above), dos2unix on Mac OSX covers this, quoting man dos2unix:
To run in Mac mode use the command-line option "-c mac" or use the
commands "mac2unix" or "unix2mac"
I settled on 'mac2unix', which got rid of my less-cmd-visible '^M' entries, introduced by an Apple 'Messages' transfer of a bash script between 2 Yosemite (OSX 10.10) Macs!
I installed 'dos2unix', trivially, on Mac OSX using the popular Homebrew package installer, I highly recommend it and it's companion command, Cask.
This is clean and simple and it works:
sed -i 's/\r//g' file
where \r of course is the equivalent for ^M.
Simply run the following command:
sed -i -e 's/\r$//' input.file
I verified this as valid in Mac OSX Monterey.
remove any \r :
nawk 'NF+=OFS=_' FS='\r'
gawk 3 ORS= RS='\r'
remove end of line \r :
mawk2 8 RS='\r?\n'
mawk -F'\r$' NF=1
Related
How can I add ",1" at the end of each of the lines using sed?
The file I am using is like this;
A
B
C
I tried this;
sed 's/$/,1/' alphabets.txt > alphabets1.txt
and got this;
A
,1
B
,1
C
,1
but what I want is this;
A,1
B,1
C,1
When I tried the best answer from "https://stackoverflow.com/questions/15978504/add-text-at-the-end-of-each-line", I got a blank page.
sed -n 's/$/,1/' alphabets.txt > alphabets2.txt
I found out that there's ^M at the end using cat -vt file. How can I delete it and change it to ",1"?
Mac sed is BSD sed. You may try this command on a file optionally ending with \r to remove \r and append 1 in the end:
sed -E $'s/\\\r?$/,1/'
A,1
B,1
C,1
Make sure you shell is bash while running this.
You may also use:
sed -E 's/'$'\r''?$/,1/' file
$'\r' is a BASH string to match \r and ? after this makes it optional.
Please note that this sed command will also run fine on gnu sed.
I want to add a string to file at certain line number in Linux. I searched and found a command like:
sed "5i helloworld" test.txt
to add helloworld at line 5, but I got an error:
sed: command garbled.
I am testing in RedHat here. Is there any other command I can use here? Any other ways?
Older seds are a bit pickier with how you type commands like i, a and c. Try an actual line continuation:
sed '5i\
helloworld' test.txt
The i text syntax is a GNU extension. POSIX sed only know about the i\ version with linebreak.
Also, notice that there is a difference between the sed i command1 (insert text) and the -i option (in-place editing).
1 Or "function".
Here is awk solution:
awk 'NR==5{1;print "Hey there this is new text added on line 5"}1' inputfile
i have the following command
sed '/.*href="\(backup[^"]*tbz\)".*/!d;s//\1/;q'
which is works on my CentOS install.
But when I try to run it on FreeBSD i got the following error:
sed: 1: "/.*href="\(backup[ ...": extra characters at the end of d command
(23) Failed writing body
What's wrong with this?
Thanks!
Try to run it like this:
sed '/.*href="\(backup[^"]*tbz\)".*/\!d;s//\1/;q'
Note the extra \ character escaping your !d command
Apparently FreeBSD sed doesn't like the semicolon as command separator. Try with multiple -e options instead:
sed -e '/.*href="\(backup[^"]*tbz\)".*/!d' -e 's//\1/' -e q
or perhaps newlines:
sed '/.*href="\(backup[^"]*tbz\)".*/!d
s//\1/
q'
(Yes, that's a long single-quoted string with two newlines in it.)
I'm on Linux command line and I have file with
127.0.0.1
128.0.0.0
121.121.33.111
I want
127.0.0.1:80
128.0.0.0:80
121.121.33.111:80
I remember my colleagues were using sed for that, but after reading sed manual still not clear how to do it on command line?
You could try using something like:
sed -n 's/$/:80/' ips.txt > new-ips.txt
Provided that your file format is just as you have described in your question.
The s/// substitution command matches (finds) the end of each line in your file (using the $ character) and then appends (replaces) the :80 to the end of each line. The ips.txt file is your input file... and new-ips.txt is your newly-created file (the final result of your changes.)
Also, if you have a list of IP numbers that happen to have port numbers attached already, (as noted by Vlad and as given by aragaer,) you could try using something like:
sed '/:[0-9]*$/ ! s/$/:80/' ips.txt > new-ips.txt
So, for example, if your input file looked something like this (note the :80):
127.0.0.1
128.0.0.0:80
121.121.33.111
The final result would look something like this:
127.0.0.1:80
128.0.0.0:80
121.121.33.111:80
Concise version of the sed command:
sed -i s/$/:80/ file.txt
Explanation:
sed stream editor
-i in-place (edit file in place)
s substitution command
/replacement_from_reg_exp/replacement_to_text/ statement
$ matches the end of line (replacement_from_reg_exp)
:80 text you want to add at the end of every line (replacement_to_text)
file.txt the file name
How can this be achieved without modifying the original file?
If you want to leave the original file unchanged and have the results in another file, then give up -i option and add the redirection (>) to another file:
sed s/$/:80/ file.txt > another_file.txt
sed 's/.*/&:80/' abcd.txt >abcde.txt
If you'd like to add text at the end of each line in-place (in the same file), you can use -i parameter, for example:
sed -i'.bak' 's/$/:80/' foo.txt
However -i option is non-standard Unix extension and may not be available on all operating systems.
So you can consider using ex (which is equivalent to vi -e/vim -e):
ex +"%s/$/:80/g" -cwq foo.txt
which will add :80 to each line, but sometimes it can append it to blank lines.
So better method is to check if the line actually contain any number, and then append it, for example:
ex +"g/[0-9]/s/$/:80/g" -cwq foo.txt
If the file has more complex format, consider using proper regex, instead of [0-9].
You can also achieve this using the backreference technique
sed -i.bak 's/\(.*\)/\1:80/' foo.txt
You can also use with awk like this
awk '{print $0":80"}' foo.txt > tmp && mv tmp foo.txt
Using a text editor, check for ^M (control-M, or carriage return) at the end of each line. You will need to remove them first, then append the additional text at the end of the line.
sed -i 's|^M||g' ips.txt
sed -i 's|$|:80|g' ips.txt
sed -i 's/$/,/g' foo.txt
I do this quite often to add a comma to the end of an output so I can just easily copy and paste it into a Python(or your fav lang) array
Another Solaris question.
This is my file.
/abc/123/gfh/hello/what/is/up <THIS WOULD BE WHERE A NEW LINE STARTS>
bhn/fda/fds/hello/the/sky/is/blue <THIS WOULD BE WHERE A NEW LINE STARTS>
...etc
I need to delete everything before "hello" include the forward slash "/" infront of it for everyline in the file...
I'm stuck -> I had used a sed -E command but Solaris doesn't recognize the "-E". sigh
I think you can grep this:
grep -o hello.*
This should delete everything up to the slash before "hello":
sed -e 's|^.*hello/|hello|' <inputfile >outputfile
That should do it:
sed -e 's/.hello(.)/hello\1/'
#user4815162342: No need for "^" in the sed solution, ".*" would suffice.
Since there was an awk tag too, here's the equivalent awk solution:
awk '{sub(/.*hello\//,"hello")}1'