As I'm new with sed, I'm having the fun of seeing that sed doesn't think that the \r character is a valid line delimiter.
Does anyone know how to tell sed which character(s) I'd like it to use as the line delimiter when processing many lines of text?
You can specify it with awk's RS (record separator) variable: awk 'BEGIN {RS = "\r"} ...
Or you can convert with: tr '\r' '\n'
(For making the examples below clearer and less ambiguous, I'll use the od util extensively.)
It is not possible to do with a flag, for example. I bet the best solution is the one cited by the previous answers: using tr. If you have a file such as the one below:
$ od -xc slashr.txt
0000000 6261 0d63 6564 0d66
a b c \r d e f \r
0000010
There are various ways of using tr; the one we wanted is to pass two parameters for it - two different chars - and tr will replace the first parameter by the second one. Sending the file content as input for tr '\r' '\n', we got the following result:
$ tr '\r' '\n' < slashr.txt | od -xc
0000000 6261 0a63 6564 0a66
a b c \n d e f \n
0000010
Great! Now we can use sed:
$ tr '\r' '\n' < slashr.txt | sed 's/^./#/'
#bc
#ef
$ tr '\r' '\n' < slashr.txt | sed 's/^./#/' | od -xc
0000000 6223 0a63 6523 0a66
# b c \n # e f \n
0000010
But I presume you need to use \r as the line delimiter, right? In this case, just use tr '\n' '\r' to reverse the conversion:
$ tr '\r' '\n' < slashr.txt | sed 's/^./#/' | tr '\n' '\r' | od -xc
0000000 6223 0d63 6523 0d66
# b c \r # e f \r
0000010
As far as I know, you can't. What's wrong with using a newline as the delimiter? If your input has DOS-style \r\n line endings it can be preprocessed to remove them and, if necessary, they can be returned afterwards.
Related
To delete/comment 3 lines befor a pattern (including the line with the pattern):
how can i achive it through sed command
Ref:
sed or awk: delete n lines following a pattern
the above ref blog help to achive the this with after a pattern match but i need to know before match
define host{
use xxx;
host_name pattern;
alias yyy;
address zzz;
}
the below sed command will comment the '#' after the pattern match for example
sed -e '/pattern/,+3 s/^/#/' file.cfg
define host{
use xxx;
#host_name pattern;
#alias yyy;
#address zzz;
#}
like this how can i do this for the before pattern?
can any one help me to resolve this
If tac is allowed :
tac|sed -e '/pattern/,+3 s/^/#/'|tac
If tac isn't allowed :
sed -e '1!G;h;$!d'|sed -e '/pattern/,+3 s/^/#/'|sed -e '1!G;h;$!d'
(source : http://sed.sourceforge.net/sed1line.txt)
Reverse the file, comment the 3 lines after, then re-reverse the file.
tac file | sed '/pattern/ {s/^/#/; N; N; s/\n/&#/g;}' | tac
#define host{
#use xxx;
#host_name pattern;
alias yyy;
address zzz;
}
Although I think awk is a little easier to read:
tac file | awk '/pattern/ {c=3} c-- > 0 {$0 = "#" $0} 1' | tac
This might work for you (GNU sed):
sed ':a;N;s/\n/&/3;Ta;/pattern[^\n]*$/s/^/#/mg;P;D' file
Gather up 4 lines in the pattern space and if the last line contains pattern insert # at the beginning of each line in the pattern space.
To delete those 4 lines, use:
sed ':a;N;s/\n/&/3;Ta;/pattern[^\n]*$/d;P;D' file
To delete the 3 lines before pattern but not the line containing pattern use:
sed ':a;N;s/\n/&/3;Ta;/pattern[^\n]*$/s/.*\n//;P;D'
My source data contains special characters not in readable format. Can anyone help on the below :
Source data:
Commands Tryed:
sed 's/../t/g' test.txt > test2.txt
you can use tr to keep only printable characters:
tr -cd "[:print:]" <test.txt > test2.txt
Uses tr delete option on non-printable (print criteria negated by -c option)
If you want to replace those special chars by something else (ex: X):
tr -c "[:print:]" "X" <test.txt > test2.txt
With sed, you could try that to replace non-printable by X:
sed -r 's/[^[:print:]]/X/g' text.txt > test2.txt
it works on some but fails on chars >127 (maybe because the one I tried is printable as ▒ !) on my machine whereas tr works perfectly.
inline examples (printf to generate special chars + filter + od to show bytes):
$ printf "\x01ABC\x05\xff\xe0" | od -c
0000000 001 A B C 005 377 340
0000007
$ printf "\x01ABC\x05\xff\xe0" | sed "s/[^[:print:]]//g" | od -c
0000000 A B C 377 340
0000005
$ printf "\x01ABC\x05\xff\xe0" | tr -cd "[:print:]" | od -c
0000000 A B C
0000003
I would be happy if anyone can suggest me command (sed or AWK one line command) to divide each line of file in equal number of part. For example divide each line in 4 part.
Input:
ATGCATHLMNPHLNTPLML
Output:
ATGCA THLMN PHLNT PLML
This should work using GNU sed:
sed -r 's/(.{4})/\1 /g'
-r is needed to use extended regular expressions
.{4} captures every four characters
\1 refers to the captured group which is surrounded by the parenthesis ( ) and adds a space behind this group
g makes sure that the replacement is done as many times as possible on each line
A test; this is the input and output in my terminal:
$ echo "ATGCATHLMNPHLNTPLML" | sed -r 's/(.{4})/\1 /g'
ATGC ATHL MNPH LNTP LML
I suspect awk is not the best tool for this, but:
gawk --posix '{ l = sprintf( "%d", 1 + (length()-1)/4);
gsub( ".{"l"}", "& " ) } 1' input-file
If you have a posix compliant awk you can omit the --posix, but --posix is necessary for gnu awk and since that seems to be the most commonly used implementation I've given the solution in terms of gawk.
This might work for you (GNU sed):
sed 'h;s/./X/g;s/^\(.*\)\1\1\1/\1 \1 \1 \1/;G;s/\n/&&/;:a;/^\n/bb;/^ /s/ \(.*\n.*\)\n\(.\)/\1 \n\2/;ta;s/^.\(.*\n.*\)\n\(.\)/\1\2\n/;ta;:b;s/\n//g' file
Explanation:
h copy the pattern space (PS) to the hold space (HS)
s/./X/g replace every character in the HS with the same non-space character (in this case X)
s/^\(.*\)\1\1\1/\1 \1 \1 \1/ split the line into 4 parts (space separated)
G append a newline followed by the contents of the HS to the PS
s/\n/&&/ double the newline (to be later used as markers)
:a introduce a loop namespace
/^\n/bb if we reach a newline we are done and branch to the b namespace
/^ /s/ \(.*\n.*\)\n\(.\)/\1 \n\2/;ta; if the first character is a space add a space to the real line at this point and repeat
s/^.\(.*\n.*\)\n\(.\)/\1\2\n/;ta any other character just bump along and repeat
:b;s/\n//g all done just remove the markers and print out the result
This work for any length of line, however is the line is not exactly divisible by 4 the last portion will contain the remainder as well.
perl
perl might be a better choice here:
export cols=4
perl -ne 'chomp; $fw = 1 + int length()/$ENV{cols}; while(/(.{1,$fw})/gm) { print $1 . " " } print "\n"'
This re-calculates field-width for every line.
coreutils
A GNU coreutils alternative, field-width is chosen based on the first line of infile:
cols=4
len=$(( $(head -n1 infile | wc -c) - 1 ))
fw=$(echo "scale=0; 1 + $len / 4" | bc)
cut_arg=$(paste -d- <(seq 1 $fw 19) <(seq $fw $fw $len) | head -c-1 | tr '\n' ',')
Value of cut_arg is in the above case:
1-5,6-10,11-15,16-
Now cut the line into appropriate chunks:
cut --output-delimiter=' ' -c $cut_arg infile
How would like to join two lines usung awk or sed?
For example, I have data like below:
abcd
12:12:12:12:12:12:12:12
efgh001_01
45:45:45:45:45:45:45:45
ijkl7464746
78:78:78:78:78:78:78:78
and I need output like below:
abcd 12:12:12:12:12:12:12:12
efgh001_01 45:45:45:45:45:45:45:45
ijkl7464746 78:78:78:78:78:78:78:78
Running this almost works, but I need the space or tab:
awk '!(NR%2){print$0p}{p=$0}'
You're almost there:
awk '(NR % 2 == 0) {print p, $0} {p = $0}'
With sed you can do that as follows:
sed -n 'N;s/\n/ /p' file
where:
N reads next line
s replaces the new line character with a space to join both lines properly
p prints the result
This might work for you:
sed '$!N;s/\n/ /' file
or this:
paste -sd' \n' file
The sed below will output the input exactly. What I'd like to do is replace all occurrences of _ with - in the first matching group (\1), but not in the second. Is this possible?
echo 'abc_foo_bar=one_two_three' | sed 's/\([^=]*\)\(=.*\)/\1\2/'
abc_foo_bar=one_two_three
So, the output I'm hoping for is:
abc-foo-bar=one_two_three
I'd prefer not to resort to awk since I'm doing a string of other sed commands too, but I'll resort to that if I have to.
Edit: Minor fix to RE
You can do this in sed using the hold space:
$ echo 'abc_foo_bar=one_two_three' | sed 'h; s/[^=]*//; x; s/=.*//; s/_/-/g; G; s/\n//g'
abc-foo-bar=one_two_three
You could use awk instead of sed as follows:
echo 'abc_foo_bar=one_two_three' | awk -F= -vOFS== '{gsub("_", "-", $1); print $1, $2}'
The output would be, as expected:
abc-foo-bar=one_two_three
You could use ghc instead of sed as follows:
echo "abc_foo_bar=one_two_three" | ghc -e "getLine >>= putStrLn . uncurry (++) . (map (\x -> if x == '_' then '-' else x) *** id) . break (== '=')"
The output would be, as expected:
abc-foo-bar=one_two_three
This might work for you:
echo 'abc_foo_bar=one_two_three' |
sed 's/^/\n/;:a;s/\n\([^_=]*\)_/\1-\n/;ta;s/\n//'
abc-foo-bar=one_two_three
Or this:
echo 'abc_foo_bar=one_two_three' |
sed 'h;s/=.*//;y/_/-/;G;s/\n.*=/=/'
abc-foo-bar=one_two_three