How to shutter the "#" characters in line to one "#" char by sed? - sed

How to shutter the "#" characters to one "#" char by sed ?
From:
param=## ### ff ## e ##44
To:
param=# # ff # e #44

One way to do it using extended regexps:
vinko#parrot:~$ echo "## ### ff ## e ##44" | sed -r s/#+/#/g
# # ff # e #44
With regular regexps:
vinko#parrot:~$ echo "## ### ff ## e ##44" | sed -e s/##*/#/g
# # ff # e #44
Only after the equal sign:
vinko#parrot:~$ echo "param=## ### ff ## e ##44" | sed s/=##*/=#/g
param=# ### ff ## e ##44

posix version (for non GNU sed)
sed 's/#\{2,\}/#/g' YourFile

Related

Hot to replace newline characters with a string in sed

First, this is not a duplicate of, e.g., How can I replace each newline (\n) with a space using sed?
What I want is to exactly replace every newline (\n) in a string, like so:
printf '%s' $'' | sed '...; s/\n/\\&/g'
should result in the empty string
printf '%s' $'a' | sed '...; s/\n/\\&/g'
should result in a (not followed by a newline)
printf '%s' $'a\n' | sed '...; s/\n/\\&/g'
should result in
a\
(the trailing \n of the final line should be replaced, too)
A solution like :a;N;$!ba; s/\n/\\&/g from the other question doesn't do that properly:
printf '%s' $'' | sed ':a;N;$!ba; s/\n/\\&/g' | hd
works;
printf '%s' $'a' | sed ':a;N;$!ba;s/\n/\\&/g' | hd
00000000 61 |a|
00000001
works;
printf '%s' $'a\nb' | sed ':a;N;$!ba;s/\n/\\&/g' | hd
00000000 61 5c 0a 62 |a\.b|
00000004
works;
but when there's a trailing \n on the last line
printf '%s' $'a\nb\n' | sed ':a;N;$!ba;s/\n/\\&/g' | hd
00000000 61 5c 0a 62 0a |a\.b.|
00000005
it doesn't get quoted.
Easier to use perl than sed, since it has (by default, at least) a more straightforward treatment of the newlines in its input:
printf '%s' '' | perl -pe 's/\n/\\\n/' # Empty string
printf '%s' a | perl -pe 's/\n/\\\n/' # a
printf '%s\n' a | perl -pe 's/\n/\\\n/' # a\<newline>
printf '%s\n' a b | perl -pe 's/\n/\\\n/' # a\<newline>b\<newline>
# etc
If your inputs aren't huge, you could use
perl -0777 -pe 's/\n/\\\n/g'
instead to read the entire input at once instead of line by line, which can be more efficient.
how to replace newline charackters with a string in sed
It's not possible. From sed script point of view, the trailing line missing or not makes no difference and is undetectable.
Aaaanyway, use GNU sed with sed -z:
sed -z 's/\n/\\\n/g'
GNU awk can use the RT variable to detect a missing record terminator:
$ printf 'a\nb\n' | gawk '{ORS=(RT != "" ? "\\" : "") RT} 1'
a\
b\
$ printf 'a\nb' | gawk '{ORS=(RT != "" ? "\\" : "") RT} 1'
a\
b$
This adds a "\" before each non-empty record terminator.
Using any awk:
$ printf 'a\nb\n\n' | awk '{printf "%s%s", sep, $0; sep="\\\n"}'
a\
b\
$ printf 'a\nb\n' | awk '{printf "%s%s", sep, $0; sep="\\\n"}'
a\
b$
Or { cat file; echo; } | awk ... – always add a newline to the input.

Trim the last character of a stream

I use sed -e '$s/.$//' to trim the last character of a stream. Is it the correct way to do so? Are there other better ways to do so with other command line tools?
$ builtin printf 'a\nb\0' | sed -e '$s/.$//' | od -c -t x1 -Ax
000000 a \n b
61 0a 62
000003
EDIT: It seems that this command is not robust. The expected output is a\nb for the following example. Better methods (but not too verbose) are needed.
$ builtin printf 'a\nb\n' | sed -e '$s/.$//' | od -c -t x1 -Ax
000000 a \n \n
61 0a 0a
000003
You may use head -c -1:
printf 'a\nb\0' | head -c -1 | od -c -t x1 -Ax
000000 a \n b
61 0a 62
000003
printf 'a\nb\n' | head -c -1 | od -c -t x1 -Ax
000000 a \n b
61 0a 62
000003
It seems you can't rely on any line-oriented tools (like sed) that automatically remove and re-add newlines.
Perl can slurp the whole stream into a string and can remove the last char:
$ printf 'a\nb\0' | perl -0777 -pe chop | od -c -t x1 -Ax
000000 a \n b
61 0a 62
000003
$ printf 'a\nb\n' | perl -0777 -pe chop | od -c -t x1 -Ax
000000 a \n b
61 0a 62
000003
The tradeoff is that you need to hold the entire stream in memory.

grep + grep + sed = sed: no input files

Can anybody help me please?
grep " 287 " file.txt | grep "HI" | sed -i 's/HIS/HID/g'
sed: no input files
Tried also xargs
grep " 287 " file.txt | grep HI | xargs sed -i 's/HIS/HID/g'
sed: invalid option -- '6'
This works fine
grep " 287 " file.txt | grep HI
If you want to keep your pipeline:
f=file.txt
tmp=$(mktemp)
grep " 287 " "$f" | grep "HI" | sed 's/HIS/HID/g' > "$tmp" && mv "$tmp" "$f"
Or, simplify:
sed -i -n '/ 287 / {/HI/ s/HIS/HID/p}' file.txt
That will filter out any line that does not contain " 287 " and "HI" -- is that what you want? I suspect you really want this:
sed -i '/ 287 / {/HI/ s/HIS/HID/}' file.txt
For lines that match / 287 /, execute the commands in braces. In there, for lines that match /HI/, search for the first "HIS" and replace with "HID". sed implicitly prints all lines if -n is not specified.
Other commands that do the same thing:
awk '/ 287 / && /HI/ {sub(/HIS/, "HID")} {print}' file.txt > new.txt
perl -i -pe '/ 287 / and /HI/ and s/HIS/HID/' file.txt
awk does not have an "in-place" option (except gawk -i inplace for recent gawk versions)

Pattern extraction using SED or AWK

How do I extract 68 from v1+r0.68?
Using awk, returns everything after the last '.'
echo "v1+r0.68" | awk -F. '{print $NF}'
Using sed to get the number after the last dot:
echo 'v1+r0.68' | sed 's/.*[.]\([0-9][0-9]*\)$/\1/'
grep is good at extracting things:
kent$ echo " v1+r0.68"|grep -oE "[0-9]+$"
68
Match the digit string before the end of the line using grep:
$ echo 'v1+r0.68' | grep -Eo '[0-9]+$'
68
Or match any digits after a .
$ echo 'v1+r0.68' | grep -Po '(?<=\.)\d+'
68
Print everything after the . with awk:
echo "v1+r0.68" | awk -F. '{print $NF}'
68
Substitute everything before the . with sed:
echo "v1+r0.68" | sed 's/.*\.//'
68
type man grep
and you will see
...
-o, --only-matching
Show only the part of a matching line that matches PATTERN.
then type echo 'v1+r0.68' | grep -o '68'
if you want it any where special do:
echo 'v1+r0.68' | grep -o '68' > anyWhereSpecial.file_ending

How can I use a Perl one-liner to decode an ASCII string encoded in hex?

I’d like to write a Perl one-liner to decode a line of ASCII characters encoded as hexadecimal numbers (for example the line 48 54 54 50 should be decoded as HTTP). I came up with this:
perl -nE 'say map(chr, map { qq/0x$_/ } split)'
It prints an empty line. What am I doing wrong and how would you write it?
It's your qq/0x$_/ trick that doesn't work. chr expects a number as argument, but gets the string literal "0x48". Use the hex function to convert 48 to a decimal number, like datageist does in his answer.
This works for me:
echo '48 54 54 50' | perl -nE 'say map(chr, map { hex } split)'
This works:
echo '48 54 54 50' | perl -nE 'say map{chr(hex)} split'
I’m assuming you want to feed the data from STDIN.
As always with Perl TIMTOWTDI.
I thought I would submit several options, and show what they would look like if they were written normally. If you want to know more about the command line options perldoc perlrun is a useful resource.
These all output the same thing. With the exception that some of them don't print a newline on the end.
echo '48 54 54 50' | perl -0x20 -pe'$_=chr hex$_'
echo '48 54 54 50' | perl -0x20 -ne'print chr hex$_'
echo '48 54 54 50' | perl -0777 -anE'say map chr,map hex,#F'
echo '48 54 54 50' | perl -0777 -anE'say map{chr hex$_}#F'
echo '48 54 54 50' | perl -0apple'$_=chr hex$_' -0x20
echo '48 54 54 50' | perl -apple'$_=join"",map{chr hex}#F'
echo '48 54 54 50' | perl -lanE'say map{chr hex}#F'
The following is what some of the examples would look like if they were written normally. If you want to figure out what the rest of them do, definitely look at perldoc perlrun.
perl -0x20 -pe'$_=chr hex$_'
This is one is fairly straight forward. It is perhaps the best example here, and is also the shortest one. It pretends that spaces are used to separate lines, so that there is only one letter to deal with inside of the loop.
# perl -0x20 -pe'$_=chr hex$_'
$/ = " "; # -0 ( input separator )
while( <> ){
$_ = chr hex $_;
} continue {
print $_;
}
perl -0apple'$_=chr hex$_' -0x20
This one has a few command line options that don't do anything useful.
The first -0 option is there so that -l sets the output separator to an empty string.
Which is actually the default for the output separator.
There are two -p options where one would have sufficed.
The -a option sets up the #F array, but we don't actually use it.
Basically I used -a -l and a second -p so that the options would spell apple. Otherwise this one is the same as the last example.
echo '48 54 54 50' | perl -0x20 -pe'$_=chr hex$_'
# perl -0apple'$_=chr hex$_' -0x20
$/ = ""; # -0 ( input separator )
$\ = $/; # -l ( output separator )
$/ = " "; # -0x20 ( input separator )
while( <> ){
#F = split " ", $_; # -a ( unused )
$_ = chr hex $_;
} continue {
print $_;
}
perl -lanE'say map{chr hex}#F'
I figured I already spelled apple, I might as well spell lanE.
-l isn't really useful, because we already are using say.
Used -E instead of -e so that we could use say.
# perl -lanE'say map{chr hex}#F'
$\ = $/; # -l ( output separator set to "\n" )
while( <> ){
#F = split " ", $_; # -a
say map { chr hex $_ } #F;
}
Play perlgolf?
-ple y/0-9A-Fa-f//cd;$_=pack"H*",$_
-ple $_=pack"H*",$_,join"",split
-nE say map chr hex,split
-naE say map chr hex,#F