So basically i have this:
while true; do printf "%s\r" $(date +"%H:%M:%S"); sleep 1; done | sed s/"\(^[^$]*$\)"/"[test] \1"/g
The goal of this example is to print the date on this same line using carriage return, then filter the ouput through sed to prepend [test] to the line as if i was using
`printf "[test] %s\r" $(date)`
but instead sed outputs nothing. I've tried stdbuf and unbuffer and they haven't worked or i'm not using them correctly. I want to get this to work using ffmpeg but more importantly any kind of output that uses a carriage return.
You can use this to workaround:
while true; do printf "%s\n" $(date +"%H:%M:%S"); sleep 1; done|awk '{printf("[test]%s\r",$0)}'
When you use printf "%s\r", you are printing string out in one line, that means the line will never ends, but sed will work only then a line ends(means this line is complete). I think that's why your code not work.
You'd better let the last command print lines into one.
Related
I want to extract strings between two patterns with GREP, but when no match is found, I would like to print a blank line instead.
Input
This is very new
This is quite old
This is not so new
Desired Output
is very
is not so
I've attempted:
grep -o -P '(?<=This).*?(?=new)'
But this does not preserve the second blank line in the above example. Have searched for over an hour, tried a few things but nothing's worked out.
Will happily used a solution in SED if that's easier!
You can use
#!/bin/bash
s='This is very new
This is quite old
This is not so new'
sed -En 's/.*This(.*)new.*|.*/\1/p' <<< "$s"
See the online demo yielding
is very
is not so
Details:
E - enables POSIX ERE regex syntax
n - suppresses default line output
s/.*This(.*)new.*|.*/\1/ - finds any text, This, any text (captured into Group 1, \1, and then any text again, or the whole string (in sed, line), and replaces with Group 1 value.
p - prints the result of the substitution.
And this is what you need for your actual data:
sed -En 's/.*"user_ip":"([^"]*).*|.*/\1/p'
See this online demo. The [^"]* matches zero or more chars other than a " char.
With your shown samples, please try following awk code.
awk -F'This\\s+|\\s+new' 'NF==3{print $2;next} NF!=3{print ""}' Input_file
OR
awk -F'This\\s+|\\s+new' 'NF==3{print $2;next} {print ""}' Input_file
Explanation: Simple explanation would be, setting This\\s+ OR \\s+new as field separators for all the lines of Input_file. Then in main program checking condition if NF(number of fields) are 3 then print 2nd field (where next will take cursor to next line). In another condition checking if NF(number of fields) is NOT equal to 3 then simply print a blank line.
sed:
sed -E '
/This.*new/! s/.*//
s/.*This(.*)new.*/\1/
' file
first line: lines not matching "This.*new", remove all characters leaving a blank line
second lnie: lines matching the pattern, keep only the "middle" text
this is not the pcre non-greedy match: the line
This is new but that is not new
will produce the output
is new but that is not
To continue to use PCRE, use perl:
perl -lpe '$_ = /This(.*?)new/ ? $1 : ""' file
This might work for you:
sed -E 's/.*This(.*)new.*|.*/\1/' file
If the first match is made, the line is replace by everything between This and new.
Otherwise the second match will remove everything.
N.B. The substitution will always match one of the conditions. The solution was suggested by Wiktor Stribiżew.
I want to remove the last part of a file, starting at a line following a certain pattern and including the preceding newline.
So, stopping at "STOP", the following file:
keep\n
STOP\n
whatever
Should output:
keep
With no trailing newline.
I tried this, and the logic seems to work, but it seems that sed adds a newline every time it prints its buffer. How can I avoid that? When sed doesn't manipulate the buffer, I don't have that problem (IE If I remove the STOP, sed outputs 'whatever' at the end of the file without a newline).
printf 'keep
STOP
Whatever' | sed 'N
/\nSTOP/ {
s/\n.*$//
P
Q
}
P
D'
I'm trying to write a git cleaning filter, and I cannot have a new newline appended every time I commit.
$ awk '/^STOP/{exit} {printf "%s%s", ors, $0; ors=RS}' file
keep$
The above prints every line without a trailing newline but preceded by a newline (\n or \r\n - whichever your environment dictates so it'll behave correctly on UNIX or Windows or whatever) for every 2nd and subsequent line. When it finds a STOP line it just exits before printing anything.
Note that the above doesn't keep anything in memory except the current line so it'll work no matter how large your input file is and no matter where the STOP appears in it - it'll even work if STOP is the first line of the file unlike the other answers you have so far.
It will also work using any awk in any shell on every UNIX box.
This might work for you (GNU sed):
sed -z 's/\nSTOP.*//' file
The -z option slurps the whole file into memory and the substitute command, removes the remainder of the file from the first newline followed by STOP.
Using awk you could:
$ awk '$0=="STOP"{exit} {b=b (b==""?"":ORS) $0} END{printf "%s",b}' file
Output:
keep$
Explained:
$ awk '
$0=="STOP" { exit } # exit at STOP, ie. go to END
{ b=b (b==""?"":ORS) $0 } # gather an output buffer, control \n
END { printf "%s",b } # in the END output output buffer
' file
... more (focusing a bit on the conditional operator):
b=b # appending to b, so b is b and ...
(b==""?"":ORS) # if b was empty, add nothing to it, if not add ORS ie. \n ...
$0 # and the current record
Suppose the following command
echo -en "abc1\ndef2\nghi1" | sed -n 'p; d;'
In this case the output is just the same as it would be without sed at all. So the last line still has no new line character. Next command
echo -en "abc1\ndef2\nghi1" | sed -n '$! {p; d;}; /1$/ {s/1$//; p; d;}'
sed prints all but the last line without modification. The last line is shortened by one character. Still there is no new line character on the last line. Next command
echo -en "abc1\ndef2\nghi1" | sed -n '$! {p; d;}; /1$/ {s/1$//; p; q1;}'
("d" replaced by "q1" in the last command block. Same output as before, but this time there is an additional new line character in the last line.
Why?
How to fix?
(For those who are interested in the intention for this command: Given a certain STDIN, I want to scan for the last character, pass on STDIN to STDOUT without this last character and set an exit code based on that character. There should no other modification. sed seems to be perfect, if there wouldn't be this newline problem
sed -n '
$! {p; d;}; #print any non last line, do next cycle
/0$/ {s/0$//; p; d}; #last line ending with 0? Remove 0, print, next cycle
/1$/ {s/1$//; p; d}; #last line ending with 1? Remove 1, print, next cycle
{p} #fall back, print last line
'
So far this script works perfect regarding to the newline issue. No new line is added. Now if i replace the "d" command with "q"
sed -n '
$! {p; d;}; #print any non last line, do next cycle
/0$/ {s/0$//; p; q0}; #last line ending with 0? Remove 0, print, exit 0
/1$/ {s/1$//; p; q1}; #last line ending with 1? Remove 1, print, exit 1
{p} #fall back, print last line
'
the newline problem suddenly arise...
Other solutions are welcome, they should be as fast as possible.
I guess this is a bug. According to the manual, q should not print the pattern space if auto-print is disabled. Thus, it should not print anything. Since you are already using GNU sed, you could avoid this problem by using Q instead of q. At least, this works for me (version 4.2.2).
I would like to delete all empty segments in my file.
The empty segment can be specified by a pair of consecutive lines starting with START and ending with END. Valid segments will have some contents between lines starting with START and ending with END
Sample Input
Header
START arguments
END
Any contents
START arguments
...
something
...
END
Footer
Desired Output
Header
Any contents
START arguments
...
something
...
END
Footer
Here I'm looking for possible one liners. Any help would be appreciated.
Trials
I tried following awk. It works to some extent but it deletes START lines even in valid segments.
awk '/^START/ && getline && /^END$/ {next} 1' file
perl -00 -pe 's/START .*?\nEND//g' file
this is a better one.
the solution I gave earlier will discard whole paragraph if they are not separated by blank lines.
Earlier response below:
how about this perl one liner ?
perl -00 -ne 'print if not /START .*\nEND/' file
read-in file in paragraph mode and discard lines matching START <string><newline>END
Meanwhile people are suggesting nice solutions, I came up with alternative solution using sed
sed '/^START/N;/^START.*END$/d' file
Or as suggested by #jthill
sed '/^START/N; /\nEND$/d' file
gawk only
awk -v RS='START[^\n]*\nEND\n' '{printf "%s", $0}' file.txt
Perhaps the following will be helpful:
perl -ne 'print /^START/?do{$x=<>;$_,$x if $x!~/^END/}:$_' inFile
Output on your dataset:
Header
Any contents
START arguments
...
something
...
END
Footer
$ awk '{rec = rec $0 RS} END{ gsub(/START[^\n]*\nEND\n/,"",rec); printf "%s", rec }' file
Header
Any contents
START arguments
...
something
...
END
Footer
/^START/ {
startline=$0
next
}
/^END$/ && startline {
startline=""
next
}
startline {
print startline
}
startline=""
1
I have 5 lines like:
typeA;pointA1
typeA;pointA2
typeA;pointA3
typeB;pointB1
typeB;pointB2
result output would be:
typeA;pointA1;typeA;pointA2
typeA;pointA2;typeA;pointA3
typeB;pointB1;typeB;pointB2
Is it possible to use sed or awk for this purpose?
This is easy with awk:
awk -F';' '$1 == prevType { printf("%s;%s;%s\n", $1, prevPoint, $0) } { prevType = $1; prevPoint = $2 }'
I've assumed that the blank lines between the records are not part of the input; if they are, just run the input through grep -v '^$' before awk.
paste could be useful in this case. it could save a lot of codes:
sed '1d' file|paste -d";" file -|awk -F';' '$1==$3'
see the test below
kent$ cat a
typeA;pointA1
typeA;pointA2
typeA;pointA3
typeB;pointB1
typeB;pointB2
kent$ sed '1d' a|paste -d";" a -|awk -F';' '$1==$3'
typeA;pointA1;typeA;pointA2
typeA;pointA2;typeA;pointA3
typeB;pointB1;typeB;pointB2
This GNU sed solution might work for you:
sed -rn '1{h;b};H;x;/^([^;]*);.*\n\1/!{s/.*\n//;x;d};s/\n/;/p' source_file
Assumes no blank lines else pipe preformat the source file with sed '/^$/d' source_file
EDIT:
On reflection the above solution is far too elaborate and can be condensed to:
sed -ne '1{h;b};H;x;/^\([^;]*\);.*\1/s/\n/;/p' source_file
Explanation:
The -n prevents any lines being implicitly printed. The first line is copied to the hold space (HS an extra register) and then a break is made that ends the iteration. All subsequent lines are appended to the HS. The HS is then swapped with the pattern space (PS - a register holding the current line). The HS at this point contains the previous and current lines which are now checked to see if the first field in each line are identical. If so, the newline separating the two lines is replaced by a ; and providing the substitution occurred the PS is printed out. The next iteration now takes place, the current line refreshes the PS and HS now holds the previous line.