selectively delete lines - sed

I have a text file with tab delimited data spread across 16 columns.
I want to delete the complete row where the values 1260, 1068 and 907 found in 6th column.
9513 2010-06-15 17:00:00 94 0 69 12 0 0 0 0.0000 0 \N \N \N 2010-06-15 18:00:02 \N
9523 2010-06-15 18:00:00 94 0 69 12 0 0 0 0.0000 0 \N \N \N 2010-06-15 19:00:02 \N
9534 2010-06-15 19:00:00 94 0 69 12 0 0 0 0.0000 0 \N \N \N 2010-06-15 20:00:02 \N
9543 2010-06-15 20:00:00 94 0 69 12 0 0 0 0.0000 0 \N \N \N 2010-06-15 21:00:02 \N
9552 2010-06-15 21:00:00 94 0 69 12 0 0 0 0.0000 0 \N \N \N 2010-06-15 22:00:02 \N
9560 2010-06-15 22:00:00 94 0 69 12 0 0 0 0.0000 0 \N \N \N 2010-06-15 23:00:02 \N
9569 2010-06-15 23:00:00 94 0 69 12 0 0 0 0.0000 0 \N \N \N 2010-06-16 00:00:02 \N
9579 2010-06-16 00:00:00 94 0 69 12 0 0 0 0.0000 0 \N \N \N 2010-06-16 01:00:02 \N
9589 2010-06-16 01:00:00 94 0 69 12 0 0 0 0.0000 0 \N \N \N 2010-06-16 02:00:01 \N
9599 2010-06-16 02:00:00 94 0 69 12 0 0 0 0.0000 0 \N \N \N 2010-06-16 03:00:02 \N
95642733 2011-10-19 19:00:00 4341 0 1263 0 11 0 0 0.0000 0 \N \N \N 2011-10-19 20:05:06 \N
95642732 2011-10-19 19:00:00 4341 0 1260 0 24635 0 0 0.0000 0 \N \N \N 2011-10-19 20:05:06 \N
95642540 2011-10-19 19:00:00 4050 0 1068 103 113 2 0 0.0000 0 \N \N \N 2011-10-19 20:05:06 \N
95642539 2011-10-19 19:00:00 4050 0 907 19 0 0 0 0.0000 0 \N \N \N 2011-10-19 20:05:06 \N


Awk is the tool you want to use.
awk '$6==1260 || $6==1068 || $6==907 {next} {print}'
What does this do?
Awk runs a block of code on each line of your file. The code starts with an expression that must evaluate true (in this case the three possible values of the 6th field), followed by commands in curly braces. In this case, the command next tells it to proceed to the next input line without running any more commands.
If the three comparisons FAIL, and we don't run the next, then we print the line.


What you want to us is awk. Awk is an amazingly powerful language inside UNIX, and if you ever run into a complicated test-streaming problem, awk is your solution.
Try this script:
awk '{
if ($6 != 1260 || $6 != 1068 || $6 != 907)
print $0;
}' file.txt >> output_file.txt


This might work for you (GNU sed?):
sed '/^\(\S*\s*\)\{5\}\(1260\|1068\|907\)\s/d' file
or generally:
sed '/^\([^[:space:]]*[[:space:]]*\)\{5\}\(1260\|1068\|907\)[[:space:]]/!d'


awk '$6!=1260 && $6!=1068 && $6!=907' file

Related

What's the format of tm->when in /proc/net/tcp?

I need to know what tm->when means, but proc(5) doesn't mention anything helpful,
So, does it store the creation time of the socket? The number seems to be decreasing each time I view the file.
root#ubuntu-vm:~# cat /proc/net/tcp
sl local_address rem_address st tx_queue rx_queue tr tm->when retrnsmt uid timeout inode
0: 00000000:0CEA 00000000:0000 0A 00000000:00000000 00:00000000 00000000 104 0 17410 1 dddb6d00 100 0 0 10 -1
1: 00000000:0016 00000000:0000 0A 00000000:00000000 00:00000000 00000000 0 0 7959 1 dddb4500 100 0 0 10 -1
2: B238A8C0:0016 0138A8C0:9C96 01 00000000:00000000 02:00061444 00000000 0 0 8243 4 daa3c000 20 4 27 10 16
3: B238A8C0:0CEA 0138A8C0:8753 01 00000000:00000000 02:0009C787 00000000 104 0 19467 2 daa3e300 20 4 18 10 -1

From Exploring the /proc/net/ Directory
The tr field indicates whether a timer is active for this socket. A value of zero indicates the timer is not active. The tm->when field indicates the time remaining (in jiffies) before timeout occurs.

replace zero with text using sed or awk

I have text file which looks like as shown below:
0 chr23:54039 0 54039
0 chr23:103278 0 103278
0 chr22:174609 0 174609
0 chr22:54039 0 54039
0 chr25:103278 0 103278
0 chr25:174609 0 174609
26 chr26:174609 0 174609
If the first column is '0' i need to replace the 0 in the first column with the number after chr. So, the output should look like:
23 chr23:54039 0 54039
23 chr23:103278 0 103278
22 chr22:174609 0 174609
22 chr22:54039 0 54039
25 chr25:103278 0 103278
25 chr25:174609 0 174609
26 chr26:174609 0 174609
Can anyone provide a simple sed or awk any linux solution?

If number in column #1 is always the same as chr number you can do this with awk
awk '{split($2,a,":|chr");$1=a[2]}1' file
23 chr23:54039 0 54039
23 chr23:103278 0 103278
22 chr22:174609 0 174609
22 chr22:54039 0 54039
25 chr25:103278 0 103278
25 chr25:174609 0 174609
26 chr26:174609 0 174609

With sed:
$ sed -r '/^0/s/0(\s*chr)([^:]*)/\2\1\2/g' file
23 chr23:54039 0 54039
23 chr23:103278 0 103278
22 chr22:174609 0 174609
22 chr22:54039 0 54039
25 chr25:103278 0 103278
25 chr25:174609 0 174609
26 chr26:174609 0 174609
Without -r:
$ sed '/^0/s/0\(\s*chr\)\([^:]*\)/\2\1\2/g' file
23 chr23:54039 0 54039
23 chr23:103278 0 103278
22 chr22:174609 0 174609
22 chr22:54039 0 54039
25 chr25:103278 0 103278
25 chr25:174609 0 174609
26 chr26:174609 0 174609
The idea is to replace lines starting with 0. In those, the 0...chrNUM:... is caught and printed back with desired format.
With awk:
$ awk '/^0/ {split($2,a,":"); gsub("chr", "", a[1]); $1=a[1]}1' file
23 chr23:54039 0 54039
23 chr23:103278 0 103278
22 chr22:174609 0 174609
22 chr22:54039 0 54039
25 chr25:103278 0 103278
25 chr25:174609 0 174609
26 chr26:174609 0 174609
Given lines starting with 0, the 2nd field is broken into pieces by : delimiter and then chr text is removes. Then it is ready to be stored as first field. 1 makes the condition true, so the full new line is printed.

sed "s/^0[[:blank:]]\{1,\}chr\([0-9]\{1,\}\):/\1 chr\1:/"

How to filter a file by n. word in line after pattern?

I've got a large file with diffrent lines.
The lines i am interested in, are looking alike:
lcl|NC_005966.1_gene_59 scaffold441.6 99.74 390 1 0 1 390 34065 34454 0.0 715
lcl|NC_005966.1_gene_59 scaffold2333.4 89.23 390 42 0 1 390 3114 2725 1e-138 488
lcl|NC_005966.1_gene_60 scaffold441.6 100.00 186 0 0 1 186 34528 34713 1e-95 344
Now i want to get the lines after the pattern 'lcl|NC_' but just if the third word(or the nth word in the line) is smaller than 100.
(In this case the first two lines, since they just got a number of 99.74 and 89.23)
Next they should be saved into a new file.

This can make it:
$ awk '$1 ~ /^lcl\|NC_/ && $3<100' file
lcl|NC_005966.1_gene_59 scaffold441.6 99.74 390 1 0 1 390 34065 34454 0.0 715
lcl|NC_005966.1_gene_59 scaffold2333.4 89.23 390 42 0 1 390 3114 2725 1e-138 488
It checks both things:
- 1st field starting with lcl|NC_: $1 ~ /^lcl\|NC_/ does it. (Thanks Ed Morton for improving the previous $1~"^lcl|NC_")
- 3rd field being <100: $3<100.
To save into a file, you can do:
awk '$1 ~ /^lcl\|NC_/ && $3<100' file > new_file
^^^^^^^^^^

need to insert datetime in every row of vmstat output

I need to insert datetime in every vmstat line that has value.
I can create a function like this:
function insert_datetime {
while read line
do
printf "$line"
date '+ %m-%d-%Y %H:%M:%S'
done
}
then call vmstat as below:
'vmstat 3 5 | insert_datetime'
but this line puts date time to every line, including dashes (--) and any rows that has text. How can I exclude rows that has dahses and text?
kthr memory page faults cpu 04-23-2013 10:19:49
----- ----------- ------------------------ ------------ ----------------------- 04-23-2013 10:19:49
r b avm fre re pi po fr sr cy in sy cs us sy id wa pc ec 04-23-2013 10:19:49
0 0 45688088 4094129 0 0 0 0 0 0 45 12172 2840 1 1 99 0 0.35 2.2 04-23-2013 10:19:49
2 0 45694135 4088082 0 0 0 0 0 0 451 56350 21818 3 1 97 0 0.73 4.5 04-23-2013 10:19:52
1 0 45694137 4088061 0 0 0 0 0 0 303 24568 951 3 1 96 0 0.82 5.1 04-23-2013 10:19:55
1 0 45694138 4087739 0 0 0 0 0 0 445 9170 1504 2 0 98 0 0.64 4.0 04-23-2013 10:19:58
4 0 45703145 4078732 0 0 0 0 0 0 335 47175 1306 4 1 95 0 1.01 6.3 04-23-2013 10:20:01
I needed to look like this:
kthr memory page faults cpu
----- ----------- ------------------------ ------------ -----------------------
r b avm fre re pi po fr sr cy in sy cs us sy id wa pc ec
0 0 45688088 4094129 0 0 0 0 0 0 45 12172 2840 1 1 99 0 0.35 2.2 04-23-2013 10:19:49
2 0 45694135 4088082 0 0 0 0 0 0 451 56350 21818 3 1 97 0 0.73 4.5 04-23-2013 10:19:52
1 0 45694137 4088061 0 0 0 0 0 0 303 24568 951 3 1 96 0 0.82 5.1 04-23-2013 10:19:55
1 0 45694138 4087739 0 0 0 0 0 0 445 9170 1504 2 0 98 0 0.64 4.0 04-23-2013 10:19:58
4 0 45703145 4078732 0 0 0 0 0 0 335 47175 1306 4 1 95 0 1.01 6.3 04-23-2013 10:20:01

Why not just use vmstat -t? It seems to be exactly what you are looking for. Here is some sample output
[root#web5 vmstat]# vmstat -t 1
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------ ---timestamp---
r b swpd free buff cache si so bi bo in cs us sy id wa st
1 0 15704 193236 189628 595868 9 3 25 16 15 20 11 1 88 1 0 2013-05-22 13:32:36 JST
0 0 15704 193212 189628 595868 0 0 0 0 22 20 0 0 100 0 0 2013-05-22 13:32:37 JST
0 0 15704 193212 189628 595868 0 0 0 0 19 12 0 0 100 0 0 2013-05-22 13:32:38 JST
0 0 15704 193212 189628 595868 0 0 0 0 10 11 0 0 100 0 0 2013-05-22 13:32:39 JST
0 0 15704 193212 189628 595868 0 0 0 96 34 25 0 1 99 0 0 2013-05-22 13:32:40 JST
0 0 15704 193212 189628 595868 0 0 0 0 10 9 0 0 100 0 0 2013-05-22 13:32:41 JST
0 0 15704 193212 189628 595868 0 0 0 0 14 23 0 0 100 0 0 2013-05-22 13:32:42 JST
executed on CentOS6.3 with procps 3.2.8
[root#web5 uptime]# vmstat -V
procps version 3.2.8

Use awk:
vmstat 3 5 | awk '/^ *[0-9]/{$0=$0 " " strftime("%m-%d-%Y %T")};1'

Try:
function insert_datetime {
while read line
do
printf "$line"
if [[ "$line" =~ [0-9].* ]]; then
date '+ %m-%d-%Y %H:%M:%S'
else
echo
fi
done
}

sed can give you answer too... in much cleaner & portable (across shells) way:
vmstat 3 5 | sed '/^ *[0-9].*/s/.*/printf "&";date "+ %m-%d-%Y %H:%M:%S"/e'
All lines starting with a number are appended date in required format.

Ghostscript postscript pswrite is encoding text

Why is Ghostscript pswrite encoding my text in its output? Consider the following MWE:
%!PS-Adobe-3.0
%%Title: mwe.ps
%%Pages: 001
%%BoundingBox: 0 0 595 842
%%EndComments
%%Page: 1 1
%%PageBoundingBox: 0 0 595 842
0 0 1 setrgbcolor
0 0 595 842 rectfill
1 0 0 setrgbcolor
247 371 100 100 rectfill
/Times-Roman findfont
72 scalefont
setfont
newpath
247 300 moveto
(Chris) show
showpage
Saving this MWE to file and viewing in GSview will display a blue page with red square and my name underneath. Now run this file through Ghostscript 9.06 with the following command line:
"c:\Program Files\gs\gs9.06\bin\gswin64c.exe" ^
-dSAFER -dBATCH -dNOPAUSE ^
-sDEVICE=pswrite -sPAPERSIZE=a4 -r72 -sOutputFile=mwe_gs.ps mwe.ps
See Ghostscript output below. Can someone please explain what is happening here. Whilst the two rectfill commands are still apparent, my text (Chris) has been encoded and is no longer distinguishable.
Is there an alternative postscript device which would retain my text please?
<snip>
%%Page: 1 1
%%PageBoundingBox: 0 0 595 842
%%BeginPageSetup
GS_pswrite_2_0_1001 begin
595 842 /a4 setpagesize
/pagesave save store 197 dict begin
1 1 scale
%%EndPageSetup
gsave mark
255 0 r6
0 0 595 842 rf
255 0 r3
247 371 100 100 rf
Q q
0 0 595 0 0 842 ^ Y
255 0 r3
249 299 43 50 /5D
$C
,6CW56m1G"ZORNkWR*rB:!c2;9rlWTH="2^^[(q"h>cG<omZ2l^=qC[XbO:8_[?kji-8^"N#3q*
jhL~>
,
289 300 41 49 /0P
$C
4r?0p$m<EkK3,0>s8W-!s8W-!s8W,u]<1irI=*p=<t0>_#<)>Is8K6,aTi'$~>
,
325 300 30 33 /5I
$C
49S"pc4+Rhs8W-!s8W)oqdD:saRZq[4+k%):]~>
,
349 300 24 49 /0T
$C
4q%Ms%;PqCs8W-!s8W%1_qkn/K?*sYFSGd:5Q~>
,
377 299 23 34 /5M
$C
-TQR7$&O'!K+D:XribR9;$mr4#sqUi.T#,dX=Y&Llg+F`d^HC#%$"]~>
,
cleartomark end end pagesave restore
showpage
%%PageTrailer
%%Trailer
%%Pages: 1
%%EOF
NOTE: This might seem an odd activity but I'm exploring the idea of using Ghostscript to 'clean up' postscript output from Matlab application..

The 'text' has been converted to images, not vector paths. This is a serious limitation of the pswrite device, and one of the reasons it is deprecated, you should use the ps2write device instead. The only reason the pswrite device is still included at all is for epswrite which uses it (which is why the pswrite and epswrite output looks the same). At some point there will be an eps2write device and pswrite will be binned.
ps2write output is, by default, compressed. If you want uncompressed output, use the -dCompressPages=false switch on the command line.
If all you want is the location of the text you might consider the txtwrite device. The default implementation of this creates a plain text representation of the input, but you can have it output a faked up XML instead which includes things like the origin of the text.

Here is a simple example of the show operator being redefined to display position information about the show, along with performing the standard show operation. With ghostscript you can run multiple files, so the header file would be a prefix to the other file, which alters standard behavior.
The redefined show could have included font name and size. The data could have been written to a disk file, rather than dumped to the console. Any of other operator could have also been redefined, like rectfill, fill, stroke... Because the original operator is also called, you can convert a .ps to .pdf using a pdfwrite device, while at the same time obtaining position information.
gswin32c.exe -dBATCH -dNOPAUSE header.ps trash.ps
gswin32c.exe -sDEVICE=pdfwrite -dCompressPages=false -sOutputFile=test.pdf header.ps trash.ps
output
currentpoint x:247.0 y:300.0 pathbbox 249.015,298.992 400.066,349.184 text:Chris currentrgbcolor:1.0,0.0,0.0( )
currentpoint x:50.0 y:90.0 pathbbox 50.8682,89.2852 181.327,139.184 text:Fred currentrgbcolor:1.0,0.0,0.0( )
currentpoint x:150.0 y:200.0 pathbbox 150.867,184.298 304.154,247.673 text:Mary currentrgbcolor:1.0,0.0,0.0( )
currentpoint x:300.0 y:350.0 pathbbox 300.867,348.993 598.79,398.681 text:Mr. Green currentrgbcolor:0.0,1.0,0.0( )
currentpoint x:100.0 y:400.0 pathbbox 100.866,399.202 358.547,449.183 text:Mr. Blue currentrgbcolor:0.0,0.0,1.0( )
Header.ps
/mydict 5 dict def
mydict begin
/show
{
(currentpoint ) print
currentpoint exch 10 string cvs ( x:) print print 10 string cvs ( y:) print print
gsave dup false charpath flattenpath
( pathbbox ) print
pathbbox
4 -1 roll 10 string cvs print (,) print
3 -1 roll 10 string cvs print ( ) print
2 -1 roll 10 string cvs print (,) print
10 string cvs print ( ) print
grestore
( text:) 10 string cvs print
dup print ( ) print
( currentrgbcolor:) print
currentrgbcolor
3 -1 roll 10 string cvs print (,) print
2 -1 roll 10 string cvs print (,) print
10 string cvs print ( ) ==
systemdict /show get exec
} def
trash.ps
%!PS-Adobe-3.0
%%Title: mwe.ps
%%Pages: 001
%%BoundingBox: 0 0 595 842
%%EndComments
%%Page: 1 1
%%PageBoundingBox: 0 0 595 842
0 0 1 setrgbcolor
0 0 595 842 rectfill
1 0 0 setrgbcolor
247 371 100 100 rectfill
/Times-Roman findfont
72 scalefont
setfont
newpath
247 300 moveto (Chris) show
50 90 moveto (Fred) show
150 200 moveto (Mary) show
0 1 0 setrgbcolor
300 350 moveto (Mr. Green) show
0 0 1 setrgbcolor
100 400 moveto (Mr. Blue) show
showpage

The text has been converted to vector paths. 249 299 43 50 /5D begins the first letter "C", then 289 300 is the "h", 289 300 the "r"....
What pswrite has done is eliminate the need for a font, so while your original code used /Times-Roman, the distilled code doesn't need any font, but rather draws the text using vectors.
I'm not sure exactly what you are after, but you could try "ps2write" or "epswrite" as alternatives to "pswrite". pswrite is used to write to ps level 1 standard and ps2write will write ps level 2 output. Nobody requires ps level 1 anymore, so level 2 would be acceptable. The epswrite will write to encapsulated postscript (eps).