I need to add a timestamp in front of the output of a long-executing command (a "tcpdump", in my use-case...).
It - very simplified - looks like this one:
(echo A1; sleep 3; echo B2) | perl -MPOSIX -pe 'print strftime "%T ", localtime $^T; s/\d//'
which gives this kind of output:
16:10:24 A
16:10:24 B
i.e.: perl's localtime is (obviously) called when perl is invoked.
Instead I need this kind of result:
16:10:24 A
16:10:27 B
i.e.: time stamp should be relative to the input's generation time...
Any smart (or no so smart :-) solution?
Just remove the $^T from your Perl command. That way, you will use the current time instead of the process start time. See the docs for $^T.
However, a more elegant formulation with Perl would be:
... | perl -MPOSIX -ne's/\d//; print strftime("%T ", localtime), $_'
You could pipe the output to:
awk '{ print strftime("%T"), $0; }'
Example:
while : ; do echo hey; sleep 1; done | awk '{ print strftime("%T"), $0; }'
20:49:58 hey
20:49:59 hey
20:50:00 hey
20:50:01 hey
20:50:02 hey
20:50:03 hey
20:50:04 hey
20:50:05 hey
Alternatively, you could use ts:
ts '%T'
(echo A1; sleep 3; echo B2) | perl -MPOSIX -pe 'print strftime "%T ", localtime; s/\d//'
Works excellent for me. Why you added $^T there?
on Linux, you can use tcpdump -tttt to print the timestamp before each output line.
# tcpdump -tttt -c 1 2>/dev/null
2013-12-13 23:42:12.044426 IP 10.0.2.15.ssh > 10.0.2.2.53466: Flags [P.], seq 464388005:464388121, ack 16648998, win 65535, length 116
If your tcpdump doesn't have the -tttt option, you should use awk from devnull
Related
Say I have a file like so:
+jaklfjdskalfjkdsaj
fkldsjafkljdkaljfsd
-jslakflkdsalfkdls;
+sdjafkdjsakfjdskal
I only want to find and count the amount of times during this file a line that starts with - is immediately followed by a line that starts with +.
Rules:
No external scripts
Must be done from within a bash script
Must be inline
I could figure out how to do this in a Python script, for instance, but I've never had to do something this extensive in Bash.
Could anyone help me out? I figure it'll end up being grep, perl, or maybe a talented sed line -- but these are things I'm still learning.
Thank you all!
grep -A1 "^-" $file | grep "^+" | wc -l
The first grep finds all of the lines starting with -, and the -A1 causes it to also output the line after the match too.
We then grep that output for any lines starting with +. Logically:
We know the output of the first grep is only the -XXX lines and the following lines
We know that a +xxx line cannot also be a -xxx line
Therefore, any +xxx lines must be following lines, and should be counted, which we do with wc -l
Easy in Perl:
perl -lne '$c++ if $p and /^\+/; $p = /^-/ }{ print $c' FILE
awk one-liner:
awk -v FS='' '{x=x sprintf("%s", $1)}END{print gsub(/-\+/,"",x)}' file
e.g.
kent$ cat file
+jaklfjdskalfjkdsaj
fkldsjafkljdkaljfsd
-jslakflkdsalfkdls;
+sdjafkdjsakfjdskal
-
-
-
+
-
+
foo
+
kent$ awk -v FS='' '{x=x sprintf("%s", $1)}END{print gsub(/-\+/,"",x)}' file
3
Another Perl example. Not as terse as choroba's, but more transparent in how it works:
perl -e'while (<>) { $last = $cur; $cur = $_; print $last, $cur if substr($last, 0, 1) eq "-" && substr($cur, 0, 1) eq "+" }' < infile
Output:
-jslakflkdsalfkdls;
+sdjafkdjsakfjdskal
Pure bash:
unset c p
while read line ; do
[[ $line == +* && $p == 0 ]] && (( c++ ))
[[ $line == -* ]]
p=$?
done < FILE
echo $c
I want to feed input to a C program with a perl script like this
./cprogram $(perl -e 'print "\xab\xcd\xef";').
However, the string must be read from a file. So I get something like this:
./cprogram $(perl -e 'open FILE, "<myfile.txt"; $file_contents = do { local $/; <FILE> }; print $file_contents'. However, now perl interprets the string as the string "\xab\xcd\xef", and I want it to interpret it as the byte sequence as in the first example.
How can this be achieved? It has to be ran on a server without File::Slurp.
In the first case, you pass the three bytes AB CD EF (produced by the string literal "\xAB\xCD\xEF") to print.
In the second case, you must be passing something other than those three bytes to print. I suspect you are passing the twelve character string \xAB\xCD\xEF to print.
So your question becomes: How does one convert the twelve-character string \xAB\xCD\xEF into the three bytes AB CD EF. Well, you'd require some kind of parser such as
s/\\x([0-9a-fA-F][0-9a-fA-F])|\\([^x])|([^\\]+)/
$1 ? chr(hex($1)) : $2 ? $2 : $3
/eg
And here it is at work:
$ perl -e'print "\\xAB\\xCD\\xEF";' >file
$ echo -n "$( perl -0777pe'
s{\\x([0-9a-fA-F][0-9a-fA-F])|\\([^x])|([^\\]+)}{
$1 ? chr(hex($1)) : $2 // $3
}eg;
' file )" | od -t x1
0000000 ab cd ef
0000003
Is Perl's eval too evil? If not, end in print eval("\"$file_contents\"");
Or can you prepare the file in advance using Perl? EG print FILE "\xAB\xCD\xED"; then read the resulting file with your existing code.
using a bash trick:
perl -e "$(echo "print \"$(cat input)"\")"
which for your example becomes:
./cprogram "$(perl -e "$(echo "print \"$(cat myfile.txt)"\")")"
In awk I can write: awk -F: 'BEGIN {OFS = FS} ...'
In Perl, what's the equivalent of FS? I'd like to write
perl -F: -lane 'BEGIN {$, = [what?]} ...'
update with an example:
echo a:b:c:d | awk -F: 'BEGIN {OFS = FS} {$2 = 42; print}'
echo a:b:c:d | perl -F: -ane 'BEGIN {$, = ":"} $F[1] = 42; print #F'
Both output a:42:c:d
I would prefer not to hard-code the : in the Perl BEGIN block, but refer to wherever the -F option saves its argument.
To sum up, what I'm looking for does not exist:
there's no variable that holds the argument for -F, and more importantly
Perl's "FS" is fundamentally a different data type (regular expression) than the "OFS" (string) -- it does not make sense to join a list of strings using a regex.
Note that the same holds true in awk: FS is a string but acts as regex:
echo a:b,c:d | awk -F'[:,]' 'BEGIN {OFS=FS} {$2=42; print}'
outputs "a[:,]42[:,]c[:,]d"
Thanks for the insight and workarounds though.
You can use perl's -s (similar to awk's -v) to pass a "FS" variable, but the split becomes manual:
echo a:b:c:d | perl -sne '
BEGIN {$, = $FS}
#F = split $FS;
$F[1] = 42;
print #F;
' -- -FS=":"
If you know the exact length of input, you could do this:
echo a:b:c:d | perl -F'(:)' -ane '$, = $F[1]; #F = #F[0,2,4,6]; $F[1] = 42; print #F'
If the input is of variable lengths, you'll need something more sophisticated than #f[0,2,4,6].
EDIT: -F seems to simply provide input to an automatic split() call, which takes a complete RE as an expression. You may be able to find something more suitable by reading the perldoc entries for split, perlre, and perlvar.
You can sort of cheat it, because perl is actually using the split function with your -F argument, and you can tell split to preserve what it splits on by including capturing parens in the regex:
$ echo a:b:c:d | perl -F'(:)' -ane 'print join("/", #F);'
a/:/b/:/c/:/d
You can see what perl's doing with some of these "magic" command-line arguments by using -MO=Deparse, like this:
$ perl -MO=Deparse -F'(:)' -ane 'print join("/", #F);'
LINE: while (defined($_ = <ARGV>)) {
our(#F) = split(/(:)/, $_, 0);
print join('/', #F);
}
-e syntax OK
You'd have to change your #F subscripts to double what they'd normally be ($F[2] = 42).
Darnit...
The best I can do is:
echo a:b:c:d | perl -ne '$v=":";#F = split("$v"); $F[1] = 42; print join("$v", #F) . "\n";'
You don't need the -F: this way, and you're only stating the colon once. I was hoping there was someway of setting variables on the command line like you can with Awk's -v switch.
For one liners, Perl is usually not as clean as Awk, but I remember using Awk before I knew of Perl and writing 1000+ line Awk scripts.
Trying things like this made people think Awk was either named after the sound someone made when they tried to decipher such a script, or stood for AWKward.
There is no input record separator in Perl. You're basically emulating awk by using the -a and -F flags. If you really don't want to hard code the value, then why not just use an environmental variable?
$ export SPLIT=":"
$ perl -F$SPLIT -lane 'BEGIN { $, = $ENV{SPLIT}; } ...'
"a004-1b","North","at006754"
"a004-1c","south","atytgh0"
"a004-1d","east","atrthh"
"a010-1a","midwest","atyu"
"a010-1b","south","rfg67"
I want to print the first column and the second column without any extra character I want eliminate all ("", and the third column) Thanks in advance
awk -F'^"|","|"$' '{print $2,$3}' ./infile.csv
The above script will even handle fields that have embedded double quotes or commas. The only downside (if you can call it that) is that the first field starts at $2
Proof of Concept
$ awk -F'^"|","|"$' '{print $2,$3}' ./infile.csv
a004-1b North
a004-1c south
a010-1a midwest
a010-1b south
You need GNU Awk 4 for this to work:
$ gawk -vFPAT='[^",]+' '{print $1,$2}'
I love this new "field pattern" feature. It's my new hammer and everything is a nail. Read up on it at http://www.gnu.org/software/gawk/manual/html_node/Splitting-By-Content.html
(Written this way it doesn't account for embedded commas or quotes, because the question implies this is not needed.)
If you're using awk for this, why put a Perl tag on it?
In Perl:
#!/usr/bin/env perl
use strict;
use warnings;
use Data::Dumper;
# Make Data::Dumper pretty
$Data::Dumper::Sortkeys = 1;
$Data::Dumper::Indent = 1;
# Set maximum depth for Data::Dumper, zero means unlimited
local $Data::Dumper::Maxdepth = 0;
use Text::CSV;
my $csv = Text::CSV->new();
while( my $row = $csv->getline( \*DATA )){
print 'row: ', Dumper $row;
}
__DATA__
"a004-1b","North","at006754"
"a004-1c","south","atytgh0""a004-1d","east","atrthh"
"a010-1a","midwest","atyu"
"a010-1b","south","rfg67"
awk -F'\"|\,' '{print $2,$5}' sample
Not handling embedded double quotes:
sed -e 's/^"\([^"]*\)","\([^"]*\)".*/\1 \2/'
To handle them:
sed -n -e 's/^"//;s/"$//;s/","/ /;s/","/\n/;P'
The above works even for a 1 or 2 field input.
If you want it "pure" awk or sed, this won't fit the bill, but otherwise it works:
awk -F, '{print $1 " " $2}' | tr -d '"'
I have a text file which contains:
Cycle code
Cycle month
Cycle year
Event type ID
Event ID
Network start time
I want to change this text so that when ever there is a space, I want to replace it with a _. And after that, I want the characters to lower case letter like below:
cycle_code
cycle_month
cycle_year
event_type_id
event_id
network_start_time
How could I accomplish this?
Another Perl method:
perl -pe 'y/A-Z /a-z_/' file
tr alone works:
tr ' [:upper:]' '_[:lower:]' < file
Looking into sed documentation some more and following advice from the comments the following command should work.
sed -r {filehere} -e 's/[A-Z]/\L&/g;s/ /_/g' -i
There is a perl tag in your question as well. So:
#!/usr/bin/perl
use strict; use warnings;
while (<DATA>) {
print join('_', split ' ', lc), "\n";
}
__DATA__
Cycle code
Cycle month
Cycle year
Event type ID
Event ID
Network start time
Or:
perl -i.bak -wple '$_ = join('_', split ' ', lc)' test.txt
sed "y/ABCDEFGHIJKLMNOPQRSTUVWXYZ /abcdefghijklmnopqrstuvwxyz_/" filename
Just use your shell, if you have Bash 4
while read -r line
do
line=${line,,} #change to lowercase
echo ${line// /_}
done < "file" > newfile
mv newfile file
With gawk:
awk '{$0=tolower($0);$1=$1}1' OFS="_" file
With Perl:
perl -ne 's/ +/_/g;print lc' file
With Python:
>>> f=open("file")
>>> for line in f:
... print '_'.join(line.split()).lower()
>>> f.close()