Parsing TCPDUMP output - perl

Im trying to parse my TCPDUMP command output to print "ok" if a specific server sends data back before a given amount of seconds ( or nanoseconds ) Example:
11:45:41.198150 IP X.X.X.X.662 > Y.Y.Y.Y.161: UDP, length 37
11:45:41.315699 IP Y.Y.Y.Y.161 > X.X.X.X.662: UDP, length 13
11:45:42.198845 IP X.X.X.X.168.662 > Y.Y.Y.Y.161: UDP, length 37
11:45:42.316745 IP Y.Y.Y.Y.161 > X.X.X.X.662: UDP, length 13
as you can see, it first outputs the first row where im sending data, then the server i sent data to repsponds, Now i want it so if the server that i sent data to doesnt respond withen a set amount of seconds, then i do nothing. but if it does, then i print "ok".
Somtimes data will look like this
11:45:41.198150 IP X.X.X.X.662 > Y.Y.Y.Y.161: UDP, length 37
11:45:41.315699 IP Y.Y.Y.Y.161 > X.X.X.X.662: UDP, length 13
11:45:42.198845 IP X.X.X.X.168.662 > Y.Y.Y.Y.161: UDP, length 37
11:45:42.198845 IP X.X.X.X.168.662 > Y.Y.Y.Y.161: UDP, length 37
11:45:42.198845 IP X.X.X.X.168.662 > Y.Y.Y.Y.161: UDP, length 37
11:45:42.316745 IP Y.Y.Y.Y.161 > X.X.X.X.662: UDP, length 13
And The ips will respond at diffrent times, how could i still parse this.

With the information from your other question Parsing TCPDUMP output and since you asked about parsing the file, there are several ways it can be done. I have generate a simple script to read in the data and get it into a hash. I'm going with the data from your other posting as the input you want to parse. It does not do data validation and expects all lines to be the same format in the file.
# Checking for errors (Good practice to always use)
use strict;
# open the file (first on on the command line)1
open my $input,$ARGV[0] or die "Unable to open file: $ARGV[0]";
# scalar/variable into which to save the line read from the file
my $line;
# Hash/mapping by machine for the time
my %machine2time;
# Array/List to store parsed line into individual list/array items
my #parsedLineSpace;
# Read line from the file. This will fail when a line cannot be read
while ( $line = <$input> )
{
# Parse the line based on spaces first element is time (index 0),
# the second is IP (index 1)
#parsedLineSpace = split('\s+',$line);
# If the IP exists in the hash/mapping, then the delta time needs to be
# computed as there is a response
if ( exists $machine2time{$parsedLineSpace[1]} )
{
# Get the times which are needed to compute the difference
# and place in scalar/variables
my $firstTime = $machine2time{$parsedLineSpace[1]};
my $responseTime = $parsedLineSpace[0];
# Compute the time difference (Exercise for the user)
# Use an array and split to break the time into individual components or
# the to do that. Make sure you use a \ to escape the . for the split
# and that you check for boundary conditions
# Remove the item from the hash/mapping as it is not needed and
# any remaining items left in the hash would be items which did
# get a response
delete $machine2time{$parsedLineSpace[1]};
}
# else this the first occurrence (or there was no response) so
# save the time for use later
else
{
$machine2time{$parsedLineSpace[1]} = $parsedLineSpace[0];
}
}
# Print out any machines which did not have a matched response
print "\nIPs which did not get a response\n";
# For each key in the hash/mapping (sorted) print out the key which
# is the IP
foreach my $machine ( sort keys %machine2time )
{
print "$machine\n";
}
Hopefully this will get you started on your effort

Related

match and print each selection from array

I have a script which collects configuration data from a database and stores it in an array. I need to print find each interface and print it's config detail for a device. I will only post the parts that I need help with.
So firstly, here is an extract from the array:
track 2 interface GigabitEthernet1/6 line-protocol
!
!
!
!
!
interface Port-channel10.1
description Enclosure 1 - 3040-1b
switchport
switchport trunk encapsulation dot
switchport trunk allowed vlan 200, 202
switchport mode trunk
logging event link-status
logging event trunk-status
logging event bundle-status
logging event spanning-tree status
shutdown
interface Ethernet1/20
description tx to something
switchport mode trunk
switchport trunk allowed vlan 200-300
spanning-tree guard root
speed 1000
duplex full
interface Bundle-Ether2.1
description Bundle link to something
service-policy input qos-pol1
vrf V17:vodanet
ipv4 address 10.1.1.1 255.0.0.0
encapsulation 250
interface Ethernet1/1
description some interface desc
switchport mode trunk
switchport trunk allowed vlan 200-299
spanning-tree guard root
duplex full
no negotiate auto
clock timezone GMT 2 0
line console
terminal length 48
...
So each interface does not necessarily have the same amount of config detail following it. I basically need to take each of these interface configurations and print them individually. I will be storing these in smaller config-let flat files. So I need to print from interface ... till the last item before the next interface, but I am lost in searching. This is my latest attempt, but only works if I search for 2 different start/end strings, like so:
foreach (#array) {
if (/^interface/) {
$counter = 1;
} elsif (/duplex/) {
$counter =0;
} elsif ($counter) {
print;
}
}
The problem here is I am missing the first and last string, and not all interfaces end with duplex. I was thinking of using the beginning of line interface and then each double white space delimited items after but unsure how. Can someone please help me in finding a solution.
Edit
So to clarify array it is simply lines grabbed from a database and pushed to #array
The problem: An array (#lines) has "interface sections" which start with an /^interface/ line and run until the next interface line. The last section stops at the first unindented line. Parse interface sections from the array.
One way: Find indices of each interface line in the array #lines. Then elements of #lines between successive indices are the interface sections. The last one is found separately.
my #idx = grep { $lines[$_] =~ /^interface/ } 0..$#lines;
for my $i (0..$#idx-1) {
say "Interface:";
say "\t$_" for #lines[$idx[$i]..$idx[$i+1]-1];
}
# The last interface section stops at the first unindented line
say "Interface:";
for my $i ($idx[-1] .. $#lines) {
last if $lines[$i] =~ /^\S/ and $lines[$i] !~ /^interface/;
say "\t$lines[$i]";
}
The last interface segment goes until the first unindented line, as clarified in comments.
Prints "Interface:" are there only to visually distinguish sections; the lines for each interface can be added to an arrayref instead of printing, for example. Tested on the posted sample.
Another way: Iterate over #lines and for each /^interface/ line add a new arrayref to the array with all sections. Then the lines are added to that, last, arrayref. Exit on the first unindented line which isn't /^interface/
The program gets lines from the submitted file (or lines.txt), with the question's sample.
use warnings;
use strict;
use feature 'say';
use Data::Dump qw(dd);
use Path::Tiny;
my $file = shift #ARGV or 'lines.txt';
my #lines = path($file)->lines({chomp=>1});
my #if_sections; # store arrayrefs with lines for each interface
for (#lines) {
if (/^interface/) {
push #if_sections, [ $_ ]; # add arrayref for new interface
next;
}
elsif (/^\S/) { last }
push #{$if_sections[-1]}, $_; # add lines to the last arrayref
}
dd \#if_sections;
Uses the handy Path::Tiny to read the file. The module runs checks and croaks on failures.
The dd from Data::Dump is used to easily show the data structure, for convenience. This is an array with elements that are array references, so to work with it
foreach my $iface (#if_sections) {
say "Interface section:";
foreach my $line (#$iface) {
say $line;
}
}
what can be written more compactly in a number of ways.
See the tutorial perlreftut and
the complex data structure cookbook perldsc.

Chilkat encryption doesn't work as expected

I was trying to test file encryption using the chilkat functionality. Based on code found on this example page, I replaced the last part with this:
# Encrypt a string...
# The input string is 44 ANSI characters (i.e. 44 bytes), so
# the output should be 48 bytes (a multiple of 16).
# Because the output is a hex string, it should
# be 96 characters long (2 chars per byte).
my $input = "sample.pdf";
# create file handle for the pdf file
open my $fh, '<', $input or die $!;
binmode ($fh);
# the output should be sample.pdf.enc.dec
open my $ffh, '>', "$input.enc.dec" or die $!;
binmode $ffh;
my $encStr;
# read 16 bytes at a time
while (read($fh,my $block,16)) {
# encrypt the 16 bytes block using encryptStringEnc sub provided by chilkat
$encStr = $crypt->encryptStringENC($block);
# Now decrypt:
# decrypt the encrypted block
my $decStr = $crypt->decryptStringENC($encStr);
# print it in the sample.pdf.enc.dec file
print $ffh $decStr;
}
close $fh;
close $ffh;
Disclaimer:
I know the CBC mode is not recommended for file encryption because if one block is lost, the other blocks are lost too.
The output file is corrupted and when I look with beyond compare at the two files, there are chunks of the file which match and there are chunks of file which doesn't. What am I doing wrong?
You're trying to use character string encryption (encryptStringENC(), decryptStringENC()) for what is, at least partly, a binary file.
This worked for me:
my $input = "sample.pdf";
# create file handle for the pdf file
open my $fh, '<', $input or die $!;
binmode $fh;
# the output should be sample.pdf.enc.dec
open my $ffh, '>', "$input.enc.dec" or die $!;
binmode $ffh;
my $inData = chilkat::CkByteData->new;
my $encData = chilkat::CkByteData->new;
my $outData = chilkat::CkByteData->new;
# read 16 bytes at a time
while ( my $len = read( $fh, my $block, 16 ) ) {
$inData->clear;
$inData->append2( $block, $len );
$crypt->EncryptBytes( $inData, $encData );
$crypt->DecryptBytes( $encData, $outData );
print $ffh $outData->getData;
}
close $fh;
close $ffh;
You likely better off perusing the Chilkat site further though, there are sample codes for binary data.
I'm going to write and post a link to a sample that is much better than the examples posted here. The examples posted here are not quite correct. There are two important Chilkat Crypt2 properties that one needs to be aware of: FirstChunk and LastChunk. By default, both of these properties are true (or the value 1 in Perl). This means that for a given call to encrypt/decrypt, such as EncryptBytes, DecryptBytes, etc. it assumes the entire amount of data was passed. For CBC mode, this is important because the IV is used for the first chunk, and for the last chunk, the output is padded to the block size of the algorithm according to the value of the PaddingScheme property.
One can instead feed the input data to the encryptor chunk-by-chunk by doing the following:
For the 1st chunk, set FirstChunk=1, LastChunk=0.
For middle chunks, set FirstChunk=0, LastChunk=0.
For the final chunk (even if a 0-byte final chunk), set FirstChunk=0, LastChunk=1. This causes a final padded output block to be emitted.
When passing chunks using FirstChunk/LastChunk, one doesn't need to worry about passing chunks matching the block size of the algorithm. If a partial block is passed in, or if the bytes are not an exact multiple of the block size (16 bytes for AES), then Chilkat will buffer the input and the partial block will be added to the data passed in the next chunk. For example:
FirstChunk=1, LastChunk=0, pass in 23 bytes, output is 16 bytes, 7 bytes buffered.
FirstChunk=0, LastChunk=0, pass in 23 bytes, output is 16 bytes, (46-32 bytes) 14 bytes buffered
FirstChunk=0, LastChunk=1, pass in 5 bytes, output is 32 bytes, (14 buffered bytes + 5 more = 19 bytes. The 19 bytes is one full block (16 bytes) plus 3 bytes remainder, which is padded to 16, and thus the output is 32 bytes and the CBC stream is ended.
This example demonstrates using FirstChunk/LastChunk. Here's the example: https://www.example-code.com/perl/encrypt_file_chunks_cbc.asp

Perl Pcap Module not capturing over 1500 Bytes

I have a small pcap listener i made in perl. And whenever i get anything over 1500 Bytes, it just prints as 1500.
66.0.X.X 1500
Now i get that reply when i sent a 2000 byte packet, i also tried sending 1600, and 10k.
No matter what i get 1500 for pcap.
How can i fix this i looked at SNAPLEN but when i set it under or over, i get the same results.
I have also tried tcpdump and i get weird results, as you can see the first "length" is 1500, but the second a line under is 5000 which is what i sent.
tcpdump: listening on eth1, link-type EN10MB (Ethernet), capture size 65535 bytes
21:59:06.142530 IP (tos 0x0, ttl 58, id 45206, offset 0, flags [+], proto UDP (17), length 1500)
37.X.X.X.48254 > XXX.54: UDP, length 5000
my code:
use Net::Pcap;
use NetPacket::Ethernet;
use NetPacket::Ethernet qw(:strip);
use NetPacket::IP;
use NetPacket::IP;
use NetPacket::TCP;
use strict;
my $err;
# Use network device passed in program arguments or if no
# argument is passed, determine an appropriate network
# device for packet sniffing using the
# Net::Pcap::lookupdev method
my $dev = $ARGV[0];
unless (defined $dev) {
$dev = Net::Pcap::lookupdev(\$err);
if (defined $err) {
die 'Unable to determine network device for monitoring - ', $err;
}
}
# Look up network address information about network
# device using Net::Pcap::lookupnet - This also acts as a
# check on bogus network device arguments that may be
# passed to the program as an argument
my ($address, $netmask);
if (Net::Pcap::lookupnet($dev, \$address, \$netmask, \$err)) {
die 'Unable to look up device information for ', $dev, ' - ', $err;
}
# Create packet capture object on device
my $object;
$object = Net::Pcap::open_live($dev, 65535, 1, 0, \$err);
unless (defined $object) {
die 'Unable to create packet capture on device ', $dev, ' - ', $err;
}
# Compile and set packet filter for packet capture
# object - For the capture of TCP packets with the SYN
# header flag set directed at the external interface of
# the local host, the packet filter of '(dst IP) && (tcp
# [13] & 2 != 0)' is used where IP is the IP address of
# the external interface of the machine. For
# illustrative purposes, the IP address of 127.0.0.1 is
# used in this example.
my $filter;
Net::Pcap::compile(
$object,
\$filter,
'(port 111)',
0,
$netmask
) && die 'Unable to compile packet capture filter';
Net::Pcap::setfilter($object, $filter) &&
die 'Unable to set packet capture filter';
# Set callback function and initiate packet capture loop
Net::Pcap::loop($object, -1, \&process_packet, '') ||
die 'Unable to perform packet capture';
Net::Pcap::close($object);
sub process_packet {
my ($user_data, $hdr, $pkt) = #_;
my $ip_obj = NetPacket::IP->decode(eth_strip($pkt));
#print("$ip_obj->{src_ip} -> $ip_obj->{dest_ip} $ip_obj->{caplen}\n");
warn "packet!\n";
my %header = %$hdr;
#process_packet(\%header, $pkt);
my $len = length $pkt;
my $fag = length $user_data;
my $fag2 = length $hdr;
warn "$header{len} $header{caplen} $len $fag $fag2\n";
}
From "listening on eth1" I infer you're capturing on an Ethernet. The largest packet size on Ethernet is 1518 bytes (except for non-standard "jumbo frames"), which is:
14 bytes of Ethernet header;
1500 bytes of Ethernet payload;
4 bytes of frame check sequence at the end.
This means that (unless the network is using jumbo frames) the largest IP packet you can send on an Ethernet is 1500 bytes.
So, if you try to send 5000 bytes of data over UDP-over-IPv4 on an Ethernet, that will become a 5008-byte UDP packet when the 8-byte UDP header is added, and that will become a 5028-byte or larger IPv4 packet when the IPv4 header is added (the minimum size of an IPv4 header is 20 bytes, and it can be bigger if there are options in the packet). That's too big for Ethernet, so the IP layer in the protocol stack on your machine will "fragment" that packet into multiple smaller IP packets, and, if all of those fragments arrive at the destination machine, its IP layer will reassemble them into a larger IP packet and then hand the IP payload of the reassembled packet to the UDP layer, so the program receiving the UDP packet will see all 5000 bytes.
Your tcpdump output is the output for the first of the fragments; the IP-layer total length of the packet, extracted from the IP header, is 1500 (IP's length field includes the length of the IP headers and the payload), but the UDP-layer length, extracted from the UDP header (which will normally fit in the first fragment), is 5000.
This has nothing to do with the snapshot length; it has to do with the way IP works, and the maximum packet size on Ethernet.
Upon advice from the comments, I decided to try Net::Pcap::Easy instead of Net::Pcap. As stated in the pod for the former module: "Net::Pcap is awesome, but it's difficult to bootstrap".
The following code solves my problem:
use strict;
use warnings;
use Net::Pcap::Easy;
# all arguments to new are optoinal
my $npe = Net::Pcap::Easy->new(
dev => "eth1",
filter => "port 111",
packets_per_loop => 10,
bytes_to_capture => 1024,
timeout_in_ms => 0, # 0ms means forever
promiscuous => 0, # true or false
udp_callback => sub {
my ($npe, $ether, $ip, $udp, $header ) = #_;
print "UDP: $ip->{src_ip}:$udp->{src_port}"
. " -> $ip->{dest_ip}:$udp->{dest_port} $udp->{len}\n";
},
# tcp_callback => sub {
# my ($npe, $ether, $ip, $tcp, $header ) = #_;
# my $xmit = localtime( $header->{tv_sec} );
#
# print "$xmit TCP: $ip->{src_ip}:$tcp->{src_port}"
# . " -> $ip->{dest_ip}:$tcp->{dest_port}\n";
#
# },
#
# icmp_callback => sub {
# my ($npe, $ether, $ip, $icmp, $header ) = #_;
# my $xmit = localtime( $header->{tv_sec} );
#
# print "$xmit ICMP: $ether->{src_mac}:$ip->{src_ip}"
# . " -> $ether->{dest_mac}:$ip->{dest_ip}\n";
# },
);
1 while $npe->loop;

Pattern matching an array with specific start and end words/characters

Have some output in an array which I am trying to pull details from where the output begins with a specific word and ends with a specific word/character. This output is then to be printed to the screen.
The output in the array which I am working with is:
router rip
version 2
redistribute bgp 45134 metric 3
passive-interface Serial1/3:1.333
passive-interface Serial3/1:3.333
passive-interface Serial3/1:5.333
passive-interface Serial3/2:1.333
passive-interface Serial3/4:1.333
passive-interface Serial3/4:17.333
passive-interface Serial6/1:1.333
no auto-summary
!
address-family ipv4 vrf TestVRF-0001
redistribute bgp 45134 metric 3
network 10.0.0.0
no auto-summary
version 2
exit-address-family
!
The perl code I have generated so far is below:
elsif ( $action eq "show_vrf1" ) {
my $cmd = "show run | begin router rip";
my #lines = $s->cmd(String => $cmd,
Prompt => "/$enableprompt/",
Timeout => 10);
foreach my $line (#lines) {
if(/address-family ipv4 vrf TestVRF-0001.*?!/){
$result=$1;
print $result;
}
}
}
Which I am wanting to only pull the below out of the array:
address-family ipv4 vrf TestVRF-0001
redistribute bgp 45134 metric 3
network 10.0.0.0
no auto-summary
version 2
exit-address-family
!
For some reason when I run the script, I just get a blank screen with no data pulled from the array.
if($line =~ /address-family ipv4 vrf TestVRF-0001.*?!/){
instead of
if(/address-family ipv4 vrf TestVRF-0001.*?!/){
?
You match $line against the regular expression. The output you expect is not on one line, so no line can match it.
You can store the whole multiline string in a scalar:
my $output = join q(), $s->cmd(...);
And then, you can retrieve the output, if you use parentheses to really capture a part of the string:
if ($output =~ /(address-family ipv4 vrf TestVRF-0001.*?!)/s) {
my $result = $1;
Note that the /s modifier is needed to make dot match a newline, which it normaly does not.

mailserver log filtering

I've got multi-GB mailserver log file and a list of ~350k messages ID.
I want to pull out from the big log file rows with IDs from the long list... and I want it faster than it is now...
Currently I do it in perl:
#!/usr/bin/perl
use warnings;
#opening file with the list - over 350k unique ID
open ID, maillog_id;
#lista_id = <ID>;
close ID;
chomp #lista_id;
open LOG, maillog;
# while - foreach would cause out of memory
while ( <LOG> ) {
$wiersz = $_;
my #wiersz_split = split ( ' ' , $wiersz );
#
foreach ( #lista_id ) {
$id = $_;
# ID in maillog is 6th column
if ( $wiersz_split[5] eq $id) {
# print whole row when matched - can be STDOUT or file or anything
print "#wiersz_split\n";
}
}
}
close LOG;
It works but it is slow... Every line from log is taken into comparison with list of ID.
Should I use database and perform a kind of join? Or compare substrings?
There are a lot of tools for log analyse - e.g. pflogsumm... but it just summarizes. E.g. I could use
grep -c "status=sent" maillog
It would be fast but useless and I would use it AFTER filtering my log file... the same is for pflogsumm etc. - just increasing variables.
Any suggestions?
-------------------- UPDATE -------------------
thank you Dallaylaen,
I succeded with this (instead internal foreach on #lista_id):
if ( exists $lista_id_hash{$wiersz_split[5]} ) { print "$wiersz"; }
where %lista_id_hash is a hash table where keys are items taken from my ID list. It works superfast.
Processing 4,6 GB log file with >350k IDs takes less than 1 minute to filter interesting logs.
Use a hash.
my %known;
$known{$_} = 1 for #lista_id;
# ...
while (<>) {
# ... determine id
if ($known{$id}) {
# process line
};
};
P.S. If your log is THAT big, you're probably better off with splitting it according to e.g. last two letters of $id into 256 (or 36**2?) smaller files. Something like a poor man's MapReduce. The number of IDs to store in memory at a time will also be reduced (i.e. when you're processing maillog.split.cf, you should only keep IDs ending in "cf" in hash).