Perl: pack int to arbitrary length byte string - perl

I want to encode numbers in N bit containers and send them in a UDP packet. A receiver will know N, and the receiver will grab a number from exactly N bits.(N <= 64)
Somethink like this:
sub to56BIT {
return pack("??", shift);
}
sub to24BIT {
return pack("??", shift);
}
my $n = 7;
to24BIT($n);
On the receiver's side:
int n = Get_val24(byte_stream, offset);
Is there any way to do this in Perl?
I think solution might be:
sub packIntN {
my $int = shift;
my $length = shift;
return pack("B" . $length, substr(unpack("B64", pack("Q>", $int)), 64 - $length));
}
But maybe there is more elegant way.
Input/Output example:
We have a script test.pl:
use strict;
use warnings;
sub to24BIT {
#???
}
my $n = 7;
print to24BIT($n);
I want this:
./test.pl | hexdump -C
00000000 00 00 07 |...|
00000003
Another script test2.pl:
use strict;
use warnings;
sub to40BIT {
#???
}
my $n = 7;
print to40BIT($n);
I want this:
./test.pl | hexdump -C
00000000 00 00 00 00 07 |.....|
00000005

Is N always going to be an integer factor of 8 (one of 8, 16, 24, 32, 40, 48, 56, 64)? If so, for speed, I recommend writing a packer for each size and use a dispatch table to find the right packer.
sub pack_8bit { pack('C', $_[0]) }
sub pack_16bit { pack('S>', $_[0]) }
sub pack_24bit { substr(pack('L>', $_[0]), 1) }
sub pack_32bit { pack('L>', $_[0]) }
sub pack_40bit { substr(pack('Q>', $_[0]), 3) }
sub pack_48bit { substr(pack('Q>', $_[0]), 2) }
sub pack_56bit { substr(pack('Q>', $_[0]), 1) }
sub pack_64bit { pack('Q>', $_[0]) }
{
my %packers = (
8 => \&pack_8bit, 40 => \&pack_40bit,
16 => \&pack_16bit, 48 => \&pack_48bit,
24 => \&pack_24bit, 56 => \&pack_56bit,
32 => \&pack_32bit, 64 => \&pack_64bit,
);
sub pack_num {
my $packer = $packers{$_[0]}
or die;
return $packer->($_[1]);
}
sub get_packer {
my $packer = $packers{$_[0]}
or die;
return $packer;
}
}
my $packed = pack_num(40, 7);
-or-
my $packer = get_packer(40);
my $packed = $packer->(7);
If you're planning on packing multiple numbers into one string (like pack('L>*', #nums)), I'd also use a dispatch table like this, though I'm not sure what would be the fastest implementation of pack_24bit, pack_40bit, pack_48bit and pack_56bit (other than a C solution).

Bearing in mind that you will always have a whole
number of bytes, I ended up with
substr(pack("Q>",$n<<(64-$len)),0,($len+7)/8);
and
unpack("Q>",$s.(0 x 8)) >> (64-$len);
as tried in this example:
#!/usr/bin/perl
$len = 40;
$n = 7;
$s = substr(pack("Q>",$n<<(64-$len)),0,($len+7)/8);
open(PIPE,"| hexdump -C");
print PIPE $s;
close PIPE;
$v = unpack("Q>",$s.(0 x 8)) >> (64-$len);
printf "%d\n",$v;

Related

unpacking a data structure whose first byte indicates length

I am trying to unpack a TLE (Tagged Logical Element) from an IBM AFP format file.
The specification (http://www.afpcinc.org/wp-content/uploads/2017/12/MODCA-Reference-09.pdf) indicates that these are two triplets (even though there are four values) that are structured as follows (with their byte offsets):
0: Tlength | 1: Tid | 2-n: Parameter (= 2: Type + 3: Format + 4-n: EBCDIC encoded String)
Example (with two triplets, one indicating the name and one the value):
0C 02 0B 00 C3 A4 99 99 85 95 83 A8 07 36 00 00 C5 E4 D9
12 KEY UID CHAR C u r r e n c y 7 VAL RESERVED E U R
I use Perl to parse it as follows (and successfully):
if ($key eq 'Data') {
my $tle = $member->{struct}->{$key};
my $k_length = hex(unpack('H2', substr($tle, 0, 1)));
my $key = decode('cp500', substr($tle, 4, $k_length - 4));
my $v_length = hex(unpack('H2', substr($tle, $k_length, 1)));
my $value = decode('cp500', substr($tle, $k_length + 4, $v_length - 4));
print("'$key' => '$value'\n");
}
Result:
'Currency' => 'EUR'
While the above is successful, I feel that my way is a bit too cpmplicated and that there's a more efficient way to do this. E.g. do pack templates support reading the first n bytes to use as a quantifier for how many successive bytes to unpack? I read the Perl pack tutorial but can't seem to find something along those lines.
If the length field didn't include itself, you could do something like the following:
(my $record, $unparsed) = unpack("C/a a*", $unparsed);
my $key = decode("cp500", unpack("x3 a*", $record));
But the length field includes itself.
(my $length, $unparsed) = unpack("C a*", $unparsed);
(my $record, $unparsed) = unpack("a".($length-1)." a*", $unparsed);
my $key = decode("cp500", unpack("x3 a*", $record));
Please see if following demo code fulfill your requirements.
This code
defines hash decoder subroutines
reads hex representation of bytes provided by OP from DATA block
converts read data into binary representation $data utilizing pack
extracts length, key/tid, type by utilizing unpack
call decoder subroutine for this particular type
gets back hash consisting two arrays keys and vals
forms new hash %data with provided keys and vals
outputs keys and values (returned keys are used to preserve byte/field order)
NOTE: Encode 'from_to' is utilized to decode EBCDIC -- alternative
use strict;
use warnings;
use feature 'say';
use utf8;
use Encode 'from_to';
my $debug = 1;
my %decoder = (
1 => \&decode_type1,
2 => \&decode_currency,
3 => \&decode_type3,
4 => \&decode_type4,
5 => \&decode_type5
);
my $bytes = read_bytes();
my($len,$key,$type) = unpack('C3',$bytes);
my $data = $decoder{$type}($bytes);
my %data;
#data{#{$data->{keys}}} = #{$data->{vals}};
say '
Unpacked data
---------------';
printf "%-8s => %s\n", $_, $data{$_} for #{$data->{keys}};
sub read_bytes {
my $hex_bytes = <DATA>;
chomp $hex_bytes;
my $bytes = pack('H*',$hex_bytes);
return $bytes;
}
sub show_bytes {
my $data = shift;
print "Bytes: ";
printf "%02X ", $_ for unpack 'C*', $data;
print "\n";
}
sub decode_type1 {
my $bytes = shift;
return { keys => 'type1', vals => 'vals1' };
}
sub decode_currency {
my $bytes = shift;
show_bytes($bytes) if $debug;
my #keys = qw/length_1 key uid char data_1 length_2 val reserved data_2/;
my #vals = unpack('C4A8C2SA3',$bytes);
from_to($vals[4], 'cp37', 'latin1');
from_to($vals[8], 'cp37', 'latin1');
return { keys => \#keys, vals => \#vals};
}
sub decode_type3 {
my $bytes = shift;
return { keys => 'type3', vals => 'vals3' };
}
sub decode_type4 {
my $bytes = shift;
return { keys => 'type4', vals => 'vals4' };
}
sub decode_type5 {
my $bytes = shift;
return { keys => 'type5', vals => 'vals5' };
}
__DATA__
0C020B00C3A49999859583A807360000C5E4D9
Output
Bytes: 0C 02 0B 00 C3 A4 99 99 85 95 83 A8 07 36 00 00 C5 E4 D9
Unpacked data
---------------
length_1 => 12
key => 2
uid => 11
char => 0
data_1 => Currency
length_2 => 7
val => 54
reserved => 0
data_2 => EUR
Note:
It looks suspicious that val occupies one byte only what gives range of 0..255 for an amount in Euro. Perhaps reserved bytes might be a part of the val amount of Euro.

Conditionally print elements in a text file

I have a text file of the following format:
1 4730 1031782 init
4 0 6 events
2190 450 0 top
21413 5928 1 sshd
22355 1970 2009 find
I want to print rows of this file only if the second column of data meets the requirement >= 2000 - how can I do this?
Currently I am reading the file and printing it like so:
sub read_file{
my $data_failed = 1;
my $file = 'task_file';
if(open (my $file, "task_file" || die "$!\n")){
my #COLUMNS = qw( memory cpu program );
my %sort_strings = ( program => sub { $a cmp $b } );
my (%process_details, %sort);
while (<$file>) {
$data_failed = 0;
my ($process_id, $memory_size, $cpu_time, $program_name) = split;
$process_details{$process_id} = { memory => $memory_size,
cpu => $cpu_time,
program => $program_name };
undef $sort{memory}{$memory_size}{$process_id};
undef $sort{cpu}{$cpu_time}{$process_id};
undef $sort{program}{$program_name}{$process_id};
}
if($option_a == 1){
if (-z $file){print "No tasks found\n";}
for my $column ($COLUMNS[2]) {
my $cmp = $sort_strings{$column} || sub { $a <=> $b };
for my $value (sort $cmp keys %{ $sort{$column} }
) {
my #pids = keys %{ $sort{$column}{$value} };
say join ' ', $_, #{ $process_details{$_} }{#COLUMNS}
for #pids;
}
}
}
} else { print "No tasks found\n"}
}
The if($option_a == 1) bit is just reading values from another function that parses command line options.
my ($process_id, $memory_size, $cpu_time, $program_name) = split;
At this point, you can complete the loop, or you can continue to the next line. Just add the line:
next if $memory_size < 2000;
right after the split, and you'll eliminate all the records in memory that fail to meet your requirements.
Filtering a list is easily done with grep:
#!/usr/bin/perl
use strict;
use feature qw{ say };
use warnings;
my #COLUMNS = qw( memory cpu program );
my (%process_details, %sort);
while (<DATA>) {
my ($process_id, $memory_size, $cpu_time, $program_name) = split;
$process_details{$process_id} = { memory => $memory_size,
cpu => $cpu_time,
program => $program_name };
undef $sort{memory}{$memory_size}{$process_id};
undef $sort{cpu}{$cpu_time}{$process_id};
undef $sort{program}{$program_name}{$process_id};
}
for my $value (sort { $a cmp $b } keys %{ $sort{program} }) {
my #pids = grep $process_details{$_}{memory} > 2000,
keys %{ $sort{program}{$value} };
say join ' ', $_, #{ $process_details{$_} }{#COLUMNS}
for #pids;
}
__DATA__
...
Something like this perhaps:
#!/usr/bin/perl
use strict;
use warnings;
while (<DATA>) {
print if (split)[1] > 2000;
}
__DATA__
1 4730 1031782 init
4 0 6 events
2190 450 0 top
21413 5928 1 sshd
22355 1970 2009 find
With no arguments, split() splits $_ on whitespace (which is what we want). We can then use a list slice to look at the second element of that and print the line if that value is greater than 2000.

Strawberry Perl: "out of memory!"

trying to run the following code:
$combs = combinations(\#set,$k);
while (my $c = $combs->next)
{
$nrc=1;
}
Gives me "out of memory!" when I hit Ctrl+C (because its taking too long and it should not) if I pass a set from, for example, (0..450) and numbers to combine ($k) of 6. This issue does not occur with, lets say, a set of 0..45 and $k=6.
Note that the while loop seems to do nothing, in the original script it printed out the combination and incremented a counter that will hold the total number of combinations. But since I was not sure what the problem was, I decided to eliminate that.
I've read the Algorithm:Combinatorics on CPAN and it states that memory usage is minimal, so I don't know what's happening.
I am using Strawberry Perl 32bit on a Windows 10 machine.
Thanks.
--------------------- COMPLETE CODE
#!/usr/bin/perl
use List::MoreUtils "uniq";
use Algorithm::Combinatorics "combinations";
my $argc = $#ARGV+1;
my #set;
if ($argc == 0)
{
print STDERR "Valor minimo de rango: "; # range min
my $minrange = int <STDIN>;
print STDERR "Valor maximo de rango: "; #range max
my $maxrange = int <STDIN>;
#set = uniq sort { $a <=> $b }($minrange...$maxrange);
}
elsif ($argc == 1)
{
open(SETFROMFILE,"<$ARGV[0]") or die "No se puedo abrir el fichero, $!";
chomp(#set = <SETFROMFILE>);
close(SETFROMFILE);
#set = uniq sort { $a <=> $b } #set;
}
else
{
print STDERR "Uso: $0 [file]\n";
exit;
}
my $nrc = 0;
print STDERR "\n";
print STDERR "Numeros a combinar: "; # get subset
my $k = <STDIN>;
if ($k == 0) { exit; }
$combs = combinations(\#set,$k);
print STDERR "\n";
while (my $c = $combs->next)
{
print join(";",#$c) . "\n";
$nrc++;
}
print STDERR "\n";
print STDERR "numero total de combinaciones: $nrc\n";
It works for me.
use strict;
use warnings;
use Algorithm::Combinatorics qw( combinations );
sub show_mem { system('ps', '--no-heading', '-o', 'rss', $$); }
my #set = (0..450);
my $k = 6;
my $count = 0;
#show_mem();
my $combs = combinations(\#set, $k);
#show_mem();
while (my $c = $combs->next) {
++$count;
if (($count % 100_000) == 0) {
print("$count\n");
#show_mem();
}
}
Output:
784
784
100000
776
200000
784
300000
788
400000
776
500000
780
600000
784
700000
768
800000
784
900000
784
1000000
776
...
Of course, it will take forever to go through all C(451, 6) = 11,303,769,578,640 combinations! (We're talking about 251 days on my machine[1].)
(Note that 11,303,769,578,640 is too large for a 32-bit integer. Fortunately, Perl will switching to using a double-precision floating-point number, and those are large enough to hold that all numbers up to and including that one.)
By the way, if you just need the number of combinations, you can use
my $count = 1; $count *= ( #set - $_ + 1 ) / $_ for 1..$k;
How I timed it:
use Algorithm::Combinatorics qw( combinations );
use Time::HiRes qw( time );
my #set = (0..450);
my $k = 6;
my $count = 0;
my $combs = combinations(\#set, $k);
my $s = time;
while (my $c = $combs->next) {
++$count;
last if $count == 1_000_000;
}
my $e = time;
print($e-$s, "\n");
There are 11.1 trillion combinations of six items out of 450. I'm not surprised it ran out of memory!

How can I print N array elements with delimiters per line?

I have an array in Perl I want to print with space delimiters between each element, except every 10th element which should be newline delimited. There aren't any spaces in the elements if that matters.
I've written a function to do it with for and a counter, but I wondered if there's a better/shorter/canonical Perl way, perhaps a special join syntax or similar.
My function to illustrate:
sub PrintArrayWithNewlines
{
my $counter = 0;
my $newlineIndex = shift #_;
foreach my $item (#_)
{
++$counter;
print "$item";
if($counter == $newlineIndex)
{
$counter = 0;
print "\n";
}
else
{
print " ";
}
}
}
I like splice for a job like this:
sub PrintArrayWithNewlines {
my $n = 10;
my $delim = " ";
while (my #x = splice #_, 0, $n) {
print join($delim, #x), "\n";
}
}
You can use List::MoreUtils::natatime:
use warnings; use strict;
use List::MoreUtils qw( natatime );
my #x = (1 .. 35);
my $it = natatime 10, #x;
while ( my #v = $it->() ) {
print "#v\n"
}
Output:
C:\Temp> x
1 2 3 4 5 6 7 8 9 10
11 12 13 14 15 16 17 18 19 20
21 22 23 24 25 26 27 28 29 30
31 32 33 34 35
If you do not want to use any external modules, you can use array slices:
use warnings; use strict;
my #x = (1 .. 95);
my $n = 10;
for my $i ( 0 .. int #x/$n ) {
no warnings 'uninitialized';
print "#x[$n * $i .. $n * ($i + 1) - 1]\n";
}
The functions by and every in my module List::Gen can solve this problem:
use List::Gen;
for (every 10 => 'a' .. 'z') {
print "#$_\n"
}
# a b c d e f g h i j
# k l m n o p q r s t
# u v w x y z
it can also be written
foreach (by 10 => 'a' .. 'z') {
print "#$_\n"
}
or using the functional form:
mapn {print "#_\n"} 10 => 'a' .. 'z'; # #_ not #$_ here
or an iterator if that's your style:
my $letters = by 10 => 'a' .. 'z';
while (my $line = $letters->next) {
print "#$line\n";
}
You can also use map with a modification to PrintArrayWithNewlines:
#!/usr/bin/perl -w
use strict;
sub PrintArrayWithNewlines
{
my #array = #_;
my $newlineIndex = 10;
foreach my $item (#array) {
++$globalCounter;
print "$item";
if ($globalCounter == $newlineIndex) {
$globalCounter = 0;
print "\n";
}
else {
print " ";
}
}
}
my $globalCounter = 0;
my #myArray = 'a' .. 'z'
map { PrintArrayWithNewlines($_) } #myArray;
print "\n";
The output would be:
$ ./test.pl
a b c d e f g h i j
k l m n o p q r s t
u v x y z

Is there a module which gives me an output of two colums ( or more ) on STDOUT?

is there a module which gives me an output of two columns ( or more ) on STDOUT?
#!/usr/bin/env perl
use warnings;
use strict;
printf "%0.3d\n", $_ for 1 .. 100;
I'd like to have 1-50 in the first column and 51-100 in the second.
Text::Column
use Text::Column qw(format_array_table);
print format_array_table([map [$_, 50+$_], 1..50], [6, 6], [qw(first second)]);
Or, if you're on a UNIX system, pipe through pr(1).
open COLUMN, '|-', qw(pr -T2);
print COLUMN "$_\n" for 1..100;
There might be a module for this sort of thing ... or you can roll your own:
use strict;
use warnings;
my $n_cols = shift #ARGV;
my #list = (1..100);
my $part_size = #list / $n_cols;
$part_size = int($part_size + 1) unless $part_size == int($part_size);
my $fmt = '%8s' x $n_cols . "\n";
for my $i (0 .. $part_size - 1){
my #vals = map { defined($list[$_]) ? $list[$_] : '' }
map { $_ * $part_size + $i }
0 .. $n_cols;
printf $fmt, #vals;
}
You can use Perl formatting to do this.
Here is a link with a tutorial on it.
Here is a snippet from that page:
format MYFILE =
Name: #<<<<<<<<<<<<<<<<<<<<<<<<<<<< Salary: ############.##
$name, $salary
.
Produces the output:
Name: John Smith Salary: 78293.22
The formatting characters have the following formatting characteristics:
> right justified
# right justified (numeric only; can include a decimal point)
< left justified
| center justified
* left justified, fill in all data from value
Edit:
Here is a more direct answer to your question:
#!/usr/bin/perl
use strict;
my #data = (1..100);
my ($v1, $v2);
# Create formats
format ONECOL =
Col1: ####
$v1
.
format TWOCOL =
Col1: #### Col2: ####
$v1, $v2
.
# Set the two column format
$~ = "TWOCOL";
my $i = 0;
my $middle = int(#data/2) + (#data % 2);
for (my $x = $middle; $x < #data; $i++, $x++)
{
$v1 = $data[$i];
$v2 = $data[$x];
write;
}
# Set the format to one column
if ($i < $middle)
{
$~ = "ONECOL";
$v1 = $data[$i];
write;
}
This is begging for iterators (not! but it's fun). Note that the revised solution allows you to customize the number of columns as well.
#!/usr/bin/perl
use strict;
use warnings;
use List::AllUtils qw( min );
print_cols('%04d', 1, 100, 11);
sub print_cols {
my ($fmt, $min, $max, $cols) = #_;
my $its = partition($min, $max, $cols);
while ( (my #vals = grep { defined } map {$_->()} #$its)) {
printf join(' ', ($fmt) x #vals) . "\n", #vals;
}
return;
}
sub make_asc_it {
my ($min, $max) = #_;
return sub {
return unless $min <= $max;
return $min ++;
}
}
sub partition {
my ($min, $max, $cols) = #_;
return unless $min <= $max;
my $rows = sprintf '%.0f', ($max - $min) / $cols;
my #its;
for my $col (1 .. $cols) {
push #its, make_asc_it(
min( $min, $max ),
min( $min + $rows - 1, $max )
);
$min += $rows;
}
push #its, make_asc_it($min, $max) if $min <= $max;
return \#its;
}
Output for eleven columns:
0001 0010 0019 0028 0037 0046 0055 0064 0073 0082 0091 0100
0002 0011 0020 0029 0038 0047 0056 0065 0074 0083 0092
0003 0012 0021 0030 0039 0048 0057 0066 0075 0084 0093
0004 0013 0022 0031 0040 0049 0058 0067 0076 0085 0094
0005 0014 0023 0032 0041 0050 0059 0068 0077 0086 0095
0006 0015 0024 0033 0042 0051 0060 0069 0078 0087 0096
0007 0016 0025 0034 0043 0052 0061 0070 0079 0088 0097
0008 0017 0026 0035 0044 0053 0062 0071 0080 0089 0098
0009 0018 0027 0036 0045 0054 0063 0072 0081 0090 0099
Have a look at Perl6::Form
use Perl6::Form;
my $col1 = [ 1..50 ];
my $col2 = [ 51..100 ];
print form '{[} {[}', $col1, $col2;
So you can create a helper sub to produce the correct form which could go something like this:
sub mk_form {
my $rows = shift;
my #form;
# calculate columns needed
use integer;
my $cols = scalar #_ / $rows;
$cols++ if scalar #_ % $rows;
# create Perl6::Form args
push #form, '{[} ' x $cols;
push #form, [ splice #_, 0, $rows ] for 1..$cols;
return #form;
}
Then to produce a columned page at 50 rows:
say form mk_form(50, 1..101);
/I3az/
PS: See SO question What other languages have features and/or libraries similar to Perl’s format? for a bit more about Perl6::Form.
Take a look at the column utility. I don't think you can ask for a specific number of columns, though.
Can you do it this way?
for ($count = 1; $count <=100; $count++) {
printf "%d %d\n", $count, $count+50;
}