Suggestions for optimizing sieve of Eratosthenes in perl - perl

use warnings;
use strict;
my $in=<STDIN>;
my #array=(1...$in);
foreach my $j(2...sqrt($in)){
for(my $i=$j*2;$i<=$in;$i+=$j){
delete($array[$i-1]);
}
}
delete($array[0]);
open FILE, ">","I:\\Perl_tests\\primes.dat";
foreach my $i (#array){
if($i){
print FILE $i,"\n";
}
}
I'm sure there is a better way to do the array of all numbers however I don't really know of a way to do it in perl. I'm pretty inexperienced in perl. Any recommendations for speeding it up are greatly appreciated. Thanks a ton!

The fastest solution is using a module (e.g. ntheory) if you just want primes and don't care how you get them. That will be massively faster and use much less memory.
I suggest looking at the RosettaCode Sieve task. There are a number of simple sieves shown, a fast string version, some weird examples, and a couple extensible sieves.
Even the basic one uses less memory than your example, and the vector and string versions shown there use significantly less memory.
Sieve of Atkin is rarely a good method unless you are Dan Bernstein. Even then, it's slower than fast SoE implementations.

That's a fairly inefficient way to look for primes, so unless you absolutely have to use the Sieve of Erastosthenes, The Sieve of Atkin is probably a faster algorithm for finding primes.
With regard to the memory usage, here's a perlified version of this python answer. Instead of striking out the numbers from a big up front array of all the integers, it tracks which primes are divisors of non-prime numbers, and then masks out the next non-prime number after each iteration. This means that you can generate as many primes as you have precision for without using all the RAM, or any more memory than you have to. More code and more indirection makes this version slower than what you have already though.
#!/usr/bin/perl
use warnings;
use strict;
sub get() {
my $this = shift;
if ($this->{next} == 2) {
$this->{next} = 3;
return 2;
}
while (1) {
my $next = $this->{next};
$this->{next} += 2;
if (not exists $this->{used}{$next}) {
$this->{used}{$next * $next} = [$next];
return $next;
} else {
foreach my $x (#{$this->{used}{$next}}) {
push (#{$this->{used}{$next + $x}}, $x);
}
delete $this->{used}{$next};
}
}
}
sub buildSieve {
my $this = {
"used" => {},
"next" => 2,
};
bless $this;
return $this;
}
my $sieve = buildSieve();
foreach $_ (1..100) {
print $sieve->get()."\n";
}
You should be able to combine the better algorithm with the more memory efficient generative version above to come up with a good solution though.
If you want to see how the algorithm works in detail, it's quite instructive to use Data::Dumper; and print out $sieve in between each call to get().

Related

Reducing code verbosity and efficiency

I came across the below where some heavy stipulations were done, finally we got a number of #hits and we need to return just one:
if ($#hits > 0)
{
my $highestScore = 0;
my $chosenMatch = "";
for $hit (#hits)
{
my $currScore = 0;
foreach $k (keys %{$hit})
{
next if $k eq $retColumn;
$currScore++ if ($hit->{$k} =~ /\S+/);
}
if ($currScore > $highestScore)
{
$chosenMatch = $hit;
$highestScore = $currScore;
}
}
return ($chosenMatch);
}
elsif ($#hits == 0)
{
return ($hits[0]);
}
That's an eye full and I was hoping to simplify the above code, I came up with:
return reduce {grep /\S+/, values %{$a} > grep /\S+/, values %{$b} ? $a : $b} #matches;
After of using of course useing, List::Util
I wonder if the terse version is any efficient and/or advantage over the original one. Also, there's one condition that's skipped: if $k eq $retColumn, how can I efficiently get that in?
There is a famous quote:
"Premature optimisation is the root of all evil" - Donald Knuth
It is almost invariably the case that making code more concise really doesn't make much difference to the efficiency, and can cause significant penalties to readability and maintainability.
Algorithm is important, code layout ... isn't really. Things like reduce, map and grep are still looping - they're just doing so behind the scenes. You've gained almost no efficiency by using them, you've just saved some bytes in your file. That's fine if they make your code more clear, but that should be your foremost consideration.
Please - keep things clear first, foremost and always. Make your algorithm good. Don't worry about replacing an explicit loop with a grep or map unless these things make your code clearer.
And in the interests of being constructive:
use strict and warnings is really important. Really really important.
To answer your original question:
I wonder if the terse version is any efficient and/or advantage over the original one
No, I think if anything the opposite. Short of profiling code speed, the rule of thumb is look at number and size of loops - a single chunk of code rarely makes much difference, but running it lots and lots of times (unnecessarily) is where you get your inefficiency.
In your first example - you have two loops, a foreach loop inside a for loop. It looks like you traverse your #hits data structure once, and 'unwrap' it to get at the inner layers.
In your second example, both your greps are loops, and your reduce is as well. If I'm reading it correctly, then it'll be traversing your data structure multiple times. (Because you are greping values $a and $b - these will be applied several times).
So I don't think you have gained either readability or efficiency by doing what you've done. But you have made a function that's going to make future maintenance programmers have to think really hard. To take another quote:
"Everyone knows that debugging is twice as hard as writing a program in the first place. So if you're as clever as you can be when you write it, how will you ever debug it?" - Brian Kernighan
I wonder if the terse version is any efficient and/or advantage over the original one
The terse version is less efficient than the original because it calculates the score of every element twice, but it does have readability advantages.
The following keeps the readability gain (and even adds some):
sub get_score {
my ($match) = #_;
my #keys = grep { $_ ne $retColumn } keys %$match;
my $score = grep { /\S/ } #{$match}{ #keys };
return $score;
}
return reduce { get_score($a) > get_score($b) ? $a : $b } #matches;
You can look at any part of that sub and understand it without looking around. The least context you need to understand code, the more readable it is.
If you did need an efficiency boost, you can avoid calling get_score on every input twice by using a Schwartzian Transform. As with many optimizations, you will take a readability hit, but at least it's idiomatic (well known and thus well recognizable).
return
map { $_->[0] }
reduce { $a->[1] > $b->[1] ? $a : $b }
map { [ $match, get_score($match) ] }
#matches;

Perl equivalent of java Iterator

In java for a class that represents a collection, if it implements the iterable interface, iterating over the collection can be done without any knowledge about the collection using the foreach loop.
Can we do something similar in perl? Suppose my perl class is a collection. What is the best way of iterating over the collection ?
If you use a Perl hash in place of a Collection, then you can use each to iterate over it.
Every call to each %hash will return the next key/value pair, or an empty list at the end.
Perl doesn't have Java-style iterators, cause it doesn't need them. Perl has rather robust hashes and lists built in, which supplant most custom collections you'll ever need, and you can easily loop over them already. And if you want to do something with each element (and particularly if you want to build a list of the results), there's functions like map that you can pass a code ref to.
If you really want this Java'ism, though, you could build your own iterator class. It doesn't have to implement or extend any particular interface, cause Perl duck-types -- and normal people don't use iterators anyway, so there's no standard interface in the first place. And it doesn't have anything to do with foreach. :P
Really, it's better to just use the standard collection types. I've never run into a case where i needed a custom collection.
You can create arbitrarily complex iterators in perl by using closures and functional programming techniques, for when you need something above and beyond perl's built in ability to iterate over arrays and hashes. Here is is simplistic example to illustrate the technique - in this instance, its pretty useless since an iterator is unnecessary for this particular case. But as I said this technique can be applied usefully in other scenarios. This example will walk over an array backwards, skipping elements that we want excluded (in this case, the letter 'c').
sub gen_iter {
my #collection = #_;
return sub {
while (my $item = pop #collection) {
next if $item eq 'c';
return $item;
}
}
}
my $it = gen_iter((qw/a b c d e f g/));
# prints 'gfedba'
while (my $item = $it->()) {
print $item;
}
I recommend grabbing the free ebook 'High-Order Perl' and digesting the chapter on iterators. It has some great, useful examples.
Perl's two builtin collections — arrays and hashes — have various ways to be iterated:
foreach loops: This loops (but not iterates lazily) through each item in the given list. Using this on hashes is probably wrong, you should either iterate over the values or the keys:
foreach my $key (keys %hash) {
do something;
}
The each iterator: You can iterate through "collections" with the each function. Each time it is called, it will return a list of a key/index, and a value. If no further items are there, it will return a false value:
while (my ($key, $value) = each %hash) {
do something;
}
You can only have one such iterator per array/hash. This usage of each on arrays is a rather new feature.
The each idiom can be extended via closures to create custom iterators:
my %hash = ...;
my #keys = keys %hash; # cache the current keys
my $i = 0;
my $iter = sub {
return unless $i < #keys;
my $key = $keys[$i++];
return ($key, $hash{$key});
};
...;
while (my ($key, $value) = $iter->()) {
do something;
}

Uniform distribution of truncated md5?

Can we say that a truncated md5 hash is still uniformly distributed?
To avoid misinterpretations: I'm aware the chance of collisions is much greater the moment you start to hack off parts from the md5 result; my use-case is actually interested in deliberate collisions. I'm also aware there are other hash methods that may be better suited to use-cases of a shorter hash (including, in fact, my own), and I'm definitely looking into those.
But I'd also really like to know whether md5's uniform distribution also applies to chunks of it. (Consider it a burning curiosity.)
Since mediawiki uses it (specifically, the left-most two hex-digits as characters of the result) to generate filepaths for images (e.g. /4/42/The-image-name-here.png) and they're probably also interested in an at least near-uniform distribution, I imagine the answer is 'yes', but I don't actually know.
Yes, not exhibiting any bias is a design requirement for a cryptographic hash. MD5 is broken from a cryptographic point of view however the distribution of the results was never in question.
If you still need to be convinced, it's not a huge undertaking to hash a bunch of files, truncate the output and use ent ( http://www.fourmilab.ch/random/ ) to analyze the result.
I wrote a little php-program to answer this question. It's not very scientific, but it shows the distribution for the first and the last 8 bits of the hashvalues using the natural numbers as hashtext. After about 40.000.000 hashes the difference between the highest and the lowest counts goes down to 1%, so I'd say the distribution is ok. I hope the code is more precise in explaining what was computed :-)
Btw, with a similar program I found that the last 8 bits seem to be distributed slightly better than the first.
<?php
// Setup count-array:
for ($y=0; $y<16; $y++) {
for ($x=0; $x<16; $x++) {
$count[dechex($x).dechex($y)] = 0;
}
}
$text = 1; // The text we will hash.
$hashCount = 0;
$steps = 10000;
while (1) {
// Calculate & count a bunch of hashes:
for ($i=0; $i<$steps; $i++) {
$hash = md5($text);
$count[substr($hash, 0, 2)]++;
$count[substr($hash, -2)]++;
$text++;
}
$hashCount += $steps;
// Output result so far:
system("clear");
$min = PHP_INT_MAX; $max = 0;
for ($y=0; $y<16; $y++) {
for ($x=0; $x<16; $x++) {
$n = $count[dechex($x).dechex($y)];
if ($n < $min) $min = $n;
if ($n > $max) $max = $n;
print $n."\t";
}
print "\n";
}
print "Hashes: $hashCount, Min: $min, Max: $max, Delta: ".((($max-$min)*100)/$max)."%\n";
}
?>

How fast is Perl's smartmatch operator when searching for a scalar in an array?

I want to repeatedly search for values in an array that does not change.
So far, I have been doing it this way: I put the values in a hash (so I have an array and a hash with essentially the same contents) and I search the hash using exists.
I don't like having two different variables (the array and the hash) that both store the same thing; however, the hash is much faster for searching.
I found out that there is a ~~ (smartmatch) operator in Perl 5.10. How efficient is it when searching for a scalar in an array?
If you want to search for a single scalar in an array, you can use List::Util's first subroutine. It stops as soon as it knows the answer. I don't expect this to be faster than a hash lookup if you already have the hash, but when you consider creating the hash and having it in memory, it might be more convenient for you to just search the array you already have.
As for the smarts of the smart-match operator, if you want to see how smart it is, test it. :)
There are at least three cases you want to examine. The worst case is that every element you want to find is at the end. The best case is that every element you want to find is at the beginning. The likely case is that the elements you want to find average out to being in the middle.
Now, before I start this benchmark, I expect that if the smart match can short circuit (and it can; its documented in perlsyn), that the best case times will stay the same despite the array size, while the other ones get increasingly worse. If it can't short circuit and has to scan the entire array every time, there should be no difference in the times because every case involves the same amount of work.
Here's a benchmark:
#!perl
use 5.12.2;
use strict;
use warnings;
use Benchmark qw(cmpthese);
my #hits = qw(A B C);
my #base = qw(one two three four five six) x ( $ARGV[0] || 1 );
my #at_end = ( #base, #hits );
my #at_beginning = ( #hits, #base );
my #in_middle = #base;
splice #in_middle, int( #in_middle / 2 ), 0, #hits;
my #random = #base;
foreach my $item ( #hits ) {
my $index = int rand #random;
splice #random, $index, 0, $item;
}
sub count {
my( $hits, $candidates ) = #_;
my $count;
foreach ( #$hits ) { when( $candidates ) { $count++ } }
$count;
}
cmpthese(-5, {
hits_beginning => sub { my $count = count( \#hits, \#at_beginning ) },
hits_end => sub { my $count = count( \#hits, \#at_end ) },
hits_middle => sub { my $count = count( \#hits, \#in_middle ) },
hits_random => sub { my $count = count( \#hits, \#random ) },
control => sub { my $count = count( [], [] ) },
}
);
Here's how the various parts did. Note that this is a logarithmic plot on both axes, so the slopes of the plunging lines aren't as close as they look:
So, it looks like the smart match operator is a bit smart, but that doesn't really help you because you still might have to scan the entire array. You probably don't know ahead of time where you'll find your elements. I expect a hash will perform the same as the best case smart match, even if you have to give up some memory for it.
Okay, so the smart match being smart times two is great, but the real question is "Should I use it?". The alternative is a hash lookup, and it's been bugging me that I haven't considered that case.
As with any benchmark, I start off thinking about what the results might be before I actually test them. I expect that if I already have the hash, looking up a value is going to be lightning fast. That case isn't a problem. I'm more interested in the case where I don't have the hash yet. How quickly can I make the hash and lookup a key? I expect that to perform not so well, but is it still better than the worst case smart match?
Before you see the benchmark, though, remember that there's almost never enough information about which technique you should use just by looking at the numbers. The context of the problem selects the best technique, not the fastest, contextless micro-benchmark. Consider a couple of cases that would select different techniques:
You have one array you will search repeatedly
You always get a new array that you only need to search once
You get very large arrays but have limited memory
Now, keeping those in mind, I add to my previous program:
my %old_hash = map {$_,1} #in_middle;
cmpthese(-5, {
...,
new_hash => sub {
my %h = map {$_,1} #in_middle;
my $count = 0;
foreach ( #hits ) { $count++ if exists $h{$_} }
$count;
},
old_hash => sub {
my $count = 0;
foreach ( #hits ) { $count++ if exists $old_hash{$_} }
$count;
},
control_hash => sub {
my $count = 0;
foreach ( #hits ) { $count++ }
$count;
},
}
);
Here's the plot. The colors are a bit difficult to distinguish. The lowest line there is the case where you have to create the hash any time you want to search it. That's pretty poor. The highest two (green) lines are the control for the hash (no hash actually there) and the existing hash lookup. This is a log/log plot; those two cases are faster than even the smart match control (which just calls a subroutine).
There are a few other things to note. The lines for the "random" case are a bit different. That's understandable because each benchmark (so, once per array scale run) randomly places the hit elements in the candidate array. Some runs put them a bit earlier and some a bit later, but since I only make the #random array once per run of the entire program, they move around a bit. That means that the bumps in the line aren't significant. If I tried all positions and averaged, I expect that "random" line to be the same as the "middle" line.
Now, looking at these results, I'd say that a smart-match is much faster in its worst case than the hash lookup is in its worst case. That makes sense. To create a hash, I have to visit every element of the array and also make the hash, which is a lot of copying. There's no copying with the smart match.
Here's a further case I won't examine though. When does the hash become better than the smart match? That is, when does the overhead of creating the hash spread out enough over repeated searches that the hash is the better choice?
Fast for small numbers of potential matches, but not faster than the hash. Hashes are really the right tool for testing set membership. Since hash access is O(log n) and smartmatch on an array is still O(n) linear scan (albeit short-circuiting, unlike grep), with larger numbers of values in the allowed matches, smartmatch gets relatively worse.
Benchmark code (matching against 3 values):
#!perl
use 5.12.0;
use Benchmark qw(cmpthese);
my #hits = qw(one two three);
my #candidates = qw(one two three four five six); # 50% hit rate
my %hash;
#hash{#hits} = ();
sub count_hits_hash {
my $count = 0;
for (#_) {
$count++ if exists $hash{$_};
}
$count;
}
sub count_hits_smartmatch {
my $count = 0;
for (#_) {
$count++ when #hits;
}
$count;
}
say count_hits_hash(#candidates);
say count_hits_smartmatch(#candidates);
cmpthese(-5, {
hash => sub { count_hits_hash((#candidates) x 1000) },
smartmatch => sub { count_hits_smartmatch((#candidates) x 1000) },
}
);
Benchmark results:
Rate smartmatch hash
smartmatch 404/s -- -65%
hash 1144/s 183% --
The "smart" in "smart match" isn't about the searching. It's about doing the right thing at the right time based on context.
The question of whether it's faster to loop through an array or index into a hash is something you'd have to benchmark, but in general, it'd have to be a pretty small array to be quicker to skim through than indexing into a hash.

Is returning a whole array from a Perl subroutine inefficient?

I often have a subroutine in Perl that fills an array with some information. Since I'm also used to hacking in C++, I find myself often do it like this in Perl, using references:
my #array;
getInfo(\#array);
sub getInfo {
my ($arrayRef) = #_;
push #$arrayRef, "obama";
# ...
}
instead of the more straightforward version:
my #array = getInfo();
sub getInfo {
my #array;
push #array, "obama";
# ...
return #array;
}
The reason, of course, is that I don't want the array to be created locally in the subroutine and then copied on return.
Is that right? Or does Perl optimize that away anyway?
What about returning an array reference in the first place?
sub getInfo {
my $array_ref = [];
push #$array_ref, 'foo';
# ...
return $array_ref;
}
my $a_ref = getInfo();
# or if you want the array expanded
my #array = #{getInfo()};
Edit according to dehmann's comment:
It's also possible to use a normal array in the function and return a reference to it.
sub getInfo {
my #array;
push #array, 'foo';
# ...
return \#array;
}
Passing references is more efficient, but the difference is not as big as in C++. The argument values themselves (that means: the values in the array) are always passed by reference anyway (returned values are copied though).
Question is: does it matter? Most of the time, it doesn't. If you're returning 5 elements, don't bother about it. If you're returning/passing 100'000 elements, use references. Only optimize it if it's a bottleneck.
If I look at your example and think about what you want to do I'm used to write it in this manner:
sub getInfo {
my #array;
push #array, 'obama';
# ...
return \#array;
}
It seems to me as straightforward version when I need return large amount of data. There is not need to allocate array outside sub as you written in your first code snippet because my do it for you. Anyway you should not do premature optimization as Leon Timmermans suggest.
To answer the final rumination, no, Perl does not optimize this away. It can't, really, because returning an array and returning a scalar are fundamentally different.
If you're dealing with large amounts of data or if performance is a major concern, then your C habits will serve you well - pass and return references to data structures rather than the structures themselves so that they won't need to be copied. But, as Leon Timmermans pointed out, the vast majority of the time, you're dealing with smaller amounts of data and performance isn't that big a deal, so do it in whatever way seems most readable.
This is the way I would normally return an array.
sub getInfo {
my #array;
push #array, 'foo';
# ...
return #array if wantarray;
return \#array;
}
This way it will work the way you want, in scalar, or list contexts.
my $array = getInfo;
my #array = getInfo;
$array->[0] == $array[0];
# same length
#$array == #array;
I wouldn't try to optimize it unless you know it is a slow part of your code. Even then I would use benchmarks to see which subroutine is actually faster.
There's two considerations. The obvious one is how big is your array going to get? If it's less than a few dozen elements, then size is not a factor (unless you're micro-optimizing for some rapidly called function, but you'd have to do some memory profiling to prove that first).
That's the easy part. The oft overlooked second consideration is the interface. How is the returned array going to be used? This is important because whole array dereferencing is kinda awful in Perl. For example:
for my $info (#{ getInfo($some, $args) }) {
...
}
That's ugly. This is much better.
for my $info ( getInfo($some, $args) ) {
...
}
It also lends itself to mapping and grepping.
my #info = grep { ... } getInfo($some, $args);
But returning an array ref can be handy if you're going to pick out individual elements:
my $address = getInfo($some, $args)->[2];
That's simpler than:
my $address = (getInfo($some, $args))[2];
Or:
my #info = getInfo($some, $args);
my $address = $info[2];
But at that point, you should question whether #info is truly a list or a hash.
my $address = getInfo($some, $args)->{address};
What you should not do is have getInfo() return an array ref in scalar context and an array in list context. This muddles the traditional use of scalar context as array length which will surprise the user.
Finally, I will plug my own module, Method::Signatures, because it offers a compromise for passing in array references without having to use the array ref syntax.
use Method::Signatures;
method foo(\#args) {
print "#args"; # #args is not a copy
push #args, 42; # this alters the caller array
}
my #nums = (1,2,3);
Class->foo(\#nums); # prints 1 2 3
print "#nums"; # prints 1 2 3 42
This is done through the magic of Data::Alias.
3 other potentially LARGE performance improvements if you are reading an entire, largish file and slicing it into an array:
Turn off BUFFERING with sysread() instead of read() (manual warns
about mixing)
Pre-extend the array by valuing the last element -
saves memory allocations
Use Unpack() to swiftly split data like uint16_t graphics channel data
Passing an array ref to the function allows the main program to deal with a simple array while the write-once-and-forget worker function uses the more complicated "$#" and arrow ->[$II] access forms. Being quite C'ish, it is likely to be fast!
I know nothing about Perl so this is a language-neutral answer.
It is, in a sense, inefficient to copy an array from a subroutine into the calling program. The inefficiency arises in the extra memory used and the time taken to copy the data from one place to another. On the other hand, for all but the largest arrays, you might not give a damn, and might prefer to copy arrays out for elegance, cussedness or any other reason.
The efficient solution is for the subroutine to pass the calling program the address of the array. As I say, I haven't a clue about Perl's default behaviour in this respect. But some languages provide the programmer the option to choose which approach.