Cant get Weighted Cosine Similarity to work - perl

Question

I'm trying to get the Weighted Cosine Similarity of two documents. I'm using Text::Document and Text::DocumentCollection. My code seems to work but it isn't returning a number as I expected.
Here is my code
use strict;
use warnings;
use Text::Document;
use Text::DocumentCollection;
my $newfile = shift #ARGV;
my $newfile2 = shift #ARGV;
##This is in another file.
my $t1 = countFreq($newfile);
my $t2 = countFreq($newfile2);
my $collection = Text::DocumentCollection->new(file => 'coll.db');
$collection->Add("One", $t1);
$collection->Add("Two", $t2);
my $wSim = $t1->WeightedCosineSimilarity( $t2,
\&Text::DocumentCollection::IDF,
$collection
);
print "\nWeighted Cosine Sim is: $wSim\n";
All this returns is Weighted Cosine Sim is: without anything following the colon.
Here is the code for countFreq:
sub countFreq{
my ($file) = #_;
my $t1 = Text::Document->new();
open (my $info, $file) or die "Could not open file.";
while (my $line = <$info>) {
chomp $line;
$line =~ s/[[:punct:]]//g;
foreach my $str (split /\s+/, $line) {
if (!defined $sp{lc($str)}) {
$t1 -> AddContent ($str);
}
}
}
return $t1;
}

Answer 1

###Update
Here's an example program that works fine. It's based on looking at the test code in the distribution for inspiration
I was expecting the test to be much less sensitive, so I was getting zeroes from two wildly different text sources. This example adds three short sentences $d1, $d1, and $d3, to a collection $c, and then compares each of the three documents to $d1
Comparing $d1 to itself produces 1 -- an exact match, as expected, while comparing $d2and $d3 gives 0.087 and 0 respectively -- a partial match and no match at all
I hope this helps you to resolve your specific issue?
use strict;
use warnings 'all';
use Text::Document;
use Text::DocumentCollection;
my $d1 = Text::Document->new;
$d1->AddContent( 'my heart belongs to sally webster' );
my $d2 = Text::Document->new;
$d2->AddContent( 'my heart belongs to the girl next door' );
my $d3 = Text::Document->new;
$d3->AddContent( 'I want nothing to do with my neighbours' );
my $c = Text::DocumentCollection->new( file => 'coll2.db' );
$c->Add('one', $d1);
$c->Add('two', $d2);
$c->Add('three', $d3);
for my $doc ( $d1, $d2, $d3 ) {
my $wcs = $d1->WeightedCosineSimilarity(
$doc,
\&Text::DocumentCollection::IDF,
$c
);
die qq{Invalid parameters for "WeightedCosineSimilarity"} unless defined $wcs;
print $wcs, "\n";
}
###output
1
0.0874311036726221
0
This is the code for Text::Document::WeightedCosineSimilarity
# this is rather rough
sub WeightedCosineSimilarity
{
my $self = shift;
my ($e,$weightFunction,$rock) = #_;
my ($Dv,$Ev) = ($self->{terms}, $e->{terms});
# compute union
my %union = %{$self->{terms}};
my #keyse = keys %{$e->{terms}};
#union{#keyse} = #keyse;
my #allkeys = keys %union;
# weighted D
my #Dw = map(( defined( $Dv->{$_} )?
&{$weightFunction}( $rock, $_ )*$Dv->{$_} : 0.0 ),
#allkeys
);
# weighted E
my #Ew = map(( defined( $Ev->{$_} )?
&{$weightFunction}( $rock, $_ )*$Ev->{$_} : 0.0 ),
#allkeys
);
# dot product of D and E
my $dotProduct = 0.0;
map( $dotProduct += $Dw[$_] * $Ew[$_] , 0..$#Dw );
# norm of D
my $nD = 0.0;
map( $nD += $Dw[$_] * $Dw[$_] , 0..$#Dw );
$nD = sqrt( $nD );
# norm of E
my $nE = 0.0;
map( $nE += $Ew[$_] * $Ew[$_] , 0..$#Ew );
$nE = sqrt( $nE );
# dot product scaled by norm
if( ($nD==0) || ($nE==0) ){
return undef;
} else {
return $dotProduct / $nD / $nE;
}
}
I'm afraid I don't understand the theory behind what it is doing, but it looks like your problem is that either $nD ("norm of D") or $nE ("norm of D") is zero
All I can suggest is that your two text samples may be too similar/different, or perhaps they are too long/short?
Either way, your code should look like this so as to catch an invalid return value from the cosine function:
my $wSim = $t1->WeightedCosineSimilarity( $t2,
\&Text::DocumentCollection::IDF,
$collection
);
die qq{Invalid parameters for "WeightedCosineSimilarity"} unless defined $wSim;
print "\nWeighted Cosine Sim is: $wSim\n";