How to use Lingua::EN::Ngram for multiple files - perl

Question

I am implementing a naive Bayesian classification algorithm. In my training set I have a number of abstracts in separate files. I want to use N-gram in order to get the term frequency weight, but the code is not taking multiple files.
I edited my code, and now the error I am getting is
cant call method tscore on an undefined value. To check this, I printed #ngrams and it is showing me junk values like hash0*29G45 or something like that.
#!c:\perl\bin\perl.exe -w
use warnings;
use Algorithm::NaiveBayes;
use Lingua::EN::Splitter qw(words);
use Lingua::StopWords qw(getStopWords);
use Lingua::Stem;
use Algorithm::NaiveBayes;
use Lingua::EN::Ngram;
use Data::Dumper;
use Text::Ngram;
use PPI::Tokenizer;
use Text::English;
use Text::TFIDF;
use File::Slurp;
my $pos_file = 'D:\aminoacids';
my $neg_file = 'D:\others';
my $test_file = 'D:\testfiles';
my #vectors = ();
my $categorizer = Algorithm::NaiveBayes->new;
my #files = <$pos_file/*>;
my #ngrams;
for my $filename (#files) {
open(FH, $filename);
my $ngram = Lingua::EN::Ngram->new($filename);
my $tscore = $ngram->tscore;
foreach (sort { $$tscore{$b} <=> $$tscore{$a} } keys %$tscore) {
print "$$tscore{ $_ }\t" . "$_\n";
}
my $trigrams = $ngram->ngram(2);
foreach my $trigram (sort { $$trigrams{$b} <=> $$trigrams{$a} } keys %$trigrams) {
print $$trigrams{$trigram}, "\t$trigram\n";
}
my %positive;
$positive{$_}++ for #files;
$categorizer->add_instance(
attributes => \%positive,
label => 'positive'
);
}
close FH;

Answer 1

Your code <$pos_file/*> should work fine ( thanks #borodir ), still, here is an alternative so as to not mess up the history.
Try
opendir (DIR, $directory) or die $!;
and then
while (my $filename = readdir(DIR)) {
open ( my $fh, $filename );
# work with filehandle
close $fh;
}
closedir DIR;
If called in list context, readdir should give you a list of files:
my #filenames = readdir(DIR);
# you could call that function you wanted to call with this list, file would need to be
# opened still, though
Another point:
If you want to pass a reference to an array, do it like so:
function( list => \#stems );
# thus, your ngram line should probably rather be
my $ngram = Lingua::EN::Ngram->new (file => \#stems );
However, the docs for Lingua::EN::Ngram only talk about scalar for file and so on, it does not seem to expect an array for input. ( Exception being the 'intersection' method )
So you would have to put it in a loop and cycle through, or use map
my #ngrams = map{ Lingua::EN::Ngram->new( file => $_ ) }#filenames
Seems unnecessary to open in filehandle first, Ngram does that by itself.
If you prefer a loop:
my #ngrams;
for my $filename ( #filenames ){
push #ngrams, Lingua::EN::Ngram->new( file => $filename );
}
I think now I got what you actually want to do.
get the tscore: you wrote $tscore = $ngram->tscore, but $ngram is not defined anymore.
Not sure how to get the tscore for a single word. ( "significance of word in text" ) kind of indicates a text.
Thus: make an ngram not for each word, but either for each sentence or each file.
Then you can determine the t-score of that word in that sentence or file ( text ).
for my $filename ( #files ){
my $ngram = Lingua::EN::Ngram->new( file => $filename );
my $tscore = $ngram->tscore();
# tscore returns a hash reference. Keys are bigrams, values are tscores
# now you can do with the tscore what you like. Note that for arbitrary length,
# tscore will not work. This you would have to do yourself.