Can I pipe a webpage's source from curl to perl? - perl

Question

I'm parsing the sourcecode of many websites, an entire huge web with thousands of pages. Now I want to search for stuff in perĺ, I want to find the number of occurrences of a keyword.
For parsing the webpages I use curl and pipe the output to "grep -c" which doesn't work, so I want to use perl. Can be perl utilised completely to crawl a page?
E.g.
cat RawJSpiderOutput.txt | grep parsed | awk -F " " '{print $2}' | xargs -I replaceStr curl replaceStr?myPara=en | perl -lne '$c++while/myKeywordToSearchFor/g;END{print$c}'
Explanation: In the textfile above I have usable and unusable URLs. With "Grep parsed" I fetch the usable URLs. With awk I select the 2nd column with contains the pure usable URL. So far so good. Now to this question: With Curl I fetch the source (appending some parameter, too) and pipe the whole source code of each page to perl in order to count "myKeywordToSearchFor" occurrences. I would love to do this in perl only if it is possible.
Thanks!

Answer 1

This uses Perl only (untested):
use strict;
use warnings;
use File::Fetch;
my $count;
open my $SPIDER, '<', 'RawJSpiderOutput.txt' or die $!;
while (<$SPIDER>) {
chomp;
if (/parsed/) {
my $url = (split)[1];
$url .= '?myPara=en';
my $ff = File::Fetch->new(uri => $url);
$ff->fetch or die $ff->error;
my $fetched = $ff->output_file;
open my $FETCHED, '<', $fetched or die $!;
while (<$FETCHED>) {
$count++ if /myKeyword/;
}
unlink $fetched;
}
}
print "$count\n";

Answer 2

Try something more like,
perl -e 'while(<>){my #words = split ' ';for my $word(#words){if(/myKeyword/){++$c}}} print "$c\n"'
i.e.
while (<>) # as long as we're getting input (into “$_”)
{ my #words = split ' '; # split $_ (implicit) into whitespace, so we examine each word
for my $word (#words) # (and don't miss two keywords on one line)
{ if (/myKeyword/) # whenever it's found,
{ ++$c } } } # increment the counter (auto-vivified)
print "$c\n" # and after end of file is reached, print the counter
or, spelled out strict-like
use strict;
my $count = 0;
while (my $line = <STDIN>) # except that <> is actually more magical than this
{ my #words = split ' ' => $line;
for my $word (#words)
{ ++$count; } } }
print "$count\n";