Efficient Way To Check 10,000s of Blog Feeds in Perl - perl

Question

We have 10,000s of blogs we want to check multiple times a day for new posts. I'd love some ideas with example code on the most efficient way to do this using Perl.
Currently we are just using LWP::UserAgent to download each RSS feed and then checking each URL in the resulting feed against a MySQL database table of already found URLs one at a time. Needless to say this doesn't scale well and is super inefficient.
Thanks in advance for your help & advice!

Answer 1

Unfortunately, there is probably no other way than do some kind of polling.
Luckily, implementing the PubSubHubbub protocol can greatly help reduce the amount of polling for the feeds who support it.
For those feeds who don't support PubSubHubbub, then you'll have to make sure you use HTTP-level protocols (like ETags or If-Modified-Since headers to know if/when a resource has been updated).
Also make sure you implement some kind of back-off mechanisms.

Answer 2

Perhaps look at AnyEvent::Feed, it is asynchronous (using the AnyEvent event loop) with configurable polling intervals as well as built in support for 'seen' articles, and support for RSS and Atom feeds. You could possibly create a single process polling every feed or multiple processes polling different sections of your feed list.
From the synopsis:
use AnyEvent;
use AnyEvent::Feed;
my $feed_reader =
AnyEvent::Feed->new (
url => 'http://example.com/atom.xml',
interval => $seconds,
on_fetch => sub {
my ($feed_reader, $new_entries, $feed, $error) = #_;
if (defined $error) {
warn "ERROR: $error\n";
return;
}
for (#$new_entries) {
my ($hash, $entry) = #_;
# $hash a unique hash describing the $entry
# $entry is the XML::Feed::Entry object of the new entries
# since the last fetch.
}
}
);

Answer 3

Seems like two questions rolled into one: fetching an comparing. Others have answered the fetch part. As for comparing:
I've been reading about redis lately and it seems like a good fit for you as it can do a lot of simple operations per second (lets say ~80k /s). So checking if you already have an url should go really fast. Never actually used it though ;)
An idea: Have you tried comparing on size before parsing the RSS? Might save you some time if the change infrequently.

Answer 4

10000 are not so many.
You could probably handle then using some simple approach like forking some worker processes that get RSS URLs from the db, fetch them and update the database:
for (1..$n) {
my $pid = fork;
if (!$pid) {
defined $pid or die "fork failed";
my $db = open_db();
while (1) {
$url = get_next_url($db) or last;
$rss = feed_rss($url);
update_rss($db, $rss);
}
exit(0);
}
}
wait_for_workers(#pid);
That, considering you are not able to use some of the existent applications already pointed by other responders.