What's wrong with my IF statement and LWP::Simple? - perl

Question

I am trying to create a simple scraper, and I am using getstore(), but the scirpt won't create the .txt file when used within an IF statement. What am I doing wrong there?
Thanks,
Carlos N.
#!/usr/bin/perl -w
use strict;
use LWP::Simple;
my $url;
my $content;
print "Enter URL:";
chomp($url = <STDIN>);
$content = get($url);
if ($content =~ s%<(style|script)[^<>]*>.*?</\1>|</?[a-z][a-z0-9]*[^<>]*>|<!--.*?-->%%g) {
$content = getstore($content,"../crawled_text.txt");
}
die "Couldn't get $url" unless defined $content;

Answer 1

From the LWP::Simple documentation:
my $code = getstore($url, $file)
Gets a document identified by a URL and stores it in the file. The
return value is the HTTP response code.
Your first parameter is a stripped HTML file and likely not a URL. You could use a debugger or print statements in your code to understand more about the contents of your variables and about whether your program goes into an if block.

Answer 2

getstore takes an URL as a parameter and stores it into a file. What you want to do is to just store content in a file, so use this instead
#!/usr/bin/perl
use strict;
use warnings;
use LWP::Simple;
use Path::Tiny;
my $url = shift || "https://perl.org";
my $content = get($url) or die "Couldn't get $url" ;
if ($content =~ s%<(style|script)[^<>]*>.*?</\1>|</?[a-z][a-z0-9]*[^<>]*>|<!--.*?-->%%g) {
my $crawled_text = path("../crawled_text.txt");
$crawled_text->spew_utf8($content)
}
I have made also some small style changes and Path::Tiny to save content to a file. You can use the default open and print (or say) if you prefer to do so. Using shift allows also to take the URL as an argument from the command line, which is more idiomatic than prompting the user for it.