Web collection from Google data page - perl

Question

I am trying to collect the data from this page:
https://datastudio.google.com/reporting/c07bc5cf-5c09-4156-a903-3e7acd02721a/page/ql6IC
I usually use PERL/LWP to GET the page then parse, but this page does not return the visible elements, just the initial Google goo.
Looking to grab the Number of Confirmed Cases and the date updated at the bottom of the page.
Thanks in advance!

Answer 1

Here is an example of how to access the data after javascript has modified the DOM using Selenium::Chrome :
use strict;
use warnings;
use Selenium::Chrome;
# Enter your driver path here. See https://sites.google.com/a/chromium.org/chromedriver/
# for download instructions
my $driver_path = '/home/hakon/chromedriver/chromedriver';
my $driver = Selenium::Chrome->new( binary => $driver_path );
$driver->get("https://datastudio.google.com/reporting/c07bc5cf-5c09-4156-a903-3e7acd02721a/page/ql6IC");
sleep 5; # modify this sleep period such that the page is fully loaded before continuing
my $elem = $driver->find_element_by_class_name('tableBody');
# Do something with the table..
Update
To avoid specifying the sleep limit above you can use the wait_until() function from Selenium::Waiter, for example:
use feature qw(say);
use strict;
use warnings;
use Selenium::Chrome;
use Selenium::Waiter;
# Enter your driver path here. See https://sites.google.com/a/chromium.org/chromedriver/
# for download instructions
my $driver_path = '/home/hakon/chromedriver/chromedriver';
my $driver = Selenium::Chrome->new(
binary => $driver_path,
# avoid printing error from find_element_by_class_name() when class is not found,
# see the below wait_until() call. The error message will be on the form:
#
# "Error while executing command: no such element: no such element:
# Unable to locate element"
#
error_handler => sub { my $msg = $_[1]; die $msg if $msg !~ /\Qno such element\E/}
);
$driver->get("https://datastudio.google.com/reporting/c07bc5cf-5c09-4156-a903-3e7acd02721a/page/ql6IC");
my $timeouts = $driver->get_timeouts();
say "Current implicit timeout = ", $timeouts->{implicit};
$driver->set_implicit_wait_timeout(0);
say "Updated implicit wait timeout to 0 ms";
my $timeout = 30;
my $start_time = time;
my $elem = wait_until {
$driver->find_element_by_class_name('tableBody')
} timeout => $timeout, interval => 1;
if ( $elem ) {
my $elapsed = time - $start_time;
say "Found element after $elapsed seconds";
}
else {
say "Could not find tableBody element after $timeout seconds";
}