Flutter scrape Google Shopping - flutter

Question

I am trying to get a product price by searching for the product in Google Shopping with a given title and storeName.
I found this question on SO, which explains how you would do it with Python. How can something like this be done with Flutter?
Also like I said, is there any way to select the store?
I couldn't find anything on this. Any help is appreciated. Let me know if you need any more info!
I tried it like this:
Future<void> getGoogleShoppingResult() async {
const String requestUrl =
'https://www.google.com/search?q=minecraft+toys&tbm=shop';
final response = await http.get(Uri.parse(requestUrl));
dom.Document document = parser.parse(response.body);
final elements = document.querySelectorAll(
'div.sh-np__product-title.translate-content',
);
print(elements.first.innerHtml);
}
With this I tried to get the title of the first found product. But this never finds any product, even though I copied the selector from Google Shopping.

Answer 1

You should use these packages:
webview_flutter
http
html
And wait till JavaScript has been loaded.
This tutorial can help:
https://www.youtube.com/watch?v=EHk66k___EY
The code snippet:
import 'dart:developer';
import 'package:flutter/material.dart';
import 'package:webview_flutter/webview_flutter.dart';
import 'package:html/dom.dart' as dom;
class WebViewPageExample extends StatefulWidget {
const WebViewPageExample({Key? key}) : super(key: key);
#override
State<WebViewPageExample> createState() => _WebViewPageExampleState();
}
class _WebViewPageExampleState extends State<WebViewPageExample> {
late WebViewController _controller;
#override
Widget build(BuildContext context) {
return WebView(
onWebViewCreated: _onWebViewCreated,
javascriptMode: JavascriptMode.unrestricted,
initialUrl: "https://www.google.com/search?q=minecraft+toys&tbm=shop",
onPageFinished: _onPageFinished,
);
}
void _onWebViewCreated(WebViewController controller) {
_controller = controller;
}
Future _onPageFinished(String url) async {
final html = await _controller.runJavascriptReturningResult(
"new XMLSerializer().serializeToString(document);");
final div = dom.Document.html(html)
.querySelectorAll('div.sh-np__product-title.translate-content');
log('Count: ${div.length}');
}
}