I need to get data from javascript in web-site.
It was successful to get data from general html web by using flutter_webscrapper dart package but looks like the webscrapper do not support to load javascript and scrap it.
I've been searching it and lots of developers recommend to use flutter_webview as headless browser.
However, flutter_webview spends too much time to create browser & load website. It takes almost over 4~5 seconds. Seems that challenge to use it in commercial product.
I also cannot use flutter_puppeteer because it does not support android & ios.
If there are any other way or other Dart package to scrap the data from javascript in web-site, please let me recommend. Waiting for your supports.
You can use web_scraper package .
may be using getAllScripts or getScriptVariables methods help you .
this sample repository used web_scraper package .
Related
I have a Flutter app which I want to get the lot size for an input address from Redfin and use the data scraped in the app for a search process. Does anyone have any idea on how to go about this or share any resource that can help?
I believe that you should handle scraping tasks on the server-side. There are great wrappers for redfin in python for example one here. But if you want to handle scraping in flutter you will need http package to load redfin pages and html package to get specific dom elements. There is a good video tutorial by Johannes Milke.
Looking for a UI integration test strategy for Flutter. We'd love to use Silenium/Ghost Inspector but seems that is not practical due to lack of html id's or CSS classes in Flutter (Add id or name property or other means of identification for Flutter Web applications?). Or has anyone found a way round that?
In the meantime Flutter Driver has only very basic documentation for simple tests like finding a button and pressing the button. Anyone know if I can do other operations like navigate to a specific page (e.g. using a # url fragment), test a link which leads to an external site, check visual setup of the page against an image, and other such tests which would be standard in Silenium and the like.
Thanks!!
Well seems Flutter Driver is still very limited so I have instead found a strategy for using Selenium, posted full details here:
Strategy to use Selenium browser testing with Flutter Web apps
I am pretty new to this tool https://learn.microsoft.com/en-us/composer/introduction
and just trying to explore its capabilities after I heard about in in Build conference last week.
I tried to update echo-bot that came in as a template and wondering about following questions
What language does this tool use to create bots conversations? C#
SDK or JS BF SDK?
How can I get access to code files (.js,.cs) dialog files? So if I
need to modify something we can use VS code later if needed.
Basically I'm trying to understand if we can use this solution to create Bots (dialogs and conversational components etc ) and, later on update code using IDE like VS code to just integrate API's, business logic so finally we get fully functional bot.
Since I basically can get the same info with both approaches, I'm wondering which is the preferred one and what are the advantages using one over the other?
$cordovaGeolocation is angular wrapper over plain javascript plugin, developed by ionic. Now question is why ngCordova was introduced,in simple words to deal it as plugin service as module and inject plugin wrapper as dependency to only particular controller or service.
On Pratical level, cordova developers were having issues with plugins on angular project. One simple issue was that $scope does not get updated sometimes in simple plugins callback.
Quoting from ionic blog post :
The services support promises to make it easier to deal with their
asynchronous nature and ensure scope data is properly updated.
So my conclusion is, you should go with $cordovaGeolocation.
As far as I remember, on Android that plugin does not do anything at all, and on iOS it prevents the OS asking for access to location over and over.
Overall the plugin totally conforms the html5 specs, so you shouldn't change anything in your calling code (in js) if you use the plugin.
And the plugin has a good documentation here: https://github.com/apache/cordova-plugin-geolocation/blob/master/doc/index.md
It is worth noting that according to the current and oficial documentation (see cordova geolocation doc), there is no need to use $cordovaGeolocation, but rather the global object navigator.geolocation.
I am building an iPhone Wikipeida game app, that requires modifying the default Wiki HTML a little bit (mostly simplifying the page).
So far I am directly downloading the HTML output from en.wikipedia.org/wiki/Article_Foo to a python Google App Engine, and then modify its CSS and HTML structure, cache it, and finally output to iPhone. It works but I find this method quite tedious, there must be a better method?
Please note that I use App Engine not just for parsing the Wiki, but the game also requires it to keep the stores...etc, hence not a overkill. Also, I would prefer doing all the work with python on App Engine, to keep the iPhone client as thin and mobile as possible (XML on iPhone is a big no fun)
Thanks a lot.
=======
Nick mentions why not use the mobile Wiki which already optimizes for iPhone. However, the issue is that it goes down quite frequently (every couple weeks or so), also its HTML structure changes quite frequently too.
You can use the MediaWiki API to download the markup text and use some API tools for Python that could make the process/modify work easier.
Caching and outputting to iPhone is fine. I believe there is not much to simplify here.
Why not just fetch the mobile version of the page from http://en.m.wikipedia.org/? This is already formatted for mobile devices.
You can set up your own copy of the server used by m.wikimedia.org:
http://github.com/hcatlin/wikimedia-mobile
It's written in Ruby, but this shouldn't be an issue if your app just uses the HTML output.