I need to take data from html page, so I am using LWP to get the page content.
the response I got is partial and not the full source of the page.
...
<div style="display:none" id="QUERY" query=""></div>
<div style="display:none" id="COLL" idcoll=""></div>
<div style="display:none" id="BROWSE" field=""></div>
<div id="center"></div>
<div id="loading"></div>
...
when using a web debugger(FIRE BUG) I can see a hidden content under:
<div id="center"></div>
<div id="loading"></div>
How can I get the hidden data using Perl ?
It breaks my mind for 3 days now !
Thanks ahead.
let's say it a JS running... How can i
see the content?
You could use WWW::Mechanize::Firefox. It seems to support Javascript.
If the content is indeed added using Javascript, you might be able to use WWW::Scripter with the Javascript or Ajax plugin.
If it is not present in the HTML source that LWP fetches, it is added in some other way. There probably is a Javascript running, or the webserver serves you and LWP different pages because of cookies or user agent string.
Install Firebug or use the Safari Develop menu to see what AJAX/XHR requests are being done to the server, and with what POST/GET parameters. You can then use LWP or any other HTTP client module to do such a request.
Related
I am trying to scrape a page that is modified by javascript after initial load using Scrapy on Raspberri Pi.
I tried to install docker and scrapinghub/splash to render the page before passing it into scrapy, but realized Splash doesn't support ARM yet. Are there other options to scrape pages using javascript with Scrapy on a Raspberry Pi?
Currently, using the normal scrapy request on the site I only get this html, which is because the site loads first, and then the javascript renders the entire content. So before the javascript the page source looks empty:
<body class="notie8 notie9 lang-{{html.lang}}">
<!--<![endif]-->
<div loading-line></div>
<div page-layout>
<div ng-view></div>
</div>
</body>
</html>
For reference, the site I am referring to is: https://www.sreality.cz/hledani/prodej/byty?region=brno
Sreality uses API, isn't this a way to go? For your URL, there's this API call: https://www.sreality.cz/api/cs/v2/estates?category_main_cb=1&category_type_cb=1&per_page=20®ion=brno&tms=1502631428897 (look for XHR requests in your browser's developer tools).
I am trying to understand what kind of web applications Apache Wicket is suitable for, and it seems to be page based from what I have seen. How can it be used to make single page web apps as well?
At our company we have 3 SPA built on Wicket. They all work :) Basiclly our structure is:
<html>
<head>
</head>
<body>
<div wicket;id="menu">
<!-- navi links -->
</div>
<div wicket:id="main">
</div>
</body>
</html>
And we replace the main panel and it's inside panels effectivly getting an SPA.
Ajax support in Wicket 6 is very good.
Yes, its basically for page based webapps. So, it can also be used easily form single page web app.
I suggest just reading this short example of Hello world
After that, you can easily edit your equivalent of HelloWorld.html and HelloWorld.java to make really easy HTML in Java.
I have a fairly simple jQuery mobile app, which works fine as a web app (tested in browsers on desktops/smartphones) but when porting it to android using phonegap, I have one major issue. It occurs both on a real phone and in an AVD.
there's a form in the app, like:
<div data-theme="a" data-role="dialog" id="a-form">
<div data-role="content">
<h3>...</h3>
<form action="http://a-live-site.com/a-form.php" method="post">
...
<div data-role="fieldcontain">
<button type="submit">Submit</button>
</div>
</form>
</div>
</div>
Its response content loads perfectly in the browser (the action is a relative link there), but within phonegap I just get a white screen and the hardware back button exits the app instead of the usual step back in history. The puzzling thing is that the request actually gets through. The server side code triggers an email, so I know the server is getting the request. I have added the appropriate <access /> rules in the phonegap.xml file, but I'm not sure that's enough.
EDIT: the form is in static content so, in phonegap, it's loaded by file:// and shouldn't have cross-domain issues.
The problem was due to data-role="dialog" in the result. Using page fixed it. It only broke in phonegap and phonegap's logs made it look like it was something else. A real PITA.
I'm trying to do a submit via ajax of a form that contains a file element.
<form id="classic_upload" enctype="multipart/form-data>
<input type="file" name="file" id="file"/>
<br/>
<!-- ...other inputs...-->
<button type="button" id="classic_save"> Send </button>
</form>
What I need to do is to submit this form and check if the file fulfills some requirements, so I wrote an ajax submit for this form
$('#classic_save').click(function(){
$.ajax({
type:"POST",
url:'<g:createLink action="classicUploadFile" controller="scan"/>',
success: function(msg){
alert("Data Loaded: " + msg);
}
});
});
However, I have no idea how to send the file through ajax.
Some context
Originally we were using swfUpload for this. However, we ran into some trouble with https and some certificate issues. So we decided to implement a basic html fallback. Plugins are nice, but we need to guarantee that this fall back is bullet proof (thinking of google mail "classic upload").
Any thoughts? Are iframes the way to go (read somewhere google mail uses them for their classic upload)
Thanks in advance.
If you like, there is jQuery Uploadify plugin to do exactly what you are looking for other than other great features.
I use the jQuery AJAX form plugin on my site to upload files.
Here you'll find a really good example/tutorial how to upload one file http://www.phpletter.com/Demo/AjaxFileUpload-Demo/
Hope it works for you.
Cheers
I'm writing a Facebook iframe/Facebook Connect application and one of the pages includes a multi-friend-selector. It renders perfectly in every browser I tried (FF/Mac, Safari/Mac, IE8/Win) but does not render at all in IE7/Win. I waited and waited (in case it was just being slow) and it never appeared.
Here's my code.
<fb:serverfbml style="width:750px;height:700px">
<script type="text/fbml">
<fb:fbml>
<fb:request-form
action="<? echo $invite_href; ?>"
method="post"
type="<? echo $app_name; ?>"
content="<? echo htmlentities($content,ENT_COMPAT,'UTF-8'); ?>">
<div class="clearfix" style="padding-right:20px;" height="500" width="750">
<fb:multi-friend-selector
actiontext="Here are your friends who don't have <? echo $app_name; ?> yet. Invite your friends and let them help too - it's free!"
exclude_ids="<? echo $friends; ?>" />
</div>
</fb:request-form>
</fb:fbml>
</script>
</fb:serverfbml>
The other required things (FeatureLoader.js and all that) are (must be) properly included, since the friend selector and every other fb tag I use around the application works perfectly. The serverfbml tag is the ONLY thing giving me trouble in IE7. Most people seem to have a problem with the selector rendering at an annoying 150px tall - I can't seem to find anyone else who has a problem with it not rendering at all.
You do not need the closing tags because you're rendering FBML insdie the serverfbml tag, not xfbml. Another thing I noticed is you're passing PHP variables into it, whereas I'm pretty sure Facebook strips out all your php from within the server fbml, I may be wrong though.
Daniel
Another SO post, which solved my problem: Why does this XFBML markup embedded within an iframe'd facebook application not display anything?
Are you using IE Tester to test ie6? If so it fails to render, this is an IE Tester bug.
Facebook is encouraging developers to switch to Requests 2.0. FBML is being deprecated and so reliance on fb:serverFbml will likely by phased out at some point this year. Fortunately there is a MUCH easier way to do this using the JavaScript SDK and it even works on websites.
FB.ui({method:'apprequests',...});
Learn more about using this here: http://af-design.com/blog/2011/02/17/using-facebook-requests-to-promote-a-website/