I'm trying to scrape prices from this site:
https://www.pensketruckrental.com/quote/start.html
I can easily enter the form data, and I can activate the "Get A Quote" button and click it.
What I can't seem to do is get the form data to submit using a web scraper (I'm just doing it in VBA). When I input text using the scraper, the button remains grayed out, and even making a .click call on the button just displays errors on the form telling you not to leave the fields blank. Apparently it only recognizes data when you use an input device?
The code for one of the required fields, pickupLocation, is the following when I enter it manually (and thus the button works and the form can be submitted):
<input
type="text"
id="pickUpLocation"
name="pickUpLocation"
class="penskeValidateField penskeGoogleTypeAhead penskeInlineError ng-isolate-scope ng-touched ng-focused ng-dirty ng-valid-penske-err_loc_empty_sa ng-valid ng-valid-parse ng-valid-required"
aria-invalid="false"
aria-required="false"
country="rentalEntryCtrl.formItems.country"
penske-validate-field="pickuplocation"
required=""
autocompelete="off"
data-penske-placeholder="rentalEntryCtrl.activePlaceHolders.pickUpLocation"
ng-model="rentalEntryCtrl.formItems.pickupLocationSearchCriteria.address"
autocomplete="off">
And when I enter the data automatically using my scraper the tag & attributes read as follows:
<input
type="text"
id="pickUpLocation"
name="pickUpLocation"
class="penskeValidateField penskeGoogleTypeAhead penskeInlineError ng-pristine ng-isolate-scope ng-invalid ng-invalid-required placeholder ng-touched"
aria-invalid="true"
aria-required="true"
country="rentalEntryCtrl.formItems.country"
penske-validate-field="pickuplocation"
required=""
autocomplete="off"
data-penske-placeholder="rentalEntryCtrl.activePlaceHolders.pickUpLocation"
ng-model="rentalEntryCtrl.formItems.pickupLocationSearchCriteria.address"
autocompelete="off">
So of course I tried to copy the fields in the first code block into the second code block using setAttribute(), but even though I could change the attributes, I still couldn't get the form to submit properly.
I've looked at others that have dealt with something somewhat similar with autocorrect; their solutions have involved looking at the header and responses and just using the straight XHR to loop through the autocomplete queries, but the pricing information I'm scraping comes after several pages of form submissions, so that's not an option here.
I'm stuck I think; any ideas on how to populate the form and click the button/submit via my scraper?
Related
The lastpass autofill icon appears on certain form fields, and on my site it has appeared unexpectedly on an input, with autocomplete="off", with no id and a name of CatName_autocomplete.
I understand I can add an attribute to disable it, but unless I do this for everything in my site, plus any other rival password managers, it could potentially crop up elsewhere. It interferes with my own custom javascript autocomplete functionality that I have on the control because it doesn't trigger the correct events to work properly.
Is there a way to find out the specific underlying logic it uses to decide whether to appear in an input? This would allow me to check I don't accidentally write misleading inputs that trigger it, or that I can know to put the attribute to disable it onto those I know would trigger.
I'm sure it's far more complicated than this, but in case it helps anyone: I disabled the icon in many fields on my page just by removing 'id' from the name attribute of the FIRST input element.
<input type="text" class="form-control" placeholder="Schedule ID"
name="scheduleId" [(ngModel)]="scheduleId" aria-label="Schedule ID"/>
triggers icons on the page, not just on this form.
<input type="text" class="form-control" placeholder="Schedule ID"
name="schedule" [(ngModel)]="scheduleId" aria-label="Schedule ID"/>
does not.
<form name="abc" id="abc" method="post" action="/test.cfm" enctype="multipart/form-data">
<input type="submit" name="btnSubmit" id="btnSubmit" value="OK" />
</form>
for some reason when I hit submit the "btnSubmit" is not showing up in the cfdump.
<cfdump var="#form#">
There aren't a lot of things that could cause it to simply not appear in your form. My short-list of culprits are:
Looking at the wrong file / server.
The page is being redirected via cflocation or otherwise (javascript location.replace() or location.href=x) -- this would cause that problem even if the redirection is returning the browser to the same page.
Form variables being stripped out somewhere further up, I would guess in onRequestStart
A local variable named "variables.form" was created and set to a structure further up - not very likely, but I suppose it's possible someone could accidentally write something like <cfset form = url />, which might cause that
Usually, when something like this has happened to me in the past, I've found that I've been viewing the wrong file in the browser. Usually it's come down to me looking at the same file on the wrong domain name, e.g. looking at the production server instead of the development server.
If you combine your code segments above into a single file like this (test.cfm):
<cfdump var="#form#" />
<form name="abc" id="abc" method="post" action="test.cfm" enctype="multipart/form-data">
<input type="submit" name="btnSubmit" id="btnSubmit" value="OK" />
</form>
That ought to give you some insight into your problem. Note that I removed the leading-slash / in the form action, so that this form will post to itself. When I first view this template, I see an empty struct (followed by the button), because I haven't put anything in the form scope yet. When I submit the form I then see two elements in the structure, fieldnames and btnSubmit. That's another good indicator, if you don't see fieldnames in the form structure, then your CFML page may not have received a form submission. If you know you're looking at the right page and you've submitted the form and you still don't have the fieldnames entry, then I'd start looking for potential browser redirection.
You might also want to add an empty Application.cfc in the same directory just to be sure that there's not an application interfering with it. It's possible that something in the onRequestStart might be stripping out form variables with the name "btnSubmit" or even any form variable with the string "submit" anywhere in the name. I wouldn't expect it though -- I'd look for other causes like cflocation tags first.
This is quite a common question but the solutions I found in other people posts are either related to a specific browser (mostly firefox) or incorrect usage of names (name="U12-678132-34")
My issues are with browsers other then Firefox (Firefox all ways works).
The form that I use is pretty standard HTML form but the submission of it is done with javascript (jQuery AJAX).
Firefox all ways asks to remember the password (if it is a new user) and refills the form if you land on that same page again. But when it comes to Chrome/Safari/IE8-9 then they never request to save a password if the form is submitted with javascript.
(By the way I did check if the browsers dont have the - never remember passwords turned on)
My submit happens when you click on the link inside the form or if you just click the "ENTER" button on your keyboard, then it initiates the $.submit() method.
Is there a specific way that the submit needs to occur so that the browser would request to save a password like firefox does? or is there a way to at least tell a specific browser like Chrome/IE to force that type of request?
Form example:
<form class="loginform" method="post" action="">
<div class="inputfield">
<input name="email" type="text" class="emailaddress inputclass" value="" title="Email Address" />
</div>
<div class="inputfield">
<input name="password" type="password" class="password inputclass" title="Password" value="" />
</div>
<div class="submit">
<div class="checking">
<img src="/preloaders/login-preloader.png"/>
</div>
<input type="submit" name="submit" value="submit" style="display:none"/>
</div>
</form>
This is browser behaviour and can't really be changed. Firefox might be "smart" enough to offer to save passwords without the form actually being submitted, but that risks having buttons in the form also trigger that option even if the button does something different. So in my opinion, it's a bad thing for Firefox to do (I've had many problems with Firefox submitting forms even though it shouldn't).
If you really want the save password option to show up, use an iframe and submit to the iframe, instead of using AJAX. You could then use AJAX from the iframe to keep the old behaviour.
attach click event to your submit button
$('#id_of_submit').click(function() {
/your ajax logic
return false;
});
and on link
$('#id_of_your_link').click(function() {
$('#id_of_submit').click();
});
this will do the trick.
Looking at the answer accepted on here - How can I get browser to prompt to save password? - it seems that a valid action might help.
But i would suggest its down to browser behaviour and cannot be controlled by HTML and/or JavaScript. If you want to remember the values entered use a Cookie
As u r doing an AJAX post, then-
Remove the <form> tags
instead of <input type="submit", use button
take the field values & AJAX post- on button click event
it might do the trick.
One of the reason is that site should have a valid certificate. If it is not secured site, password save prompt will not appear after login.
I have a little problem here. Actually, more of an annoyance.
I have a form on my index page that has a small search form:
<form action="search.php" method="post">
<input name="search" type="text" />
<input type="submit" name="submit">
now on the search.php file I just use the $_POST['search'] to retrieve the value that I'm searching. This file also displays the actual search results, which I can click on to go to that page.The search is actual done on the database. Also, because my search returns the top 10 random results from the DB, if I hit the 'back' button and confirm the warning, the search will be executed again, and will return a different set of results.
The problem, as you know, is that when you click on one of the search results and then hit 'back' on the browser, you get that browser warning about 'To display this page, Firefox must send information that will repeat any action (such as a search or order confirmation) that was performed earlier.'.
What's the easiest way to prevent this from happening? I looked into that PRG technique but I'm not sure how to implement that.
Any help would be great.
Thanks.
Bruno.
Is it OK to use GET method here, users won't see warning if form sent by GET method.
Last night, I did a quick spike to try and implement username/password autocomplete in my GXT application. By "autocomplete", I don't mean Ajax-style autocomplete, but rather browser-based autocomplete. The best information I found on this via google is in the following post:
http://osdir.com/ml/GoogleWebToolkit/2009-04/msg01838.html
I didn't use this technique because I'm using GXT and didn't want to lose the look-and-feel of my login form.
I was successful in getting everything to work in Firefox (it populates both the username and password). In IE, it only populates the username, not the password. In Safari/Chrome, it doesn't work at all.
Here's how I did it:
Created a hidden HTML form on my HTML page that embeds GWT.
<form method="post" action="javascript:void(0)" style="display: none">
<input type="text" id="username" name="username" value=""/>
<input type="password" id="password" name="password" value=""/>
<input type="submit" value="Login" id="login"/>
</form>
When a user clicks on the "Login" button in my GWT application, populate the fields in this hidden form and "click" on the Login button (which will do nothing since the action="javascript:void(0)".
// Set the hidden fields to trigger the browser to remember
DOM.getElementById("username").setAttribute("value", username.getValue());
DOM.getElementById("password").setAttribute("value", password.getValue());
clickFormLogin();
...
public static native void clickFormLogin() /*-{
$doc.getElementById("login").click();
}-*/;
This works in Firefox 3.5 and prompts me to save the user/pass at the top of the screen. I believe I know why this doesn't work in Safari/Chrome and that's because the form's action doesn't go anywhere and the form is not submitted. If I change the action to be an actual URL and show the form, clicking on the form's Login button will save it in those browsers.
After typing this up as a question here, I got to thinking this might make a good blog post. Therefore, I copied everything and added a bit to my blog:
http://raibledesigns.com/rd/entry/browser_based_username_password_autocomplete
Summary and Question
While I'm glad I got it working in Firefox, I'm disappointed with IE's lack of password autocompletion. More than anything, I can't help but think there's a way to make this work in WebKit-based browsers.
Anyone know how to implement cross-browser username/password autocomplete in GWT (specifically GXT)?
Use persistent Cookies instead.
IE do save passwords, if user chooses to, but it works different. You need to type at least the username so it will autocomplete the password.
You need a plain vanilla html submit button. I think that will fix it.
http://groups.google.com/group/Google-Web-Toolkit/browse_thread/thread/2b2ce0b6aaa82461