I am trying to update status on facebook using Mechanize.I am able to login using the script but unable to update.I verified the id of form for status update is "u_0_w".
But selecting the form_id method says "There is no form with ID "u_0_w"".
My script is this:
use WWW::Mechanize;
use strict;
use warnings;
use Data::Dumper;
use HTTP::Cookies::Netscape;
my $cookiesfilename='/home/xxx/xxx/cookies.txt';
my $out;
my $mech = WWW::Mechanize->new( cookie_jar => HTTP::Cookies::Netscape->new( file => $cookiesfilename ) );
$mech->get("https://www.facebook.com/login.php");
my $response=$mech->submit_form(
fields => {
email => 'xxxx#xxxx.com',
pass => 'xxxxx',
}
);
#my $array=$mech->forms();
#$mech->get('/home.php');
print Dumper($mech->forms());
#$mech->form_id("u_0_w");
$mech->submit_form(
fields => {
xhpc_message_text=>'Why so serious'
}
);
print $response->status_line;
open($out, ">", "output_page.html") or die "Can't open output_page.html: $!";
print $out $response->decoded_content;
Then I tried to print all the forms on the page using Dumper the output is:
$VAR1 = bless( {
'default_charset' => 'UTF-8',
'enctype' => 'application/x-www-form-urlencoded',
'accept_charset' => 'UNKNOWN',
'action' => bless( do{\(my $o = 'https://www.facebook.com/search/web/direct_search.php')}, 'URI::https' ),
'method' => 'GET',
'attr' => {
'method' => 'get'
},
'inputs' => [
bless( {
'tabindex' => '-1',
'value' => '1',
'class' => '_42ft _42fu _4w98',
'type' => 'submit'
}, 'HTML::Form::SubmitInput' ),
bless( {
'/' => '/',
'autocomplete' => 'off',
'tabindex' => '1',
'name' => 'q',
'aria-label' => 'Search Facebook',
'value_name' => '',
'class' => 'inputtext _586f',
'type' => 'text',
'id' => 'u_0_b',
'role' => 'combobox',
'placeholder' => 'Search Facebook'
}, 'HTML::Form::TextInput' )
]
}, 'HTML::Form' );
It means it is not detecting the status update form it is detecting only Facebook search form.
What may be the problem for mechanize not detecting all the form elements?
The form contains <button type="submit">. Do Mechanize support it?
Why do you have to use Mechanize for this? There's already a module available for this on CPAN.
Take a look at WWW::Facebook::API.
Also see a related question: How do I use Perl's WWW::Facebook::API to publish to a user's newsfeed?
Synopsis:
use WWW::Facebook::API;
my $facebook = WWW::Facebook::API->new(
desktop => 0,
api_key => $fb_api_key,
secret => $fb_secret,
session_key => $query->cookie($fb_api_key.'_session_key'),
session_expires => $query->cookie($fb_api_key.'_expires'),
session_uid => $query->cookie($fb_api_key.'_user')
);
my $response = $facebook->stream->publish(
message => qq|Test status message|,
);
Related
I have the following code to scrape a form for inputs and get the attributes id and name.
#!/usr/bin/perl
use warnings;
use strict;
use URI;
use Data::Dumper::Simple;
use Web::Scraper;
my $urlToScrape = "http://digitalarkivet.arkivverket.no/finn_kilde";
my $scrap = scraper {
process 'div.listGroup.open > ul.grouped > li.expandable', 'data[]' => scraper {
process 'input', 'id' => '#id', name => '#name';
process 'label', 'label_for' => '#for';
process 'span.listExpander ', 'Text' => 'TEXT';
process 'ul.sublist1', 'sublist[]' => scraper {
process 'input', 'id' => '#id', name => '#name';
process 'label', 'label_for' => '#for';
process 'span', 'label' => 'TEXT';
};
};
};
my $res = $scrap->scrape(URI->new($urlToScrape));
print Dumper($res);
which gives me (shortend $res to fit screen better)
$res = {
'data' => [
{
'label_for' => 'ka0',
'sublist' => [
{
'label' => 'Statlig folketelling',
'label_for' => 'ka0kt0',
'name' => 'kt[]',
'id' => 'ka0kt0'
}
],
'name' => 'ka[]',
'id' => 'ka0>',
'Text' => 'Folketellinger'
},
{
'sublist' => [
{
'label' => 'Manntall',
'name' => 'kt[]',
'label_for' => 'ka1kt0',
'id' => 'ka1kt0'
}
],
'label_for' => 'ka1',
'id' => 'ka1>',
'name' => 'ka[]',
'Text' => 'Manntall'
},
....
{
'label_for' => 'r0',
'sublist' => [
{
'label_for' => 'r0f0',
'id' => 'r0f0',
'name' => 'f[]',
'label' => "01 Østfold"
}
],
'id' => 'r0',
'name' => 'r[]',
'Text' => "Østlandet"
},
{
'Text' => "Sørlandet",
'id' => 'r1',
'sublist' => [
{
'label_for' => 'r1f0',
'name' => 'f[]',
'id' => 'r1f0',
'label' => '09 Aust-Agder'
}
],
'label_for' => 'r1',
'name' => 'r[]'
}
]
};
I' have 2 issues I need to fix. First, I only want to get data for inputs having 'name' = ka[] (at top level).
Second, I only get data for first ul.sublist1 (If you study the page I'm scraping you can see that several "Kildekategori" have subsets of data, which are revealed if expanded/ clicked upon. Putting brackets on Text[] only gets me the sublist textnames, but not their attributes.
I'm thinking I might have to grab data in 2 scrapes instead, since nested values are revealed by id and label_for.
Solved it by scraping three times, foreach "level"
#!/usr/bin/perl
use strict;
use warnings;
use URI;
use Web::Scraper;
use Data::Dumper::Simple;
my %site;
my #res;
my $i;
my $j;
my $label_for;
my #scrape;
$site{'siteID'} = 1;
$site{'url'} = "http://digitalarkivet.arkivverket.no/finn_kilde";
$site{'name'} = "finn_kilde";
open FIL, ">$site{'name'}.csv" or die $!;
my $seperator=";";
$scrape[0] = scraper {
process 'div.listGroup.open > ul.grouped > li.expandable', 'data[]' => scraper {
process 'input',
'id' => '#id',
'value' => '#value',
'type' => '#type',
'name' => '#name';
process 'label', 'label_for' => '#for';
process 'span.listExpander ', 'text' => 'TEXT';
};
};
$scrape[1] = scraper {
process 'ul.sublist1 > li', 'data[]' => scraper {
process 'input',
'id' => '#id',
'value' => '#value',
'type' => '#type',
'name' => '#name';
process 'label', 'label_for' => '#for';
process 'span', 'text' => 'TEXT';
}
};
$scrape[2] = scraper {
process 'ul.sublist2 > li', 'data[]' => scraper {
process 'input',
'id' => '#id',
'value' => '#value',
'type' => '#type',
'name' => '#name';
process 'label', 'label_for' => '#for';
process 'span', 'text' => 'TEXT';
}
};
for $i (0 .. $#scrape){
$res[$i] = $scrape[$i]->scrape(URI->new($site{'url'}));
unless ($i) {
print FIL join($seperator,"label_for","text","name","value","id","type")."\n";
}
for $j (0 .. $#{$res[$i]->{data}}) {
if (defined($res[$i]->{data}[$j]->{label_for})){
$label_for=$res[$i]->{data}[$j]->{label_for};
} else {
$label_for="";
}
if (length($label_for)>0) {
my $name=$res[$i]->{data}[$j]->{name};
my $text=$res[$i]->{data}[$j]->{text};
my $value=$res[$i]->{data}[$j]->{value};
my $id=$res[$i]->{data}[$j]->{id};
my $type=$res[$i]->{data}[$j]->{type};
my #row=($label_for,$text,$name,$value,$id,$type);
print FIL join($seperator,#row);
print FIL "\n";
}
}
sleep(2);
}
close FIL;
print Dumper(\#res);
1;
I'm a non-programmer attemting to retrieve useful info from our InfoBlox DHCP boxes. I've installed the Perl API and can make some use of it.
I've got an output from the Data::Dumper "thingie" that appears to have some of the info I want. I'd like to directly reference some of that data but I'm unsure how.
print Dumper(\$object)
Here is part of the Data::Dumper output;
$VAR1 = \bless( {
'network' => '10.183.1.0/24',
'override_lease_scavenge_time' => 'false',
'enable_ifmap_publishing' => 'false',
'low_water_mark_reset' => '10',
'use_lease_time' => 0,
'use_enable_option81' => 0,
'network_container' => '/',
'override_ddns_ttl' => 'false',
'rir' => 'NONE',
'network_view' => bless( {
<snip> --------------------------------------
'extattrs' => {
'Use' => bless( {
'value' => 'Voip'
}, 'Infoblox::Grid::Extattr' )
},
<snip> --------------------------------------
'members' => [
bless( {
'ipv4addr' => '10.85.9.242',
'name' => 'ig3-app3.my.net'
}, 'Infoblox::DHCP::Member' ),
bless( {
'ipv4addr' => '10.85.9.210',
'name' => 'ig3-app1.my.net'
}, 'Infoblox::DHCP::Member' ),
bless( {
'ipv4addr' => '10.85.9.226',
'name' => 'ig3-app2.my.net'
}, 'Infoblox::DHCP::Member' )
],
'override_ignore_client_identifier' => 'false',
'email_list' => undef,
'rir_registration_status' => '??
}, 'Infoblox::DHCP::Network' );
How do I view the elements? ie ...
print $object{members->name};
print $object{members->ipv4addr};
print $object{extattrs->Use->value};
I've found the API dox insufficiant for my skill level:) The data I'd like to pull remains just out of reach.
my #retrieved_objs = $session->search (
object => "Infoblox::DHCP::Network",
network => '.*\.*\.*\..*',
);
foreach $object ( #retrieved_objs ) {
my $network = $object->network;
my $comment = $object->comment;
my $extattrs = $object->extattrs;
my $options = $object->options;
print $network, " network ", $comment, " ", $extattrs, " ", $options, "\n";
}
-------- output ---
10.183.2.0/24 network HASH(0x6a2f038) ARRAY(0x1d20eb0)
10.192.1.0/24 network HASH(0x9df6540) ARRAY(0x9df5468)
10.192.2.0/24 network HASH(0xa088fc8) ARRAY(0xa089718)
You shouldn't try to access the internal values of an object directly. The module - in this case Infoblox::DHCP::Network will provide methods that allow you to read or manipulate the values properly.
This post is follow on work related to LWP GET large file download. That post was regarding an error from LWP when trying to pass arguments in the header incorrectly. Now I am posting the changes I made and how I am trying to debug the approach. This discussion should be very informative for those interested in POST vs GET header formation, and what the server receives while using the CGI package. It is not information easily found on the net.
Here is my client code snip:
my $bytes_received = 0; # vars used below are set prior to this point
my $filename = $opt{t}."/$srcfile";
open (FH, ">", "$filename") or $logger->error( "Couldn't open $filename for writing: $!" );
my $ua = LWP::UserAgent->new();
my $target = $srcfile;
my $res = $ua->get(
$url,
':content_cb' => \&callback,
'api' => 'olfs', # Note attempted use of different types of quotes had no impact
"cmd" => 'rfile',
"target" => $target,
"bs" => $bs
);
print $logger->info("$bytes_received bytes received");
sub callback{
my($chunk, $res) = #_;
$bytes_received += length($chunk);
print FH $chunk;
}
Here is the server snip (cgi script):
my $query = new CGI;
my $rcvd_data = Dumper($query);
print $rcvd_data;
Here is the output from a GET:
$VAR1 = bless( {
'.parameters' => [],
'use_tempfile' => 1,
'.charset' => 'ISO-8859-1',
'.fieldnames' => {},
'param' => {},
'.header_printed' => 1,
'escape' => 1
}, 'CGI' );
Here is a client with a POST request:
my $ua = new LWP::UserAgent();
local $HTTP::Request::Common::DYNAMIC_FILE_UPLOAD = 1;
my $req =
POST
$url,
'Content_Type' => 'form-data',
'Content' => {
"api" => 'olfs',
"cmd" => 'wfile',
"target" => $target,
"tsize" => $file_size,
"bs" => $bs,
"filename" => [ $file ] };
# HTTP::Message calls set_content, which appears to set the subroutine for content
# LWP::UserAgent
# LWP::Protocol::file::request sends content in chunks
#
$req->content( $req->content() );
$logger->info("Uploading: $file");
my $resp = $ua->request($req);
Here is the output on the server, just like before but now from the POST:
'.parameters' => [
'cmd',
'bs',
'api',
'target',
'filename',
'tsize'
],
'use_tempfile' => 1,
'.tmpfiles' => {
'*Fh::fh00001random23' => {
'info' => {
'Content-Type' => 'text/plain',
'Content-Disposition' => 'form-data; name="filename"; filename="random23"'
},
'name' => bless( do{\(my $o = '/usr/tmp/CGItemp33113')}, 'CGITempFile' ),
'hndl' => bless( \*Fh::fh00001random23, 'Fh' )
}
},
'.charset' => 'ISO-8859-1',
'.fieldnames' => {},
'param' => {
'cmd' => [
'wfile'
],
'bs' => [
'buffer1'
],
'api' => [
'olfs'
],
'target' => [
'random23'
],
'tsize' => [
'1073741824'
],
'filename' => [
$VAR1->{'.tmpfiles'}{'*Fh::fh00001random23'}{'hndl'}
},
'escape' => 1,
'.header_printed' => 1
}, 'CGI' );
In short, you can see in the POST dump the "key" / "value" pairs, ie "target => random23". In the GET dump I do not find any keys or values from what I submitted on the client side. Can that be explained, or what do I need to do so as to extract key / value pairs in the CGI script?
You're passing your form variables as HTTP headers.
Like I previously mentioned, if you want to build a url, you can use URI.
$url = URI->new($url);
$url->query_form(
api => 'olfs',
cmd => 'rfile',
target => $target,
bs => $bs,
);
Here is my dilemma: I am trying to fill out a web form and get a result back from that form using LWP::UserAgent. Here is an example of my code:
#!/usr/bin/perl -w
use strict;
use LWP;
use HTTP::Request::Common;
use LWP::Debug qw(+);
my $ua = LWP::UserAgent->new(protocols_allowed=>["https"]);
my $req = POST 'https://their.securesite.com/index.php',
[ 'firstName' => 'Me',
'lastName' => 'Testing',
'addressLine1' => '123 Main Street',
'addressLine2' => '',
'city' => 'Anyplace',
'state' => 'MN',
'zipCode' => '55555',
'card' => 'visa',
'cardNumber' => '41111111111111111',
'ccv2' => '123',
'exp_month' => '07',
'exp_year' => '2015',
'shared_key' => 'hellos',
];
my $response = $ua->request($req);
print $response->is_success() . "\n";
print $response->status_line . "\n";
print $response->content . "\n";
When I run this, I get back a 200 OK and a "1" for success, but not the response page from the form. Just the closing tags:
</body>
</html>
Could this possibly be due to the fact that the form page and response page both have the same URL? I am new to LWP, so I am grasping at straws here. It may still be on the clients end, but I want to rule out any issues on my end as well.
Thanks in advance for any help you guys can give - I am Googled out.
If you can use Mojo::UserAgent (part of the Mojolicious suite of tools) the code would look like this. Note that you might need IO::Socket::SSL in order to use HTTPS.
#!/usr/bin/env perl
use strict;
use warnings;
use Mojo::UserAgent;
my $ua = Mojo::UserAgent->new;
my $tx = $ua->post('https://their.securesite.com/index.php', form =>
{ 'firstName' => 'Me',
'lastName' => 'Testing',
'addressLine1' => '123 Main Street',
'addressLine2' => '',
'city' => 'Anyplace',
'state' => 'MN',
'zipCode' => '55555',
'card' => 'visa',
'cardNumber' => '41111111111111111',
'ccv2' => '123',
'exp_month' => '07',
'exp_year' => '2015',
'shared_key' => 'hellos',
});
if ( $tx->success ) {
print $tx->res->body;
# or work with the resulting DOM
# my $dom = $tx->res->dom;
} else {
my ($err, $code) = $tx->error;
print $code ? "$code response: $err\n" : "Connection error: $err\n";
}
The interface is a little different, but it has lots of nice features, including Mojo::DOM integration for parsing the response HTML.
Use $response->decoded_content to get the content without the headers. See HTTP::Message for more information.
#!/usr/bin/perl -w
use strict;
use URI;
use LWP::UserAgent;
use HTTP::Request;
my $url = URI->new('https://their.securesite.com/index.php');
my $ua = LWP::UserAgent->new();
my $request = HTTP::Request->new(
'POST',
$url,
HTTP::Headers->new(
'User-Agent' => "perl ua/ v0.001",
'Accept' => "text/xml, multipart/*, application/soap"
),
[ 'firstName' => 'Me',
'lastName' => 'Testing',
'addressLine1' => '123 Main Street',
'addressLine2' => '',
'city' => 'Anyplace',
'state' => 'MN',
'zipCode' => '55555',
'card' => 'visa',
'cardNumber' => '41111111111111111',
'ccv2' => '123',
'exp_month' => '07',
'exp_year' => '2015',
'shared_key' => 'hellos',
]
) or die "Error initiating Request: $#\n";
my $response = $ua->request( $request );
if ($response->is_success) {
print $response->decoded_content, "\n";
} else {
die $response->status_line;
}
Check the value of $response->as_string
It'll show you full http response with headers
there is a bit of code I'm trying to replicate in Perl using either LWP::UserAgent or WWW::Mechanize from an existing script.
The original script actually does more than I'm looking to do. I'd just like to log into the Nest website (the part I need help with) and then parse out some data for historical logging (I'm good there).
My current script I would expect to work, but I'm not sure if the authResult/access_token from the Ruby example us actually understood/used by either Perl module.
My code in Perl:
#!/usr/bin/perl
use WWW::Mechanize;
#use HTTP::Request::Common qw(POST);
use HTTP::Cookies;
use LWP::UserAgent;
use Data::Dumper;
use CGI;
my $email; #stores our mail
my $password; #stores our password
my $user_agent = 'Nest/1.1.0.10 CFNetwork/548.0.4';
$email = "email#email";
$password = "mypassword";
my #headers = (
'User-Agent' => 'Nest/1.1.0.10 CFNetwork/548.0.4',
'X-nl-user-id' => $email,
'X-nl-protocol-version' => '1',
'Accept-Language' => 'en-us',
'Connection' => 'keep-alive',
'Accept' => '*/*'
);
# print "Content-type: text/html\n\n";
my $cookie = HTTP::Cookies->new(file => 'cookie',autosave => 1,);
my $browser = WWW::Mechanize->new(cookie_jar => $cookie, autocheck => 1,);
# tell it to get the main page
$browser->get("https://home.nest.com/user/login");
print Dumper($browser->forms);
# okay, fill in the box with the name of the
# module we want to look up
$browser->form_number(1);
$browser->field("username", $email);
$browser->field("password", $password);
$browser->submit();
print $browser->content();
When I submit the form, I just get the same page returned back to me, and I don't know what exactly is causing Nest to not like what I'm submitting. There are two additional fields in the form on their log-in page:
'inputs' => [
bless( {
'maxlength' => '75',
'/' => '/',
'value_name' => 'E-mail address',
'name' => 'username',
'id' => 'id_username',
'type' => 'text'
}, 'HTML::Form::TextInput' ),
bless( {
'/' => '/',
'value_name' => 'Password',
'name' => 'password',
'id' => 'id_password',
'type' => 'password',
'minlength' => '6'
}, 'HTML::Form::TextInput' ),
bless( {
'readonly' => 1,
'/' => '/',
'value_name' => '',
'value' => '',
'name' => 'next',
'type' => 'hidden'
}, 'HTML::Form::TextInput' ),
bless( {
'readonly' => 1,
'/' => '/',
'value_name' => '',
'value' => 'dbbadca7910c5290a13d30785ac7fb79',
'name' => 'csrfmiddlewaretoken',
'type' => 'hidden'
}, 'HTML::Form::TextInput' )
Do I need to use the csrfmiddlewaretoken value in each submission? It appears to change. I thought getting a cookie upon a successful login would be enough.
Any suggestions on what I'm doing wrong?
Shot in the blue:
perl -E'use warnings; $email = "email#email"; say "<$email>"'
Possible unintended interpolation of #email in string at -e line 1.
<email>
I suspect it fails because the form gets the wrong user name, print it out to confirm. Always enable the pragmas strict and warnings to make many common mistakes visible.