Bellow is a script which grabs a cookie from "example.com" and encodes it base64,
It usually works, although for some reason I will have random days where it acts up and does not grab any cookies at all.
I've checked myself at times where the script was failing, and the site would still send a cookie to the client.
Same method, nothing changed on the sites behalf, and nothing would change on the script, though it would still just act up some times.
Anyone know what this could possibly be?
Do I have to change my method of grabbing cookies incase this method may be obsolete or ancient?
my $ua = new LWP::UserAgent;
$ua->cookie_jar({});
use Data::Dumper;
$ua->get("http://www.example.com");
my $cookie = encode_base64($ua->cookie_jar->as_string);
Other info: It's apart of a perl cgi script, hosted on a website.
I'm still unsure of the error thats causing this, Although, when I had turned off "CloudFlare" for my website, the problem resolved itself immediately.
Oh well there goes 3 hours of my life over CloudFlare...
Related
I'm using HttRequestMessage, and adding a cookie in an IHttpFilter as follows:
public IAsyncOperationWithProgress<HttpResponseMessage, HttpProgress> SendRequestAsync(HttpRequestMessage request)
{
var c = new HttpCookiePairHeaderValue("SSOAutologonCertificate", "true");
request.Headers.Cookie.Add(c);
return InnerFilter.SendRequestAsync(request);
}
Using Fiddler I can see that the cookie is correctly added to the outgoing request. However, the reply is an auto-redirect. When fiddler captures that outgoing request, that cookie is no longer set in the http header - it seems to have been forgotten.
Is this expected behavior? Or a bug? If it is expected behavior, is there anything I can do to alter it?
If, on the other hand, I add the cookie to the Cookie collection on a HttpBaseProtocolFilter object, then everything works as expected, and the cookie is there for every auto-redirect query. Of course, while that will work for now, that totally defeats the purpose of a IHttpFilter. :-) This behavior was observed on a windows store app (running on Windows 10 tech preview).
BTW, I spent some time trying to figure out how to step into the .NET source for the Windows.Web.Http namespace, but nothing seemed to work (I can into other .NET source). Is there a special trick for these windows component files (the module is Windows.Web.winmd).
Thanks in advance!
I am using WWW::Mechanize::Firefox along with MozRepl plugin in Firefox. The code works properly to fetch content from sites by sending them an HTTP GET request.
I am going through a list of URLs and sending an HTTP GET Request to each of them.
However, if the request hangs on a particular URL, it keeps waiting.
Please note that I am referring to cases where a part of the web page content is loaded while some of the content is still pending. It happens in cases where a web page loads a lot of content from third party sites and if one of the resources (an image for instance) could not be loaded, the browser keeps waiting for it.
I want the request to timeout after 'n' seconds so that I can read the next URL from the list and continue with the code execution.
In WWW::Mechanize perl module, the constructor supported the timeout option as shown below:
$mech=WWW::Mechanize->new(timeout => 10);
However, I could not find a similar option in the documentation for the Perl Module, WWW::Mechanize::Firefox here:
http://metacpan.org/pod/WWW::Mechanize::Firefox
I tried this:
$mech=WWW::Mechanize::Firefox->new(timeout => 10);
But I think it does not work as there are still some sites for which the request hangs.
WWW::Mechanize::Firefox uses Mozrepl to connect with Firefox browser, so you don't need to declare timeout parameter, because Firefox will wait for page load.
If you want to check if site is really fully loaded, you should check that the element that you want (e.g. div) is present:
while (!$mech->is_visible( xpath => '//div[#class="myDivClassAtHtml"]')) {
sleep 1;
};
# do something with your page
I have a few problems authenticating to a website via Perl.
I can login to the website without any problems using a browser, if I do, it shows me a little prompt, where I have to supply my credentials.
Easy going!
But now I wanted to connect to the site by using the script and I am always running into a 401 unauthorized error.
The server I am trying to connect to runs IIS7 and the WWW-Authenticate header told me, the realm would be something like 'negotiate, NTLM' (I tried all three possibilities, one of the words or both, nothing happened).
So far I really don't know how to figure out what information is needed, my last try was to get the form, where I have to supply the credentials and then use the submit_form function, but to get the forms, I have to establish the connection.
I am really confused because I tried most of the things I found on google, but I don't know what is going on, especially from the technical view, which could give me a hint.
Well, I would really appreciate it, if someone could help me and I thank you in advance for doing it.
#!/usr/bin/perl
use strict;
use warnings;
use LWP::UserAgent;
use WWW::Mechanize;
use Authen::NTLM;
my $url = "some.url";
my $usr = 'bernd';
my $pwd = 'bernd';
my $mech = WWW::Mechanize->new(ssl_opts => {verify_hostname => 0});
$mech -> credentials($url.":443","NTLM",$usr,$pwd);
$mech->get("https://".$url); #added the https, because $url doesn't have it
print $mech->forms;
PS:
The bypassing of ssl verification was due to a certificate problem I had at the beginning.
I speculate that in your use of the 4 parameter form of the credentials method, the netloc and user are wrong.
When the URL is https://www.example.com/foo/bar?quux;baz=42#fnord, the netloc is www.example.com:443.
The user is a string of the NTLM domain, a single backslash, and the NTLM username, e.g. internal\bernd. Mind that if you hardcode that value as a string literal as you did in the example, you need to escape the backslash with a backslash.
Check for these possible errors.
Alright, the problem is not a script or Perl problem as it seems, some of the developers of the site did a really shitty job with the permissions, so you have to be authorized to view everything else, in order to view the single site and the service account I were told to use, does not have this permissions.
Very sad, but thanks for the help so far!
Based on this question I asked earlier on setting up cookies in Perl, I successfully got an answer and way to do this but am now faced with a new interesting challenge.
In Perl CGI scripts, it demands you setup a cookie on the header statement which ought to be the first statement in your script. To clarify, you need to have a CGI->header() statement at the top of your script for the CGI script to work!
My scenario:
I have a login screen with a user name and password, upon successful login I need to setup a cookie with the current user name, then redirect the user to another form that checks if the cookie is set before allowing any transactions.
Please note if I set the cookie after the CGI->header() statement, it never get set, and if I set it at the top of my script, it has a bogus value for obvious reasons (user has not logged in)
How do I achieve this? does the CGI->header() statement need to be at the top of my script always?
Gath
Your assertion is wrong: print CGI->header can appear anywhere in your script. However, headers must be output once and only once and before any other output is emitted.
For your purposes, I would recommend using CGI::Application along with CGI::Session (via CGI::Application::Plugin::Session).
You're gonna end up writing yourself some code that can be hijacked if you keep heading in that direction, given your current level of knowledge ("just enough to be dangerous").
I'd suggest reading my article on how to "brand" a browser using cookies. It's an oldie-but-a-goodie.
For the impatient, the trick is to use a cookie only to distinguish one browser from another, and keep everything important as server-side state.
I'm writing a Catalyst application that's required to have a fairly short session expiration (15 minutes). I'm using the standard Catalyst framework authentication modules, so the user data is stored in the session -- i.e., when your session expires, you get logged out.
Many of the uses of this application will require >15 minutes to complete, so users will frequently submit a form only to find their session state is gone and they're required to log back in.
If this happens I want to preserve the original form submission, and if they log in successfully, continue on and carry out the form submission just as if the session had not expired.
I've got the authentication stuff being handled by an auto() method in the controller -- if you request an action that requires authentication and you're not currently logged in, you get redirected to the login() method, which displays the login form and then processes it once it's submitted. It seems like it should be possible to store the request and any form parameters when the auto method redirects to the login(), and then pull them back out if the login() succeeds -- but I'm not entirely sure of the best way to grab or store this information in a generic/standard/reusable way. (I'm figuring on storing it in the session and then deleting it once it's pulled back out; if that seems like a bad idea, that's something else to address.)
Is there a standard "best practices" or cookbook way to do this?
(One wrinkle: these forms are being submitted via POST.)
I can't help thinking that there's a fundamental flaw in mandating a 15 minute timeout in an app that routinely requires >15 minutes between actions.
Be that as it may, I would look at over-riding the Catalyst::Plugin::Session->delete_session method so that any contents of $c->request->body_parameters are serialised and saved (presumably to the database) for later recovery. You would probably want some rudimentary check of the POST arguments to ensure they're what you're expecting.
Similarly, create_session needs to take responsibility for pulling this data back out of the database and making it available to the original form action.
It does seem like a messy situation, and I'm inclined to repeat my first sentence...
UPDATE:
Whether you use delete_session or auto, the paradoxical issue remains: you can't store this info in the session because the time-out event will destroy the session. You've got to store it somewhere more permanent so it survives the session re-initialization. Catalyst::Plugin::Session itself is using Storable, and you should be able to with something along these lines:
use Storable;
...
sub auto {
...
unless (...) { #ie don't do this if processing the login action
my $formitems = freeze $c->request->body_parameters;
my $freezer = $rs->update_or_create(
{user => $c->user, formitems => $formitems} );
# Don't quote me on the exact syntax, I don't use DBIx::Class
}
...
my $formitems = $c->request->body_parameters
|| thaw $rs->find({$user => $c->user})->formitems
|| {} ;
# use formitems instead of $c->request->body_parameters from here on in
The underlying table probably has (user CHAR(x), formitems TEXT) or similar. Perhaps a timestamp so that nothing too stale gets recovered. You might also want to store the action you were processing, to be sure the retrieved form items belong to the right form. You know the issues for your app better than me.
I would store the form data as some sort of per user data in the model.
Catalyst::Plugin::Session::PerUser is one way of doing that (albeit somewhat hackishly). I would reccomend using the session plugin only for authentication and storing all the state info in the model that stores your user data instead.
And I totally agree with RET's opinion that the 15 minute limit seems really counter productive in this context.
I came across this whilst searching CPAN for something entirely unrelated.
Catalyst::Plugin::Wizard purports to do exactly what you need. The documentation suggests it can redirect to a login page whilst retaining the state of the previous action.
NB: I haven't used it, so can't vouch for its effectiveness.
In the end, we ended up grabbing the pending request (URL+params) in the auto(), serializing and encrypting it, and passing it via a hidden form element on the login page. If we got a login request with the hidden element populated, we decrypted and deserialized it and then redirected appropriately (making sure to pass through the standard "can this user do this thing" code paths).
You could always have some javascript on the client that keeps the session from expiring by making a small request every few minutes.
Or you could have AJAX check for an active session before posting the form and presenting the user with a new login box at that time if needed.