I am trying to scrape a website using WWW::Mechanize module. I have configured the mechanize agent with a proxy URL and setting the proxy credentials using credentials method.
Code snippet :
my $url = 'https://abcde.com';
my $proxy_username = 'abc';
my $proxy_password = 'xyz';
my $proxy_url = 'http://xx.xxx.xxx.xxx:13228';
my $mechanize_agent = new WWW::Mechanize('cookie_jar' => {}, 'noproxy' => 1, 'ssl_opts' => { 'verify_hostname' => 0 });
$mechanize_agent->credentials( $proxy_username, $proxy_password );
$mechanize_agent->proxy(['http', 'https'], $proxy_url);
$mechanize_agent->get($url) or die 'Error in get request of $url: $#';
When URL is a plain HTTP, the script fetches and gives back the result. But when I try to hit HTTPS url I get the error
establishing SSL tunnel failed: 407 Proxy Authentication Required
I credentials are valid and I can view the website using proxy URL in Mozilla browser. Also i need t avoid using call to env_proxy() function since the proxy URL is dynamic. How can I get let my script fetch HTTPS request?
All suggestions are welcome!
thanks in advance.
Related
Server application is using following code to get the request from client to process. Now I got a requirement to make this call secure using HTTPS (2-way-tls). I need to convert this call to secure call in Perl and implement client authentication for trusted client. I am not sure how can I do that and how to provide certificates.
my $daemonname = "SOAP::Transport::HTTP::Daemon";
$daemon = $daemonname
->new(Scheme => 'http', LocalPort => 8080, Listen => 128)
->dispatch_to('MyCode')
->options({compress_threshold => 10000});
$daemon->handle;
Here is my issue : I have got an API using mojolicious, an external script perl and a JS file, and I would like to connect them this way: the external script launch a random POST request, if it is a success it have to send the message "Success" throught a websocket. If an error occures, It will have to send "Error". The websocket on the API will just relay the message for the JS which will use it.
How I imagine the code to be :
Inside the Mojolicious launcher script:
websocket '/foo' => sub {
$self->on(message => sub {
my ($self, $msg) = #_;
$self->send($msg);
});
};
when get a message send it
Inside the JS file:
var ws = new WebSocket('ws://api/foo');
ws.onmessage = function(msg){
if(msg == "Error") {console.log("got an error")};
else if(msg == "Success") {console.log("got a success")};
};
So, how can I connect my external script to the websocket, and be able to send "Error" or "Success"? (This external script has nothing to do with the web server, it s somewhere else, doing something else).
I understand that you want a plain perl-script who connects to the Mojolicious webserver where you have a websocket endpoint.
The external perl script should connect to the websocket server and send some messages. The websocket server can then redistribute those messages to other clients.
Check my github where you can find the above screnario.
Perl Mojolicious websockets
Using node-red on Bluemix, I was able to get my application running on both http://example.com and https://example.com. However, I want my application to be secured and can be accessed only with https. I can not find the way to redirect http to https on node-red running on Bluemix. I found only the way to do it on node.js/express and other web servers.
I also tried to do the redirect on my application by detecting msg.req.headers.$wssc and use http request node to redirect to https url but it did not work (still got http response).
Configure redirect on DNS is also not an option because AFAIK, you can not redirect http://example.com to https://example.com but you can redirect http://example.com to https://www.example.com.
Please suggest.
You should be able to do this by using the msg.req.headers.x-forwarded-proto value then sending a response with a 301 status e.g.
if (msg.req.headers['x-forwarded-proto'] == 'http') {
msg.statusCode = 301;
msg.headers = {
location: 'https://' + msg.req.get('host') + msg.req.originalUrl
}
msg.payload = "";
return [null, msg]
} else {
return [msg, null]
}
If you put that in a function node with 2 outputs it will send the redirect out the second output else send it out the first.
You could also do this by adding your own httpNodeMiddleware function in settings.js like this:
....
httpNodeMiddleware: function(req, res, next){
if (req.headers['x-forwarded-proto'] == 'http') {
res.redirect(301, 'https://' + req.get('host') + req.originalUrl);
} else {
next();
}
},
...
This method means you don't need any of the redirect logic in your flow
Side Note:
You probably need to test for msg.req.headers['$wssc'] rather than msg.req.headers.$wssc.
I do not use a proxy.
Here is my code:
use JIRA::Client::Automated;
my $jira = JIRA::Client::Automated->new(https://myserver.com, "user", "password");
And the error response is:
Unable to GET /jira/rest/api/latest/issue/DCS-51191: 500 Can't connect
to myserver.com:443 Can't connect to myserver.com:443
Bad file descriptor at
C:/Users/Fred/applis_portables/Strawberry_Perl/perl/vendor/lib/LWP/Protocol/http.pm
line 47.
at createPage2.pl line 16.
Thank you for your help.
It seems that there is a self signed certificate on JIRA server. To bypass, I added following code:
my $jira_ua = $jira->ua();
$jira_ua->ssl_opts( verify_hostname => 0 );
The error doesn't look like a JIRA::Client::Automated error. It's generated by LWP::UserAgent and usually means exactly what is shown.
Do you have a self signed certificate on your server?
Did you try to open that URL in in your browser? https://myserver.com:443 (exactly as you provide it to the module).
Try using curl from your webserver:
curl -vvv https://myserver.com/jira/rest/api/latest/issue/DCS-51191
Maybe it's just a missing www. prefix in your server URL?
I have a perl script which will post HTTP request to specified server URL (Say: http://some-ip/here_action_url ). My problem is, Sometimes I am getting the below error.
Error:
500 SSL read timeout.
Sample Code:
my $ua = new LWP::UserAgent;
$ua->timeout(30);
my $res = $ua->post( $url, { 'data' => $my_data } );
if(! $res->is_success ) {
# Error Logging
print $res->status_line."\n";
}
else {
$response_content = $res->content;
}
I read about the error. Most of the documents are saying that it is because of the response delay in server side.
I just want to confirm, whether this error is coming because of server response delay? (or) Can be the problem with my perl script?
If you get a result some of the time, and the error at other times, then it looks like your code is working.
If you alway get the 500 error, it indicates a connection problem. Would need to know more about the service you are trying to connect to, does it require certificates or other authentication (which may be needed for secure socket layer connection)