Capture both resolved and rejected in Perl's Mojo::Promises - perl

I'm playing with Mojo::UserAgent and Mojo::Promise to run non-blocking calls to 3 services A, B, and C. The problem is it works fine when all the services connect/resolve, but if one of those, say, service C is unable to connect, the whole thing fail. Is there a way to capture all services (connect and Not-connect)? Any insight is greatly appreciated. Thanks!
my #urls = (
'https://hostA/serviceA', # ServcieA connects and returns some text
'https://hostB/serviceB', # ServiceB connects and returns some text
'https://hostC/serviceC', # ServiceC refuses to connect
);
my $ua = Mojo::UserAgent->new;
my #promises = map { $ua->get_p($_) } #urls;
Mojo::Promise->all( #promises )->then(
sub {
for my $tx (map { $_->[0] } #_) {
print "Service result: $tx->res->text";
}#end for
}#end sub
)->catch(
sub {
for my $err (map { $_->[0] } #_) {
print "ERROR: $err";
}#end for
}#end sub
)->wait;

I think I'd make it simpler. Give each Promise its own handlers, then simply put all of those together. Inside the code refs in then, do whatever you need to do:
#!perl
use v5.10;
use Mojo::Promise;
use Mojo::UserAgent;
my #urls = qw(
https://www.yahoo.com
https://hostB/serviceB
https://hostC/serviceC
);
my $ua = Mojo::UserAgent->new;
my #promises = map {
my $url = $_;
$ua->get_p( $url )->then(
sub { say "$url connected" },
sub { say "$url failed" },
);
} #urls;
Mojo::Promise->all( #promises )->wait;
This outputs which connected or failed, although I could have also marked their status in some data structure or database:
https://hostB/serviceB failed
https://hostC/serviceC failed
https://www.yahoo.com connected
I have many other Promises examples in Mojo Web Clients.

Related

Perl WebService increase max connections using HTTP::Server::Simple::CGI

I'm running a little Perl Webservice, based on the example i found on this page : https://www.perlmonks.org/?node_id=1078567 (first example)
However, when a lot of clients are calling it at once, it looks like the requests are suddenly crashing, and there's a lot of TIME_WAIT tcp connections left on the server running the webservice, as if the webservice was not able to handle that many connections at once.
is there a parameter in that module or other that i could use to extend this ?
or a way to put some kind of queue for the incoming requests ?
some parts of my code, to help :
{
package TACWebService;
use HTTP::Server::Simple::CGI;
use base qw(HTTP::Server::Simple::CGI);
use Cwd 'abs_path';
use POSIX;
use DBI;
use warnings;
.........
my %dispatch = (
'/insertunix' => \&resp_insertunix,
'/insertwin' => \&resp_insertwin,
'/getpwdate' => \&resp_getpwdate,
);
# ---------------------------------------------------------------------
# Requests Handling
# ---------------------------------------------------------------------
sub handle_request {
my $self = shift;
my $cgi = shift;
my $path = $cgi->path_info();
my $handler = $dispatch{$path};
if (ref($handler) eq "CODE") {
print "HTTP/1.0 200 OK\r\n";
$handler->($cgi);
} else {
print "HTTP/1.0 404 Not found\r\n";
print $cgi->header,
$cgi->start_html('Not found'),
$cgi->h1('Not found'),
$cgi->end_html;
}
}
sub resp_insertwin {
my $cgi = shift; # CGI.pm object
return if !ref $cgi;
....
} else {
print $cgi->header("text/plain"), "INSERT";
}
.....
# ---------------------------------------------------------------------
# WebService Start in background
# ---------------------------------------------------------------------
my $pid = TACWebService->new($TACWebService::conf{tac_ws_port})->background();
print "Use 'kill $pid' to stop TAC WebService.\n";
the clients themselves are using use LWP::UserAgent like this :
my $ua = LWP::UserAgent->new();
$ua->timeout($timeout);
my $response = $ua->post($TAC_Url,
[
'args' => $here,
]
if (!$response->is_success) {
print "Timeout while connecting to $TAC_Url\n";
} else {
my $content = $response->as_string();
print $content if (grep(/INSERT_/,$content));
}
to describe the exact issue would be complicated. In short : the clients are Unix servers sending their user database (user accounts). and when lots of clients are sending this user db at once, i can see the webservice receiving half of the data, and answering "timeout" after a couple of accounts (probably because it's overloaded in some way)
thanks again
The problem is, that the client waits to long for the server to respond. To solve this you have to start the server multiple times. The easiest Solution to this is to add
sub net_server { 'Net::Server::PreFork' }
to your package TACWebService and the HTTP::Server::Simple::CGI will do the rest of the magick.
Or you can use HTTP::Server::Simple::CGI::PreFork instead. See https://metacpan.org/pod/HTTP::Server::Simple::CGI::PreFork

Mojo::UserAgent non-blocking vs blocking performance

I have the following code:
my $ua = Mojo::UserAgent->new ();
my #ids = qw(id1 id2 id3);
foreach (#ids) {
my $input = $_;
my $res = $ua->get('http://my_site/rest/id/'.$input.'.json' => sub {
my ($ua, $res) = #_;
print "$input =>" . $res->result->json('/net/id/desc'), "\n";
});
}
Mojo::IOLoop->start unless Mojo::IOLoop->is_running;
Why when I run the above code (non-blocking) does it take about 6 seconds while when running the code as blocking, i.e. inside the loop something like:
my $res = $ua->get('http://my_site/rest/id/'.$input.'.json');
print "$input =>" . $res->result->json('/net/id/desc'), "\n";
without the latest line it takes about 1 second?
Why is the blocking code faster than the non-blocking code?
The first thing to check when things happened. I couldn't get the same delay. Remember to try each way several times to spot outliers where there's a network hiccup. Note that the second argument to the non-blocking sub is a transaction object, normally written as $tx, where the response object is normally written res:
use Mojo::Util qw(steady_time);
say "Begin: " . steady_time();
END { say "End: " . steady_time() }
my $ua = Mojo::UserAgent->new ();
my #ids = qw(id1 id2 id3);
foreach (#ids) {
my $input = $_;
my $res = $ua->get(
$url =>
sub {
my ($ua, $tx) = #_;
print "Fetched\n";
}
);
}
One possibility is that keep-alive is holding an open connection. What happens if you turn that off?
my $res = $ua->get(
$url =>
{ Connection => 'close' }
sub {
my ($ua, $tx) = #_;
print "Fetched\n";
}
);
Here's a version that uses promises, which you'll want to get used to as more Mojo stuff moves to it:
use feature qw(signatures);
no warnings qw(experimental::signatures);
use Mojo::Promise;
use Mojo::Util qw(steady_time);
use Mojo::UserAgent;
my $ua = Mojo::UserAgent->new;
say "Begin: " . steady_time();
END { say "End: " . steady_time() }
my #ids = qw(id1 id2 id3);
my #gets = map {
$ua->get_p( 'http://www.perl.com' )->then(
sub ( $tx ) { say "Fetched: " . steady_time() },
sub { print "Error: #_" }
);
} #ids;
Mojo::Promise->all( #gets )->wait;

AnyEvent::STOMP::Client + AnyEvent::ForkManger = Intermittent Error

I'm trying to write a process that listens to ActiveMQ and based on the message, goes out and grabs data from a webservice, does some processing and then puts the process data to another webservice. (REST/JSON)
The module below works fine until one of the wonky webservices I talk to returns an error. I've tried many things to catch the error but to no avail, yet. Once the webservice error happens though I get the following message:
unhandled callback exception on event (MESSAGE,
AnyEvent::STOMP::Client=HASH(0x3ad5e48), HASH(0x3a6bbb0)
{"action":"created","data":{"id":40578737,"type":"alert","who":null},"guid":"ADCCEE0C-73A7-11E6-8084-74B346D1CA67","hostname":"myserver","pid":48632}):
$fork_manager->start() should be called within the manager process
OK, I conceptually understand that child process is trying to start another process and that fork manager is saying that is a no no. But given the module below, what is the proper way to start a new process to handle the long running processing. Or why is an child process dying causing this exception and how can I prevent this
Here's the module (stripped down)
package consumer;
use AnyEvent::ForkManager;
use AnyEvent::STOMP::Client;
use JSON;
use Data::Dumper;
use v5.18;
use Moose;
sub run {
my $self = shift;
my $pm = AnyEvent::ForkManager->new(max_workers => 20);
my $stomp = AnyEvent::STOMP::Client->new();
$stomp->connect();
$stomp->on_connected(sub {
my $stomp = shift;
$stomp->subscribe('/topic/test');
say "Connected to STOMP";
});
$pm->on_start(sub {
my ($pm,$pid,#params) = #_;
say "Starting $pid worker";
});
$pm->on_finish(sub {
my ($pm, $pid,#params) = #_;
say "Finished $pid worker";
});
$pm->on_error(sub {
say Dumper(\#_);
});
$stomp->on_message(sub {
my ($stomp, $header, $body) = #_;
my $href = decode_json $body;
$pm->start(cb => sub {
my ($pm, #params) = #_;
$self->process(#params);
},
args => [ $href->{id}, $href->{data}->{type}, $href->{data}->{who} ],
);
});
my $cv = AnyEvent->condvar;
$cv->recv;
}
sub process {
say "Processing ".Dumper(\#_);
sleep 5;
if ( int(rand(10)) < 5 ) {
die "OOPS"; # this triggers the error message above
}
say "Done Processing $_[1]";
}
1;
Heres the driver for the module above:
#!/usr/bin/env perl
use v5.18;
use lib '.';
use consumer;
my $c = consumer->new();
$c->run;
Finally a traffic generator that you can use to see this in action:
#!/usr/bin/env perl
use lib '../lib';
use lib '../../lib';
use v5.18;
use Data::Dumper;
use JSON;
use Net::STOMP::Client;
$ENV{'scot_mode'} = "testing";
my $stomp = Net::STOMP::Client->new(
host => "127.0.0.1",
port => 61613
);
$stomp->connect();
for (my $i = 1; $i < 1000000; $i++) {
my $href = {
id => $i,
type => "event",
what => "foo",
};
my $json = encode_json $href;
say "Sending ".Dumper($href);
$stomp->send(
destination => "/topic/test",
body => $json,
);
}
$stomp->disconnect();
I was able to solve this by using Try::Catch and wrapping the call to self->process with a try catch like this:
$stomp->on_message(sub {
my ($stomp, $header, $body) = #_;
my $href = decode_json $body;
$pm->start(cb => sub {
my ($pm, #params) = #_;
try {
$self->process(#params);
}
catch {
# error handling stuff
};
},
args => [ $href->{id}, $href->{data}->{type}, $href->{data}->{who} ],
);
}
);

Can Net::LDAP and Parallel::Forkmanager work together?

I need to query different LDAP servers in perl via Net::LDAP. I have something that works well. However, in an attempt to speed up things, I tried to query the different servers in parallel, using Parallel::Forkmanager - and things do not work when I do that.
I get the following types of errors:
decode error 02<=>30 0 8 at /Users/myname/perl5/perlbrew/perls/perl-5.18.2/lib/site_perl/5.18.2/Convert/ASN1/_decode.pm line 113, <> line 18.
decode error 43<=>30 0 8 at /Users/myname/perl5/perlbrew/perls/perl-5.18.2/lib/site_perl/5.18.2/Convert/ASN1/_decode.pm line 113, <> line 25.
at the line that gets the search response:
$mesg = $ldap->search( base => $dn, filter => '(CN=*)');
I am puzzled.
Telling it in other other words, why does this fail:
use Net::LDAP;
use Parallel::Forkmanager;
...; # bind LDAP servers
while (<>) {
chop;
my $dn = $_;
foreach my $ldap (#servers) {
my $pid;
$pid = $pm->start and next; # do the fork
print $dn, $pid;
my $mesg;
try {
$mesg = $ldap->search( base => $dn, filter => '(CN=*)');
} catch {
...;
}
$pm->finish;
}
}
while this:
use Net::LDAP;
...; # bind LDAP servers
while (<>) {
chop;
my $dn = $_;
foreach my $ldap (#servers) {
print $dn;
my $mesg;
try {
$mesg = $ldap->search( base => $dn, filter => '(CN=*)');
} catch {
...;
}
}
}
works perfectly?
Whilst forking doesn't have quite the same thread safety problems of threading - there are still a few places you have gotchas. I think this is what's biting you - your Net::LDAP objects are created in the parent thread, but then (effectively) cloned to each when you fork.
Which means in your code - there's a very real possibility that if you've got the list of names coming in fast enough, that a new fork will try to reuse an existing Net::LDAP connection before a previous one is finished with it.
The easy way of preventing this is call wait_all_children to ensure all your parallel LDAP queries are finished before the next one starts.
If you put your LDAP bind within the ForkManager loop, do you still have the same problem? I appreciate that's a potential overhead as you'll be binding each iteration, but if that addresses it, I'd suggest that it's because Net::LDAP is sharing the same file descriptors between forks.
The next best solution there would be to adopt a 'worker' model, where you've got a bunch of 'workers' each with their on LDAP connections to do the querying. That's easier with threading, than forking - goes a bit like this:
#!/usr/bin/perl
use strict;
use warnings;
use threads;
use Thread::Queue;
sub server_worker {
my ( $hostname, $q_ref ) = #_;
## connect LDAP to $hostname;
while ( my $dn = $$q_ref->dequeue ) {
#query $dn
}
}
my #queues;
foreach my $server (#list_of_servers) {
my $server_q = Threads::Queue->new();
push( #queues, $server_q );
threads->create( \&server_worker, $hostname, \$server_q );
}
while ( my $dn = <STDIN> ) {
chomp($dn);
foreach my $q (#queues) {
$q->enqueue($dn);
}
}
foreach my $q ( #queues ) {
$q -> end;
}
foreach my $thr ( threads->list ) {
$thr->join();
}
Doing something similar with forking should work:
#!/usr/bin/perl
use strict;
use warnings;
use IO::Pipe;
use Parallel::ForkManager;
use Net::LDAP;
my #list_of_servers = qw ( servername servenama anotherserver );
my $pm = Parallel::ForkManager -> new ( scalar #list_of_servers );
my %pipe_for;
foreach my $server ( #list_of_servers ) {
my $pipe = IO::Pipe -> new();
my $pid = pm -> start;
if ( $pid ) {
print "$$: parent\n";
$pipe -> writer -> autoflush;
$pipe_for{$server} = $pipe;
}
else {
print "$$ child connecting to $server\n";
$pipe -> reader -> autoflush;
close ( STDIN ); #because this is a child.
#Net::LDAP setup
while ( my $item = <$pipe> ) {
chomp ( $item );
#ldap_search $item;
}
}
$pm -> finish;
}
And then send stuff:
for my $number ( 1..10 ) {
foreach my $pipe ( values %pipe_for ) {
print {$pipe} "test$number\n";
}
}
$pm -> wait_all_children();
Edit: Note - autoflush is important, otherwise the IO buffers and doesn't look like it's working. I'm pretty sure closing STDIN is probably a good idea in the child, but perhaps not vitally necessary if they don't use it.

How do I make parallel HTTP requests in Perl, and receive them back in order?

Using Perl, I'm looking for a simple way to perform a handful of HTTP requests in parallel, where I get responses back in the same order I sent them after they complete, e.g.:
my ($google, $perl) = foobar(GET => 'http://www.google.com/',
GET => 'http://www.perl.org/');
Is there a module I should be looking at?
I know I can do the bookkeeping by hand, but I feel spoiled after being able to do this using jQuery's when method, and I'd love to have as simple a solution using Perl.
Thanks for your help.
use threads;
use LWP::UserAgent qw( );
my $ua = LWP::UserAgent->new();
my #threads;
for my $url ('http://www.google.com/', 'http://www.perl.org/') {
push #threads, async { $ua->get($url) };
}
for my $thread (#threads) {
my $response = $thread->join;
...
}
The best part is that the parent doesn't wait for all requests to be completed. As soon as the right request is completed, the parent will unblock to process it.
If you used Parallel::ForkManager or something else where you can't wait for a specific child, you can use the following code to order the results:
for my $id (0..$#urls) {
create_task($id, $urls[$id]);
}
my %responses;
for my $id (0..$#urls) {
if (!exists($responses{$id})) {
my ($id, $response) = wait_for_a_child_to_complete();
$responses{$id} = $response;
redo;
}
my $response = delete($responses{$id});
...
}
I am a fan of Mojo!
From the Mojo::UserAgent documentation:
use Mojo;
use Mojo::UserAgent;
# Parallel requests
my $ua = Mojo::UserAgent->new;
$ua->max_redirects(5);
my $delay = Mojo::IOLoop->delay;
for my $url ('http://www.google.com/', 'http://www.perl.org/') {
$delay->begin;
$ua->get($url => sub {
my ($ua, $tx) = #_;
$delay->end($tx->res->dom);
});
}
my #responses = $delay->wait;
print join "\n", #responses
Enjoy!
EDIT
Btw. you do not have to process the responses at the end, you may do it in between:
# ...
$ua->get($url => sub {
my ($ua, $tx) = #_;
$delay->end(1);
# process $tx->res here
});
# ...
$delay->wait;