Using Redis.pm pipeline in perl - perl

My attempt to Redis pipeline in perl, using Redis.pm, Is this correct approach? Snipped of code below:
...
my $redis = Redis->new(server => '127.0.0.1:6379', reconnect => 60);
foreach my $key (keys %hval) {
my $ok = $redis->zadd($key, $hval{ $key }, &process);
}
sub process {
my ($reply, $error) = #_;
my $cr = sub {
my ($r, $e) = #_;
if ($e) {
warn Dumper('Redis pipelining crapped out', $e);
{
}
}
Have you tried this before? I looked around but could not found any suitable example, Please let me know. I am using all required module and this code is for here only. Actual code is much complex? Thanks in advance.

Here is an example:
use Redis;
my $redis = Redis->new(server => '127.0.0.1:6379', reconnect => 60);
my %hval = ( 'foo', 1, 'bar', 2, 'foobar', 3 );
foreach my $key (keys %hval) {
my $ok = $redis->zadd("myzset", $hval{ $key }, $key, sub {
my ($reply, $error) = #_;
print "Returned $reply with error = [$error]\n" ;
});
}
print "Waiting replies ...\n";
$redis->wait_all_responses;
Please note that:
a wait_all_responses clause is required to put a synchronization point with the server.
zadd requires 3 parameters (zset name, score, key) in that order

Related

Mojo::UserAgent non-blocking vs blocking performance

I have the following code:
my $ua = Mojo::UserAgent->new ();
my #ids = qw(id1 id2 id3);
foreach (#ids) {
my $input = $_;
my $res = $ua->get('http://my_site/rest/id/'.$input.'.json' => sub {
my ($ua, $res) = #_;
print "$input =>" . $res->result->json('/net/id/desc'), "\n";
});
}
Mojo::IOLoop->start unless Mojo::IOLoop->is_running;
Why when I run the above code (non-blocking) does it take about 6 seconds while when running the code as blocking, i.e. inside the loop something like:
my $res = $ua->get('http://my_site/rest/id/'.$input.'.json');
print "$input =>" . $res->result->json('/net/id/desc'), "\n";
without the latest line it takes about 1 second?
Why is the blocking code faster than the non-blocking code?
The first thing to check when things happened. I couldn't get the same delay. Remember to try each way several times to spot outliers where there's a network hiccup. Note that the second argument to the non-blocking sub is a transaction object, normally written as $tx, where the response object is normally written res:
use Mojo::Util qw(steady_time);
say "Begin: " . steady_time();
END { say "End: " . steady_time() }
my $ua = Mojo::UserAgent->new ();
my #ids = qw(id1 id2 id3);
foreach (#ids) {
my $input = $_;
my $res = $ua->get(
$url =>
sub {
my ($ua, $tx) = #_;
print "Fetched\n";
}
);
}
One possibility is that keep-alive is holding an open connection. What happens if you turn that off?
my $res = $ua->get(
$url =>
{ Connection => 'close' }
sub {
my ($ua, $tx) = #_;
print "Fetched\n";
}
);
Here's a version that uses promises, which you'll want to get used to as more Mojo stuff moves to it:
use feature qw(signatures);
no warnings qw(experimental::signatures);
use Mojo::Promise;
use Mojo::Util qw(steady_time);
use Mojo::UserAgent;
my $ua = Mojo::UserAgent->new;
say "Begin: " . steady_time();
END { say "End: " . steady_time() }
my #ids = qw(id1 id2 id3);
my #gets = map {
$ua->get_p( 'http://www.perl.com' )->then(
sub ( $tx ) { say "Fetched: " . steady_time() },
sub { print "Error: #_" }
);
} #ids;
Mojo::Promise->all( #gets )->wait;

Perl: Issue with blessed object

I am creating a bot that connects to a Matrix server. For that I use Net::Async::Matrix.
The code:
#!/usr/bin/perl
use strict;
use warnings;
use Net::Async::Matrix;
use Net::Async::Matrix::Utils qw ( parse_formatted_message );
use IO::Async::Loop;
use Data::Dumper;
my $loop = IO::Async::Loop->new;
my $matrix = Net::Async::Matrix->new(
server => 'matrix.server.net',
on_error => sub {
my ( undef, $message ) = #_;
warn "error: $message\n";
},
);
$loop->add( $matrix );
$matrix->login(
user_id => '#bot:matrix.server.net',
password => 'password',
)->get;
my $room = $matrix->join_room( '#Lobby:matrix.server.net' )->get;
$room->configure(
on_message => sub {
my ( undef, $member, $content, $event ) = #_;
my $msg = parse_formatted_message( $content );
my $sendername = $member->displayname;
print Dumper $sendername;
&sendmsg("$sendername said: $msg");
},
);
my $stream = $matrix->start;
sub sendmsg {
my $input = shift;
if ($input) {
$room->send_message(
type => "m.text",
body => $input,
),
}
}
$loop->run;
Basically, I want the bot to echo what was said.
I get following output:
$VAR1 = 'm1ndgames'; Longpoll failed - encountered object 'm1ndgames
said: test', but neither allow_blessed, convert_blessed nor
allow_tags settings are enabled (or TO_JSON/FREEZE method missing) at
/usr/local/share/perl/5.24.1/Net/Async/Matrix.pm line 292.
and I don't understand it. When I enter a string like test into the body, it gets sent to the room.
parse_formatted_message returns a String::Tagged object. This class overloads concatenation so that "$sendername said: $msg" also returns a String::Tagged object. This object is passed to sendmsg which tries to serialize it into JSON, but it refuses to serialize objects.
Fix: Replace
my $msg = parse_formatted_message( $content );
with
my $msg = parse_formatted_message( $content )->str;
I'd guess that this is a quoting error. If you look at Net::Async::Matrix::Room:
sub send_message
{
my $self = shift;
my %args = ( #_ == 1 ) ? ( type => "m.text", body => shift ) : #_;
my $type = $args{msgtype} = delete $args{type} or
croak "Require a 'type' field";
$MSG_REQUIRED_FIELDS{$type} or
croak "Unrecognised message type '$type'";
foreach (#{ $MSG_REQUIRED_FIELDS{$type} } ) {
$args{$_} or croak "'$type' messages require a '$_' field";
}
if( defined( my $txn_id = $args{txn_id} ) ) {
$self->_do_PUT_json( "/send/m.room.message/$txn_id", \%args )
->then_done()
}
else {
$self->_do_POST_json( "/send/m.room.message", \%args )
->then_done()
}
}
The type you sent is handled by this sub, and then the actual message gets handed off to _do_POST_json in Net::Async::Matrix.
But you've sent a string containing a :.
So I think what's happening is it's encoding like this:
use JSON;
use Data::Dumper;
my $json = encode_json ( {body => "m1ndgames: said test"});
print Dumper $json;
But the response that's coming back, at line 292 which is:
if( length $content and $content ne q("") ) {
eval {
$content = decode_json( $content );
1;
} or
return Future->fail( "Unable to parse JSON response $content" );
return Future->done( $content, $response );
}
So I think is what is happening is the remote server is sending you a broken error code, and the module isn't handling it properly - it's expecting JSON but it isn't actually getting it.
My best guess would be - try dropping the : out of your message, because I would guess there's some bad quoting happening. But without seeing the code on the server side, I can't quite tell.

perl - Resolve hostname and AP status from SNMP

The idea is that the first get_table gets the AP status off a WLAN controller, then it uses get_request to get the AP's hostname as it's printing out the status table. The problem I'm having is the $ap_name comes back as an array, when I just want the single value.
my ($session, $error) = Net::SNMP->session(
-hostname => "$hostaddr",
-community => "$community",
-timeout => "30",
-version => "2c",
-port => "161");
if (!defined($session)) {
printf("ERROR: %s.\n", $error);
exit 1;
}
my $ap_stat = $session->get_table( -baseoid => $ap_stat_oid );
my $ap_name = $session->get_table( -baseoid => $ap_name_oid);
if (! defined $ap_stat || ! defined $ap_name) {
die "Failed to get OID '$ap_stat_oid': " . $session->error;
$session->close();
}
my #ap_name_array;
foreach my $ap_name_key (keys %$ap_name) {
push(#ap_name_array,$ap_name->{$ap_name_key});
}
my #ap_stat_array;
foreach my $ap_stat_key (keys %$ap_stat) {
push(#ap_stat_array,$ap_stat->{$ap_stat_key});
}
Edit: I changed it up a bit but still can't figure out what's next. I think I want to store the print output's into arrays and then join them and print for the joined array but I'm not sure how.
Edit: Here's my desired output:
AP-01 = 1
AP-02 = 1
AP-03 = 2
AP-04 = 1
etc..
More edits: I got the values into an arrays, now I'm just trying to get the output right.
Figured it out using use List::MoreUtils qw(pairwise); from here
if (! defined $ap_stat || ! defined $ap_name) {
die "Failed to get OID '$ap_stat_oid': " . $session->error;
$session->close();
}
my #ap_name_array;
foreach my $ap_name_key (keys %$ap_name) {
push(#ap_name_array,$ap_name->{$ap_name_key});
}
my #ap_stat_array;
foreach my $ap_stat_key (keys %$ap_stat) {
push(#ap_stat_array,$ap_stat->{$ap_stat_key});
}
print pairwise { "$a = $b\n" } #ap_name_array, #ap_stat_array;
Beat you to it #ThisSuitIsBlackNot, thanks anyways!

Perl LDAP search - over 1500 member in a group

I want to search with an Perl script and ldap connection all members of a group with over 10.000 member.
I can only find results, if i set $first=0 and $last=1499 and than i get only the first 1500 member of the group.
If i use other parameter for $first and $last, then i got no results.
"$ldapsearchresult = $ldapconnect->search (
Sizelimit => 0,
base => 'any_base',
filter => '(objectClass=*)',
attr => ['member;range=$first-$last'],
);"
Thanks for your help!
You need to search the attribute range as a subtype again and again until the last return '*'.
Here is the code I am using, it is also use paged search in AD.
use Net::LDAP;
use Net::LDAP qw(LDAP_CONTROL_PAGED);
use Net::LDAP::Util qw(ldap_error_name canonical_dn ldap_explode_dn ldap_error_text);
use Net::LDAP::Control::Paged;
my $page_page = Net::LDAP::Control::Paged->new( 'size' => $input{'page'} );
my $finished_search = 0;
my $page_cookie;
my $result;
my #page_search_args = (
'base' => $input{"base"},
'scope' => $input{'scope'},
'filter' => $input{'filter'},
'attrs' => $input{'attrs'},
'control' => [ $page_page ],
'deref' => 'never',
'raw' => qr!^DO_NOT_MATCH!,
);
while (!$finished_search) {
my $msg = $ldap->search(#page_search_args);
if ($msg->is_error()) {
die "ERROR: ",$msg->error,"\n";
last;
} else {
my ($response) = $msg->control(LDAP_CONTROL_PAGED);
$page_cookie = $response->cookie();
$finished_search = 1 if !$page_cookie;
$page_page->cookie($page_cookie);
while (my $entry = $msg->pop_entry()){
$ldap_searches++;
print_all_attributes($entry);
}
}
}
if ($page_cookie) {
$page_page->cookie($page_cookie);
$page_page->size(0);
$ldap->search(#page_search_args);
}
sub add_result {
my $dn = shift;
my $attr = shift;
my $data = shift;
my $res = shift;
$attr =~ s!(;range\=\d+\-\d+)!!i;
#print "removed $1 from $attr" if $1;
foreach my $subtype (keys %{$data}){
$attr = $attr.$subtype if $subtype ne '';
$attr =~ s!(;range\=\d+\-\d+)!!i;
if (defined $$res->{$dn}->{$attr}){
push(#{$$res->{$dn}->{$attr}},#{$data->{$subtype}});
} else {
push(#{$$res->{$dn}->{$attr}},#{$data->{$subtype}});
}
}
return $res;
}
sub print_all_attributes {
my $entry = shift;
foreach my $attr ($entry->attributes()) {
if ($attr =~ /;range=/) {
my $last = 0;my $first = 0;
### $var will look like this --> "member;range=0-1499"
(my $pure_attr,my $range) = split /;/, $attr,2;
(my $junk,$range) = split /=/, $range,2;
($first,$last) = split /-/, $range,2;
$i++;
add_result($entry->dn(),$pure_attr,$entry->get_value($attr,alloptions => 1, asref => 1),\$result) if $last eq '*' or $last >= $parms{'attribute_page'};
### if $last eq "*", indicates this is the last range increment, and
### we do not need to perform another supplemental search
if ($last ne "*") {
my $range_diff = ($last - $first) + 1;
my $increment = $last + $range_diff;
$last = $last + 1;
$attr = "$pure_attr;range=$last-$increment";
$parms{'attrs'} = [$attr];
search_nonpaged(%parms);
}
} else {### if $attr matches range pattern
add_result($entry->dn(),$attr,$entry->get_value($attr,alloptions => 1, asref => 1),\$result);
}
}
return 1;
}
sub search_nonpaged{
my %input = #_;
my #page_search_args = (
'base' => $input{"base"},
'scope' => $input{'scope'},
'filter' => $input{'filter'},
'attrs' => $input{'attrs'},
'deref' => 'never',
'raw' => qr!^DO_NOT_MATCH!,
);
my $msg = $ldap->search(#page_search_args);
if ($msg->is_error()) {
die "ERROR: ",$msg->error,"\n";
}
while (my $entry = $msg->pop_entry()){
$ldap_searches++;
print_all_attributes($entry);
}
}
You maybe able to simplify the program by searching for:
memberOf=CN=GroupOne,OU=Security Groups,OU=Groups,DC=YOURDOMAIN,DC=NET
You will still need to use the paged results control but will not need the range control.
Microsoft Active Directory uses the MaxValRange to control the number of values that are returned in the retrieval of multi-valued attributes of an entry.
By using the filter above, you can avoid the MaxValRange settings.
BY THE WAY: if you want to obtain nested members also, try:
(memberOf:1.2.840.113556.1.4.1941:=CN=GroupOne,OU=Security Groups,OU=Groups,DC=YOURDOMAIN,DC=NET)
This filter uses the LDAP_MATCHING_RULE_IN_CHAIN extensible match.
-jim
I found an easier way to search all member of a AD group:
http://permalink.gmane.org/gmane.comp.lang.perl.modules.ldap/246
use Net::LDAP; use Net::LDAP::Util;
# Connect to AD make sure to specify version 3
$ldap = new Net::LDAP("myGC.yy.xx.com",
port => 3268,
debug => 0,
version => 3
) or die "New failed:$ <at> ";
# Do an anonymous bind. You MAY have to do an authenticated bind in your configuration
$result=$ldap->ldapbind() || die "Bind Failed:$ <at> ";
# Some error trapping
$err=$result->code;
if ($err){
$errname=Net::LDAP::Util::ldap_error_name($err);
$errtxt=Net::LDAP::Util::ldap_error_text($err);
if ($errtxt){
print "($err) $errtxt\n";
}
else
{
if ($errname){
print "($err) $errname\n";
}
else
{
print "ERR: $err\n";
}
}
exit;
}
# The combination of the search base and filter determine which object that you
# retrieve
# set search filter to groups of objects. This is what you want to enumerate NT groups.
$filter="(objectClass=group)";
# Set the search base to the DN of the object that you want to retrieve. BTW, using this method on
# groups with less than 1000 members works as well.
$base='CN=mygroup,DCyyy,DC=xxx,DC=com';
# Set the initial attribute indexes and name
$found=1; $startr=0; $endr=-1; $startattr="member";
while($found){
# Create the attribute range specification
$startr=$endr+1;
$endr=$startr+999;
$attr="$startattr;range=$startr-$endr";
$saveattr=$attr;
<at> attr=("$attr");
# Perform the search
$result=$mesg = $ldap->search(base => "$base",filter => $filter,
attrs => [ <at> attr],
scope => "sub") or die "search died";
# Some error trapping
$err=$result->code;
if ($err){
if (!($err == 1)){
$errname=Net::LDAP::Util::ldap_error_name($err);
$errtxt=Net::LDAP::Util::ldap_error_text($err);
if ($errtxt){
print "($err) $errtxt\n";
}
else
{
if ($errname){
print "($err) $errname\n";
}
else
{
print "ERR: $err\n";
}
}
}
else
{
print "COUNT=$cnt\n";
}
exit;
}
$found=0;
# OK, get the attribute range...so we can update the value of the attribute
# on the next pass
foreach $entry ($mesg->all_entries) {
<at> attr=$entry->attributes;
foreach( <at> attr){
$curattr=$_;
}
}
# Print out the current chunk of members
foreach $entry ($mesg->all_entries) {
$ar=$entry->get("$curattr");
foreach( <at> $ar){
$cnt++;
print "$_\n";
}
$found=1;
if (! <at> $ar[0]){
$found=0;
}
}
# Check to see if we got the last chunk. If we did print toe total and set
# the found flag so we don't search for anymore members
if ($curattr=~/\;range=/){
if ($curattr=~/\-\*/){
print "LASTCOUNT:$cnt\n";
$found=0;
}
}
}

DNS Resolver result from multiple nameservers

I am passing two name servers to the Net::DNS::Resolver constructor but I am getting only one result back.
How should I change the code to receive result from all the name servers?
sub resolve_dns()
{
my $dns = $_[0];
my $res = Net::DNS::Resolver->new(
nameservers => [qw(24.116.197.232 114.130.11.67 )],
recurse => 0,
debug => 1,
tcp_timeout => 3
);
my $query = $res->search($dns);
if ($query) {
foreach my $rr ($query->answer) {
next unless $rr->type eq "A";
print $rr->address, "\n";
}
} else {
warn "query failed: ", $res->errorstring, "\n";
}
}
I presume the DNS servers after the first are there for fallback purposes and only a single reply will ever be returned.
The best way seems to be to manipulate the Net::DNS::Resolver server list and explicitly make a request to each of them.
This example code demonstrates the principle
sub resolve_dns {
my $address = shift;
my $res = Net::DNS::Resolver->new
recurse => 0,
debug => 1,
tcp_timeout => 3,
);
for my $ns (qw( 24.116.197.232 114.130.11.67 )) {
$res->nameservers($ns);
my $reply = $res->send($address);
if ($reply) {
my #type_a = grep $_->type eq 'A', $reply->answer;
print $_->address, "\n" for #type_a;
}
else {
warn sprintf "Query to %s failed: %s\n", $ns, $res->errorstring;
}
}
}