Use Mango to search MongoDB by multiple ObjectIDs - perl

How can I find multiple MongoDB documents by their ObjectIDs using Mango in one go. I currently have a non-blocking sub which I'm using to filter out the docs that don't match in a grep statement but was wondering if it's possible to pass the ObjectIDs (maybe an array of instances of Mango::BSON::ObjectID) to the find method?
I also think it's not very efficient this way since big collections will bare a hefty price!
#items;
$self->mango->db->collection('items')->find()->all(sub {
my ($collection, $err, $items) = #_;
return $self->render_exception($err) if $err;
my #oids = $self->req->params->param('ids[]');
foreach my $item (#$items) {
push #items, $item if grep (/$item->{_id}/, #oids);
}
$self->render(json => {items => \#items});
});
This sub is inside one of my Mojolicous controllers which responds to a JSON call.
I'm using the following:
Mojolicous 4.91
Mango 0.24
MongoDB 2.4.9
Thanks in advance.
Update
I have applied Neil's logic and now it works fine.
my #oids = map { Mango::BSON::ObjectID->new($_) } ($self->req->params->param('ids[]'));
print Dumper(#oids),"\n";
$self->mango->db->collection('items')->find({ "_id" => {'$in' => \#oids} })->all(sub {
my ($collection, $err, $items) = #_;
return $self->render_exception($err) if $err;
print Dumper($items),"\n";
$self->render(json => {items => $items});
});
The dumped OIDs are as follows:
$VAR1 = bless( {
'oid' => '52faf6de10d041d196cb545b'
}, 'Mango::BSON::ObjectID' );
$VAR2 = bless( {
'oid' => '5300409310d041d196cb545d'
}, 'Mango::BSON::ObjectID' );
and the found objects will look something like this (depending on your structure):
$VAR1 = [
{
'_id' => bless( {
'oid' => '52faf6de10d041d196cb545b'
}, 'Mango::BSON::ObjectID' ),
'class' => [
'Sport'
],
'make' => '4321',
'year' => 2012
},
{
'_id' => bless( {
'oid' => '5300409310d041d196cb545d'
}, 'Mango::BSON::ObjectID' ),
'class' => [
'Classic'
],
'make' => '1234',
'year' => 2014
}
]

You seem to be looking for the $in operator. So rather than looping all the items to find the ones that match you pass in the _id values sent to your controller within the call to find():
my #oids = $self->req->params->param('ids[]');
$self->mango->db->collection('items')
->find({ "_id" => { '$in' => \#oids } })->all(sub {
my ($collection, $err, $items) = #_;
return $self->render_exception($err) if $err;
$self->render(json => {items => $items});
});
Update
Actually had some time to test this with Mango and managed to get a working case:
{ "_id" : ObjectId("5345faf8a1f97e61848485e8"), "prerequisites" : [ "a" ] }
{ "_id" : ObjectId("5345fafea1f97e61848485e9"), "prerequisites" : [ "a", "b" ] }
{ "_id" : ObjectId("5345fb02a1f97e61848485ea"), "prerequisites" : [ "a", "c" ] }
And then running with the code:
#!/usr/bin/env perl
use Modern::Perl;
use EV;
use AnyEvent;
use Data::Dumper;
use Mango;
use Mango::BSON::ObjectID;
my $mango = Mango->new();
my $cv = AE::cv;
my $col = $mango->db('test')->collection('courses');
my #oids = (
"5345fafea1f97e61848485e9",
"5345fb02a1f97e61848485ea"
);
#oids = map { Mango::BSON::ObjectID->new($_) } #oids;
$col->find({ '_id' => { '$in' => \#oids } })->all( sub {
my ( $cursor, $err, $docs ) = #_;
$cv->send( Dumper( $docs ) );
});
say $cv->recv;
So that actually will select the correct documents from the sample, and it does seem that the ObjectID values need to be passed in that way in order to match.
At least there should be enough there to debug from.

Related

Turning a set of parent-child relationships into a hierarchical structure

I have an LDAP directory that I'm querying using Net::LDAP. This gives me a set of parent-child relationships.
It's a directory of people - and includes a 'manager' DN (which is another field within the directory).
I'm having real trouble turning this manager->person set of records into a hierarchical structure.
What I've got so far is:
#!/usr/bin/env perl
use strict;
use warnings;
use Net::LDAP;
use Data::Dumper;
my %people;
my $ldap = Net::LDAP->new('my_ldap_server');
my $result = $ldap->bind('bind_dn');
die if $result->code;
my $search = $ldap->search(
base => 'ou=yaddayadda',
scope => 'subtree',
filter => 'objectClass=person',
attrs => ['manager'],
);
foreach my $found ( $search->entries ) {
my $mgr = $found->get_value('manager');
my $dn = $result->dn;
push( #{ $people{$mgr} }, $dn );
}
What this gives me is a hash of managers and the people who work for them (using DN, which is unique).
An entry from %people looks like:
$VAR1 = {
'cn=Firstname Lastname,ou=OrgUnit' => [
'cn=Personame Lastname,ou=OrgUnit',
'cn=AnotherPerson NameHere,ou=OrgUnit',
],
'cn=AnotherPerson NameHere,ou=OrgUnit' => [
'cn=Someone Else,ou=OrgUnit',
]
};
But I'm having trouble with turning that parent-child mapping into a hierarchical structure.
e.g.:
'ceo' => [
'pa' => [],
'head_of_dept' => [
'person' => [],
'person_with_staff' => [ 'person3', 'person4' ]
]
]
I'm at something of a loss for how to accomplish this. It seems it shouldn't be too hard to do, given that each person is unique within the organisation structure.
NB - in the above, I've got cn=AnotherPerson NameHere,ou=OrgUnit who has a subordinate, and I'm after making a nested mapping out of this:
e.g.:
$VAR1 = {
'cn=Firstname Lastname,ou=OrgUnit' => [
'cn=Personame Lastname,ou=OrgUnit',
'cn=AnotherPerson NameHere,ou=OrgUnit',
[
'cn=Someone Else,ou=OrgUnit'
]
]
};
What you need is a directed graph, and I suggest using the Graph::Directed module, whose methods are documented in Graph
This program will build the graph for you, but without any data I couldn't test it beyond making sure it compiles
use strict;
use warnings 'all';
use feature 'say';
use Net::LDAP;
use Graph::Directed;
use Data::Dumper;
my $ldap = Net::LDAP->new('my_ldap_server');
my $result = $ldap->bind('bind_dn');
die if $result->code;
my $search = $ldap->search(
base => 'ou=yaddayadda',
scope => 'subtree',
filter => 'objectClass=person',
attrs => ['manager'],
);
my $g = Graph::Directed->new;
for my $found ( $search->entries ) {
my $mgr = $found->get_value('manager');
my $dn = $result->dn;
$g->add_edge($mgr, $dn);
}
say $g;
The resulting Graph::Directed object has a stringification overload so you can examine it superficially by simply printing it, but when you want to interrogate the structure further you will need to know some of the terms of graph theory. For instance, $g->source_vertices will return a list of all nodes that have descendants but no parents—in this case, a list of senior management, or $g->is_cyclic will return true if your data has any loops anywhere
Here's an example of a program that uses your brief sample data to display a hierarchical tree of nodes
use strict;
use warnings 'all';
use Graph::Directed;
my $data = {
'cn=Firstname Lastname,ou=OrgUnit' => [
'cn=Personame Lastname,ou=OrgUnit',
'cn=AnotherPerson NameHere,ou=OrgUnit',
],
'cn=AnotherPerson NameHere,ou=OrgUnit' =>
[ 'cn=Someone Else,ou=OrgUnit', ]
};
my $g = Graph::Directed->new;
for my $mgr ( keys %$data ) {
$g->add_edge($mgr, $_) for #{ $data->{$mgr} };
}
dump_tree($_) for $g->source_vertices;
sub dump_tree {
my ($node, $level) = ( #_, 0);
print ' ' x $level, $node, "\n";
dump_tree($_, $level+1) for $g->successors($node);
}
output
cn=Firstname Lastname,ou=OrgUnit
cn=AnotherPerson NameHere,ou=OrgUnit
cn=Someone Else,ou=OrgUnit
cn=Personame Lastname,ou=OrgUnit
#Hunter McMillen unfortunately deleted his very good but slightly off answer. Here is my attempt to augment his code by turning the relationship from underling -> boss towards boss -> underlings.
To simulate the LDAP responses, I created a simple Moose class.
package Person;
use Moose;
has name => ( is => 'ro' );
has boss => ( is => 'ro', predicate => 'has_boss' );
package main;
use strict;
use warnings;
use Data::Printer;
# make a randomized list of people
my %people = map { $_->name => $_ }
map {
Person->new(
name => $_->[0], ( $_->[1] ? ( boss => $_->[1] ) : () )
)
} (
[qw( ceo )], [qw( head_of_dept ceo)],
[qw( person head_of_dept)], [qw( person_with_staff head_of_dept )],
[qw( person3 person_with_staff )], [qw( person4 person_with_staff )],
);
my %manages;
foreach my $p (values %people) {
push #{ $manages{ $p->boss } }, $p->name if $p->has_boss;
}
# this part shamelessly stolen from #HunterMcMillen's deleted answer
sub build_tree {
my ($person) = #_;
my #subtrees;
foreach my $managee ( #{ $manages{$person} } ) {
push #subtrees, build_tree($managee);
}
return { $person => \#subtrees };
}
p build_tree 'ceo';
Here's the output.
\ {
ceo [
[0] {
head_of_dept [
[0] {
person []
},
[1] {
person_with_staff [
[0] {
person4 []
},
[1] {
person3 []
}
]
}
]
}
]
}
This should be more or less what you want.

executing a function within an array within a hash in perl

I have a Perl data structurte like so
%myhash = (
k1 => v1,
kArray => [
{
name => "anonymous hash",
...
},
\&funcThatReturnsHash,
{
name => "another anonymous hash",
...
}
]
);
Elsewhere I iterate through the list in kArray which contains a bunch of hashes. I would like to either process the actual hash OR the hash returned by the function.
foreach my $elem( #{myhash{kArray}} ) {
if (ref($elem) == "CODE") {
%thisHash = &$elem;
}
else {
%thisHash = %$elem;
}
...
}
However ref ($elem) is always scalar or undefined. I tried func, &func, \&func, \%{&func}, in %myhash to no effect.
how do I extract the hash within the function in the main body?
Apart from the code sample you give being invalid Perl, the main problems seem to be that you are using == to compare strings instead of eq, and you are assigning a hash reference to a hash variable %thishash. I assure you that ref $elem never returns SCALAR with the data you show
It would help you enormously if you followed the common advice to use strict and use warnings at the top of your code
This will work for you
for my $elem ( #{ $myhash{kArray} } ) {
my $this_hash;
if ( ref $elem eq 'CODE' ) {
$this_hash = $elem->();
}
else {
$this_hash = $elem;
}
# Do stuff with $this_hash
}
or you could just use a map like this
use strict;
use warnings;
use 5.010;
use Data::Dump;
my %myhash = (
k1 => v1,
kArray => [
{
name => "anonymous hash",
},
\&funcThatReturnsHash,
{
name => "another anonymous hash",
}
]
);
for my $hash ( map { ref eq 'CODE' ? $_->() : $_ } #{ $myhash{kArray} } ) {
say $hash->{name};
}
sub funcThatReturnsHash {
{ name => 'a third anonymous hash' };
}
output
anonymous hash
a third anonymous hash
another anonymous hash
If you turn on strict and warnings, you'll see that:
foreach my $elem(#{mynahs{kArray}}) {
Isn't valid. You need at the very least a $ before mynahs.
But given something like this - your approach works - here's an example using map to 'run' the code references:
#!/usr/bin/perl
use strict;
use warnings;
use Data::Dumper;
sub gimme_hash {
return { 'fish' => 'paste' };
}
my $stuff =
[ { 'anon1' => 'value' },
\&gimme_hash,
{ 'anon2' => 'anothervalue' }, ];
my $newstuff = [ map { ref $_ eq "CODE" ? $_->() : $_ } #$stuff ];
print Dumper $newstuff;
Turns that hash into:
$VAR1 = [
{
'anon1' => 'value'
},
{
'fish' => 'paste'
},
{
'anon2' => 'anothervalue'
}
];
But your approach does work:
foreach my $element ( #$stuff ) {
my %myhash;
if ( ref $element eq "CODE" ) {
%myhash = %{$element -> ()};
}
else {
%myhash = %$element;
}
print Dumper \%myhash;
}
Gives:
$VAR1 = {
'anon1' => 'value'
};
$VAR1 = {
'fish' => 'paste'
};
$VAR1 = {
'anon2' => 'anothervalue'
};

Perl data structure to xml

I have got perl data something like below:
$data = {
id => 1,
name => "A",
users => [ { id => 1, name => "u1" }, { id => 2, name => "u2" } ],
groups => [ { id => 1, name => "g1" } ]
};
I would like to convert this into an xml something like below:
<map>
<item id="1" name="A">
<users>
<user id="1" name="u1"/>
<user id="2" name="u2"/>
</users>
<groups>
<group id="1" name="g1"/>
</groups>
</item>
</map>
I could do that manually creating each line explicitly. However I am looking for any CPAN Module base solution.
I tried XML::Twig but didn't go anywhere. I have used XML::Simple in the past for such thing but this time wanted to try something else as XML::Simple has been getting bad reviews.
You can do it similarly to Sobrique's method but with less hardcoded strings, like this:
#!/usr/bin/env perl
use strict; use warnings;
use XML::Twig;
my $data = {
id => 1,
name => "A",
users => [ { id => 1, name => "u1" }, { id => 2, name => "u2" } ],
groups => [ { id => 1, name => "g1" } ]
};
sub array_to_elts {
my ( $root, $name, $arrayref ) = #_;
map { $root->insert_new_elt($name, $_) } #{ $arrayref };
}
my $twig = XML::Twig
->new()
->set_xml_version("1.0")
->set_encoding('utf-8');
my $map = XML::Twig::Elt->new('map');
$twig->set_root($map);
my $item = $map->insert_new_elt(
'item',
{ id => $data->{'id'}, name => $data->{'name'} },
);
my $lines = $item->insert_new_elt('groups');
my $links = $item->insert_new_elt('users' );
array_to_elts($lines, 'group', $data->{'groups'});
array_to_elts($links, 'user', $data->{'users' });
$twig->set_pretty_print('indented');
$twig->print;
You could go to extreme lengths to reduce the hardcoded vals and base more off the raw data, but it quickly gets harder to read..
"Generic" way using XML::LibXML. You might need to add new code to the "else" part to handle other types of structures.
#!/usr/bin/perl
use warnings;
use strict;
use XML::LibXML;
my $data = {
id => 1,
name => "A",
users => [ { id => 1, name => "u1" },
{ id => 2, name => "u2" } ],
groups => [ { id => 1, name => "g1" } ],
};
sub to_xml {
my ($data, $xml) = #_;
for my $entry (keys %$data) {
my $ref = ref $data->{$entry};
if (not $ref) {
$xml->setAttribute($entry, $data->{$entry});
} elsif ('ARRAY' eq $ref) {
(my $name = $entry) =~ s/s$// or die "Can't guess the element name.\n";
my $list = $xml->addNewChild(q(), $entry);
for my $inner (#{ $data->{$entry} }) {
to_xml($inner, $list->addNewChild(q(), $name));
}
} else {
die "Unhandled structure $ref.\n";
}
}
}
my $xml = 'XML::LibXML::Document'->createDocument;
my $root = $xml->createElement('map');
$xml->setDocumentElement($root);
for my $entry ($data) {
my $item = $root->addNewChild(q(), 'item');
to_xml($entry, $item);
}
print $xml;
Yes, wise choice. XML::Simple ... isn't. It's for simple XML.
As noted in the comments though - your data is a little ambiguous - specifically, how do you tell what the elements should be called within 'groups' or 'users'.
This looks like you might have parsed some JSON. (Indeed, you can turn it straight back into JSON:
print to_json ( $data, { pretty => 1 } );
The core problem is - where JSON supports arrays, XML does not. So there's really very little you could do that will directly turn your data structure into XML.
However if you don't mind doing a bit of work yourself:
Here's how you assemble some XML using XML::Twig
Assembling XML in Perl
use strict;
use warnings;
use XML::Twig;
my $twig = XML::Twig->new( 'pretty_print' => 'indented' );
$twig->set_root(
XML::Twig::Elt->new(
'map',
)
);
my $item = $twig->root->insert_new_elt('item', { 'id' => 1, 'name' => 'A' } );
my $users = $item ->insert_new_elt( 'users' );
$users -> insert_new_elt ( 'user', { 'id' => 1, 'name' => 'u1' } );
$users -> insert_new_elt ( 'user', { 'id' => 2, 'name' => 'u2' } );
my $groups = $item -> insert_new_elt ('last_child', 'groups');
$groups -> insert_new_elt ( 'group', { 'id' => 1, 'name' => 'g1' } );
$twig->set_xml_version("1.0");
$twig->set_encoding('utf-8');
$twig->print;
Which prints:
<?xml version="1.0" encoding="utf-8"?>
<map>
<item id="1" name="A">
<users>
<user id="2" name="u2"/>
<user id="1" name="u1"/>
</users>
<groups>
<group id="1" name="g1"/>
</groups>
</item>
</map>
Iterating your data structure is left as an exercise for the reader.
As Borodin correctly notes - you have no way to infer map item group or user from your data structure. The latter two you can perhaps infer based on plurals, but given your data set, the best I can come up with is something like this:
use strict;
use warnings;
use XML::Twig;
my $data = {
id => 1,
name => "A",
users => [ { id => 1, name => "u1" }, { id => 2, name => "u2" } ],
groups => [ { id => 1, name => "g1" } ]
};
my $twig = XML::Twig->new( 'pretty_print' => 'indented' );
$twig->set_root( XML::Twig::Elt->new( 'map', ) );
my $item = $twig->root->insert_new_elt('item');
foreach my $key ( keys %$data ) {
if ( not ref $data->{$key} ) {
$item->set_att( $key, $data->{$key} );
next;
}
if ( ref( $data->{$key} ) eq "ARRAY" ) {
my $fakearray = $item->insert_new_elt($key);
foreach my $element ( #{ $data->{$key} } ) {
my $name = $key;
$name =~ s/s$//g;
$fakearray->insert_new_elt( $name, $element );
}
next;
}
if ( ref ( $data -> {$key} ) eq "HASH" ) {
$item -> insert_new_elt( $key, $data -> {$key} );
next;
}
}
$twig->set_xml_version("1.0");
$twig->set_encoding('utf-8');
$twig->print;
This isn't ideal because - map is hardcoded, as is item. And I take the very simplistic approach of assuming the array has an s on the end, to pluralise it.

Anomalous push behaviour under Catalyst MVC

I would expect the following code
my #array;
for my $rapport ( qw( value1 value2 value3 ) ) {
push #array, { key => $rapport };
}
to produce:
$VAR1 = [
{
'key' => 'value1'
},
{
'key' => 'value2'
},
{
'key' => 'value3'
}
];
However, running this code segment under Catalyst MVC I get:
$VAR1 = [
{
'key' => [ 'value', 'value2', 'value3' ]
},
];
Can someone please explain to me why?
EDIT: could anyone with the same issue please add an example? I cannot reproduce after some code changes, but as it has been upvoted 5 times I assume some other users have also experienced this issue?
This code example...
#!/usr/bin/perl
use Data::Dumper;
my #input = ( "var1", "var2", "var3" );
my #array;
for my $rapport ( #input ) {
push #array, { key => $rapport };
}
print Dumper( \#array );
exit;
produces ...
$VAR1 = [
{
'key' => 'var1'
},
{
'key' => 'var2'
},
{
'key' => 'var3'
}
];
But the following...
#!/usr/bin/perl
use Data::Dumper;
my #input = [ "var1", "var2", "var3" ]; # sometimes people forget to dereference their variables
my #array;
for my $rapport ( #input ) {
push #array, { key => $rapport };
}
print Dumper( \#array );
exit;
shows...
$VAR1 = [
{
'key' => [
'var1',
'var2',
'var3'
]
}
];
As you can see both examples loop through an array but the second one is an array, that was initialized with a reference value. Since in Catalyst you normally ship various values through your application via stash or similar constructs, you could check weather your array really contains scalar values : )

How can I convert one Perl hash to another using the keys?

I've just started diving in to the crazy world that is perl and have come across a problem that I cannot seem to wrap my head around. Specifically I need to be able to convert from one hash structure to another, that uses the keys/values as the new hashes keys/values.
An example:
Input hash:
my %original_hash = (
first_key => { some_key => "apples",
another_key => "chips",
#potentially more here
},
second_key => { more_of_same => "dogs",
its_another => "cats",
#potentially more here
}
);
Output hash:
my %final_hash = (
some_key => {
apples => {
more_of_same => "dogs",
its_another => "cats",
}
} ,
another_key => {
chips => {
more_of_same => "dogs",
its_another => "cats",
}
}
);
And yes I do want the second_key's data repeated in the final_hash, as there will be an array of the original_hashes that are inputted. The first element becomes the base-case, and all other elements may append or remove from that list.
If anyone has any suggestions on how to go about doing this that would be greatly appreciated!
Here is another way
my %final_hash;
my %tmp = %{$original_hash{first_key}};
my $val = $original_hash{second_key};
while ( my ($k,$v) = each %tmp) {
$final_hash{$k} = { $v => $val };
}
print Dumper (\%final_hash);
Okay, Sinan is right, it's very hard to guess your problem, but the following code seems to do what you want ... or at least it produces the listed output.... :)
use strict;
use warnings;
use Data::Dumper;
$Data::Dumper::Deepcopy = 1;
my %original_hash = (
first_key => { some_key => "apples",
another_key => "chips",
#potentially more here
},
second_key => { more_of_same => "dogs",
its_another => "cats",
#potentially more here
}
);
my %final_hash;
for my $key ( keys %{ $original_hash{first_key} } ) {
$final_hash{$key} = {
$original_hash{first_key}->{$key}
=> $original_hash{second_key}
};
}
print Dumper(\%final_hash);