Access element across multiple hash of hash of arrays - perl

So I have created a hash of hash of arrays. This data structure (lets call it "amey") is a collection of python tests that i wish to control the execution of.
It looks something like this:
'amey'=
{
'test_type1' = {
'class_test1' => [
'test_1'
],
'class_test2' => [
'test_1',
'test_2'
],
'class_test3' => [
'test_1'
]
};
'test_type2' = {
'class_test1' => [
'test_1',
'test_2'
]
};
'test_type3' = {
'class_test1' => [
'test_1',
'test_2',
'test_3',
'test_4',
'test_5',
'test_6',
'test_7'
],
}
}
The intention is to somehow iterate through this hash and run each test of a type in parallel with a test of another test type.
*Also knowing which class the test belongs to is important as there are similar named tests. (Example "test_bat")*
So for example:
Start executing the below tests in parallel
run test test_type1/class_test1/test_1 &;
run test test_type2/class_test1/test_1 &;
run test test_type3/class_test1/test_1 &;
Wait for the test of a type to finish running and then start the next of that type.
So for example if test_type1/class_test1/test_1 completes then start with
test_type1/class_test2/test_1.
The number of classes are not the same across each type and nor are the number of tests of each class.
I realize this is kind of a complex requirement (or may be not for the perl monks :)), but would love to hear some suggestions on how I should go about doing this.

With Forks::Super, create each test in a separate fork call and set up dependencies between them.
use Forks::Super;
$amey = { ... };
foreach my $class (keys %$amey) {
foreach my $type (keys %{$amey->{$class}}) {
my $last_job;
foreach my $name (keys %{$amey->{$class}{$type}}) {
my $job = fork {
name => "$class/$type/$name",
cmd => "run test $class/$type/$name",
depend_on => $last_job
};
$last_job = "$class/$type/$name";
}
}
}

Related

Setting the "finish" event on a Mojolicious/Minion::Job

I am trying to have Minion jobs feed back to the Mojolicious web app upon completion (probably by posting a message to an API). The underlying idea here is then for the web app to feed back to the client that has uploaded/started the job.
I have tried doing this:
get '/' => sub {
my $c = shift;
my $id = $c->minion->enqueue('thing', [ qw/a b 1/, { foo => 'bar' } ]);
my $job = $c->minion->job($id);
$job->on(finish => sub ($job) {
my $id = $job->id;
my $task = $job->task;
$job->app->log->info(qq{Job "$id" was performed with task "$task"});
});
$c->render(template => 'index');
};
which doesn't work - I guess because the event is only emitted in the process performing the job, and the event does not get serialized and queued.
If I do this:
app->minion->add_task
(thing => sub ($job, $c, $sub, #args) {
$job->on(finish => sub ($job) {
my $id = $job->id;
my $task = $job->task;
$job->app->log->info(qq{Job "$id" was performed with task "$task"});
});
sleep 2;
});
it works ok, but it means I have to add the event handling to every task - which adds complexity to the code.
Is there a way to avoid having to do this?
I am thinking of:
being able to set a default class for jobs (so that jobs are all of a subclass of Mojo::Job - for example Minion::Job::WithFeedback)
better yet, being able to inject roles into task creation (so that you can do $c->minion->enqueue('thing', [ qw/a b 1/, { foo => 'bar' } ], { roles => qw/+WithFeedback +WithTimeout/);
I know I could poll all jobs regularly and see what changed status - this is what the Minion::Admin plugin does - but I would like to see if there is a different way that doesn't require polling the database.
Is this possible? and while we're at it - is this a bad idea in and of itself?

Join attempt throwing exceptions

I'm sure I'm overlooking something glaringly obvious and I apologize for the newbie question, but I've spent several hours back and forth through documentation for DBIx::Class and Catalyst and am not finding the answer I need...
What I'm trying to do is automate creation of sub-menus based on the contents of my database. I have three tables in the database to do so: maps (in which sub-menu items are found), menus (contains names of top-level menus), maps_menus (assigns maps to top-level menus). I've written a subroutine to return a hash of resultsets, with the plan of using a Template Toolkit nested loop to build the top-level and sub-menus.
Basically, for each top-level menu in menus, I'm trying to run the following query and (eventually) build a sub-menu based on the result:
select * FROM maps JOIN maps_menus ON maps.id_maps = maps_menus.id_maps WHERE maps_menus.id_menus = (current id_menus);
Here is the subroutine, located in lib/MyApp/Schema/ResultSet/Menus.pm
# Build a hash of hashes for menu generation
sub build_menu {
my ($self, $maps, $maps_menus) = #_;
my %menus;
while (my $row = $self->next) {
my $id = $row->get_column('id_menus');
my $name = $row->get_column('name');
my $sub = $maps_menus->search(
{ 'id_maps' => $id },
{ join => 'maps',
'+select' => ['maps.id_maps'],
'+as' => ['id_maps'],
'+select' => ['maps.name'],
'+as' => ['name'],
'+select' => ['maps.map_file'],
'+as' => ['map_file']
}
);
$menus{$name} = $sub;
# See if it worked...
print STDERR "$name\n";
while (my $m = $sub->next) {
my $m_id = $m->get_column('id_maps');
my $m_name = $m->get_column('name');
my $m_file = $m->get_column('map_file');
print STDERR "\t$m_id, $m_name, $m_file\n";
}
}
return \%menus;
}
I am calling this from lib/MyApp/Controller/Maps.pm thusly...
$c->stash(menus => [$c->model('DB::Menus')->build_menu($c->model('DB::Map'), $c->model('DB::MapsMenus'))]);
When I attempt to pull up the page, I get all sorts of exceptions, the top-most of which is:
[error] No such relationship maps on MapsMenus at /home/catalyst/perl5/lib/perl5/DBIx/Class/Schema.pm line 1078
Which, as far as I can tell, originates from the call to $sub->next. I take this as meaning I'm doing my query incorrectly and not getting the results I think I should be. However, I'm not sure what I'm missing.
I found the following lines, defining the relationship to maps, in lib/MyApp/Schema/Result/MapsMenus.pm
__PACKAGE__->belongs_to(
"id_map",
"MyApp::Schema::Result::Map",
{ id_maps => "id_maps" },
{ is_deferrable => 1, on_delete => "CASCADE", on_update => "CASCADE" },
);
...and in lib/MyApp/Schema/Result/Map.pm
__PACKAGE__->has_many(
"maps_menuses",
"MyApp::Schema::Result::MapsMenus",
{ "foreign.id_maps" => "self.id_maps" },
{ cascade_copy => 0, cascade_delete => 0 },
);
No idea why it's calling it "maps_menuses" -- that was generated by Catalyst. Could that be the problem?
Any help would be greatly appreciated!
I'd suggest using prefetch of the two relationships which form the many-to-many relationship helper and maybe using HashRefInflator if you don't need access to the row objects.
Note that Catalyst doesn't generate a DBIC (which is btw the official abbreviation for DBIx::Class, DBIx is a whole namespace) schema, SQL::Translator or DBIx::Class::Schema::Loader do. Looks at the docs of the module you've used to find out how to influence its naming.
Also feel free to change the names if they don't fit you.

DBIx::Class: sub resultset in Template Toolkit presented as an array, not a resultset

I am developing a Catalyst application using DBIx::Class and Template Toolkit; in the particular part I'm having issues with, I have a resultset obtained using by calling the following function in my ResultSet schema:
sub divisions_and_teams_in_season {
my ( $self, $season, $grid ) = #_;
return $self->search({
"division_seasons.season" => $season->id,
"division_seasons.fixtures_grid" => $grid->id,
}, {
prefetch => [
"division_seasons",
{
"team_seasons" => {
"team" => [{
"club" => "venue"
},
"home_night"
]
}
}
],
order_by => {
-asc => [ qw( division_seasons.rank team_seasons.grid_position club.short_name team.name ) ]
}
});
}
This returns the data as I would expect and I'm able to do the following in my Controller code to get back my resultset and iterate through the team_seasons:
my $divisions = [ $c->model("DB::Division")->divisions_and_teams_in_season($current_season, $c->stash->{grid}) ];
foreach my $division ( #{ $divisions } ) {
$c->log->debug( $division->team_seasons->grid_positions_filled ); # This works because $division->team_seasons is a resultset object
}
However, in my template (having stashed $divisions), I'm unable to access the grid_positions_filled object because division.team_seaons gives me an arrayref of team resultsets in that division:
[%
# Loop through our divisions
FOREACH division IN divisions;
CALL c.log.debug(division.team_seasons); # The output of this is something like: ARRAY(0x6f8318c)
END;
-%]
The output I get for the same debug log in my controller is more like a list of resultset objects:
TopTable::Model::DB::TeamSeason=HASH(0x6eea94c)
TopTable::Model::DB::TeamSeason=HASH(0x6f01834)
TopTable::Model::DB::TeamSeason=HASH(0x6ef5284)
TopTable::Model::DB::TeamSeason=HASH(0x6efec9c)
TopTable::Model::DB::TeamSeason=HASH(0x6ef4dc4)
TopTable::Model::DB::TeamSeason=HASH(0x6faf0ac)
TopTable::Model::DB::TeamSeason=HASH(0x6eefa04)
Hope all this makes sense! Does anyone know how I can get the behaviour from the controller into the template so that I can access methods on the team_season ResultSet?
Thank you very much in advance.
Try $self->search_rs rather than $self->search. "This method does the same exact thing as search() except it will always return a resultset, even in list context."
See the docs for more info.
The team_seasons accessor returns a resultset in scalar context, and an array of rows in list context. It seems that template toolkit code evaluates the accessor in list context.
As a work-around, DBIC installs a special accessor, postfixed with _rs that always returns a resultset regardless of context. So the following should work:
CALL c.log.debug(division.team_seasons_rs.grid_positions_filled);
This is documented under DBIx::Class::Relationship.

perl CGI, differences with param vs Var

I do this in my cgi perl script:
my %USER_HTML_INPUT = Vars();
I noticed that if have an array of data assigned to one key it becomes like this:
$VAR= {'tempvalue' => '0�25�85�125' };
If i do #DATA = param('tempvalue'); It splits the values into the array.
How can I do the same operation without using param function.?
If you really, really want to do this without param, you could try something like (untested):
my $vars = Vars();
my %USER_HTML_INPUT = map { $_ => [ split(m{\0}, $vars->{$_}) ] } keys %$vars;
but slightly less ugly:
my %USER_HTML_INPUT = map { $_ => [ param($_) ] } param();
It is much cleaner, though, to simply use param on the parameters that you need.
Also read the section DEBUGGING of the CGI documentation to see how you can pass CGI parameters to a script using CGI.pm from the command-line.

How can I create a hash of hashes from an array of hashes in Perl?

I have an array of hashes, all with the same set of keys, e.g.:
my $aoa= [
{NAME=>'Dave', AGE=>12, SEX=>'M', ID=>123456, NATIONALITY=>'Swedish'},
{NAME=>'Susan', AGE=>36, SEX=>'F', ID=>543210, NATIONALITY=>'Swedish'},
{NAME=>'Bart', AGE=>120, SEX=>'M', ID=>987654, NATIONALITY=>'British'},
]
I would like to write a subroutine that will convert this into a hash of hashes using a given key hierarchy:
my $key_hierarchy_a = ['SEX', 'NATIONALITY'];
aoh_to_hoh ($aoa, $key_hierarchy_a) = #_;
...
}
will return
{M=>
{Swedish=>{{NAME=>'Dave', AGE=>12, ID=>123456}},
British=>{{NAME=>'Bart', AGE=>120, ID=>987654}}},
F=>
{Swedish=>{{NAME=>'Susan', AGE=>36, ID=>543210}}
}
Note this not only creates the correct key hierarchy but also remove the now redundant keys.
I'm getting stuck at the point where I need to create the new, most inner hash in its correct hierarchical location.
The problem is I don't know the "depth" (i.e. the number of keys). If I has a constant number, I could do something like:
%h{$inner_hash{$PRIMARY_KEY}}{$inner_hash{$SECONDARY_KEY}}{...} = filter_copy($inner_hash,[$PRIMARY_KEY,$SECONDARY_KEY])
so perhaps I can write a loop that will add one level at a time, remove that key from the hash, than add the remaining hash to the "current" location, but it's a bit cumbersome and also I'm not sure how to keep a 'location' in a hash of hashes...
use Data::Dumper;
my $aoa= [
{NAME=>'Dave', AGE=>12, SEX=>'M', ID=>123456, NATIONALITY=>'Swedish'},
{NAME=>'Susan', AGE=>36, SEX=>'F', ID=>543210, NATIONALITY=>'Swedish'},
{NAME=>'Bart', AGE=>120, SEX=>'M', ID=>987654, NATIONALITY=>'British'},
];
sub aoh_to_hoh {
my ($aoa, $key_hierarchy_a) = #_;
my $result = {};
my $last_key = $key_hierarchy_a->[-1];
foreach my $orig_element (#$aoa) {
my $cur = $result;
# song and dance to clone an element
my %element = %$orig_element;
foreach my $key (#$key_hierarchy_a) {
my $value = delete $element{$key};
if ($key eq $last_key) {
$cur->{$value} ||= [];
push #{$cur->{$value}}, \%element;
} else {
$cur->{$value} ||= {};
$cur = $cur->{$value};
}
}
}
return $result;
}
my $key_hierarchy_a = ['SEX', 'NATIONALITY'];
print Dumper(aoh_to_hoh($aoa, $key_hierarchy_a));
As per #FM's comment, you really want an extra array level in there.
The output:
$VAR1 = {
'F' => {
'Swedish' => [
{
'ID' => 543210,
'NAME' => 'Susan',
'AGE' => 36
}
]
},
'M' => {
'British' => [
{
'ID' => 987654,
'NAME' => 'Bart',
'AGE' => 120
}
],
'Swedish' => [
{
'ID' => 123456,
'NAME' => 'Dave',
'AGE' => 12
}
]
}
};
EDIT: Oh, BTW - if anyone knows how to elegantly clone contents of a reference, please teach. Thanks!
EDIT EDIT: #FM helped. All better now :D
As you've experienced, writing code to create hash structures of arbitrary depth is a bit tricky. And the code to access such structures is equally tricky. Which makes one wonder: Do you really want to do this?
A simpler approach might be to put the original information in a database. As long as the keys you care about are indexed, the DB engine will be able to retrieve rows of interest very quickly: Give me all persons where SEX = female and NATIONALITY = Swedish. Now that sounds promising!
You might also find this loosely related question of interest.