Perl Catalyst DBIx Cursor Cache - how to clear? - perl

apologises and thanks in advance for what, even as I type, seems likely silly question, but here goes anyway.
I have basic Catalyst application using DBIx::Class with an 'Author' and associated 'Book' table. In addition I also use DBIx::Class::Cursor::Cached to cache data as appropriate.
The issue is that, following an edit, I need to clear cached data BEFORE it has actually expired.
1.) Author->show_author_and_books which fetchs and caches resultset.
2.) the Book->edit_do which needs to clear the cached data from the Author->show_author_and_books request.
See basic/appropriate setup below.
-- MyApp.pm definition including backend 'Cache::FileCache' cache.
__PACKAGE__->config(
name => 'MyApp',
...
'Plugin::Cache' => { 'backend' => { class => 'Cache::FileCache',
cache_root => "./cache",
namespace => "dbix",
default_expires_in => '8 hours',
auto_remove_stale => 1
}
},
...
-- MyApp::Model::DB definition with 'Caching' traits set using 'DBIx::Class::Cursor::Cached'.
...
__PACKAGE__->config(
schema_class => 'MyApp::Schema',
traits => [ 'Caching' ],
connect_info => { dsn => '<dsn>',
user => '<user>',
password => '<password>',
cursor_class => 'DBIx::Class::Cursor::Cached'
}
);
...
-- MyApp::Controller::Author.pm definition with 'show_author_and_books' method - resultset is cached.
...
sub show_author_and_books :Chained('base') :PathPart('') :Args(0)
{
my ( $self, $c ) = #_;
my $author_id = $c->request->params->{author_id};
my $author_and_books_rs = $c->stash->{'DB::Author'}->search({ author_id => $author_id },
{ prefetch => 'book' },
cache_for => 600 } ); # Cache results for 10 minutes.
# More interesting stuff, but no point calling $author_and_books_rs->clear_cache here, it would make no sense:s
...
}
...
-- MyApp::Controller::Book.pm definition with 'edit_do' method which updates book entry and so invalidates the cached data in show_author_and_books.
...
sub edit_do :Chained('base') :PathPart('') :Args(0)
{
my ( $self, $c ) = #_;
# Assume stash contains a book for some author, and that we want to update the description.
my $book = $c->stash->{'book'}->update({ desc => $c->request->params->{desc} });
# How do I now clear the cached DB::Author data to ensure the new desc is displayed on next request to 'Author->show_author_and_books'?
# HOW DO I CLEAR CACHED DB::Author DATA?
...
}
Naturally I'm aware that $author_and_books_rs, as defined in Author->show_author_and_books, contains a method 'clear_cache', but obviously this is out of scope in Book->edit_do ( not to mention another problem there might be).
So, is the correct approach to make the DBIx request again , as per ...show_author_and_books and then call the 'clear_cache' again that or is there a more direct way where I can just say something like this $c->cache->('DB::Author')->clear_cache?
Thank you again.
PS. I'm sure when I look at this tomorrow, the full silliness of the question will hit me:s

Try
$c->model( 'DB::Author' )->clear_cache() ;

The solution I went for in the end was to NOT use 'DBIx::Class::Cursor::Cached', but instead directly use the Catalyst Cache plugin defining multiple
backend caches to handle the different namespaces I trying to manage in the real-world scenario.
I backed away from D::C::Cursor::Cached as all data was/is held in the same namespace plus there doesn't appear to be a method to expire data in advance of
time already set.
So for completeness, from the code above, the MyApp::Model::DB.pm definition would lose the 'traits' and 'cursor_class' key/values.
Then...
The MyApp.pm Plugin::Cache' would expand to contain multiple cache namespaces...
-- MyApp.pm definition including backend 'Cache::FileCache' cache.
...
'Plugin::Cache' => { 'backends' => { Authors => { class => 'Cache::FileCache',
cache_root => "./cache",
namespace => "Authors",
default_expires_in => '8 hours',
auto_remove_stale => 1
},
CDs => { class => 'Cache::FileCache',
cache_root => "./cache",
namespace => "CDs",
default_expires_in => '8 hours',
auto_remove_stale => 1
},
...
}
...
-- MyApp::Controller::Author.pm definition with 'show_author_and_books' method - resultset is cached.
...
sub show_author_and_books :Chained('base') :PathPart('') :Args(0)
{
my ( $self, $c ) = #_;
my $author_id = $c->request->params->{author_id};
my $author = $c->get_cache_backend('Authors')->get( $author_id );
if( !defined($author) )
{
$author = $c->stash->{'DB::Author'}->search({ author_id => $author_id },
{ prefetch => 'book', rows => 1 } )->single;
$c->get_cache_backend('Authors')->set( $author_id, $author, "10 minutes" );
}
# More interesting stuff, ...
...
}
...
-- MyApp::Controller::Book.pm definition with 'edit_do' method which updates book entry and so invalidates the cached data in show_author_and_books.
...
sub edit_do :Chained('base') :PathPart('') :Args(0)
{
my ( $self, $c ) = #_;
# Assume stash contains a book for some author, and that we want to update the description.
my $book = $c->stash->{'book'}->update({ desc => $c->request->params->{desc} });
# How do I now clear the cached DB::Author data to ensure the new desc is displayed on next request to 'Author->show_author_and_books'?
# HOW DO I CLEAR CACHED DB::Author DATA? THIS IS HOW, EITHER...
$c->get_cache_backend('Authors')->set( $c->stash->{'book'}->author_id, {}, "now" ); # Expire now.
# ... OR ... THE WHOLE Authors namespace...
$c->get_cache_backend('Authors')->clear;
...
}
NOTE : as you'll expect from the use of Author and CDs, this isn't the real world scenario I'm working, but should serve to show my intent.
As I'm relatively new to the wonder of DBIx and indeed Catalyst, I'd be interested to hear if there a better approach to this (I very much expect there is), but it will serve for the moment as I'm attempting to update a legacy application.

The plugin could probably be patched to make per result set caches easy to namespace and clear independently, and alternatively it would probably not be so hard to add a namespace to the attributes. If you want to work on that hit #dbix-class and I'd be willing to mentor you - jnap

Related

perl - searching in list of objects which are an accessor of another object

I am a Perl-OO beginner and I am encountering a design-challenge. I hope you can give me some hints to get to an elegant solution. I am working with Mouse Object System here.
For a minimal example lets say I have a User-Object. A user has a name.
package User;
use Mouse;
has "name" => (
is => "rw",
isa => "Str|Undef",
);
Then I have a User-Cache-Object, which gets a list of all Users (from an LDAP-Server). You can say this is a "has-a" Relationship between the User Cache and the User.
package UserCache;
use Mouse;
has "users" => (
is => 'rw',
isa => 'ArrayRef|Undef',
default => sub { [] },
);
I store this list of Users as an Array of User-Objects in the accessor of the User-Cache.
my $cache = UserCache->new();
foreach my $entry ( $ldap->searchGetEntries() ) {
my $user = User->new();
$user->name($entry->get_value('userdn'));
push #{ $cache->users }, $user;
}
Now this is where my Problem comes in. If I want to find a User-Object with specific attributes (e.g. a User named John), I have to loop over this whole Array of User-Objects and query each object for its name. When given a list of names, this gets a really inefficient process.
foreach my $user ( #{ $cache->users } ) {
if ( $user->name eq 'John' ) {
#do something with John
}...
}
Is there a way of storing Lists of Objects in other Objects in a way, that I can efficently search? Like $cache->get_users->get_name('John') and that returns the object I need?
You don't really have to write the UserCache class yourself. Instead, use CHI to cache users you want to cache under the key you want to use for lookups. If you want, you can wrap your cache class to abstract away from the specific cache implementation.
Also, you have this:
push #{ $cache->users }, $user;
where you leak implementation details. Instead, your UserCache object needs something like a save_user method so the code it uses does not depend on the implementation details.
$cache->save_user( $user );
For Moose objects, you get Moose::Meta::Attribute::Native::Trait::Array; for Mouse, you get MouseX::NativeTraits::ArrayRef.
No. At least not universally. You can of course build indexes for common things. Or you could cache searches once you have done them.
Lookups are best implemented as hashes. Those could be attached to the UserCache object. Something like:
my #users = $cache->find( name => 'John' );
That would internally map to a hashref with search fields.
package UserCache;
#...
has _search_index => (
is => 'ro',
isa => 'HashRef',
default => sub { {} },
);
And the hash reference would look something like this:
{
name => {
John => [
User->new( name => 'John', last_name => 'Smith' ),
User->new( name => 'John', last_name => 'Wayne' ),
User->new( name => 'John', last_name => 'Bon Jovi' ),
],
James => [ ... ],
},
id => {
# ...
},
),
But again, you'd have to build those. So you need to do the lookup once. But I think the lookup should be done inside UserCache and stored there too.
sub find {
my ($self, $key, $value) = #_;
# get operation
return #{ $self->_search_index->{$key}->{$value} }
if exists $self->_search_index->{$key}->{$value};
# set operation
foreach my $user ( #{ $self->users } ) {
push #{ $self->_search_index->{$key}->{$value} }, $user
if $user->$key eq $value
}
return #{ $self->_search_index->{$key}->{$value} }
}
This is a very naive implementation and it doesn't support multiple lookups, but it's a start.
Note that if you have a lot of users and a lot of indexes, the data structure might become large.
To make it easier, Moose's built-in traits might be helpful. If you want a stronger cache behavior, look at CHI.

DBIx::Class and overloading accessors

(Similar to, but with more concrete details that, #11526999)
My Result Classes have been built using dbicdump, however I wish to overload the default accessor for a date field.
Works, but a bodge
To hackytest my idea, I simply added an accessor attribute to the created date key of the add_columns call:
__PACKAGE__->add_columns(
"stamp_id",
{
data_type => "integer",
is_auto_increment => 1,
is_nullable => 0,
sequence => "timestamp_stamp_id_seq",
},
"date",
{ data_type => "date", is_nullable => 0, accessor => '_date' },
);
... and created my accessor routine below the Schema::Loader checksum line:
# DO NOT MODIFY THIS OR ANYTHING ABOVE! md5sum:nB5koMYAhBwz4ET77Q8qlA
sub date {
my $self = shift;
warn "overloaded date\n"; # Added for debugging
my $date;
# The date needs to be just the date, not the time
if ( #_ ) {
$date = shift;
if ( $date =~ /^([\d\-]+)/ ) {
$date = $1
}
return $self->_date($date)
}
# Fetch the column value & remove the time part.
$date = $self->_date;
if ( $date =~ /^([\d\-]+)/ ) {
$date = $1
}
return $date;
}
This works, as it returns an expected 2014-10-04, but is a bodge.
Do it the right way
The problem is that I've hacked the checksum'd code, so I can't neatly re-generate my Class objects.
Reading ResultSource and the CookBook the correct approach appears to be:
Have the ResultSource built by dbicdump as standard:
__PACKAGE__->add_columns(
"stamp_id",
{
data_type => "integer",
is_auto_increment => 1,
is_nullable => 0,
sequence => "timestamp_stamp_id_seq",
},
"date",
{ data_type => "date", is_nullable => 0 },
);
.... add a change the accessor below the line, using the + to indicate it's an alteration to an existing definition:
# DO NOT MODIFY THIS OR ANYTHING ABOVE! md5sum:nB5koMYAhBwz4ET77Q8qlA
__PACKAGE__->add_columns(
"+date", { accessor => '_date' },
);
.... use the overload method as before
Not working.
I've double-checked my spelling, I've tried add_column rather than add_columns, and I've tried putting the second add_columns to directly below the first - all to now avail.... the code uses the default accessor, and returns 2014-10-04T00:00:00
How do I over-ride the default accessor, so I can use my own method?
Thankee...
What you need here is a col_accessor_map passed in as a loader option.
col_accessor_map => {
table_name => {
date => _date,
}
}
You can pass loader options to dbicdump with -o.
$ dbicdump -o col_accessor_map="{ table_name => { date => _date } }" ... other options ...
(Replace table_name above with the name of your table - that's obvious, right?)
Update: This was posted untested, and when I finally got round to testing it, I found it didn't work. After a conversation with the author on IRC I was told that the col_accessor_map option doesn't support this nested hash approach, so if you wanted to use this approach you would need to use a coderef.
However, the author also agreed that adding this support would be a good idea and I've just got back from lunch to find this Github commit which adds the feature. I don't know how soon it will get to CPAN though.
This may be the first time that CPAN has been updated to make a SO answer correct :-)
At a different level of abstraction I believe you could use a method modifier
use Class::Method::Modifier; # or Moose/Moo
around date => sub {...};

mojolicious helper storing an elasticsearch connection

i'm experimenting with elasticsearch within mojolicious.
I'm reasonably new at both.
I wanted to create a helper to store the ES connection and I was hoping to pass the helper configuration relating to ES (for example the node info, trace_on file etc).
If I write the following very simple helper, it works;
has elasticsearch => sub {
return Search::Elasticsearch->new( nodes => '192.168.56.21:9200', trace_to => ['File','/tmp/elasticsearch.log'] );
};
and then in startup
$self->helper(es => sub { $self->app->elasticsearch() });
however if I try to extend that to take config - like the following -
it fails. I get an error "cannot find index on package" when the application calls $self->es->index
has elasticsearch => sub {
my $config = shift;
my $params->{nodes} = '192.168.56.21:' . $config->{port};
$params->{trace_to} = $config->{trace_to} if $config->{trace_to};
my $es = Search::Elasticsearch->new( $params );
return $es;
};
and in startup
$self->helper(es => sub { $self->app->elasticsearch($self->config->{es}) });
I assume I'm simply misunderstanding helpers or config or both - can someone enlighten me?
Just fyi, in a separate controller file I use the helper as follows;
$self->es->index(
index => $self->_create_index_name($index),
type => 'crawl_data',
id => $esid,
body => {
content => encode_json $data,
}
);
that works fine if I create the helper using the simple (1st) form above.
I hope this is sufficient info? please let me know if anything else is required?
First of all, has and helper are not the same. has is a lazily built instance attribute. The only argument to an attribute constructor is the instance. For an app, it would look like:
package MyApp;
has elasticsearch => sub {
my $app = shift;
Search::ElasticSearch->new($app->config->{es});
};
sub startup {
my $app = shift;
...
}
This instance is then persistent for the life of the application after first use. I'm not sure if S::ES has any reconnect-on-drop logic, so you might need to think about it a permanent object is really what you want.
In contrast a helper is just a method, available to the app, all controllers and all templates (in the latter case, as a function). The first argument to a helper is a controller instance, whether the current one or a new one, depending on context. Therefore you need to build your helper like:
has (elasticsearch => sub {
my ($c, $config) = #_;
$config ||= $c->app->config->{es};
Search::ElasticSearch->new($config);
});
This mechanism will build the instance on demand and can accept pass-in arguments, perhaps for optional configuration override as I have shown in that example.
I hope this answers your questions.

Join attempt throwing exceptions

I'm sure I'm overlooking something glaringly obvious and I apologize for the newbie question, but I've spent several hours back and forth through documentation for DBIx::Class and Catalyst and am not finding the answer I need...
What I'm trying to do is automate creation of sub-menus based on the contents of my database. I have three tables in the database to do so: maps (in which sub-menu items are found), menus (contains names of top-level menus), maps_menus (assigns maps to top-level menus). I've written a subroutine to return a hash of resultsets, with the plan of using a Template Toolkit nested loop to build the top-level and sub-menus.
Basically, for each top-level menu in menus, I'm trying to run the following query and (eventually) build a sub-menu based on the result:
select * FROM maps JOIN maps_menus ON maps.id_maps = maps_menus.id_maps WHERE maps_menus.id_menus = (current id_menus);
Here is the subroutine, located in lib/MyApp/Schema/ResultSet/Menus.pm
# Build a hash of hashes for menu generation
sub build_menu {
my ($self, $maps, $maps_menus) = #_;
my %menus;
while (my $row = $self->next) {
my $id = $row->get_column('id_menus');
my $name = $row->get_column('name');
my $sub = $maps_menus->search(
{ 'id_maps' => $id },
{ join => 'maps',
'+select' => ['maps.id_maps'],
'+as' => ['id_maps'],
'+select' => ['maps.name'],
'+as' => ['name'],
'+select' => ['maps.map_file'],
'+as' => ['map_file']
}
);
$menus{$name} = $sub;
# See if it worked...
print STDERR "$name\n";
while (my $m = $sub->next) {
my $m_id = $m->get_column('id_maps');
my $m_name = $m->get_column('name');
my $m_file = $m->get_column('map_file');
print STDERR "\t$m_id, $m_name, $m_file\n";
}
}
return \%menus;
}
I am calling this from lib/MyApp/Controller/Maps.pm thusly...
$c->stash(menus => [$c->model('DB::Menus')->build_menu($c->model('DB::Map'), $c->model('DB::MapsMenus'))]);
When I attempt to pull up the page, I get all sorts of exceptions, the top-most of which is:
[error] No such relationship maps on MapsMenus at /home/catalyst/perl5/lib/perl5/DBIx/Class/Schema.pm line 1078
Which, as far as I can tell, originates from the call to $sub->next. I take this as meaning I'm doing my query incorrectly and not getting the results I think I should be. However, I'm not sure what I'm missing.
I found the following lines, defining the relationship to maps, in lib/MyApp/Schema/Result/MapsMenus.pm
__PACKAGE__->belongs_to(
"id_map",
"MyApp::Schema::Result::Map",
{ id_maps => "id_maps" },
{ is_deferrable => 1, on_delete => "CASCADE", on_update => "CASCADE" },
);
...and in lib/MyApp/Schema/Result/Map.pm
__PACKAGE__->has_many(
"maps_menuses",
"MyApp::Schema::Result::MapsMenus",
{ "foreign.id_maps" => "self.id_maps" },
{ cascade_copy => 0, cascade_delete => 0 },
);
No idea why it's calling it "maps_menuses" -- that was generated by Catalyst. Could that be the problem?
Any help would be greatly appreciated!
I'd suggest using prefetch of the two relationships which form the many-to-many relationship helper and maybe using HashRefInflator if you don't need access to the row objects.
Note that Catalyst doesn't generate a DBIC (which is btw the official abbreviation for DBIx::Class, DBIx is a whole namespace) schema, SQL::Translator or DBIx::Class::Schema::Loader do. Looks at the docs of the module you've used to find out how to influence its naming.
Also feel free to change the names if they don't fit you.

DBIx:Class many-to-many relationship with additional attribute

I'm working with DBIx::Class in Catalyst framework. My local goal is to add a new many-to-many relationship between users and, let's say, tasks. But there's one little trick I need. User can have different roles in task (like 'worker' or 'spectator').
So I have users table with these fields:
id
name
I have task table with these fields:
id
title
description
And I have relationship table user_tasks with these fields:
user_id
task_id
role
I have set up has_many from users to user_tasks, has_many from tasks to user_tasks and corresponding many_to_many relationships between users and tasks. And that plain part works as it should.
Then, for example, I want to get my user list including user's role in task identified by $task_id:
my $users = $schema->resultset('User')->with_task_role($task_id);
while (my $u = $users->next) {
print "User: " . $u->name . ", role: " . $u->get_column('task_role');
}
So how should I code this with_task_role custom resultset to get this additional field with user's task role in my query?
First of all many-to-many is not a relationship. It's a accessor (a relationship bridge).
Second, the DBIx::Class has an excellent documentation. Take a look at join/prefetch.
In your ResultSet/User.pm file you should have something like:
sub with_task_role {
my ($self, $task_id) = #_;
return $self->search({
'task.task_id' => $task_id,
},
{
join => { 'user_task' => 'task' },
prefetch => { 'user_task' => 'task' },
},
);
}
PS: Sorry, I didn't see that Ashley already answered
PS2: Before the last line "})" should be ")" only (fixed it)
This is some User resultset code, unchanged, from the XUL (which is
no longer supported by any browsers) slideshow of the DBIC
master class (.xul resource). I highly recommend downloading the slides, which read fine as plain text, to review. They were a great eye-opener for resultsets for me.
You'll have to adjust the result source names to match your own but
this should be what you want and a bit more flexible to boot with the
_role_to_id which allows you to pass role objects or ids.
sub with_role {
my ($self, $role) = #_;
$self->search({
'role_links.role_id' => $role->id
},
{ join => 'role_links' }
);
}
sub _role_to_id {
my ($self, $role) = #_;
return blessed($role) ? $role->id : $role;
}
sub with_any_role {
my ($self, #roles) = #_;
$self->search({
'role_links.role_id' => {
-in => [
map { $self->_role_to_id($_) } #roles
]
}
},
{ join => 'role_links' }
);
}