I'm currently working on a large implementation of Class::DBI for an existing database structure, and am running into a problem with clearing the cache from Class::DBI. This is a mod_perl implementation, so an instance of a class can be quite old between times that it is accessed.
From the man pages I found two options:
Music::DBI->clear_object_index();
And:
Music::Artist->purge_object_index_every(2000);
Now, when I add clear_object_index() to the DESTROY method, it seems to run, but doesn't actually empty the cache. I am able to manually change the database, re-run the request, and it is still the old version.
purge_object_index_every says that it clears the index every n requests. Setting this to "1" or "0", seems to clear the index... sometimes. I'd expect one of those two to work, but for some reason it doesn't do it every time. More like 1 in 5 times.
Any suggestions for clearing this out?
The "common problems" page on the Class::DBI wiki has a section on this subject. The simplest solution is to disable the live object index entirely using:
$Class::DBI::Weaken_Is_Available = 0;
$obj->dbi_commit(); may be what you are looking for if you have uncompleted transactions. However, this is not very likely the case, as it tends to complete any lingering transactions automatically on destruction.
When you do this:
Music::Artist->purge_object_index_every(2000);
You are telling it to examine the object cache every 2000 object loads and remove any dead references to conserve memory use. I don't think that is what you want at all.
Furthermore,
Music::DBI->clear_object_index();
Removes all objects form the live object index. I don't know how this would help at all; it's not flushing them to disk, really.
It sounds like what you are trying to do should work just fine the way you have it, but there may be a problem with your SQL or elsewhere that is preventing the INSERT or UPDATE from working. Are you doing error checking for each database query as the perldoc suggests? Perhaps you can begin there or in your database error logs, watching the queries to see why they aren't being completed or if they ever arrive.
Hope this helps!
I've used remove_from_object_index successfully in the past, so that when a page is called that modifies the database, it always explicitly reset that object in the cache as part of the confirmation page.
I should note that Class::DBI is deprecated and you should port your code to DBIx::Class instead.
Related
I know that this feature is not implemented by mongodb. I am thinking what can be best way to achieve that.
Using caching service ? The approach will work but there is one problem, when the query You watch on is too big, like whole collection, You will never have the first before value, because You start caching only when first change appear on watch.
Service started watching.
Received object id 1, no cache for previous change, caching value.
Received object id 1, cache for previous change, can do comparison, caching value
I see another problem here, if I have 2 watchers which could potentially receive information about the same object, this will cause sync problems, as one process may update cache and second will already receive wrong data, hm. I mean the second process could be in a situation that cached previous value is already the same as the one in mongodb change stream.
I was thinking as well about mongodb replicas, but not sure if the problem can be solved with it.
Best,
Igor
I am building a large private (ie used behind a firewall) PWA and wondering how to improve the diagnostics if/when my users hit issues. I already have an error manager which uses navigator.sendBeacon to log the error on the server, but that lacks detailed info of what led up to that point.
A thought I had was to liberally mark the code with performance.mark() statements and on an error dump the performance buffer to the server. It would give me an ordered list of recent activity.
However it only makes sense to do this if the browser throws away the oldest entry to make way for the new when the internal buffer is full. However all the documentation I found with a google search doesn’t mention it. I am aware I can get an event when it is full and could use that to copy and clear it but I can find no words on what happens if I ignore the event. Neither can I find a typical size. I don’t want to keep getting entries filling up the entire computer memory either
Can anyone give me a definite answer
Edit: The more I look into this, the more confused I become. It appears that you can control the size of the resourceTimingBuffer but "resource" performance entries are related to fetch and not Performance.mark(). I can't find any statement on limitations.
There are no meaningful limitations I could find. I did a test and generated more than 4000 marks and they were all there and the memory usage did not increase in any measurable way.
In the context of a REST API, I've been using DBIx::Class to create related rows, i.e.
POST /artist
{ "name":"Bob Marley", "cds":[{"title":"Exodus"}] }
That ultimately calls $artist->new($data)->insert() which creates an Artist AND creates the related row(s) in the CD table. It then send back the resulting object to the user (via DBIC::ResultClass::HashRefInflator), including the created primary keys and default values. The problem arises when the user makes changes to those objects and send them back to the API again:
POST /artist/7
{ "name":"Robert Nesta Marley", "artistid":"7",
"cds":[{"title":"Exodus", "cdid":"1", "artistid":"7", "year":"1977"}] }
Now what? From what I can see from testing, DBIC::Row::update doesn't deal with changes in related rows, so in this case the name change would work, but the update to the CD's year would not. DBIC::ResultSet::update_or_create just calls DBIC::Row::update. So I went searching for some alternatives, and they do seem to exist, i.e. DBIC::ResultSet::RecursiveUpdate, but it hasn't been updated in 4 years, and references to it seem to suggest it should/would be folded into DBIC. Did that happen?
Am I missing something easier?
Obviously, I can handle this particular situation, but I have many APIs to write and it would be easier to handle it generically for all of them. I'm certainly tempted to use RecursiveUpdate, but wary due to its apparent abandonment.
Ribasushi promised to core the RecursiveUpdate feature if someone steps up and migrates its test suite to the DBIC schema and comes up with a sane API because he didn't like how RU handles this currently.
Catalyst::Controller::DBIC::API is using RU under the hood as well as HTML::Formhandler::Model::DBIC both here in production since years without any issues.
So currently RecursiveUpdate is the way to go.
I've just discovered an assumption in my use of SimpleDB. I suspect it's safe but would like other opinions since the docs don't seem to cover it.
So say Process 1 stores an item with x attributes. When Process 2 tries to access said item (without consistent read) & finds it, is it guaranteed to have all the attributes stored by Process 1?
I'm excluding the possibility that another process could have changed the data.
I also know that Process 2 has no guarantee of seeing the item unless consistent read is used, I'm just talking about the point when it does eventually see it.
I guess the question is, once I can get an item & am not changing it anywhere else can I assume it has an ad-hoc fixed schema and access all my expected attributes without checking they actually exist?
I don't want to be in a situation where I need to keep requesting items until they have all the attributes I need to use them.
Thanks.
Although Amazon makes no such guarantees in the documentation the current implementation of their eventual consistency guarantees that you'll see all the properties stored by Process 1 or none of them.
See this thread over at the AWS forums and more specifically this answer by an Amazon employee confirming the behavior (emphasis mine).
I don't think we make that guarantee in the documentation, but the
current implementation treats each Put request as a bundle. It won't
split the request up and apply the operations piecemeal. You'll get
either step-1 responses or step-2 responses until eventual consistency
shakes out and leaves you with step-2 responses.
While this is undocumented behavior I suspect quite a few SimpleDB users are relying on it now and as such Amazon won't be likely to change it anytime soon, but that's ju my guess.
I have a set of rather complex ORM modules that inherit from Class::DBI. Since the data changes quite infrequently, I am considering using a Caching/Memoization layer on top of this to speed things up. I found a module: Class::DBI::Cacheable but no rating or any reviews on RT. I would appreciate hearing from people who have used this or any other Class::DBI caching scheme.
Thanks a ton.
I too have rolled my own ORM plenty of times I hate to say! Caching/Memoization is pretty easy if all your fetches happen through a single api (or subclasses thereof).
For any fetch based on a unique key you can just cache based on a concatenation of the keys. A naive approach might be:
my %_cache;
sub get_object_from_db {
my ($self, $table, %table_lookup_key) = #_;
# concatenate a unique key for this object
my $cache_key = join('|', map { "$_|$table_lookup_key{$_}" }
sort keys %table_lookup_key
return $_cache{$cache_key}
if exists $_cache{$cache_key};
# otherwise get the object from the db and cache it in the hash
# before returning
}
Instead of a hash, you can use the Cache:: suite of modules on CPAN to implement time and memory limits in your cache.
If you're going to cache for some time you might want to think about a way to expire objects in the cache. If for instance all your updates also go through the ORM you can clear (or update) the cache entry in your update() ORM method.
A final point to think carefully about - you're returning the same object each time which has implications. If, eg., one piece of code retrieves an object and updates a value but doesn't commit that change to the db, all other code retrieving that object will see that change. This can be very useful if you're stringing together a series of operations - they can all update the object and then you can commit it at the end - but it may not be what you intend. I usually set a flag on the object when it is fresh from the database and then in your setter method invalidate that flag if the object is updated - that way you can always check that flag if you really want a fresh object.
On a few occasions we've rolled our own, but we limited it to special cases where profiling indicated we needed a boost (for example large joins). Since our applications typically use a custom abstraction layer (akin to a home-grown ORM) on top of the DB access, that's where we implemented the caching. We achieved good results that we were satisfied with and it didn't take a whole lot of effort. Of course, since we weren't using a CPAN ORM, we didn't really have any choice about using a CPAN caching module, either.
It was strictly case-by-case and opt-in. Whether you end up using a CPAN solution or rolling your own, it's probably a good idea to restrict it to cases where profiling indicates you need help, and make sure that it's opt-in so your caching doesn't undermine your application in subtle ways by being active when you didn't expect it.
I have used memcached before to cache objects, but not with Class::DBI (ORM makes me feel dirty).