I'm working on some reporting/analytics for a site. The goal is to filter to a group of #loans that I query from the DB and run analytics against. The desire is to use the DB to grab the filtered #loans, run metrics against those records, and then return the metrics to display on the page.
The relationships are not very complex but what I'm doing feels like it is:
Model relationships
# loan.rb
belongs_to :loan_category
belongs_to :loan_type
belongs_to :partner
# loan_category.rb
has_many :loans
# loan_type.rb
has_many :loans
# partner.rb
has_many :loans
belongs_to :company
# company.rb
has_many :partners
Filter form
I'm treating reports as a resource since users can save filters as a saved_report.
A user filters by 2 given criteria:
what type of report they want (partner, company, loan_category, loan_type), accomplished by selecting ONLY one of the four (ie, filter by parter ID#3, or loan_type ID#6, but never more than one)
date range (from, to)
At first it seems fairly trivial, until I start trying to implement this. The issue is that the various situations must be accounted for:
need an overview report that ONLY filters by date range (basically a dashboard page), so that if a user does NOT select any report type and only provides dates, you display the overview report
need to filter by provided type (partner, company, loan_category, loan_type) AND date range
need to allow users to save a filter as a saved_report to quickly filter by the criteria when they select that report
I'm trying to accomplish this in a pretty ugly fashion. These examples will demonstrate what I mean:
/reports => overview report, default to current month
/reports?from=2014-01-01 => overview report, from given date to current day
/reports/3 => a saved_report, which COULD filter by any of the four (partner, company, loan_category, loan_type) and a date range
/reports/partner?partner_id=3 => using params[:id] in a hackish fashion, my controller code knows I want to filter by a given partner
/reports/loan_type?loan_type_id=7&from=2014-01-01&to=2014-01-31 => again using params[:id] I know to filter by a given loan_type with a specified date range
This leads to some very ugly controller code:
Reports controller
# always overview report; just filter by specified date range
def index
# Dummy code to show what I'm doing...
#loans = Loan.where(created_at >= params[:from]).where(created_at <= params[:to])
# ...use #loans to calculate metrics for report...
end
def show
is_saved_report = false
#saved_report = nil
# feels very "hackish" to do this...
case params[:id]
when "partner"
#loans = Loan.where(partner_id: params[:partner_id])
when "company"
#loans = Loan.includes(:partner).where(partner.company_id: params[:company_id])
when "loan_category"
#loans = Loan.where(loan_category_id: params[:loan_category_id])
when "loan_type"
#loans = Loan.where(loan_type_id: params[:loan_type_id])
else
# if none of the 4 "keywords" was used, assume it's a saved_report ID
is_saved_report = true
#saved_report = SavedReport.find(params[:id])
# grab filters from #saved_report and use them to run ANOTHER case statement...
end
# add date range filters
if is_saved_report == true
#loans.where(created_at >= #saved_report.from).where(created_at <= #saved_report.to)
else
#loans.where(created_at >= params[:from]).where(created_at <= params[:to])
end
# at this point, should have filtered list of #loans to run metrics on
#total_loans = #loans.size
# ...have to have more conditionals to display metrics for specific type of report...
end
As you can see, the code gets VERY UGLY very quickly. There has to be a better way of approaching this. I simply cannot think of a cleaner solution. I've been working on this problem so long now that I'm not thinking clearly on it.
I'm open to any suggestions or thoughts on this situation. Something that at first seemed trivial is becoming a bit of a nightmare.
Related
I am newbie to drools. Need understanding how to cascade filtered data from one condition to next one as shown below.
rule "rulename"
when
$var1 : MainObj(filter1)
$var2 : SubObj() from $var1.getSubObj // need all SubObj without any filter here for then clause.
$var3 : SubObj(conditions) from $var2 // Need a subset of SubObj for another then clause
then
$var2.doSomething()
$var3.doVar3Stuff()
end
How can I achieve that?
Continuation of my original post..
i have below rule computing statistics of a rule execution as below.
rule "myrule"
when
$a:wrapper(isActive("ruleName") && dataTypeCode.equals("typeCodeinDB"))
$total : Number() from accumulate($entity: MyObj(ObjAttribute=='testValue') from $a.getMyObj() , count( $entity ))
$filter: MyObj(ObjAttribute=="testValue" && ObjAttribute2.equals("ValidValue")) from $a.getMyObj()
then
$filter.addToResults($filter.getRuleConstituents(),1,drools.getRule().getName(),null); // Add data for insertion in DB at later stage
$filter.addPayloadExecution($a.dataTypeCode(),$a.getMyObj().toString(),$a.getMyObj().size()); //Total Count of MyObj
$filter.addRuleExecutions(drools.getRule().getName(), $total); // anotehr slice of data
end
Here.. i am computing two stats.. Total count of MYObj(), apply some filter for data validation and compute count of valid list ($filter is valid data)
When I run that code I am getting error as below
Exception executing consequence for rule "MyRule" in moneta.rules: [Error: $filter.addRuleExecutions(drools.getRule().getName(), $total): null]
[Near : {... $filter.addToResults($filter.get ....}]
^
[Line: 1, Column: 1]
Need advice on resolving this.
I can't tell if you're trying to change the data in one rule and make it visible in another, or if you're trying to do a single complex condition in a single rule. So I'll outline how both work.
Complex conditions
The pseudo-code in your rule indicates that you have a main object with a filter and a collection of sub-objects. You need to get a list of all sub-objects without a filter, and also to get a list of sub-objects that meet some sort of condition.
To collect these sublists, we have two available operations: accumulate and collect. Since you didn't give a real example, just some psuedo-code, I can't say for sure which is the correct one for you to use. The example I present below uses collect because it is appropriate for this use case. For more information about these operations, refer to the Drools documentation. I've linked to the section on operators; scroll down to see the details for collect and accumulate.
So imagine I have an app that is modeling a school, and my KindergartenClass object has a List of Student objects. In this case, KindergartenClass is the main object, and Student is the sub-object. My Students have names, ages, and an Allergies object that indicates food allergies. Students without Allergies have no allergy object. This is going to be the filter for the example rule -- finding the students without allergies.
The rule would then look like:
rule "EXAMPLE"
when
KindergartenClass( $students: students ) // $var1
// Find all students with no allergies, $var2
$var2: List() from collect( Student( allergies == null ) from $students )
// Find all students who are older than 5, $var3
$var3: List() from collect( Student( age > 5 ) from $students )
then
...
end
Obviously your right-hand-side would have to be adjusted because 'List' has no doSomething() and doVar3Stuff() methods. But without an actual example rule for what you're attempting I can't be more specific here.
You could get the individual students from the lists like this:
$studentWithoutAllergies: Student() from $var2
$studentOlderThan5: Student() from $var3
... but then your then-clause would trigger once for every single $studentWithoutAllergies and $studentOlderThan5. Most people don't want a cartesian product all elements in two lists and then having their consequences firing for each of those products.
Of course, you also talk about "cascading", though it's rather unclear what you mean. If you mean that you want $var3 to be a subset of what you found in $var2, then it's as simple as changing the from clause. So in the example I've been working with, if you actually want all students older than 5 who have no allergies, you can simply change the when clause to be:
// Find all students with no allergies, $var2
$var2: List() from collect( Student( allergies == null ) from $students )
// Find the subset of the non-allergy students who are older than 5, $var3
$var3: List() from collect( Student( age > 5 ) from $var2 )
Changing the data that triggers the rule
There are two keywords available for changing the data available to the left hand side (when clause): update and insert. Which one you use depends on what you're trying to do, and come with very different considerations in terms of performance.
When you ask Drools to fire a set of rules, it first goes through all of the rules and determines if the rule fires, given the data you've inputted. Basically it just goes through, in order, all of the when clauses and decides if the rule is activated or not. Then, once it has the sub-set of rules, in order, it goes through, one by one, executing the right hand side.
If you call update in your right hand side (then clause), then this process repeats: all rules are evaluated again, from the top, including the rule you just fired. If you call insert in your then clause, then the new data is put into working memory, and Drools re-evaluates all of the remaining rules in its execution list. (So, for example, if Drools had decided that rules A, B, C, and D met their when criteria, and B inserts a new fact into working memory, then C and D would be re-evaluated to make sure that they were still valid.)
This is how you'd use them:
rule "Example rule with Update"
when
$input: MyAwesomeInputObject(someValue == 0)
then
$input.setSomeValue(99);
$input.doSomeStuff();
update($input);
end
rule "Example rule with Insert"
when
Person(name == "Bob")
not( Age() )
then
Age age = new Age(18);
insert(age); // there is now an Age object in working memory
end
I'm not 100% familiar with your use case, but I'm going to assume you want to fire all of the filtering rules a second time after your initial rule fires, with the updated data. So I'd do something like this:
rule "rulename"
when
$var1 : MainObj(filter1)
$var2 : SubObj() from $var1.getSubObj // need all SubObj without any filter here for then clause.
$var3 : SubObj(conditions) from $var2 // Need a subset of SubObj for another then clause
then
$var2.doSomething();
$var3.doVar3Stuff();
update($var1); // update the var1 in working memory
end
I have no idea what your when clause is trying to do, since it doesn't appear to match the comments and the syntax is completely wonky. So only the right hand side (then clause) has been adjusted. I have added update($var1) so it will refire the rules with the new instance of $var1 that should have your changes in it.
Of course, once you start re-firing all rules, you run the risk of having rules loop. If this does happen, you will see spikes of CPU and apparent thread deadlocks (this is useful to alert on.) Note that there is a no-loop property, but it won't help when you're calling update.
I need to generate a business report using perl + Template Toookit and LaTeX.
Things are working really well, but I am struggling with the problem of having breaks (for example page breaks, or special headers) and subtotals whenever a field changes.
So, for example, every time the field "category" changes, I'd need to have a total of sales for that category, and a header showing that another category listing is starting; and then do the same when the field "group" - with the added interest that "group" is made up of categories, so the two things should nest.
I guess anyone that has built reports with Microsoft Access (or probably any other business reporting application) should be familiar with the problem.
Ideally this would be solved at a meta-level, so I don't have to rebuild the code every time, but only to specify what fields should generate breaks or subtotals.
I am (voluntarily) constrained to LaTeX and TT: LaTeX because of the control it gives over typography, and the possibility of generating custom graphics, and TT (or anything else that works in perl) because of learning curves.
There's no built-in subtotaling feature in TT, but you could possibly put your data into a Data::Table object, that would give you some ability to handle subtotaling at the 'meta' level, as you say.
Depending on the number of columns involved though, it might be just as simple to create local hashes to maintain running totals: NB: untested, example code only
[%-
MACRO printrow(rowtype, line) BLOCK;
# however you print the row as LaTeX
# rowtype is 'row', 'subtotal' or 'grandtotal' for formatting purposes
END;
SET sumcols = [ 'col3', 'col4', 'col5' ]; # cols to be accumulated
SET s_tot = {}; SET g_tot = {};
FOREACH i IN sumcols;
SET s_tot.$i = 0; # initialise
SET g_tot.$i = 0;
END;
FOREACH row IN data;
IF s_tot.col2 AND s_tot.col2 <> row.col2; # start of new group
printrow('subtotal', s_tot);
FOREACH i IN sumcols;
SET s_tot.$i = 0; #re-init
END;
END;
printrow('row', row);
SET s_tot.col2 = row.col2; # keep track of group level
FOREACH i IN sumcols;
SET s_tot.$i = s_tot.$i + row.$i;
SET g_tot.$i = g_tot.$i + row.$i;
END;
END;
printrow('grandtotal', g_tot);
-%]
Of course, if you have more than a couple of grouping levels, this can get quite messy. You could make s_tot an array of hashes to manage each level, to avoid hard-coding the levels. That's left as an exercise for the reader, as they say.
During dehydration I create a custom value:
def dehydrate(self, bundle):
bundle.data['custom_field'] = ["add lots of stuff and return an int"]
return bundle
that I would like to filter on.
/?format=json&custom_field__gt=0...
however I get an error that the "[custom_field] field has no 'attribute' for searching with."
Maybe I'm misunderstanding custom filters, but in both build_filters and apply_filters I can't seem to get access to my custom field to filter on it. On the examples I've seen, it seems like I'd have to redo all the work done in dehydrate in build_filters, e.g.
for all the items:
item['custom_field'] = ["add lots of stuff and return an int"]
filter on item and add to pk_list
orm_filters["pk__in"] = [i.pk for i in pk_list]
which seems wrong, as I'm doing the work twice. What am I missing?
The problem is that dehydration is "per object" by design, while filters are per object_list. That's why you will have to filter it manually and redo work in dehydration.
You can imagine it like this:
# Whole table
[obj, obj1, obj2, obj3, obj4, obj5, obj5]
# filter operations
[...]
# After filtering
[obj1, obj3, obj6]
# Returning
[dehydrate(obj), dehydrate(obj3), dehydrate(obj5)]
In addition you can imagine if you fetch by filtering and you get let say 100 objects. It would be quite inefficient to trigger dehydrate on whole table for instance 100000 records.
And maybe creating new column in model could be candidate solution if you plan to use a lot of filters, ordering etc. I guess its kind of statistic information in this field so if not new column then maybe django aggregation could ease your pain a little.
I am converting a Rails app from using acts_as_solr to sunspot.
The app uses the field search capability in solr that was exposed in acts_as_solr. You could give it a query string like this:
title:"The thing to search"
and it would search for that string in the title field.
In converting to sunspot I am parsing out field specific portions of the query string and I need to dynamically generate the search block. Something like this:
Sunspot.search(table_clazz) do
keywords(first_string, :fields => :title)
keywords(second_string, :fields => :description)
...
paginate(:page => page, :per_page => per_page)
end
This is complicated by also needing to do duration (seconds, integer) ranges and negation if the query requires it.
On the current system users can search for something in the title, excluding records with something else in another field and scoping by duration.
In a nutshell, how do I generate these blocks dynamically?
I recently did this kind of thing using instance_eval to evaluate procs (created elsewhere) in the context of the Sunspot search block.
The advantage is that these procs can be created anywhere in your application yet you can write them with the same syntax as if you were inside a sunspot search block.
Here's a quick example to get you started for your particular case:
def build_sunspot_query(conditions)
condition_procs = conditions.map{|c| build_condition c}
Sunspot.search(table_clazz) do
condition_procs.each{|c| instance_eval &c}
paginate(:page => page, :per_page => per_page)
end
end
def build_condition(condition)
Proc.new do
# write this code as if it was inside the sunspot search block
keywords condition['words'], :fields => condition[:field].to_sym
end
end
conditions = [{words: "tasty pizza", field: "title"},
{words: "cheap", field: "description"}]
build_sunspot_query conditions
By the way, if you need to, you can even instance_eval a proc inside of another proc (in my case I composed arbitrarily-nested 'and'/'or' conditions).
Sunspot provides a method called Sunspot.new_search which lets you build the search conditions incrementally and execute it on demand.
An example provided by the Sunspot's source code:
search = Sunspot.new_search do
with(:blog_id, 1)
end
search.build do
keywords('some keywords')
end
search.build do
order_by(:published_at, :desc)
end
search.execute
# This is equivalent to:
Sunspot.search do
with(:blog_id, 1)
keywords('some keywords')
order_by(:published_at, :desc)
end
With this flexibility, you should be able to build your query dynamically. Also, you can extract common conditions to a method, like so:
def blog_facets
lambda { |s|
s.facet(:published_year)
s.facet(:author)
}
end
search = Sunspot.new_search(Blog)
search.build(&blog_facets)
search.execute
I have solved this myself. The solution I used was to compiled the required scopes as strings, concatenate them, and then eval them inside the search block.
This required a separate query builder library that interrogates the solr indexes to ensure that a scope is not created for a non existent index field.
The code is very specific to my project, and too long to post in full, but this is what I do:
1. Split the search terms
this gives me an array of the terms or terms plus fields:
['field:term', 'non field terms']
2. This is passed to the query builder.
The builder converts the array to scopes, based on what indexes are available. This method is an example that takes the model class, field and value and returns the scope if the field is indexed.
def convert_text_query_to_search_scope(model_clazz, field, value)
if field_is_indexed?(model_clazz, field)
escaped_value = value.gsub(/'/, "\\\\'")
"keywords('#{escaped_value}', :fields => [:#{field}])"
else
""
end
end
3. Join all the scopes
The generated scopes are joined join("\n") and that is evaled.
This approach allows the user to selected the models they want to search, and optionally to do field specific searching. The system will then only search the models with any specified fields (or common fields), ignoring the rest.
The method to check if the field is indexed is:
# based on http://blog.locomotivellc.com/post/6321969631/sunspot-introspection
def field_is_indexed?(model_clazz, field)
# first part returns an array of all indexed fields - text and other types - plus ':class'
Sunspot::Setup.for(model_clazz).all_field_factories.map(&:name).include?(field.to_sym)
end
And if anyone needs it, a check for sortability:
def field_is_sortable?(classes_to_check, field)
if field.present?
classes_to_check.each do |table_clazz|
return false if ! Sunspot::Setup.for(table_clazz).field_factories.map(&:name).include?(field.to_sym)
end
return true
end
false
end
I have been trying to work out what is the best way to search for gather all of the documents in a database that have a certain date.
Originally I was trying to use FTsearch or search to move through a document collection, but I changed over to processing a view and associated documents.
My first question is what is the easiest way to spin through a set of documents and find if a date stored in the documents is greater than or less than a specified date?
So, to continue working I implemented the following code.
If (doc.creationDate(0) > cdat(parm1))
And (doc.creationDate(0) < CDat(parm2)) then
...
end if
but the results are off
Included! Date:3/12/10 11:07:08 P1:3/1/10 P2: 3/5/10
Included! Date:3/13/10 9:15:09 P1:3/1/10 P2: 3/5/10
Included! Date:3/17/10 16:22:07P1:3/1/10 P2: 3/5/10
You can see that the date stored in the doc is not between P1 and P2. BUT! it does limit the documents with a date less than P1 correctly. So I won't get a result for a document with a date less than 3/1/10
If there isn't a better way than the if statement, can someone help me understand why the two examples from above are included?
Hi you can try something like this:
searchStr = {(Form = "yourForm" & ((#Created > [} & parm1 & {]) & (#Created < [} & parm2 & {])))}
Set docCollection = currentDB.Search(searchStr, Nothing, 0)
If(docCollection.Count > 0)Then
'do your stuff with the collection returned
End If
Carlos' response is pretty good.
If you have a lot of documents, you can also use a full-text search which will is much faster. The method call is very similar (db.ftsearch(), online help can be found here).
The standard DB Search method operates in the same way as view index updates, so it can get a little slow if you have thousands of documents to search through.
Just make sure you enable full text index for your database in the database properties, (last tab).
Syntax on this approach is very similar, this link provides a good reference for FTsearch. Using Carlos' syntax, you can substitute FTSearch and searchStr assignment for faster searching.