DBIx::Class joins generate a large number of select statements - perl

I've noticed that when I do a join in a select in DBIx::Class, I get a ton of SELECT statements as a result. For instance, I have a query that looks like:
my $testSetRs = $db->resultset('Testset')->search(
{%searchCriteria},
{
prefetch => [
{
'ax_group_testsets' => {'ax_group' => 'ax_user_groups'},
},
{'ax_user_testsets' => 'ax_user'},
],
'+columns' => [
{
'me.ax_group_testsets.ax_group_id' => 'ax_group_testsets.ax_group_id',
'me.ax_group_testsets.ax_group.public' => 'ax_group.public',
'me.ax_group_testsets.ax_group.name' => 'ax_group.name',
},
],
collapse => 1,
group_by => ['me.ts_id', 'me.ts_name'],
cache => 1,
}
);
generates:
SELECT me.ts_id, me.ts_name
FROM testset me
LEFT JOIN ax_group_testset ax_group_testsets ON ax_group_testsets.ts_id = me.ts_id
LEFT JOIN ax_group ax_group ON ax_group.ax_group_id = ax_group_testsets.ax_group_id
LEFT JOIN ax_user_group ax_user_groups ON ax_user_groups.ax_group_id = ax_group.ax_group_id
LEFT JOIN ax_user_testset ax_user_testsets ON ax_user_testsets.ts_id = me.ts_id
LEFT JOIN ax_user ax_user ON ax_user.ax_user_id = ax_user_testsets.ax_user_id
WHERE ( ( ax_user.ax_user_id = ? OR ax_user_groups.ax_user_id = ? ) )
GROUP BY me.ts_id, me.ts_name: '349', '349'
SELECT me.ax_group_id, ax_group.public, ax_group.name FROM ax_group_testset me JOIN ax_group ax_group ON ax_group.ax_group_id = me.ax_group_id WHERE ( me.ts_id = ? ): '4476'
SELECT me.ax_group_id, ax_group.public, ax_group.name FROM ax_group_testset me JOIN ax_group ax_group ON ax_group.ax_group_id = me.ax_group_id WHERE ( me.ts_id = ? ): '4566'
SELECT me.ax_group_id, ax_group.public, ax_group.name FROM ax_group_testset me JOIN ax_group ax_group ON ax_group.ax_group_id = me.ax_group_id WHERE ( me.ts_id = ? ): '4701'
etc.
The first SELECT, I want. Its the ones that come after that bother me. There is one for every group (over 40). This happens on all of my joins and can jump into the hundreds with the right query. Why are so many queries generated and what can I do about it?
I'm running Perl ActiveState 5.16 and DBIx .08250.

Your answer lies in the prefetch atribute as described in the DBIx::Class::Manual::Joining and DBIx::Class::ResultSet docs.
Add
prefetch => {'ax_group_testsets' => {'ax_group' => 'ax_user_groups'} },
and, assuming that you want all three related record columns:
'+columns' => [
{
'me.ax_group_testsets.ax_group_id' => 'ax_group_testsets.ax_group_id',
'me.ax_group_testsets.ax_group.public' => 'ax_group.public',
'me.ax_group_testsets.ax_group.name' => 'ax_group.name',
},
],
collapse => 1,
to the query attributes hash.

Related

Postgres json aggregate function calls cannot be nested

I have this select statement, I want to export them to json array with an another nested json agregate function but postgres says that "aggregate function calls cannot be nested", I can not figure aout how i can do this.
select json_agg(json_build_object(
'plan_number', plan.id,
'plan_carrier_code', carrier_plan.carrier_code,
'plan_name', plan.plan_name,
'plan_mac', mac.mac_name,
'plan_termination_date', plan.termination_date,
'plan_mod_start_date', plan.mod_start_date,
'plan_mod_user', plan.mod_user,
'plan_opt_brandcode_g_on_mn_as_generic_copay', plan.opt_brandcode_g_on_mn_as_generic_copay,
'plan_opt_exclude_daw2_from_ded_calculations', plan.opt_exclude_daw2_from_ded_calculations,
'plan_opt_exclude_daw2_from_oop_calculations', plan.opt_exclude_daw2_from_oop_calculations,
'plan_opt_limit_patient_pay_to_pay', plan.opt_limit_patient_pay_to_pay,
'plan_opt_only_pay_primary_claims', plan.opt_only_pay_primary_claims,
'plan_opt_allow_discontinued_drugs', plan.opt_allow_discontinued_drugs,
'plan_opt_allow_negative_payments_to_pharmacy', plan.opt_allow_negative_payments_to_pharmacy,
'plan_opt_process_y_drugs_as_preferred', plan.opt_process_y_drugs_as_preferred,
'plan_opt_reject_otc', plan.opt_reject_otc,
'plan_opt_reject_repackaged_drugs', plan.opt_reject_repackaged_drugs,
'plan_opt_test_only', plan.opt_test_only,
'plan_opt_cost_effective_pricing', plan.opt_cost_effective_pricing,
'plan_opt_original_mony_for_copay', plan.opt_original_mony_for_copay,
'plan_daw_differential', json_agg(json_build_object(
'plan_daw_differential_daw_code', plan_daw_differential.daw_code,
'plan_daw_differential_claim_type', plan_daw_differential.claim_type,
'plan_daw_differential_updated_at', plan_daw_differential.updated_at,
'plan_daw_differential_updated_by', plan_daw_differential.updated_by
))
)) as plan
from splan.groups_plan_list gpl
left join splan.plan plan on plan.id = gpl.plan_id
left join splan.carrier carrier_plan on carrier_plan.id = plan.carrier_id
left join splan.plan_daw_differential plan_daw_differential on plan_daw_differential.parent_id = plan.id
left join sdrug.mac mac on mac.id = plan.mac_id
where gpl.parent_id = 69;
but it throws me an error that said "aggregate function calls cannot be nested"
the expected result can be:
[
{
"plan_number":1,
"plan_carrier_code":"lltest",
"plan_daw_differential":[
{
"plan_daw_differential_daw_code":"0505",
"plan_daw_differential_claim_type":"02"
},
{
"plan_daw_differential_daw_code":"0505",
"plan_daw_differential_claim_type":"02"
}
]
}
]
Options:
use subquery (or join lateral) to evaluate plan_daw_differential element for every plan element.
using GROUP BY plan... gpl... carrier_plan... mac...
Because the solutions differ in execution plan you will have to pick one that fits your requirements the most.
I would go with first option in most cases. It is more compact easier to write and maintain. The only reason to pick the second would be requirement to filter the plan using data from plan_daw_differential table.
Examples:
Subquery:
select json_agg(json_build_object(
'plan_number', plan.id,
'plan_carrier_code', carrier_plan.carrier_code,
'plan_name', plan.plan_name,
'plan_mac', mac.mac_name,
'plan_termination_date', plan.termination_date,
'plan_mod_start_date', plan.mod_start_date,
'plan_mod_user', plan.mod_user,
'plan_opt_brandcode_g_on_mn_as_generic_copay', plan.opt_brandcode_g_on_mn_as_generic_copay,
'plan_opt_exclude_daw2_from_ded_calculations', plan.opt_exclude_daw2_from_ded_calculations,
'plan_opt_exclude_daw2_from_oop_calculations', plan.opt_exclude_daw2_from_oop_calculations,
'plan_opt_limit_patient_pay_to_pay', plan.opt_limit_patient_pay_to_pay,
'plan_opt_only_pay_primary_claims', plan.opt_only_pay_primary_claims,
'plan_opt_allow_discontinued_drugs', plan.opt_allow_discontinued_drugs,
'plan_opt_allow_negative_payments_to_pharmacy', plan.opt_allow_negative_payments_to_pharmacy,
'plan_opt_process_y_drugs_as_preferred', plan.opt_process_y_drugs_as_preferred,
'plan_opt_reject_otc', plan.opt_reject_otc,
'plan_opt_reject_repackaged_drugs', plan.opt_reject_repackaged_drugs,
'plan_opt_test_only', plan.opt_test_only,
'plan_opt_cost_effective_pricing', plan.opt_cost_effective_pricing,
'plan_opt_original_mony_for_copay', plan.opt_original_mony_for_copay,
'plan_daw_differential', (
SELECT
json_agg(json_build_object(
'plan_daw_differential_daw_code', plan_daw_differential.daw_code,
'plan_daw_differential_claim_type', plan_daw_differential.claim_type,
'plan_daw_differential_updated_at', plan_daw_differential.updated_at,
'plan_daw_differential_updated_by', plan_daw_differential.updated_by
))
FROM
splan.plan_daw_differential plan_daw_differential
WHERE
plan_daw_differential.parent_id = plan.id )
)) as plan
from
splan.groups_plan_list gpl
left join splan.plan plan on plan.id = gpl.plan_id
left join splan.carrier carrier_plan on carrier_plan.id = plan.carrier_id
left join sdrug.mac mac on mac.id = plan.mac_id
where
gpl.parent_id = 69;
Grouping:
select json_agg(json_build_object(
'plan_number', plan_number,
'plan_carrier_code', plan_carrier_code,
'plan_name', plan_name,
'plan_mac', plan_mac,
'plan_termination_date', plan_termination_date,
'plan_mod_start_date', plan_mod_start_date,
'plan_mod_user', plan_mod_user,
'plan_opt_brandcode_g_on_mn_as_generic_copay', plan_opt_brandcode_g_on_mn_as_generic_copay,
'plan_opt_exclude_daw2_from_ded_calculations', plan_opt_exclude_daw2_from_ded_calculations,
'plan_opt_exclude_daw2_from_oop_calculations', plan_opt_exclude_daw2_from_oop_calculations,
'plan_opt_limit_patient_pay_to_pay', plan_opt_limit_patient_pay_to_pay,
'plan_opt_only_pay_primary_claims', plan_opt_only_pay_primary_claims,
'plan_opt_allow_discontinued_drugs', plan_opt_allow_discontinued_drugs,
'plan_opt_allow_negative_payments_to_pharmacy', plan_opt_allow_negative_payments_to_pharmacy,
'plan_opt_process_y_drugs_as_preferred', plan_opt_process_y_drugs_as_preferred,
'plan_opt_reject_otc', plan_opt_reject_otc,
'plan_opt_reject_repackaged_drugs', plan_opt_reject_repackaged_drugs,
'plan_opt_test_only', plan_opt_test_only,
'plan_opt_cost_effective_pricing', plan_opt_cost_effective_pricing,
'plan_opt_original_mony_for_copay', plan_opt_original_mony_for_copay,
'plan_daw_differential', draw_differential_arr
)) as plan
FROM (
SELECT
plan.id AS plan_number,
carrier_plan.carrier_code AS plan_carrier_code,
plan.plan_name AS plan_name,
mac.mac_name AS plan_mac,
plan.termination_date AS plan_termination_date,
plan.mod_start_date AS plan_mod_start_date,
plan.mod_user AS plan_mod_user,
plan.opt_brandcode_g_on_mn_as_generic_copay AS plan_opt_brandcode_g_on_mn_as_generic_copay,
plan.opt_exclude_daw2_from_ded_calculations AS plan_opt_exclude_daw2_from_ded_calculations,
plan.opt_exclude_daw2_from_oop_calculations AS plan_opt_exclude_daw2_from_oop_calculations,
plan.opt_limit_patient_pay_to_pay AS plan_opt_limit_patient_pay_to_pay,
plan.opt_only_pay_primary_claims AS plan_opt_only_pay_primary_claims,
plan.opt_allow_discontinued_drugs AS plan_opt_allow_discontinued_drugs,
plan.opt_allow_negative_payments_to_pharmacy AS plan_opt_allow_negative_payments_to_pharmacy,
plan.opt_process_y_drugs_as_preferred AS plan_opt_process_y_drugs_as_preferred,
plan.opt_reject_otc AS plan_opt_reject_otc,
plan.opt_reject_repackaged_drugs AS plan_opt_reject_repackaged_drugs,
plan.opt_test_only AS plan_opt_test_only,
plan.opt_cost_effective_pricing AS plan_opt_cost_effective_pricing,
plan.opt_original_mony_for_copay AS plan_opt_original_mony_for_copay,
json_agg(json_build_object(
'plan_daw_differential_daw_code', plan_daw_differential.daw_code,
'plan_daw_differential_claim_type', plan_daw_differential.claim_type,
'plan_daw_differential_updated_at', plan_daw_differential.updated_at,
'plan_daw_differential_updated_by', plan_daw_differential.updated_by
)) as draw_differential_arr
FROM
splan.groups_plan_list gpl
left join splan.plan plan on plan.id = gpl.plan_id
left join splan.carrier carrier_plan on carrier_plan.id = plan.carrier_id
left join splan.plan_daw_differential plan_daw_differential on plan_daw_differential.parent_id = plan.id
left join sdrug.mac mac on mac.id = plan.mac_id
WHERE
gpl.parent_id = 69
GROUP BY
plan.id,
carrier_plan.carrier_code,
plan.plan_name,
mac.mac_name,
plan.termination_date,
plan.mod_start_date,
plan.mod_user,
plan.opt_brandcode_g_on_mn_as_generic_copay,
plan.opt_exclude_daw2_from_ded_calculations,
plan.opt_exclude_daw2_from_oop_calculations,
plan.opt_limit_patient_pay_to_pay,
plan.opt_only_pay_primary_claims,
plan.opt_allow_discontinued_drugs,
plan.opt_allow_negative_payments_to_pharmacy,
plan.opt_process_y_drugs_as_preferred,
plan.opt_reject_otc,
plan.opt_reject_repackaged_drugs,
plan.opt_test_only,
plan.opt_cost_effective_pricing,
plan.opt_original_mony_for_copay
) AS plan_row

How to get multiple sums that are subqueries

Im using Linqpad to test out my EF query and I cant seem to get my end result to include a few extra columns that represent sums of a field based on different conditions
StorePaymentInvoices table contains a FK over to CustomerStatementBatchPayments. So I need to sum the CustomerStatementBatchPayment.net field if there is a corresponding value in StorePaymentInvoices
Getting the sums is turning out to be a real mess. Any suggestions?
Sometimes what is hard to do in one statement, ends up being easier done in multiple steps.
var retval = (
from a in CustomerStatementBatches
join b in CustomerStatementBatchPayments on a.ID equals b.CustomerStatementBatchID into grp1
from c in grp1
where a.CustomerStatementID == StatementId
group c by c.CustomerStatementBatchID into grp2
from e in grp2
select new {
StatementId = e.CustomerStatementBatch.CustomerStatementID,
BatchId = e.CustomerStatementBatchID,
Applied = CustomerStatementBatchPayments.Where(csbp => !StorePaymentInvoices.Select (pi => pi.CustomerStatementBatchPaymentID ).ToList().Contains(e.ID)).Sum (csbp => csbp.Net )
}
).ToList();
retval.Dump();
[ UPDATE 1]
This is what Ive done to get the "conditional" sum values and I seem to be getting the correct numbers. The resulting SQL that it generates is kinda ugly, but executes in < 1 second.
var retval1 = (
from a in CustomerStatementBatches
join b in CustomerStatementBatchPayments on a.ID equals b.CustomerStatementBatchID into grp1
from c in grp1
where a.CustomerStatementID == StatementId
group c by new { a.CustomerStatementID, c.CustomerStatementBatchID} into grp2
from e in grp2.Distinct()
select new {
StatementId = e.CustomerStatementBatch.CustomerStatementID,
BatchId = e.CustomerStatementBatchID
}
).ToList()
.Distinct()
.Select(a => new
{
StatementId = a.StatementId,
BatchId = a.BatchId,
AppliedTotal = (from b in CustomerStatementBatchPayments.Where(r => r.CustomerStatementBatchID == a.BatchId)
join c in StorePaymentInvoices on b.ID equals c.CustomerStatementBatchPaymentID
group b by b.CustomerStatementBatchID into g1
from d in g1
select new{ Total = (decimal?)d.Net}).DefaultIfEmpty().Sum (at => (decimal?)at.Total ) ?? 0.0m,
Unappliedtotal = (from b in CustomerStatementBatchPayments.Where(r => r.CustomerStatementBatchID == a.BatchId)
.Where(s => !StorePaymentInvoices.Any (pi => pi.CustomerStatementBatchPaymentID == s.ID ) )
select new{ Total = (decimal?)b.Net}).DefaultIfEmpty().Sum (at => (decimal?)at.Total ) ?? 0.0m
})
.ToList();
Try this
from a in db.CustomerStatementBatches
join b in db.CustomerStatementBatchPayments
//.Where(i => ...)
.GroupBy(i => i.CustomerStatementBatchesId)
.Select(i => new {
CustomerStatementBatchesId = i.Key,
SumOfPayments = i.Sum(t => t.Net)
}
)
into tmp from b in tmp.DefaultIfEmpty()
on a.CustomerStatementBatchesId equals b.CustomerStatementBatchesId
select new
{
StatementId = a.CustomerStatementId,
BatchId = a.CustomerStatementBatchId,
Applied = ((b == null) ? 0 : b.SumOfPayments)
}

Additional conditions in JOIN

I have tables with articles and users, both have many-to-many mapping to third table - reads.
What I am trying to do here is to get all unread articles for particular user ( user_id not present in table reads ).
My query is getting all articles but those read are marked, which if fine as I can filter them out (user_id field contains id of user in question).
I have an SQL query like this:
SELECT articles.id, reads.user_id
FROM articles
LEFT JOIN
reads
ON articles.id = reads.article_id AND reads.user_id = 9
ORDER BY articles.last_update DESC LIMIT 5;
Which yields following:
articles.id | reads.user_id
-------------------+-----------------
57125839 | 9
57065456 |
56945065 |
56945066 |
56763090 |
(5 rows)
This is fine. This is what I want.
I'd like to get same result in Catalyst using my article model, but I cannot find any option to add conditions to a JOIN clause.
Do you know any way how to add AND X = Y to DBIx JOIN?
I know this can be done with custom resoult source and virtual view, but I have some other queries that could benefit from it and I'd like to avoid creating virtual view for each of them.
Thanks,
Canto
I don't even know what Catalyst is but I can hack the SQL query:
select articles.id, reads.user_id
from
articles
left join
(
select *
from reads
where user_id = 9
) reads on articles.id = reads.article_id
order by articles.last_update desc
limit 5;
I got an solution.
It's not straight forward, but it's better than virtual view.
http://search.cpan.org/dist/DBIx-Class/lib/DBIx/Class/Relationship/Base.pm#condition
Above describes how to use conditions in JOIN clause.
However, my case needs an variable in those conditions, which is not available by default in model.
So getting around a bit of model concept and introducing variable to it, we have the following.
In model file
our $USER_ID;
__PACKAGE__->has_many(
pindols => "My::MyDB::Result::Read",
sub {
my $args = shift;
die "no user_id specified!" unless $USER_ID;
return ({
"$args->{self_alias}.id" => { -ident => "$args->{foreign_alias}.article_id" },
"$args->{foreign_alias}.user_id" => { -ident => $USER_ID },
});
}
);
in controller
$My::MyDB::Result::Article::USER_ID = $c->user->id;
$articles = $channel->search(
{ "pindols.user_id" => undef } ,
{
page => int($page),
rows => 20,
order_by => 'last_update DESC',
prefetch => "pindols"
}
);
Will fetch all unread articles and yield following SQL.
SELECT me.id, me.url, me.title, me.content, me.last_update, me.author, me.thumbnail, pindols.article_id, pindols.user_id FROM (SELECT me.id, me.url, me.title, me.content, me.last_update, me.author, me.thumbnail FROM articles me LEFT JOIN reads pindols ON ( me.id = pindols.article_id AND pindols.user_id = 9 ) WHERE ( pindols.user_id IS NULL ) GROUP BY me.id, me.url, me.title, me.content, me.last_update, me.author, me.thumbnail ORDER BY last_update DESC LIMIT ?) me LEFT JOIN reads pindols ON ( me.id = pindols.article_id AND pindols.user_id = 9 ) WHERE ( pindols.user_id IS NULL ) ORDER BY last_update DESC: '20'
Of course you can skip the paging but I had it in my code so I included it here.
Special thanks goes to deg from #dbix-class on irc.perl.org and https://blog.afoolishmanifesto.com/posts/dbix-class-parameterized-relationships/.
Thanks,
Canto

How can disable quote join Zend db

I've sql query
select * from table1
left join (values (4),(1800),(103500)) AS "filter (id) on table1.id=filter.id
By default Zend_Db_Select table quoted.
For example:
$result = '(values (4),(1800),(103500)) AS filter (id)';
$select->joinInner($result, "table1.id = filter.id", '');
result:
SELECT * FROM "table1"
INNER JOIN "(values (4),(1800),(103500)) filter (id)" ON table1.id=filter.id
Me need
SELECT * FROM "table1"
INNER JOIN (values (4),(1800),(103500)) filter (id) ON table1.id=filter.id
How can disable quote table?
Try adding $result to your $select as a Zend_Db_Expr.
This is a little tricky. Look at the code below.
$dbh = Zend_Db_Table::getDefaultAdapter();
$select = $dbh->select();
$select->from('table1');
$select->joinInner(
array('filter (id)' => new Zend_Db_Expr('(values (4),(1800),(103500))')),
"table1.id = filter.id",
array()
);
echo $select->assemble() . PHP_EOL;
This code by default outputs the following statement which is not what we really want because identifier filter (id) is quoted. Here is the output.
SELECT `table1`.* FROM `table1`
INNER JOIN (values (4),(1800),(103500)) AS `filter (id)` ON table1.id = filter.id
We need to disable autoQuoteIdentifiers in configuration options. For example:
'db' => array(
'adapter' => 'pdo_mysql',
'isDefaultTableAdapter' => true,
'params' => array(
'host' => '<host>',
'username' => '<user>',
'password' => '<pass>',
'dbname' => '<db>',
'options' => array(
'autoQuoteIdentifiers' => false,
),
),
)
We get the following output
SELECT table1.* FROM table1
INNER JOIN (values (4),(1800),(103500)) AS filter (id) ON table1.id = filter.id
Note that in this case developer is responsible for quoting the identifiers when needed.
I think it's impossible to selectively disable quoting for one of the table alias. Well at least I found this impossible when reviewed 1.x Zend Framework code I have here locally ;)

Zend_Db_Select: LEFT JOIN on a subselect

I have a query, that does a LEFT JOIN on a subselect. This query is run in a high load environment and performs within the set requirements. The query (highly simplified) looks like:
SELECT
table_A.pKey
, table_A.uKey
, table_A.aaa
, table_B.bbb
, alias_C.ccc
, alias_C.ddd
FROM table_A
INNER JOIN table_B ON table_A.pKey = table_B.pKey
LEFT JOIN (
SELECT
table_X.pKey
, table_X.ccc
, table_Y.ddd
FROM table_X
INNER JOIN table_Y ON table_X.pKey = table_Y.pKey
) AS alias_C ON table_A.uKey = alias_C.pKey;
(for various reasons, it is not possible to rewrite the subselect as a (direct) LEFT JOIN).
Now, I cannot get the LEFT JOIN on subselect to work with Zend_Db_Select. I've tried everything I could come up with, but it does not work.
So my question is:
Is it not possible to do a query as described above with Zend_Db_Select?
What syntax do I need to get it to work within Zend Framework?
I think that it should work like this:
$subselect = $db->select->from(array('x' => 'table_X'), array('x.pKey', 'x.ccc', 'y.ddd'), 'dbname')
->join(array('Y' => 'table_Y'), 'x.pkey = y.pkey', array(), 'dbname');
$select = $db->select->from(array('a' => 'table_A'), array(/*needed columns*/), 'dbname')
->join(array('b' => 'table_B'), 'a.pkey = b.pkey', array(), 'dbname')
->joinLeft(array('c' => new Zend_Db_Expr('('.$subselect.')'), 'c.pkey = a.ukey', array())
I haven't tried it but I believe it'll work.
...
->joinLeft(array('c' => new Zend_Db_Expr('(' . $subselect->assemble() . ')'), 'c.pkey = a.ukey', array())