Grouping results when nesting queries - tsql

Question

I am trying to get a 6 month trend of entities in my database that match some criteria, however the issue is that I need to nest a few levels deep to determine if an entity qualifies.
The entities are "members" who may have multiple "accounts" and I need to make sure that none of their accounts have certain flags set before I include them.
If I wanted to just get a count as of a specific date (we keep historical data), I would do something like:
SELECT COUNT(sup.SSN)
FROM MemberSuppTable as sup
WHERE (
sup.ProcessDate = #PROCESSDATE
AND sup.MemberSuppID IN (
SELECT summ.MemberSuppID
FROM MemberSummaryTable as summ
WHERE (
summ.ProcessDate = #PROCESSDATE
AND summ.AccountNumber IN (
SELECT acct.AccountNumber
FROM AccountTable as acct
WHERE (
acct.ProcessDate = #PROCESSDATE
--other criteria for account exclusion go here.
)
)
)
)
)
MemberSuppTable has high level info on members:
(ID, FirstAccountOpenDate, status, etc)
MemberSummaryTable ties accounts to members in the MemberSuppTable:
(AccountNumber, MemberSuppID, ...)
Now, I'm trying to get a count for month end process dates, grouped by process date in a single query.
So, where the above query would return
ssn count
----------
1,000,000
I want:
process date | ssn count
------------------------
20160430 | 8,000,000
20160551 | 8,500,000
... | ...
20160331 | 1,000,000
so far I've come up with the following (see below for why it doesn't work):
WITH valid_dates AS (
SELECT D.ProcessDate
FROM arcu.vwARCUProcessDates AS D
WHERE d.FullDate = D.MonthEndDate
AND d.ProcessDate >= #SDATE
)
SELECT sup.ProcessDate, COUNT(DISTINCT sup.SSN)
FROM MemberSuppTable as sup
WHERE (
AND sup.ProcessDate IN (SELECT * FROM valid_dates)
AND sup.MemberSuppID IN (
SELECT summ.MemberSuppID
FROM MemberSummaryTable as summ
WHERE (
summ.ProcessDate IN (SELECT * FROM valid_dates)
AND summ.AccountNumber IN (
SELECT acct.AccountNumber
FROM AccountTable as acct
WHERE (
acct.ProcessDate IN (SELECT * FROM valid_dates)
...
)
)
)
)
)
GROUP BY (sup.ProcessDate)
With the above query though, I believe that a member would be included in ALL groups if they matched the criteria for ANY process date in the valid_dates table.
Can anyone help me out? (I'm new to SQL, so forgive me if I'm missing something simple.)

Answer 1

First i would rewrite your first query using INNER JOIN instead of WHERE .. IN:
SELECT COUNT(DISTINCT sup.SSN)
FROM MemberSuppTable as sup
INNER JOIN MemberSummaryTable AS summ
ON summ.MemberSuppID = sup.MemberSuppID
INNER JOIN AccountTable AS acct
ON acct.AccountNumber = summ.AccountNumber
WHERE sup.ProcessDate = #PROCESSDATE
AND summ.ProcessDate = #PROCESSDATE
AND acct.ProcessDate = #PROCESSDATE
-- other criteria for account exclusion go here.
This looks more compact and is (IMHO) more readable.
Now I would change the query the way, that #PROCESSDATE occures only once
SELECT COUNT(DISTINCT sup.SSN)
FROM MemberSuppTable as sup
INNER JOIN MemberSummaryTable AS summ
ON summ.MemberSuppID = sup.MemberSuppID
INNER JOIN AccountTable AS acct
ON acct.AccountNumber = summ.AccountNumber
WHERE sup.ProcessDate = #PROCESSDATE
AND summ.ProcessDate = sup.ProcessDate
AND acct.ProcessDate = sup.ProcessDate
-- other criteria for account exclusion go here.
You can keep the conditions in the WHERE clause, but i more like them to be in the ON clause
SELECT COUNT(DISTINCT sup.SSN)
FROM MemberSuppTable AS sup
INNER JOIN MemberSummaryTable AS summ
ON summ.MemberSuppID = sup.MemberSuppID
AND summ.ProcessDate = sup.ProcessDate
INNER JOIN AccountTable AS acct
ON acct.AccountNumber = summ.AccountNumber
AND acct.ProcessDate = sup.ProcessDate
WHERE sup.ProcessDate = #PROCESSDATE
-- other criteria for account exclusion go here.
Now it's easy to get the COUNT for each ProcessDate
SELECT sup.ProcessDate, COUNT(DISTINCT sup.SSN)
FROM MemberSuppTable as sup
INNER JOIN MemberSummaryTable AS summ
ON summ.MemberSuppID = sup.MemberSuppID
AND summ.ProcessDate = sup.ProcessDate
INNER JOIN AccountTable AS acct
ON acct.AccountNumber = summ.AccountNumber
AND acct.ProcessDate = sup.ProcessDate
-- WHERE criteria for account exclusion go here.
GROUP BY sup.ProcessDate
To also filter by "valid_dates" it would be just an additional JOIN and some WHERE conditions
SELECT sup.ProcessDate, COUNT(DISTINCT sup.SSN)
FROM MemberSuppTable as sup
INNER JOIN MemberSummaryTable AS summ
ON summ.MemberSuppID = sup.MemberSuppID
AND summ.ProcessDate = sup.ProcessDate
INNER JOIN AccountTable AS acct
ON acct.AccountNumber = summ.AccountNumber
AND acct.ProcessDate = sup.ProcessDate
INNER JOIN arcu.vwARCUProcessDates AS d
ON d.ProcessDate = sup.ProcessDate
WHERE d.FullDate = d.MonthEndDate
AND d.ProcessDate >= #SDATE
-- AND criteria for account exclusion go here.
GROUP BY sup.ProcessDate
For better performance it might be better to GROUP BY d.ProcessDate, but don't forget to also ajust the SELECT part.
Edit:
As noted in the comments, DISTINCT keyword has to be used, if every SSN hast to be counted once. So i edited the solution.
It also has to be noted, that even with DISTINCT the first query is not allways equivalent to the original one. If sup.SSN is not unique, the queries could return different results.

Answer 2

IN clauses are perfectly fine for such a query. More readable than joins, as you show clearly which table you select data from and which tables are only accessed to check record existence. This is well structured and shows you have given the query some thought.
Your query would get even more readable, however, without the unneccessary alias names and parentheses.
Anyway, you want to use the same process date that you find in the subqueries, I guess, so enhance your IN clauses accordingly:
select processdate, count(distinct ssn)
from membersupptable
where (processdate, membersuppid) in
(
select processdate, membersuppid
from membersummarytable
where (processdate, accountnumber) in
(
select processdate, accountnumber
from accounttable
where processdate in
(
select processdate
from vwarcuprocessdates
where fulldate = monthenddate
and processdate >= #sdate
)
)
)
group by processdate;