select thresholds dynamically from sql - postgresql

Question

I have a range of data on search queries across diffrent merchants.
I have a python script that 1st creates the head, torso & tail query sets from the main table in qsql, based on count(query) instances as 1000, 100 etc.
Since the number of merchants I my script runs of could have/not have queries that meet that threshold, the script does not log the "head.csv" "torso.csv" .. tail.csv always being produced.
How can I break the queries into head, torso & tail groups by respecting the logic above.
I also tried ntile to break the groups by percentiles(33, 33, 33), but that skews both the head & torso, if a merchant has a very long tail.
Current :
# head
select trim(query) as query, count(*)
from my_merchant_table
-- other conditions & date range
GROUP BY trim(query)
having count(*) >=1000
#torso
select trim(query) as query, count(*)
from my_merchant_table
-- other conditions & date range
GROUP BY trim(query)
having count(*) <1000 and count(*) >=100
#tail
select trim(query) as query, count(*)
from my_merchant_table
-- other conditions & date range
GROUP BY trim(query)
having count(*) <100
# using ntile - but note that I have percentiles of "3" , 33.#% each, which introduces the skew
select trim(query), count(*) as query_count,
ntile(3) over(order by query_count desc) AS group_ntile
from my_merchant_table
group by trim(query)
order by query_count desc limit 100;
Ideally the solution can build on top of this -:
select trim(query), count(*) as query_count,
ntile(100) over(order by query_count desc) AS group_ntile
from my_merchant_table
-- other conditions & date range
group by trim(query)
order by query_count desc
This gives,
btrim query_count group_ntile
q0 1277 1
q1 495 1
q2 357 1
q3 246 1
# so on till group_ntile =100 , while the query_count reduces.
Question :
What is the best way for the logic, to make the overall logic merchant agnostic/no hard-coding the configs ?
Note : I am fetching the data in Redshift, the solution should be compatible to postgres 8.0 & redshift in particular.

Answer 1

I imagine that you from some programming language invokes its queries to process information. My recommendation in this regard is get all the records and apply a filter over they. Consider that if you queries the database where there are several operations over the data this would result that the response time of the application is affected.

Answer 2

Assuming that the main challenge is to create the 'tiles' from a list of values, here is some sample code. It takes the 13 provinces of Canada and breaks it into a requested number of groups. It uses the province names, but numbers would work just as well.
SELECT * FROM Provinces ORDER BY province; -- To see what we are working with
+---------------------------+
| province |
+---------------------------+
| Alberta |
| British Columbia |
| Manitoba |
| New Brunswick |
| Newfoundland and Labrador |
| Northwest Territories |
| Nova Scotia |
| Nunavut |
| Ontario |
| Prince Edward Island |
| Quebec |
| Saskatchewan |
| Yukon |
+---------------------------+
13 rows in set (0.00 sec)
Now for the code:
SELECT #n := COUNT(*), -- Find total count (13)
#j := 0.5, -- 'trust me'
#tiles := 3 -- The number of groupings
FROM Provinces;
SELECT group_start
FROM (
SELECT
IF((#j * #tiles) % #n < #tiles, province, NULL) AS group_start,
#j := #j + 1
FROM Provinces
ORDER BY province
) x
WHERE group_start IS NOT NULL;
+---------------------------+
| group_start |
+---------------------------+
| Alberta |
| Newfoundland and Labrador |
| Prince Edward Island |
+---------------------------+
3 rows in set (0.00 sec)
With #tiles set to 4:
+---------------+
| group_start |
+---------------+
| Alberta |
| New Brunswick |
| Nova Scotia |
| Quebec |
+---------------+
4 rows in set (0.00 sec)
It is reasonably efficient: 1 pass to count the number of rows, 1 pass to do the computation, 1 pass to filter out the non-break values.