results of except query change after repeated consecutive runs in redshift - postgresql

Question

I'm running the postgresql query below in aws redshift. Each time I run this query I'm getting a different result for the number of records that are different on daily_table.product_repeat_sub_query side, using the except operator. Neither the daily_table.product_repeat_sub_query table or the daily_table.daily_sku_t are being updated during this time. the daily_table.product_repeat_sub_query table and the product_repeat_sub_query query both have the same record count. the schema for the daily_table.daily_sku_t is below, the matching fields in the daily_table.product_repeat_sub_query have the same data types. I've also included some sample records from the tables below. does anyone have an idea how the results of the except query can come out differently each time this query is run, when the underlying tables aren't changing?
daily_table.daily_sku_t schema:
customer_uuid string
boardname_12 string
producttype string
productsubtype string
storeid int
product_id string
dateclosed date
Size string
query:
with product_repeat_sub_query as
(
select
dateclosed, t.product_id, t.storeid, t.producttype, t.productsubtype, t.size, t.boardname_12,
case
when ticketid = first_value(ticketid) over (partition by t.product_id, customer_uuid
ORDER BY
dateclosed ASC rows between unbounded preceding and unbounded following) then 0
else grossreceipts
end as product_repeat_gross, datediff(day,
lag(dateclosed, 1) over (partition by t.boardname_12, customer_uuid, t.product_id
ORDER BY
dateclosed ASC ),
dateclosed) as product_cycle_days
from
daily_table.daily_sku_t t )
select count(*) from
(
select dateclosed, storeid, boardname_12, producttype, productsubtype, size, product_id, product_cycle_days from daily_table.product_repeat_sub_query
except
select dateclosed, storeid, boardname_12, producttype, productsubtype, size, product_id, product_cycle_days from product_repeat_sub_query
);
-- 36843
-- 36887
-- 36188
data:
daily_table.product_repeat_sub_query
dateclosed storeid boardname_12 producttype productsubtype size product_id product_cycle_days
2021-04-23 427 22RED DRUMER 1T 000011aa-4f03-4f0b-a621-xxxxxxxxxxxx 2
2021-04-24 427 22RED DRUMER 1T 000011aa-4f03-4f0b-a621-xxxxxxxxxxxx 6
2021-04-26 427 22RED DRUMER 1T 000011aa-4f03-4f0b-a621-xxxxxxxxxxxx 8
2021-04-26 427 22RED DRUMER 1T 000011aa-4f03-4f0b-a621-xxxxxxxxxxxx 3
2021-05-01 427 22RED DRUMER 1T 000011aa-4f03-4f0b-a621-xxxxxxxxxxxx 13
2020-06-18 61 FLAV RX WINGER BEVERAGE 100MT 0000265d-6b81-4d79-90cf-xxxxxxxxxxxx 5
2020-06-29
product_repeat_subquery
dateclosed storeid boardname_12 producttype productsubtype size product_id product_cycle_days
2021-04-23 427 22RED DRUMER 1T 000011aa-4f03-4f0b-a621-xxxxxxxxxxxx 2
2021-04-24 427 22RED DRUMER 1T 000011aa-4f03-4f0b-a621-xxxxxxxxxxxx 6
2021-04-26 427 22RED DRUMER 1T 000011aa-4f03-4f0b-a621-xxxxxxxxxxxx 8
2021-04-26 427 22RED DRUMER 1T 000011aa-4f03-4f0b-a621-xxxxxxxxxxxx 3
2021-05-01 427 22RED DRUMER 1T 000011aa-4f03-4f0b-a621-xxxxxxxxxxxx 13
2020-06-18 61 FLAV RX WINGER BEVERAGE 100MT 0000265d-6b81-4d79-90cf-xxxxxxxxxxxx 5
2020-06-29
update:
with product_repeat_sub_query as
(
select customer_uuid,
dateclosed, t.product_id, t.storeid, t.producttype, t.productsubtype, t.size, t.boardname_12,
case
when ticketid = first_value(ticketid) over (partition by t.product_id, customer_uuid
ORDER BY
dateclosed ASC rows between unbounded preceding and unbounded following) then 0
else grossreceipts
end as product_repeat_gross, datediff(day,
lag(dateclosed, 1) over (partition by t.boardname_12, customer_uuid, t.product_id
ORDER BY
dateclosed ASC,t.boardname_12, customer_uuid, t.product_id ),
dateclosed) as product_cycle_days
from
daily_table.daily_sku_t t
where (t.customer_uuid is not null)
and (trim(t.customer_uuid) != '')
and (t.product_id is not null)
and (trim(t.product_id) != '')
)
select count(*) from
(
select customer_uuid, dateclosed, storeid, boardname_12, producttype, productsubtype, size, product_id, product_cycle_days from daily_table.product_repeat_sub_query
except
select customer_uuid, dateclosed, storeid, boardname_12, producttype, productsubtype, size, product_id, product_cycle_days from product_repeat_sub_query
);
even after adding all the fields from the partition to the order by and filtering our nulls or blanks in the id fields, I'm still getting a different count each time.

Answer 1

Your window functions don't have fully qualified order by clauses. You have repeated "dateclosed" values within partitions. This means that Redshift can have different row orders for the lag and first-value functions. I expect that these "random" ordering differences are causing your changing results.