Clickhouse materialized view aggregate ghost rows - apache-zookeeper

Question

So I'm using clickhouse and here is my current tables architecture.
I have a main table containing my data:
CREATE TABLE default.Liquidity
(
`Date` Date,
`LiquidityId` UInt64,
`TreeId_LQ` UInt64,
`AggregateId` UInt64,
`ClientId` UInt64,
`InstrumentId` UInt64,
`IsIn` String,
`Currency` String,
`Scenario` String,
`Price` String,
`Leg` Int8,
`commit` Int64,
`factor` Int8,
`nb_aggregated` UInt64,
`stream_id` Int64
)
ENGINE = Distributed('{cluster}', '', 'shard_Liquidity', TreeId_LQ)
And I also have a materialized view that aggregate data store it in other table
CREATE MATERIALIZED VIEW default.mv_Liquidity_facet TO default.shard_state_Liquidity_facet
(
`Date` Date,
`TreeId_LQ` UInt64,
`AggregateId` UInt64,
`ClientId` UInt64,
`InstrumentId` UInt64,
`Currency` String,
`Scenario` String,
`commit` Int64,
`factor` Int8,
`nb_aggregated` AggregateFunction(sum, UInt64)
) AS
SELECT
Date,
TreeId_LQ,
AggregateId,
ClientId,
InstrumentId,
Currency,
Scenario,
commit,
factor,
sumState(nb_aggregated) AS nb_aggregated
FROM default.shard_Liquidity
GROUP BY
Date,
TreeId_LQ,
AggregateId,
ClientId,
InstrumentId,
Currency,
Scenario,
commit,
factor
----------------
CREATE TABLE default.shard_state_Liquidity_facet
(
`Date` Date,
`TreeId_LQ` UInt64,
`AggregateId` UInt64,
`ClientId` UInt64,
`InstrumentId` UInt64,
`Currency` String,
`Scenario` String,
`commit` Int64,
`factor` Int8,
`nb_aggregated` AggregateFunction(sum, UInt64)
)
ENGINE = ReplicatedAggregatingMergeTree('{zoo_prefix}/tables/{shard}/shard_state_Liquidity_facet', '{host}')
PARTITION BY Date
ORDER BY (commit, TreeId_LQ, ClientId, AggregateId, InstrumentId, Scenario)
SETTINGS index_granularity = 8192
As you might have guessed, the nb_aggregated column represents the number of rows that were aggregated to achieve this result.
If I make that query on my Distributed query with a lot of filter in order to find one row
select
sum(nb_aggregated) AS nb_aggregated
from Liquidity
where Date = '2022-10-17'
and TreeId_LQ = 1129
and AggregateId = 999999999999
and ClientId = 1
and InstrumentId = 593
and Currency = 'AUD'
and Scenario = 'BAU'
and commit = -2695401333399944382
and factor = 1;
--- Result
1
I end up with only one row, therefore if I make the same query with the same filter but one the aggregated version of my table that have been created with the materialize view I should also end up with only one line and with the nb_aggregated = 1 however I end up with nb_aggregated = 2 as if he had aggregated my row with another and most of the other value are wrong too.
I understand that my exemple is hard to understand but if you have any lead it will be nice.

Answer 1

Well, I have ask the same question one the clickhouse repository on github and Denny Crane give me this answer that is working for me here:
https://github.com/ClickHouse/ClickHouse/issues/43988#issuecomment-1339731917
In the most of cases MatView group by should match to a storage table ORDER BY
CREATE MATERIALIZED VIEW default.mv_Liquidity_facet:
GROUP BY Date, TreeId_LQ, AggregateId, ClientId, InstrumentId, Currency, Scenario, commit, factor
CREATE TABLE default.shard_state_Liquidity_facet
PARTITION BY Date
ORDER BY (commit, TreeId_LQ, ClientId, AggregateId, InstrumentId, Scenario)
Your ReplicatedAggregatingMergeTree "CORRUPTS" Currency / factor columns using ANY function
the solution is
ORDER BY (commit, TreeId_LQ, ClientId, AggregateId, InstrumentId, Scenario, Currency , factor)
https://den-crane.github.io/Everything_you_should_know_about_materialized_views_commented.pdf