Aggregate to compacted topic with unlimited retention - apache-kafka

Question

I'm setting up a Kafka Streams application that consumes from a topic (retention: 14 days, cleanup.policy: delete, partitions: 1).
I wish to consume the messages and output it into another topic (retention: -1, cleanup.policy: compact, partitions: 3).
The grouping is by key on the input topic.
So:
Input-topic:
Key: A Value: { SomeJson }
Key: A Value: { Other Json}
Key: B Value: { TestJson }
Output:
Key: A Value: {[ { SomeJson }, { Other Json } ]}
Key: B Value: {[ { TestJson } ]}
It's important that the content on the output topic is never lost, so it's ack: all and 3x replicas.
Each key in the compacted topic will have around 100 json records. Estimated to less than 20kb for each key.
I was also hoping that the output topic worked as a state-topic, so that it wouldn't have to create another topic to contain the same information.
Anyone know how to do this? Most of the examples I find relate to windowing: https://github.com/confluentinc/kafka-streams-examples/tree/5.3.1-post/src/main/java/io/confluent/examples/streams
Current code:
val mapper = new ObjectMapper();
builder.stream(properties.getInputTopic(), Consumed.with(Serdes.String(), Serdes.String()))
.groupByKey()
.aggregate(
() -> new GroupedIdenthendelser(Collections.emptyList()),
(key, value, currentAggregate) -> {
val items = new ArrayList<>(currentAggregate.getIdenthendelser());
items.add(value);
return new GroupedIdenthendelser(items);
},
Materialized.with(Serdes.String(), new JsonSerde<>(GroupedIdenthendelser.class, mapper)))
.toStream()
.to(properties.getOutputTopic(), Produced.with(Serdes.String(), new JsonSerde<>(mapper)));
If someone has some other tips to give, please do tell as this data is customer information so if there is some config's I should tweak do tell. Or if you know some blog posts/examples out there it's appreciated.
Edit: The code example above seems to work but it creates its own state topic which is something that is not needed as the output topic will always contain the same state. There will also only every be 1 application running this since input topic has 1 partition and as it's related to people in a rather fixed size (10 000 000 people give or take), the size of the data won't increase above 20kb per person either. Event's per second is estimated to be around 1/s, so the load it not much either.
The Topology:
Sub-topology: 0
Source: KSTREAM-SOURCE-0000000000 (topics: [input-topic])
--> KSTREAM-AGGREGATE-0000000002
Processor: KSTREAM-AGGREGATE-0000000002 (stores: [KSTREAM-AGGREGATE-STATE-STORE-0000000001])
--> KTABLE-TOSTREAM-0000000003
<-- KSTREAM-SOURCE-0000000000
Processor: KTABLE-TOSTREAM-0000000003 (stores: [])
--> KSTREAM-SINK-0000000004
<-- KSTREAM-AGGREGATE-0000000002
Sink: KSTREAM-SINK-0000000004 (topic: output-topic)
<-- KTABLE-TOSTREAM-0000000003

Answer 1

Looking at your example dataset, I guess what you might need is the realtime aggregation. Please take a look at this blog post of Confluent as a starting point.