Kafka Streams and Spring Cloud Stream - Processor Efficiency - apache-kafka

Question

I would like to confirm my understanding of the efficiency of having multiple processors reading from one Kafka Stream source. I believe the following in Example 1 is the most efficient if I want 2 different processes performed depending on Predicate logic. The Predicate looks at the content of the Value (the Notification object here). If you have a breakpoint in each of the following processors in Example 1, it shows each Function is called for each incoming Notification. Whereas in Example 2, you only call the process2 Function if the predicate logic is met.
Example 1
#Bean
public Function<KStream<String, Notification>,KStream<String, Notification>> process1() {
return input -> input
.branch(PREDICATE_FOR_OUT_0, PREDICATE_FOR_OUT_1);
}
#Bean
public Function<KStream<String, Notification>,KStream<String, EnrichedNotification>> process2() {
return input -> input
.filter(PREDICATE_FOR_OUT_2);
.map((key, value) ->.........; //different additional processing to map to EnrichedNotification type
}
There is no need for the following and attempt to route the output of one processor into another? (Not sure that it is even possible)
Example 2 (conceptual)
I am probably thinking this way because I am coming from using pure Kafka. Here process1 has a 3 way branch. Two of the branches go to their respective stream and then topic, but the third requires further processing before it can be routed to a topic.
#Bean
public Function<KStream<String, Notification>,KStream<String, Notification>[]> process1() {
return input -> input
.branch(PREDICATE_FOR_OUT_0, PREDICATE_FOR_OUT_1, PREDICATE_FOR_OUT_2);
}
Could we potentially route the branch for PREDICATE_FOR_OUT_2 into process2. This would mean process2 would only be called if PREDICATE_FOR_OUT_2 was met
#Bean
public Function<KStream<String, Notification>,KStream<String, EnrichedNotification>> process2() {
return input -> input
.map((key, value) ->.........; //different additional processing to map to EnrichedNotification type
}
My thinking is example 2 is redundant (and not actually possible anyway) due to the abstraction and functionality that Kafka Streams gives

Answer 1

I think both cases of your examples can get the job done, but there are some differences. In the first example, you have two functions, both receiving data from the same Kafka topic and the second function performs some additional logic before getting routed to the output topic. In the second example, you again have two functions. In the first function, you have 3 branches, each of them sending data to a Kafka topic (I assume they are 3 different topics). Then in the second function, you receive data from the 3rd output topic from the first function. After performing the logic in that second function of example 2, you send it to the final destination for this branch. You are introducing an extra topic for this second example. I think your first example is more readable and clean.