How process() method in kafka-stream-processor is called automatically? - apache-kafka

Question

I am learning kafka streams and have written a simple app, snippet below:
MainApp:
Topology topology = new Topology();
topology.addSource("SOURCE", "source-topic");
topology.addProcessor("Processor1", () -> new Processor1(), "SOURCE");
topology.addProcessor("Processor2", () -> new Processor2(), "Processor1");
topology.addProcessor("Processor3", () -> new Processor3(), "Processor2");
topology.addSink("SINK", "sink-topic", "Processor3");
KafkaStreams streams = new KafkaStreams(topology, config);
streams.start();
Snippet of Individual stream proccesor:
public class Processor1 implements Processor<String, String> {
// Rest of code
#Override
public void process(String key, String value) {
System.out.println("Inside Processor1#process() method");
context.forward(key, value);
}
I understood that we need to create Topology and then to initiate it, we invoke streams.start();
I am not able to understand how process() method is being invoked automatically and who calls it?

Answer 1

Processor process() method invoked by ProcessorContextImpl class automatically on each incoming message for specific topology node.
For your built topology, when a message arrived at the incoming topic, SOURCE node consumes it and forwards (propagates) message to child node by internally calling forward method (you could debug/take a look at code from class ProcessorContextImpl). In your case, SOURCE node forwards key and value to child node Processor1. After that, process() method from class Processor1 triggered. When code reaches context.forward(), message forwards to the next child node, Processor2. After that message propagates to Processor3 and SINK nodes in a similar manner, and finally, message produced to outbound topic. Such pipeline for specific message executes on a single thread (and if you have a default value for config num.stream.threads = 1, all messages will be processed on a single thread per app instance).