The precedence of polling of the consumer in batch mode - apache-kafka

Question

spring.kafka.consumer.max-poll-records = 2000 //each record of size 5kb takes 100 ms so to process entire batch takes 500 sec i.e 8 min 20 sec
spring.kafka.consumer.properties.max.poll.interval.ms = 900000 //15 min
spring.kafka.consumer.properties.fetch.max.wait.ms = 600000 //10 min
spring.kafka.consumer.properties.max.partition.fetch.bytes = 10485760 //10MB
spring.kafka.consumer.fetch-min-size = 5242880 // fetch.min.bytes - 5mb
spring.kafka.listener.concurrency = 1
With the above configuration the consumer is continously polling the records its respective intervals like sometimes 2 mins, 3 mins eventhough fetch.max.wait.ms has been set it as as 10 min.
How this is happening ? May i know the precedence of polling ? is it based on the max-poll-records, fetch.max.wait.ms ,max.partition.fetch.bytes or fetch-min-size ?
EDIT:
I tried to maximize the below attribute values to fetch maximum records, but still seeing only 200 to 300 records are getting processed.
spring.kafka.consumer.properties.max.poll.interval.ms = 900000 //15 min
spring.kafka.consumer.max-poll-records = 2000 //each record of size 5kb takes 100 ms so to process entire batch, takes 2000*100 ms =200sec i.e 3 min 20 sec which is way less than the max poll interval (10min)
spring.kafka.consumer.properties.fetch.max.wait.ms = 600000 //10 min
spring.kafka.consumer.properties.max.partition.fetch.bytes = 20971520
//20MB
spring.kafka.consumer.fetch-min-size = 104857600 // fetch.min.bytes - 20mb
Am i doing something wrong?

Answer 1

Assuming the records are exactly 5kb, the poll will return when 1k records are received or 10 minutes elapse, whichever happens first.
You will only ever get max.poll.records if they are immediately available.
EDIT
It looks to me like these "smaller" batches are the remnants of the previous fetch; with this code:
#SpringBootApplication
public class So68201599Application {
private static final Logger log = LoggerFactory.getLogger(So68201599Application.class);
public static void main(String[] args) {
SpringApplication.run(So68201599Application.class, args);
}
#KafkaListener(id = "so68201599", topics = "so68201599", autoStartup = "false")
public void listen(ConsumerRecords<?, ?> in) {
log.info("" + in.count() + "\n"
+ in.partitions().stream()
.map(part -> "" + part.partition() + "(" + in.records(part).size() + ")")
.collect(Collectors.toList()));
}
#Bean
public NewTopic topic() {
return TopicBuilder.name("so68201599").partitions(10).replicas(1).build();
}
#Bean
public ApplicationRunner runner(KafkaTemplate<String, String> template, KafkaListenerEndpointRegistry registry) {
String msg = new String(new byte[1024*5]);
return args -> {
List<Future<?>> futures = new ArrayList<>();
IntStream.range(0, 9000).forEach(i -> futures.add(template.send("so68201599", msg)));
futures.forEach(fut -> {
try {
fut.get(10, TimeUnit.SECONDS);
}
catch (InterruptedException e) {
Thread.currentThread().interrupt();
e.printStackTrace();
}
catch (ExecutionException e) {
e.printStackTrace();
}
catch (TimeoutException e) {
e.printStackTrace();
}
});
log.info("Sent");
registry.getListenerContainer("so68201599").start();
};
}
}
and
spring.kafka.consumer.max-poll-records = 2000
spring.kafka.consumer.properties.fetch.max.wait.ms = 10000
spring.kafka.consumer.fetch-min-size = 10240000
spring.kafka.consumer.auto-offset-reset=earliest
spring.kafka.listener.type=batch
I get
2021-07-08 15:04:11.131 INFO 45792 --- [o68201599-0-C-1] com.example.demo.So68201599Application : 2000
[1(201), 0(201), 5(201), 4(201), 3(201), 2(201), 9(201), 8(201), 7(201), 6(191)]
2021-07-08 15:04:11.137 INFO 45792 --- [o68201599-0-C-1] com.example.demo.So68201599Application : 10
[6(10)]
2021-07-08 15:04:21.170 INFO 45792 --- [o68201599-0-C-1] com.example.demo.So68201599Application : 1809
[1(201), 0(201), 5(201), 4(201), 3(201), 2(201), 9(201), 8(201), 7(201)]
2021-07-08 15:04:21.214 INFO 45792 --- [o68201599-0-C-1] com.example.demo.So68201599Application : 2000
[1(201), 0(201), 5(201), 4(201), 3(201), 2(201), 9(201), 8(201), 7(201), 6(191)]
2021-07-08 15:04:21.215 INFO 45792 --- [o68201599-0-C-1] com.example.demo.So68201599Application : 10
[6(10)]
2021-07-08 15:04:31.248 INFO 45792 --- [o68201599-0-C-1] com.example.demo.So68201599Application : 1809
[1(201), 0(201), 5(201), 4(201), 3(201), 2(201), 9(201), 8(201), 7(201)]
2021-07-08 15:04:41.267 INFO 45792 --- [o68201599-0-C-1] com.example.demo.So68201599Application : 1083
[1(27), 0(87), 5(189), 4(93), 3(114), 2(129), 9(108), 8(93), 7(42), 6(201)]
2021-07-08 15:04:51.276 INFO 45792 --- [o68201599-0-C-1] com.example.demo.So68201599Application : 201
[6(201)]
2021-07-08 15:05:01.279 INFO 45792 --- [o68201599-0-C-1] com.example.demo.So68201599Application : 78
[6(78)]
I don't know why the second and the three penultimate fetches timed out, though.