Kafkaconsumer.poll consume just from one partition - apache-kafka

Question

I created a class called ConsumerConfig and Service class that contain function that gets records from a topic.
consumer.poll consume just from one partition , i think i need to add a loop but while(true) in this case its not the best choice
when I debug I evaluate this function "records = consumer.poll(Duration.ofMillis(4000))" and each time I evaluate it it reads from the next partition
here is the code of the ConsumerConfig class and the functions that gets records from specific topic :
#Component
public class ConsumerConfig {
#Autowired
KafkaProperties kafkaProperties;
private Map<String, Object> buildDefaultConfig() {
final Map<String, Object> defaultClientConfig = new HashMap<>();
return defaultClientConfig;
}
#Bean
#RequestScope
public <K, V> KafkaConsumer<K, V> getKafkaConsumer() throws JsonProcessingException {
// Build config
final Map<String, Object> kafkaConsumerConfig = buildDefaultConfig();
String ss = new ObjectMapper().writeValueAsString(kafkaProperties.buildConsumerProperties());
log.info("config {}",ss);
kafkaConsumerConfig.putAll(kafkaProperties.buildAdminProperties());
kafkaConsumerConfig.put("key.deserializer", StringDeserializer.class);
kafkaConsumerConfig.put("value.deserializer", StringDeserializer.class);
kafkaConsumerConfig.put("max.poll.records", 500000);
kafkaConsumerConfig.put("max.poll.interval.ms",600000);
//kafkaConsumerConfig.put("fetch.max.wait.ms",2000);
//kafkaConsumerConfig.put("fetch.min.bytes",50000);
//kafkaConsumerConfig.put("groupId","mygr");
return new KafkaConsumer<K, V>(kafkaConsumerConfig);
}
}
function that get records :
public List<Record> recordsFromTopic(final String topic) {
// Find all partitions on topic.
final TopicDescription topicDescription = (TopicDescription) adminService.
topicDescription(topic);
final Collection<Integer> partitions = topicDescription
.partitions()
.stream()
.map(TopicPartitionInfo::partition)
.collect(Collectors.toList());
var list = consumeAllRecordsFromTopic(topic, partitions);
var element = list.stream().filter(Objects::nonNull).map(x -> Record
.builder()
.values(x.value())
.offset(x.offset())
.partition(x.partition()).build())
.collect(Collectors.toList());
return element;
}
public <K, V> List<ConsumerRecord<K, V>> consumeAllRecordsFromTopic(final String topic,
final Collection<Integer> partitionIds) {
// Create topic Partitions
final List<TopicPartition> topicPartitions = partitionIds
.stream()
.map((partitionId) -> new TopicPartition(topic, partitionId))
.collect(Collectors.toList());
final List<ConsumerRecord<K, V>> allRecords = new ArrayList<>();
ConsumerRecords<K, V> records;
// Assign topic partitions
consumer.assign(topicPartitions);
consumer.seekToBeginning(topicPartitions);
// Pull records from kafka
records = consumer.poll(Duration.ofMillis(4000));
records.forEach(allRecords::add);
return allRecords;
}

Answer 1

How many partitions do you have in the source topic and how many consumers are you running. Also while(true) is must required one for the continues polling of the records through consumer.poll() and iterate it through individual consumer records. Unless you have more than one consumer instance under a single consumer group that listening to a topic, consumer instance is going to read from more than one topic partition (if topic partition > 1). Provided, you also don't have some partitioning strategies implemented.