Issue while sending the Flux of data to the reactive kafka - apache-kafka

Question

I am using reactor library to fetch the large stream of data from the network and sending it to the kafka broker using the reactive kafka approach.
Below is the Kafka Producer I am using
public class LogProducer {
private final KafkaSender<String, String> sender;
public LogProducer(String bootstrapServers) {
Map<String, Object> props = new HashMap<>();
props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, bootstrapServers);
props.put(ProducerConfig.CLIENT_ID_CONFIG, "log-producer");
props.put(ProducerConfig.ACKS_CONFIG, "all");
props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
SenderOptions<String, String> senderOptions = SenderOptions.create(props);
sender = KafkaSender.create(senderOptions);
}
public void sendMessages(String topic, Flux<Logs.Data> records) {
AtomicInteger sentCount = new AtomicInteger(0);
AtomicInteger fCount = new AtomicInteger(0);
records.doOnNext(r -> fCount.incrementAndGet()).subscribe();
System.out.println("Total Records: " + fCount);
sender.send(records.doOnNext(r -> sentCount.incrementAndGet())
.map(record -> {
LogRecord lrec = record.getRecords().get(0);
String id = lrec.getId();
return SenderRecord.create(new ProducerRecord<>(topic, id,
lrec.toString()), id);
})).then()
.doOnError(e -> {
log.error("[FAIL]: Send to the topic: '{}' failed. "
+ e, topic);
})
.doOnSuccess(s -> {
log.info("[SUCCESS]: {} records sent to the topic: '{}'", sentCount, topic);
})
.subscribe();
}
}
The total number of records in the Flux (fCount) and the records sent to the Kafka topic (sentCount) does not match, it does not give any error and completes successfully.
For example: In one of the case total number of records in the Flux is 2758, while the count sent to the kafka is 256. Is there any kafka configuration, which needs to be modified, or Am I missing anything?
===========================================================
Updated based on the comments
sender.send(records
.map(record -> {
LogRecord lrec = record.getRecords().get(0);
String id = lrec.getId();
sleep(5); // sleep for 5 ns
return SenderRecord.create(new ProducerRecord<>(topic, id,
lrec.toString()), id);
})).then()
.doOnError(e -> {
log.error("[FAIL]: Send to the topic: '{}' failed. "
+ e, topic);
})
.doOnSuccess(s -> {
log.info("[SUCCESS]: {} records sent to the topic: '{}'", sentCount, topic);
})
.subscribe();
sleep(10); // sleep for 10 ns
The above code worked fine in one system but failed to send all messages in other.