Sending events with Reactor Kafka seems very slow - apache-kafka

Question

My project uses Reactor Kafka 1.2.2.RELEASE to send events serialized with Avro to my Kafka broker. This works well, however sending events seems to be quite slow.
We plugged a custom metric to the lifecycle of the Mono sending the event through KafkaSender.send(), and noticed it took around 100ms to deliver a message.
We did it this way:
send(event, code, getKafkaHeaders()).transform(withMetric("eventName"));
The send method just builds the record and sends it:
private Mono<Void> send(SpecificRecord value, String code, final List<Header> headers) {
final var producerRecord = new ProducerRecord<>("myTopic", null, code, value, headers);
final var record = Mono.just(SenderRecord.create(producerRecord, code));
return Optional.ofNullable(kafkaSender.send(record)).orElseGet(Flux::empty)
.switchMap(this::errorIfAnyException)
.doOnNext(res -> log.info("Successfully sent event on topic {}", res.recordMetadata().topic()))
.then();
}
And the withMetric transformer links a metric to the send mono lifecycle:
private Function<Mono<Void>, Mono<Void>> withMetric(final String methodName) {
return mono -> Mono.justOrEmpty(this.metricProvider)
.map(provider -> provider.buildMethodExecutionTimeMetric(methodName, "kafka"))
.flatMap(metric -> mono.doOnSubscribe(subscription -> metric.start())
.doOnTerminate(metric::end));
}
That is this custom metric that returns an average of 100ms.
We compared it to our Kafka producer metrics, and noticed that those ones returned and average of 40ms to deliver a message (0ms of queueing, and 40ms of request latency).
We have difficulties to understand the delta, an wonder if it could come from the Reactor Kafka method to send events.
Can anybody help please?
UPDATE
Here's a sample of my producer config:
acks = all
batch.size = 16384
buffer.memory = 33554432
client.dns.lookup = default
compression.type = none
connections.max.idle.ms = 540000
delivery.timeout.ms = 120000
enable.idempotence = true
key.serializer = class org.apache.kafka.common.serialization.StringSerializer
linger.ms = 0
max.block.ms = 60000
max.in.flight.requests.per.connection = 5
max.request.size = 1048576
metadata.max.age.ms = 300000
metrics.num.samples = 2
metrics.recording.level = INFO
metrics.sample.window.ms = 30000
partitioner.class = class org.apache.kafka.clients.producer.internals.DefaultPartitioner
receive.buffer.bytes = 32768
reconnect.backoff.max.ms = 1000
reconnect.backoff.ms = 50
request.timeout.ms = 30000
retries = 2147483647
retry.backoff.ms = 100
sasl.login.refresh.buffer.seconds = 300
sasl.login.refresh.min.period.seconds = 60
sasl.login.refresh.window.factor = 0.8
sasl.login.refresh.window.jitter = 0.05
sasl.mechanism = GSSAPI
security.protocol = PLAINTEXT
send.buffer.bytes = 131072
ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1]
ssl.endpoint.identification.algorithm = https
ssl.keystore.type = JKS
ssl.protocol = TLS
ssl.trustmanager.algorithm = PKIX
ssl.truststore.type = JKS
transaction.timeout.ms = 60000
value.serializer = class io.confluent.kafka.serializers.KafkaAvroSerializer
Also, the maxInFlight is 256 and the scheduler is single, I didn't configure anything special here.