Consuming from a Kafka topic that requires authentication using reactor kafka - apache-kafka

Question

I have a micro-service that consumes from a Kafka topic that requires authentication. Below is the code I wrote for that. I am fetching the username and password from environment variables that I am sure is working as expected.
val receiverOptions = ReceiverOptions.create<ByteBuffer, ByteBuffer(defaultKafkaBrokerConfig.getAsProperties())
val kafkaJaasConfig = String.format(
"org.apache.kafka.common.security.scram.ScramLoginModule required username='%s' password='%s';",
kafkaUsername,
kafkaPassword
)
val schedulerKafkaConsumer = Schedulers.newSingle("consumer")
val options = receiverOptions
.subscription(topicConfig.topics)
.pollTimeout(Duration.ofMillis(topicConfig.pollWaitTimeoutMs.toLong()))
.consumerProperty(SaslConfigs.SASL_MECHANISM, "SCRAM-SHA-512")
.consumerProperty(SaslConfigs.SASL_JAAS_CONFIG, kafkaJaasConfig)
.consumerProperty(CommonClientConfigs.SECURITY_PROTOCOL_CONFIG, "SASL_PLAINTEXT")
return KafkaReceiver.create(options)
.receive()
.subscribeOn(schedulerKafkaConsumer)
.map { record: ReceiverRecord<ByteBuffer, ByteBuffer> -> handleConsumerRecord(record) }
.onErrorContinue { throwable: Throwable?, _: Any? ->
log.error(
"Error consuming and deserializing messages",
throwable
)
}
The code works fine when I run it locally. However, on GCP development environment, i get the following error:
Bootstrap broker <<some_ip>>:9093 (id: -2 rack: null) disconnected
org.apache.kafka.clients.NetworkClient : [Consumer clientId=consumer-service_gcp-edge_6ef14fcc-3443-4782-aa38-0910a5aea9b9-2, groupId=service_gcp-edge_6ef14fcc-3443-4782-aa38-0910a5aea9b9] Connection to node -1 (kafka0-data-europe-west4-kafka.internal/<<some_ip>>:9093) terminated during authentication. This may happen due to any of the following reasons: (1) Authentication failed due to invalid credentials with brokers older than 1.0.0, (2) Firewall blocking Kafka TLS traffic (eg it may only allow HTTPS traffic), (3) Transient network issue.
Upon bashing into the cluster, I could connect to the topic and consume messages with a command line tool and so it removes the possibility of any infra related issue.
Can someone please help me figure out what am I doing wrong here and how can I fix this?