How to stream messages from Databricks to a Kafka client using Azure Event Hubs [closed] - apache-kafka

Question

Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 22 days ago.
Improve this question
I have a process that reads from a Kafka queue and writes into a DWH. The kafka queue is currently receiving data from a Java application, that reads from a local storage and writes into the Kafka queue.
We need to implement the following:
replace the local storage with an Azure Storage Account (DONE)
replace the Kafka queue with Azure Event Hubs
replace the java application with a Databricks simple job that does a readStream using Autloader from the Azure DataLake, and writes into the Azure Event Hubs
Constraint: the kafka client consumer cannot be changed, rather than its connection string.
Now, The good news is that Azure Event Hubs is Kafka-compliant (let's consider that the json body of each message is smaller than 10Kb), so my question is how to configure this architecture. More specifically:
A) how should Azure EH be configured to be kafka-compliant towards its consumer?
B) should I use kafka protocol also from Databricks to SEND the messages, or can I use it an Azure Event Hubs trusting the fact that it exposes itself with Kafka interface to the consumer, and with Event Hubs interface to the sender?
C) where can I retrieve the kafka endpoint to be used from the consumer, and what should I care of in addition to the new connection string? In the listen policy I find Primary Key, Connection String and SAS Policy ARM ID but I'm not sure how to convert them to a Kafka enpoint

Answer 1

To use EventHubs from Kafka protocol you need just configure Kafka options correctly. You need following:
we need to get Shared Access Signatures (SAS) to authenticate to Event Hubs topic - it should look like Endpoint=sb://<....>.windows.net/;?... and will be used as a password. For security reasons it's recommended to put it into a Databricks secret scope (update variables secret_scope and secret_name with your actual values).
we need to form the correct string (the eh_sasl variable) for SASL (Simple Authentication and Security Layer) authentication - as a user name we're using static value $ConnectionString, and Event Hubs SAS is used as a password. SASL string looks a bit different on Databricks - instead of org.apache.kafka.common.security.plain.PlainLoginModule... it should be prefixed with kafkashaded. as the original Java package is shaded to avoid conflicts with other packages.
you need to provide the name of the Event Hubs namespace & topic from which to read data in eh_namespace_name and topic_name variables.
secret_scope = "scope"
secret_name = "eventhub_sas"
topic_name = "topic1"
eh_namespace_name = "<eh-ns-name>"
readConnectionString = dbutils.secrets.get(secret_scope, secret_name)
eh_sasl = 'kafkashaded.org.apache.kafka.common.security.plain.PlainLoginModule' \
+ f' required username="$ConnectionString" password="{readConnectionString}";'
bootstrap_servers = f"{eh_namespace_name}.servicebus.windows.net:9093"
kafka_options = {
"kafka.bootstrap.servers": bootstrap_servers,
"kafka.sasl.mechanism": "PLAIN",
"kafka.security.protocol": "SASL_SSL",
"kafka.request.timeout.ms": "60000",
"kafka.session.timeout.ms": "30000",
"startingOffsets": "earliest",
"kafka.sasl.jaas.config": eh_sasl,
"subscribe": topic_name,
}
df = spark.readStream.format("kafka") \
.options(**kafka_options).load()
Writing is done with the similar configuration.
from pyspark.sql.functions import struct, to_json
# work with your dataframe
kafka_options = {
"kafka.bootstrap.servers": bootstrap_servers,
"kafka.sasl.mechanism": "PLAIN",
"kafka.security.protocol": "SASL_SSL",
"kafka.sasl.jaas.config": eh_sasl,
"topic": topic_name,
}
df.select(to_json(struct("*")).alias("value")) \
.write.format("kafka").options(**kafka_options).save()
See more details about Spark + Kafka in the Spark & Databricks documentation.