Kafka Topic Name in parameterized way - scala

Question

val envi = "test1"
val Q_stream = spark
.readStream
.format("kafka")
.option("kafka.bootstrap.servers", "Kafka.Server")
.option("subscribe", "topicname_${envi}")
.option("startingOffsets", "latest")
.option("failOnDataLoss", "false")
.option("kafka.group.id", "myConsumerGroup")
.load()
.toDF()
.select(col("key").cast("string").as("key"), col("value").cast("string"))
val value2Stream = Q_stream
.filter(col("key") === "AB")
.select(functions.from_json(col("value"), ABSchema).as("value"))
.select("value.*")
value2Stream.writeStream.format("orc")
.option("metastoreUri", "hive.warehouse.metastoreUri")
.option("checkpointLocation", "/tmp/strlocation_${envi}/AB/checkpointtest1")
.option("path", "/tmp/str2/AB")
.trigger(Trigger.ProcessingTime("5 Seconds"))
.partitionBy("jobid")
.start()
This code is working fine, but it failed when we are subscribing another topic. So requirement is when submit a job in test1 environment, then kafka topic should subscribe topicname_test1. And when submit a job in test2 environment, then kafka topic should subscribe topicname_test2.
I tried this also which is not working.
var isQA: Boolean = false
if (args.length > 0) {
isQA = args(0).toString.trim().toUpperCase == "Q"
getEnvironment = args(1).toString.trim().toUpperCase == "TESTDEVC"
}
var env = ""
var envi = ""
if (isQA) {
configManager.setup("test.conf")
env = "QA"
if (getEnvironment == "TESTDEVC") { // TODO: implement
envi = "TESTDEVC"
} else if (getEnvironment == "TESTDEVC1") {
envi = "TESTDEVC1"
}else if (getEnvironment == "TESTDEVC2") {
envi = "TESTDEVC2"
}
else {
throw new RuntimeException("unknown environment")
}
}
val Q_stream = spark
.readStream
.format("kafka")
.option("kafka.bootstrap.servers", "Kafka.Server")
.option("subscribe", "test_${envi}")
.option("startingOffsets", "latest")
.option("failOnDataLoss", "false")
.option("kafka.group.id", "myConsumerGroup")
.load()
.toDF()
.select(col("key").cast("string").as("key"), col("value").cast("string"))
val value2Stream = Q_stream
.filter(col("key") === "AB")
.select(functions.from_json(col("value"), ABSchema).as("value"))
.select("value.*")
value2Stream.writeStream.format("orc")
.option("metastoreUri", "hive.warehouse.metastoreUri")
.option("checkpointLocation", "/tmp/strlocation_${envi}/AB/checkpointtest1")
.option("path", "/tmp/str2/AB")
.trigger(Trigger.ProcessingTime("5 Seconds"))
.partitionBy("jobid")
.start()

Answer 1

subscribe option can only be provided once, and that's not how you do string interpolation in Scala. You're missing the leading s
Sounds like you just need to check your environment, which is a scala question, not related to Spark...
def main(args: Array[String]) {
var env: String
if (getEnvironment() == "env1") {
env = "test1"
} else if (getEnvironment() == "env2") {
env = "test2"
} else {
throw new RuntimeException("unknown environment")
}
val Q_stream = spark
.readStream
.format("kafka")
.option("kafka.bootstrap.servers", "Kafka.Server")
.option("subscribe", s"topicname_${env}")
...
}
def getEnvironment() : String {
// TODO: implement
}
Same applies for checkpointLocation
Alternatively, you can provide the environment or topic suffix as an argument to spark-submit - https://spark.apache.org/docs/latest/submitting-applications.html#launching-applications-with-spark-submit