How to run multiple structured streams in parallel? - scala

Question

I am trying to write to streams parallel, but just the last one is working.
Here is my code.
val trump_topic = filteredQueryTrump.select($"ID".as("key"), $"title".as("value"))
.writeStream.format("kafka")
.trigger(Trigger.ProcessingTime("10 seconds"))
.option("kafka.security.protocol", "SASL_SSL")
.option("kafka.sasl.mechanism", "SCRAM-SHA-256")
.option("kafka.sasl.jaas.config", """org.apache.kafka.common.security.scram.ScramLoginModule required username="" password="";""")
.option("kafka.bootstrap.servers", "rocket-01.srvs.cloudkafka.com:9094")
.option("topic", "trump")
.option("checkpointLocation", "/tmp").start()
val biden_topic = filteredQueryBiden.select($"ID".as("key"), $"title".as("value"))
.writeStream.format("kafka")
.trigger(Trigger.ProcessingTime("4 seconds"))
.option("kafka.security.protocol", "SASL_SSL")
.option("kafka.sasl.mechanism", "SCRAM-SHA-256")
.option("kafka.sasl.jaas.config", """org.apache.kafka.common.security.scram.ScramLoginModule required username="" password="";""")
.option("kafka.bootstrap.servers", "rocket-01.srvs.cloudkafka.com:9094")
.option("topic", "biden")
.option("checkpointLocation", "/tmp").start()
I tried spark.streams.awaitAnyTermination() but it didn't help either.

Answer 1

You need to have two different checkpoint locations for each output stream.
Also, do add awaitAnyTermination as you already have tried out to ensure the jobs are running continously.
spark.streams.awaitAnyTermination()
As an alternative you could combine those two queries into one by doing
val combinedDf = filteredQueryTrump.select($"ID".as("key"), $"title".as("value"), lit("trump").as("topic"))
.union(filteredQueryBiden.select($"ID".as("key"), $"title".as("value"), lit("biden").as("topic")))
The content of the column topic will be interpreted by the spark-sql-kafka-0-10 library to decide into which topic the row should be written to. Theoretically, you can use that technique to write to multiple Kafka topics within one stream. When doing this it is important to not use topic option when calling writeStream. However, in that way you will not be able to have two different Trigger durations but only one for the overall stream.
An example you code could look like this:
val outputQuery = combinedDf.select($"key", $"value", $"topic")
.writeStream
.format("kafka")
.trigger(Trigger.ProcessingTime("4 seconds"))
.option("kafka.security.protocol", "SASL_SSL")
.option("kafka.sasl.mechanism", "SCRAM-SHA-256")
.option("kafka.sasl.jaas.config", """org.apache.kafka.common.security.scram.ScramLoginModule required username="" password="";""")
.option("kafka.bootstrap.servers", "rocket-01.srvs.cloudkafka.com:9094")
.option("checkpointLocation", "/tmp")
outputQuery.start()
outputQuery.awaitTermination()