How to save or print receiverStream data in Spark scala - scala

Question

Im using the below custom receiver to consume data from Rabbitmq in Spark-Scala.
import org.apache.spark.streaming.rabbitmq.RabbitMQUtils
import java.io.{BufferedReader, InputStreamReader}
import java.net.Socket
import java.nio.charset.StandardCharsets
import org.apache.spark.SparkConf
import org.apache.spark.rdd.RDD
import org.apache.spark.storage.StorageLevel
import org.apache.spark.streaming.{Seconds, StreamingContext}
import org.apache.spark.streaming.dstream.{InputDStream, ReceiverInputDStream}
import org.apache.spark.SparkContext
import org.apache.spark.streaming._
import org.apache.spark.streaming.receiver._
import org.apache.spark.internal.Logging
import org.apache.spark._
import org.apache.spark.streaming._
import org.apache.spark.internal.Logging
import org.slf4j.LoggerFactory
val host = "10.*.*.*"
val port = "5**"
val queueName = "test"
val vHost = "sd"
val userName = "user1"
val password = "***"
var batchInterval = Seconds(500)
var ssc = new StreamingContext(sc, batchInterval)
val receiverStream = RabbitMQUtils.createStream(ssc, Map(
"host" -> host,
"port" -> port,
"queueName" -> queueName,
"vHost" -> vHost,
"userName" -> userName,
"password" -> password
))
receiverStream.start()
println("started receiverStream")
ssc.start()
ssc.awaitTermination()
How to print the receiverstream data and later save the file to any specific location.
Could someone please assist.
Thank you.

Answer 1

You should be able to print the stream data with:
receiverStream
.writeStream
.outputMode("complete")
.format("console")
.start()
Reference: https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html#quick-example
You can save to parquet (or other) files with:
receiverStream
.writeStream
.format("parquet") // can be "orc", "json", "csv", etc.
.option("path", "path/to/destination/dir")
.start()
See: https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html#output-sinks