storing the string from kafkaStream into a variable for processing - scala

Question

I need to get my messages from a Kafka producer and from the messages I need to find the words that contain % and generate a message for different % values. Finally I need to send it to ElasticSearch.
I am able to see the values in console using kafkaStream.print() but I need to process the string to match with required keywords and generate the message.
My code:
package rnd
import org.apache.spark.SparkConf
import kafka.serializer.StringDecoder
import org.apache.spark.sql.SQLContext
import org.apache.spark.streaming.dstream.DStream
import org.apache.spark.streaming.kafka.KafkaUtils
import org.apache.spark.streaming.{Minutes, Seconds, StreamingContext}
import org.apache.spark.{SparkConf, SparkContext}
object WordFind {
def main(args: Array[String]) {
val conf = new SparkConf().setMaster("local").setAppName("KafkaReceiver")
val checkpointDir = "/usr/local/kafka/kafka_2.11-0.11.0.2/checkpoint/"
import org.apache.spark.streaming.StreamingContext
import org.apache.spark.streaming.Seconds
val batchIntervalSeconds = 2
val ssc = new StreamingContext(conf, Seconds(10))
import org.apache.spark.streaming.kafka.KafkaUtils
val kafkaStream = KafkaUtils.createStream(ssc, "localhost:2181", "spark-streaming-consumer-group", Map("wordcounttopic" -> 5))
val s = kafkaStream.print()
println(" the words are: " + s)
ssc.remember(Minutes(1))
ssc.checkpoint(checkpointDir)
ssc
ssc.start()
ssc.awaitTerminationOrTimeout(batchIntervalSeconds * 5 * 1000)
}
}
If I pass "The usage is 75%" through the Lafka producer I should generate a message saying "Increase ram by 25%" in ElasticSearch.
The output that I am getting is:
18/02/09 16:38:27 INFO BlockManagerMasterEndpoint: Registering block manager localhost:37879 with 2.4 GB RAM, BlockManagerId(driver, localhost, 37879)
18/02/09 16:38:27 INFO BlockManagerMaster: Registered BlockManager
18/02/09 16:38:27 WARN StreamingContext: spark.master should be set as local[n], n > 1 in local mode if you have receivers to get data, otherwise Spark jobs will not get resources to process the received data.
***the words are: ()***
I want the String that i am passing in place of () in 's'.

Answer 1

The val kafkaStream is a RecieverInputDStream[(String, String)], where the data is (kafkaMetaData, kafkaMessage)
for more information see [https://github.com/apache/spark/blob/f830bb9170f6b853565d9dd30ca7418b93a54fe3/external/kafka-0-8/src/main/scala/org/apache/spark/streaming/kafka/KafkaInputDStream.scala#L135 ].
We need to extract the second of the tuple and do the pattern matching (i.e filter RecieverInputDStream find the words that contain %) and then use map to generate output (i.e a message for different % values). And as mentioned by #stefanobaghino, the print() function just prints the output to the console and doesn't return any string of the record.
for example:
import org.apache.spark.streaming.dstream.ReceiverInputDStream
val kafkaStream: ReceiverInputDStream[(String, String)] = KafkaUtils.createStream(sparkStreamingContext, "localhost:2181",
"spark-streaming-consumer-group", Map("wordcounttopic" -> 5))
import org.apache.spark.streaming.dstream.DStream
val filteredStream: DStream[(String, String)] = kafkaStream
.filter(record => record._2.contains("%")) // TODO : pattern matching here
val outputDStream: DStream[String] = filteredStream
.map(record => record._2.toUpperCase()) // just assuming some operation
outputDStream.print()
Use the outputDStream to be written into ElasticSearch. Hope this helps.