Spark Structured Streaming output not displaying in inteliJ console - scala

Question

I'm trying to emulate this example from Jacek Laskowski Book of read a CSV file and aggregate the data in the console, but for some reason, the output is not displaying in the InteliJ console.
scala> spark.version
res4: String = 2.2.0
I found some reference in some places (1, 2, 3, 4, 5) here in SO, I tried everything but I didn't solve the problem.
This is the code:
package org.sample
import org.apache.spark.SparkConf
import org.apache.spark.sql.SparkSession
import org.apache.spark.sql.streaming.{OutputMode, Trigger}
object App {
def main(args : Array[String]): Unit = {
val DIR = new java.io.File(".").getCanonicalPath + "dataset/stream_in"
val conf = new SparkConf()
.setMaster("local[*]")
.setAppName("Spark Structured Streaming Job")
val spark = SparkSession.builder()
.appName("Spark Structured Streaming Job")
.master("local[*]")
.getOrCreate()
val reader = spark.readStream
.format("csv")
.option("header", true)
.option("delimiter", ";")
.option("latestFirst", "true")
.schema(SchemaDefinition.csvSchema)
.load(DIR + "/*")
reader.createOrReplaceTempView("user_records")
val tranformation = spark.sql(
"""
SELECT carrier, marital_status, COUNT(1) as num_users
FROM user_records
GROUP BY carrier, marital_status
"""
)
val consoleStream = tranformation
.writeStream
.format("console")
.option("truncate", false)
.outputMode("complete")
.start()
consoleStream.awaitTermination()
}
}
My output it is only:
18/11/30 15:40:31 INFO StreamExecution: Streaming query made progress: {
"id" : "9420f826-0daf-40c9-a427-e89ed42ee738",
"runId" : "991c9085-3425-4ea6-82af-4cef20007a66",
"name" : null,
"timestamp" : "2018-11-30T14:40:31.117Z",
"numInputRows" : 0,
"inputRowsPerSecond" : 0.0,
"processedRowsPerSecond" : 0.0,
"durationMs" : {
"getOffset" : 2,
"triggerExecution" : 2
},
"eventTime" : {
"watermark" : "1970-01-01T00:00:00.000Z"
},
"stateOperators" : [ ],
"sources" : [ {
"description" : "FileStreamSource[file:/structured-streamming-taskdataset/stream_in/*]",
"startOffset" : null,
"endOffset" : null,
"numInputRows" : 0,
"inputRowsPerSecond" : 0.0,
"processedRowsPerSecond" : 0.0
} ],
"sink" : {
"description" : "org.apache.spark.sql.execution.streaming.ConsoleSink#6a62e7ef"
}
}

Answer 1

I redefine the file and right now worked for me:
Differences:
remove the unnecessary conf. With SparkSession we do not need to
call the conf
The .load(/*) didn't work. What worked was the keep only the path
dataset/stream_in;
The data to tranformation was wrong (the fields didn't match the
file)
Final code:
package org.sample
import org.apache.spark.sql.SparkSession
import org.apache.log4j.{Level, Logger}
object StreamCities {
def main(args : Array[String]): Unit = {
// Turn off logs in console
Logger.getLogger("org").setLevel(Level.OFF)
Logger.getLogger("akka").setLevel(Level.OFF)
val spark = SparkSession.builder()
.appName("Spark Structured Streaming get CSV and agregate")
.master("local[*]")
.getOrCreate()
// 01. Schema Definition: We'll put the structure of our
// CSV file. Can be done using a class, but for simplicity
// I'll keep it here
import org.apache.spark.sql.types._
def csvSchema = StructType {
StructType(Array(
StructField("id", StringType, true),
StructField("name", StringType, true),
StructField("city", StringType, true)
))
}
// 02. Read the Stream: Create DataFrame representing the
// stream of the CSV according our Schema. The source it is
// the folder in the .load() option
val users = spark.readStream
.format("csv")
.option("sep", ",")
.option("header", true)
.schema(csvSchema)
.load("dataset/stream_in")
// 03. Aggregation of the Stream: To use the .writeStream()
// we must pass a DF aggregated. We can do this using the
// Untyped API or SparkSQL
// 03.1: Aggregation using untyped API
//val aggUsers = users.groupBy("city").count()
// 03.2: Aggregation using Spark SQL
users.createOrReplaceTempView("user_records")
val aggUsers = spark.sql(
"""
SELECT city, COUNT(1) as num_users
FROM user_records
GROUP BY city"""
)
// Print the schema of our aggregation
println(aggUsers.printSchema())
// 04. Output the stream: Now we'll write our stream in
// console and as new files will be included in the folder
// that Spark it's listening the results will be updated
val consoleStream = aggUsers.writeStream
.outputMode("complete")
.format("console")
.start()
.awaitTermination()
}
}

Answer 2

I had the same issue, I solved it with:
.option("startingOffsets", "earliest") \
in the
def read_from_kafka(spark):
df_sales = spark.readStream \
.format("kafka") \
.option("kafka.bootstrap.servers", "localhost:9092") \
.option("subscribe", "spark_topic_sales") \
.option("startingOffsets", "earliest") \
.load()
return df_sales
this will make sure the data is read from the topic from the earliest. hope it will help someone and he would not spend 2 hours like I did!