Spark Structured Stream null in output dataset - spark-structured-streaming

Question

i run a scala code which aggregates data and print output to the console. Unfortunately, i got a nulls after group operation. Current output:
|Id |Date | Count |
|null|null | 35471|
I realised, that the bottle neck is the point, when i group data - when i try to use column other than numeric, output returns nulls. Any advice will be welcome - i lost hours to find solution.
My code:
// create schema
val sensorsSchema = new StructType()
.add("SensorId", IntegerType)
.add("Timestamp", TimestampType)
.add("Value", DoubleType)
.add("State", StringType)
// read streaming data from csv...
// aggregate streaming data
val streamAgg = streamIn
.withColumn("Date", to_date(unix_timestamp($"Timestamp", "dd/MM/yyyy").cast(TimestampType)))
.groupBy("SensorId", "Date")
.count()
// write streaming data...

Answer 1

I change the code - now works perfect:
/****************************************
* STREAMING APP
* 1.0 beta
*****************************************
* read data from csv (local)
* and save as parquet (local)
****************************************/
package tk.streaming
import org.apache.spark.SparkConf
import org.apache.spark.sql._
// import org.apache.spark.sql.functions._
case class SensorsSchema(SensorId: Int, Timestamp: String, Value: Double, State: String, OperatorId: Int)
object Runner {
def main(args: Array[String]): Unit = {
// Configuration parameters (to create spark session and contexts)
val appName = "StreamingApp" // app name
val master = "local[*]" // master configuration
val dataDir = "/home/usr_spark/Projects/SparkStreaming/data"
val refreshInterval = 30 // seconds
// initialize context
val conf = new SparkConf().setMaster(master).setAppName(appName)
val spark = SparkSession.builder.config(conf).getOrCreate()
import spark.implicits._
// TODO change file source to Kafka (must)
// read streaming data
val sensorsSchema = Encoders.product[SensorsSchema].schema
val streamIn = spark.readStream
.format("csv")
.schema(sensorsSchema)
.load(dataDir + "/input")
.select("SensorId", "Timestamp", "State", "Value") // remove "OperatorId" column
// TODO save result in S3 (nice to have)
// write streaming data
import org.apache.spark.sql.streaming.Trigger
val streamOut = streamIn.writeStream
.queryName("streamingOutput")
.format("parquet")
.option("checkpointLocation", dataDir + "/output/checkpoint")
.option("path", dataDir + "/output")
.start()
streamOut.awaitTermination() // start streaming data
}
}