transform dataset into case class via (wrapped) encoders - scala

Question

I am new to Scala. Excuse my lack of knowledge.
This is my dataset:
val bfDS = sessions.select("bf")
sessions.select("bf").printSchema
|-- bf: array (nullable = true)
| |-- element: struct (containsNull = true)
| | |-- s: struct (nullable = true)
| | | |-- a: string (nullable = true)
| | | |-- b: string (nullable = true)
| | | |-- c: string (nullable = true)
| | |-- a: struct (nullable = true)
| | | |-- a: integer (nullable = true)
| | | |-- b: long (nullable = true)
| | | |-- c: integer (nullable = true)
| | | |-- d: array (nullable = true)
| | | | |-- element: struct (containsNull = true)
| | | | | |-- a: string (nullable = true)
| | | | | |-- b: integer (nullable = true)
| | | | | |-- c: long (nullable = true)
| | |-- tr: struct (nullable = true)
| | | |-- a: integer (nullable = true)
| | | |-- b: long (nullable = true)
| | | |-- c: integer (nullable = true)
| | | |-- d: array (nullable = true)
| | | | |-- element: struct (containsNull = true)
| | | | | |-- e: string (nullable = true)
| | | | | |-- f: integer (nullable = true)
| | | | | |-- g: long (nullable = true)
| | |-- cs: struct (nullable = true)
| | | |-- a: integer (nullable = true)
| | | |-- b: long (nullable = true)
| | | |-- c: integer (nullable = true)
| | | |-- d: array (nullable = true)
| | | | |-- element: struct (containsNull = true)
| | | | | |-- e: string (nullable = true)
| | | | | |-- f: integer (nullable = true)
| | | | | |-- g: long (nullable = true)
1) I don't think I understand Scala datasets very well. A dataset is composed of rows, but when I print the schema, it shows an array. How does the dataset map to the array? Is each row is an element in the array?
2) I want to convert my dataset into a case class.
case class Features( s: Iterable[CustomType], a: Iterable[CustomType], tr: Iterable[CustomType], cs: Iterable[CustomType])
How do I convert my dataset and how do I use encoders?
Many thanks.

Answer 1

Welcome to StackOverflow. Sadly this question is too board for SO, take a look at "how to ask" to improve this and future questions.
However I will try to answer a few of your questions.
First, Spark Rows can encode a variety of values, including Arrays & Structures.
Second, your dataframe's rows are composed of only one column of type Array[...].
Third, if you want to create a Dataset from your df, your case class must match your schema, in such case it should be something like:
case class Features(array: Array[Elements])
case class Elements(s: CustomType, a: CustomType, tr: CustomType, cs: CustomType)
Finally, an Encoder is used to transform your case classes and their values to the Spark internal representation. You shouldn't bother too much about them yet - you just need to import spark.implicits._ and all the encoders you need will be there automatically.
val spark = SparkSession.builder.getOrCreate()
import spark.implicits._
val ds: Dataset[Features] = df.as[Features]
Also, you should take a look to this as a reference.