Trouble getting Spark aggregators to work - scala

Question

I am wanting to try out Aggregators in Scala Spark, but I cannot seem to get them to work using both the select function and the groupBy/agg functions (with my current implementation the agg function fails to compile). My aggregator is written below and should be self explanatory.
import org.apache.spark.sql.expressions.Aggregator
import org.apache.spark.sql.{Encoder, Encoders}
/** Stores the number of true counts (tc) and false counts (fc) */
case class Counts(var tc: Long, var fc: Long)
/** Count the number of true and false occurances of a function */
class BooleanCounter[A](f: A => Boolean) extends Aggregator[A, Counts, Counts] with Serializable {
// Initialize both counts to zero
def zero: Counts = Counts(0L, 0L)
// Sum counts for intermediate value and new value
def reduce(acc: Counts, other: A): Counts = {
if (f(other)) acc.tc += 1 else acc.fc += 1
acc
}
// Sum counts for intermediate values
def merge(acc1: Counts, acc2: Counts): Counts = {
acc1.tc += acc2.tc
acc1.fc += acc2.fc
acc1
}
// Return results
def finish(acc: Counts): Counts = acc
// Encoder for intermediate value type
def bufferEncoder: Encoder[Counts] = Encoders.product[Counts]
// Encoder for return type
def outputEncoder: Encoder[Counts] = Encoders.product[Counts]
}
Below is my test code.
val ds: Dataset[Employee] = Seq(
Employee("John", 110),
Employee("Paul", 100),
Employee("George", 0),
Employee("Ringo", 80)
).toDS()
val salaryCounter = new BooleanCounter[Employee]((r: Employee) => r.salary < 10).toColumn
// Usage works fine
ds.select(salaryCounter).show()
// Causes an error
ds.groupBy($"name").agg(salaryCounter).show()
The first usage of salaryCounter works fine but the second results in the following compilation error.
java.lang.ClassCastException: org.apache.spark.sql.catalyst.expressions.GenericRowWithSchema cannot be cast to Employee
Databricks has a tutorial that is rather complicated but appears to be Spark 2.3. There is also this older tutorial that uses an experimental feature from Spark 1.6.

Answer 1

You're incorrectly mixing "statically typed" and "dynamically typed" APIs. To use the former version you should call agg on KeyValueGroupedDataset, not RelationalGroupedDataset:
ds.groupByKey(_.name).agg(salaryCounter)