How to group by a dataframe of a specific class - scala

Question

I have one dataframe with this schema:
|-- Agreement_A1: string (nullable = true)
|-- Line_A1: string (nullable = true)
|-- Line_A2: string (nullable = true)
I create a new dataframe with this code:
val df2 = df.map(row => new MapResultRequestLine().apply(row))(Encoders.bean(classOf[AgreementLine]))
Function apply() is this:
public AgreementLine apply(Row row) {
AgreementLine agrLine = new AgreementLine();
agrLine.Agreement_A1 = row.getAs("Agreement_A1");
Line res = new Line();
res.Line_A1 = row.getAs("Line_A1");
res.Line_A2 = row.getAs("Line_A2");
agrLine.line = res
return agrLine;
}
Class AgreementLine looks like this:
public class AgreementLine{
public String agreementCrocCode;
public Line line;
}
Class Line is this:
public class Line{
public String Line_A1;
public String Line_A2;
}
How to group df2 so the result dataframe had Agreement_A1 column and the list of Line?
I have tried it this way:
val groupedDF = df2.groupBy($"Agreement_A1").agg(collect_set((array($"line"))).as("lines"))
But it shows an error "cannot resolve 'Agreement_A1' given input columns: [];"

Answer 1

The issue is here:
val df2 = df.map(row => new MapResultRequestLine().apply(row))(Encoders.bean(classOf[AgreementLine]))
scala doesn't show the data type, so you think it's a DataFrame(as DataSet[Row]).
But actually, it's a DataSet(as DataSet[AgreementLine]). And thanks for your encoder, it has lost all the schema, that's the reason why your df2.printSchema return the empty result.
Thus, when you call df2.groupBy($"Agreement_A1"), it will throw the Exception because there is no column named "Agreement_A1".
Obviously, the solution is to update schema of your DataSet(df2 in your case).
And sadly, I have no idea how to do with this(I'm a Rookie as well).
My only solution is to convert the dataset back to RDD[Row](mention it is RDD[AgreementLine] if you want to use df2.rdd), and build a new DataFrame with custom schema.
Hope you'll get a better solution.