array_sort function sorting the data based on first numerical element in Array<Struct> - scala

Question

I need to sort the array<struct> based on a particular element from a struct. I am trying to use the array_sort function and could see that by default it is sorting the array but based on the first numerical element. Is this the expected behavior? PFB sample code and output.
val jsonData = """
{
"topping":
[
{ "id": "5001", "id1": "5001", "type": "None" },
{ "id": "5002", "id1": "5008", "type": "Glazed" },
{ "id": "5005", "id1": "5007", "type": "Sugar" },
{ "id": "5007", "id1": "5002", "type": "Powdered Sugar" },
{ "id": "5006", "id1": "5005", "type": "Chocolate with Sprinkles" },
{ "id": "5003", "id1": "5004", "type": "Chocolate" },
{ "id": "5004", "id1": "5003", "type": "Maple" }
]
}
"""
val json_df = spark.read.json(Seq(jsonData).toDS)
val sort_df = json_df.select(array_sort($"topping").as("sort_col"))
display(sort_df)
OUTPUT
As you could see the above output is sorted based on the id element which is the first numerical element in the struct.
Is there any way to specify the element based on which sorting can be done?

Answer 1

Is this the expected behavior?
Short answer, yes!
For arrays with struct type elements, It compares first fields to determine the order and if they are equal it compares the second fields and so on. You can see that easily by modifying your input data example to have the same value in id for 2 rows, you'll then notice the order is determined by the second field.
The array_sort function uses collection operation ArraySort. If you look into the code you'll find how it handles complex DataTypes like StructType.
Is there any way to specify the element based on which sorting can be done?
One way is using a tranform function to change the positions of the struct fields so that you can have the first field containing the values you want the ordering to be based on. Example: if you want to order by the field type:
val transform_expr = "TRANSFORM(topping, x -> struct(x.type as type, x.id as id, x.id1 as id1))"
val transform_df = json_df.select(expr(transform_expr).alias("topping_transform"))
transfrom_df.show(fasle)
//+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
//|topping_transform |
//+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
//|[[None, 5001, 5001], [Glazed, 5002, 5008], [Sugar, 5005, 5007], [Powdered Sugar, 5007, 5002], [Chocolate with Sprinkles, 5006, 5005], [Chocolate, 5003, 5004], [Maple, 5004, 5003]]|
//+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
val sort_df = transform_df.select(array_sort($"topping_transform").as("sort_col"))
sort_df.show(false)
//+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
//|sort_col |
//+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
//|[[Chocolate, 5003, 5004], [Chocolate with Sprinkles, 5006, 5005], [Glazed, 5002, 5008], [Maple, 5004, 5003], [None, 5001, 5001], [Powdered Sugar, 5007, 5002], [Sugar, 5005, 5007]]|
//+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+