How to create array of structs in spark - scala

Question

I'm trying to create a array of struct(col, col) in spark dataframe but getting error.
Using sample data to arrive at same error here.
Dataframe
val df = Seq((1, "One", "uno", true), (2, "Two", "Dos", true), (3, "Three", "Tres", false)).toDF("number", "English", "Spanish", "include_spanish")
scala> df.show
+------+-------+-------+---------------+
|number|English|Spanish|include_spanish|
+------+-------+-------+---------------+
| 1| One| uno| true|
| 2| Two| Dos| true|
| 3| Three| Tres| false|
+------+-------+-------+---------------+
Now, here is I'm trying to create struct out of existing columns and then create an array out of it.
val df1 = df.withColumn("numberToEnglish", struct(col("number"), col("English"))).withColumn("numberToSpanish", struct("number", "Spanish")).withColumn("numberToLanguage", when(col("include_spanish") === true, array("numberToEnglish", "numberToSpanish")).otherwise(array("numberToEnglish"))
Getting below error,
org.apache.spark.sql.AnalysisException: cannot resolve 'array(`numberToEnglish`, `numberToSpanish`)' due to data type mismatch: input to function array should all be the same type, but it's [struct<number:int,English:string>, struct<number:int,Spanish:string>];;
'Project [number#200, English#201, Spanish#202, include_spanish#203, numberToEnglish#253, numberToSpanish#259, CASE WHEN (include_spanish#203 = true) THEN array(numberToEnglish#253, numberToSpanish#259) ELSE array(numberToEnglish#253) END AS numberToLanguage#266]
what would be the best way achieve this functionality ?

Answer 1

In order for the array method to view struct($"number", $"English") and struct($"number", $"Spanish") as same data type, you'll need to name the struct elements, as shown below:
val df = Seq(
(1, "One", "uno", true), (2, "Two", "Dos", true), (3, "Three", "Tres", false)
).toDF("number", "English", "Spanish", "include_spanish")
df.
withColumn("numberToEnglish", struct($"number".as("num"), $"English".as("lang"))).
withColumn("numberToSpanish", struct($"number".as("num"), $"Spanish".as("lang"))).
withColumn("numberToLanguage",
when($"include_spanish", array($"numberToEnglish", $"numberToSpanish")).
otherwise(array($"numberToEnglish"))
).
show
// +------+-------+-------+---------------+---------------+---------------+--------------------+
// |number|English|Spanish|include_spanish|numberToEnglish|numberToSpanish| numberToLanguage|
// +------+-------+-------+---------------+---------------+---------------+--------------------+
// | 1| One| uno| true| [1, One]| [1, uno]|[[1, One], [1, uno]]|
// | 2| Two| Dos| true| [2, Two]| [2, Dos]|[[2, Two], [2, Dos]]|
// | 3| Three| Tres| false| [3, Three]| [3, Tres]| [[3, Three]]|
// +------+-------+-------+---------------+---------------+---------------+--------------------+