Spark VectorAssembler Error With Array of Column Input - scala

Question

I have the following code that constructs a VectorAssembler:
val allColsExceptOceanProximity: Array[String] = dfRaw.drop("ocean_proximity").columns
val assembler = new VectorAssembler()
//.setInputCols(Array("longitude", "latitude", "housing_median_age", "total_rooms", "population", "households", "population", "median_income", "median_house_value"))
.setInputCols(allColsExceptOceanProximity)
.setOutputCol("features")
If I use the Array with the columns statically types, it works. When I pass in the Array dynamically, it kind of fails with the following error:
org.apache.spark.SparkException: Job aborted due to stage failure: Task 2 in stage 1763.0 failed 1 times, most recent failure: Lost task 2.0 in stage 1763.0 (TID 100292) (192.168.0.35 executor driver): org.apache.spark.SparkException: Failed to execute user defined function(VectorAssembler$$Lambda$4540/1415205968: (struct<longitude:double,latitude:double,housing_median_age:double,total_rooms:double,total_bedrooms:double,population:double,households:double,median_income:double,median_house_value:double>) => struct<type:tinyint,size:int,indices:array<int>,values:array<double>>)
at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage2.agg_doAggregateWithKeysOutput_0$(Unknown Source)
at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage2.processNext(Unknown Source)
at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
at org.apache.spark.sql.execution.WholeStageCodegenExec$$anon$1.hasNext(WholeStageCodegenExec.scala:755)
at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:458)
at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:458)
at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:458)
at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:458)
at scala.collection.Iterator.foreach(Iterator.scala:941)
Why is this? Any ideas. Is is expecting a Varargs type? I'm on my notebook and do not have the benefits of the IDE.
EDIT: Added the schema:
root
|-- longitude: double (nullable = true)
|-- latitude: double (nullable = true)
|-- housing_median_age: double (nullable = true)
|-- total_rooms: double (nullable = true)
|-- total_bedrooms: double (nullable = true)
|-- population: double (nullable = true)
|-- households: double (nullable = true)
|-- median_income: double (nullable = true)
|-- median_house_value: double (nullable = true)
|-- ocean_proximity: string (nullable = true)