Spark Scala VectorAssembler IllegalArgumentException: XX does not exist. Available: number - scala

Question

I'm using Scala on Spark and I have dense matrix like this:
vp
res63: org.apache.spark.ml.linalg.DenseMatrix =
-0.26035262239241047 -0.9349256695883184
0.08719326360909431 -0.06393629243008418
0.006698866707269257 0.04124873027993731
0.011979122705128064 -0.005430767154896149
0.049075485175059275 0.04810618828561001
0.001605411530424612 0.015016736357799364
0.9587151228619724 -0.2534046936998956
-0.04668498310146597 0.06015550772431999
-0.022360873382091598 -0.22147143481166376
-0.014153052584280682 -0.025947327705852636
I want use VectorAssembler to create feature column so I transform vp:
import org.apache.spark.ml.feature.VectorAssembler
import org.apache.spark.ml.linalg.Vectors
val c = vp.toArray.toSeq
val vp_df = c.toDF("number")
vp_df.createOrReplaceTempView("vp_df")
val vp_list = vp_df.collect.map(_.toSeq).flatten
val vp_string = vp_list.map(_.toString)
vp_string
res64: Array[String] = Array(-0.26035262239241047, 0.08719326360909431, 0.006698866707269257, 0.011979122705128064, 0.049075485175059275, 0.001605411530424612, 0.9587151228619724, -0.04668498310146597, -0.022360873382091598, -0.014153052584280682, -0.9349256695883184, -0.06393629243008418, 0.04124873027993731, -0.005430767154896149, 0.04810618828561001, 0.015016736357799364, -0.2534046936998956, 0.06015550772431999, -0.22147143481166376, -0.025947327705852636)
Then I use VectorAssembler:
val assembler = new VectorAssembler().setInputCols(vp_string).setOutputCol("features")
val output = assembler.transform(vp_df)
output.select("features").show(false)
But I have an error and I don't understand why
IllegalArgumentException: -0.26035262239241047 does not exist. Available: number
I don't know how this is possible, I've done the AssemblerVector several times and this is the first time I've seen this

Answer 1

Your "number" column is already in an array of double, so all you need is to convert this column into a dense vector.
val arrayToVectorUDF = udf((array : Seq[Double]) => {
Vectors.dense(array.toArray)
})
vp_df.withColumn("vector", arrayToVectorUDF(col("number")))
Update: I misunderstood your code.
The number column is a DoubleType column, so all you need to do is pass the column name to the vector assembler.
import org.apache.spark.ml.linalg.DenseMatrix
import org.apache.spark.ml.feature.VectorAssembler
import org.apache.spark.ml.linalg.Vectors
val data = (1 to 20).map(_.toDouble).toArray
val dm = new DenseMatrix(2, 10, data)
val vp_df = dm.toArray.toSeq.toDF("number")
val assembler = new VectorAssembler().setInputCols(Array("number")).setOutputCol("features")
val output = assembler.transform(vp_df)
output.select("features").show(false)
+--------+
|features|
+--------+
|[1.0] |
|[2.0] |
|[3.0] |
|[4.0] |
|[5.0] |
|[6.0] |
|[7.0] |
|[8.0] |
|[9.0] |
|[10.0] |
|[11.0] |
|[12.0] |
|[13.0] |
|[14.0] |
|[15.0] |
|[16.0] |
|[17.0] |
|[18.0] |
|[19.0] |
|[20.0] |
+--------+