Writing output of the Principal Components Analysis to text file - scala

Question

I have performed a Principal Component Analysis on a matrix I previously loaded with sc.textFile. The output being a org.apache.spark.mllib.linalg.Matrix I then converted it to a RDD[Vector[Double]].
with:
import java.io.PrintWriter
I did:
val pw = new PrintWriter("Matrix.csv")
rows3.collect().foreach(line => pw.println(line))
pw.flush
The output csv is promising. the only problem is that each line is a DenseVector(some values). How do I split each line into the corresponding coefficients?
Thanks a lot

Answer 1

You can use results of the computePrincipalComponents and breeze.linalg.csvwrite:
import java.io.File
import breeze.linalg.{DenseMatrix => BDM, csvwrite}
val mat: RowMatrix = ...
val pca = mat.computePrincipalComponents(...)
csvwrite(
new File("Matrix.csv"),
new BDM[Double](mat.numRows, mat.numCols, mat.toArray))

Answer 2

convert each vector to a string (you can do it either on the driver or the executers)
val pw = new PrintWriter("Matrix.csv")
rows3.map(_.mkString(",")).collect().foreach(line => pw.println(line))
pw.flush
edit:
if your data is too big to fit in the memory of the driver, you can try something like that:
val rdd = rows3.map(_.mkString(",")).zipWithIndex.cache
val total = rdd.count
val step = 10000 //rows in each chunk
val range = 0 to total by step
val limits = ranges.zip(range.drop(1))
limits.foreach { case(start, end) =>
rdd.filter(x => x._2 >= start && x._2 < end)
.map(_._1)
.collect
.foreach(pw.println(_))
}
I can't try this out, but that is the general idea