How to parallelize loading data into an RDD in Spark - scala

Question

I am using the MLLib library and the MLUtils.loadLibSVMFile() method. I have a file that is 10GB and cluster with 5 slaves, with 2 cores and 6 GB memory each. According to the documentation found here, the method has the following parameters:
loadLibSVMFile(sc, path, multiclass=False, numFeatures=-1, minPartitions=None)
I would like to have 10 partitions and I don't know the number of features. When I run the method without specifying the minimum partitions, I get the java.lang.OutOfMemoryError: Java heap space as would be expected.
However, when I specify the numFeatures to be -1 as is default in the documentation, and the number to partitions to 10, based on the WebUI it does distribute the work, but after some time I get a java.lang.ArrayIndexOutOfBoundsException. The rest of the code looks identical to the example code written here.
I am brand new to Spark so please tell me if I am making some obvious mistake. Thanks!
EDIT 2: I ran the same thing with a sample dataset that was tiny and dense, and it works fine. Seems to be a problem with the number of features?
EDIT 3:
Here is the code I am trying to run:
object MLLibExample {
def main(args: Array[String]) {
val sc = new SparkContext()
//loading the RDD
val data = MLUtils.loadLibSVMFile(sc, "s3n://file.libsvm", -1, 10)
// Run training algorithm to build the model
val numIterations = 10
val model = SVMWithSGD.train(data, numIterations)
}
}
Here is the exception:
15/07/20 20:42:43 WARN TaskSetManager: Lost task 0.0 in stage 4.0 (TID 35, 10.249.67.97): java.lang.ArrayIndexOutOfBoundsException: -1
at org.apache.spark.mllib.linalg.BLAS$.dot(BLAS.scala:136)
at org.apache.spark.mllib.linalg.BLAS$.dot(BLAS.scala:106)
at org.apache.spark.mllib.optimization.HingeGradient.compute(Gradient.scala:317)
at org.apache.spark.mllib.optimization.GradientDescent$$anonfun$runMiniBatchSGD$1$$anonfun$1.apply(GradientDescent.scala:192)
at org.apache.spark.mllib.optimization.GradientDescent$$anonfun$runMiniBatchSGD$1$$anonfun$1.apply(GradientDescent.scala:190)
-I have made sure the input file is correctly formatted (0/1 labels and space-delimited)
-Tried a subsample of the original file that is 10 MB
-Set both driver and executor memory
-Added this tag to my spark-submit command: --conf "spark.driver.maxResultSize=6G"