training a classifier with data within a partition - scala

Question

How can I train a classifier with instances within a partition while its classification algorithm depends on the partition index? For example, suppose the following code snippet:
val data = MLUtils.loadLibSVMFile(sc, "path to SVM file")
val r = data.mapPartitionsWithIndex((index,localdata)=>{
if (index % 2 == 0)
{
// train a NaiveBayes with localdata
NaiveBayes.train(localdata) // Error => found:iterator[LabeledPoint] , required: RDD[labeledPoint]
}
else
{
// train a DecisionTree classifier with localdata
DecisionTree.train(localdata) // Error => found:iterator[LabeledPoint] , required: RDD[labeledPoint]
}
})
It sounds to me that the error is right, because the tasks are executed within their separated JVM and could not be distributed from a map task. That is why I can not access the SparkContext in my tasks. However, does anyone have an alternate suggestion for doing my purpose?

Answer 1

based on the discussion in above comments section -
you can give a try to this-
val rdd = MLUtils.loadLibSVMFile(sc, "path to SVM file")
// approach -1
val nb = rdd.sample(withReplacement = false, fraction = 0.5) // sample 50% of the record
val dt = rdd.sample(withReplacement = false, fraction = 0.5) // sample 50% of the record
//or approach-2
val (nb, dt) = rdd.randomSplit(Array(0.5, 0.5))
// apply algo
NaiveBayes.train(nb)
DecisionTree.train(dt, strategy= ..)