Scala: k fold cross validation - scala

Question

I want to do k-fold cross validation. Essentially we are given a bunch of data allData. Suppose we partition our input into "k" cluster and put it in groups.
The desired output is a trainAndTestDataList: List[(Iterable[T], Iterable[T])], where the Listis of size "k". The "i"th element of the trainAndTestDataList is a tuple like (A, B), where A should be the "i"th element of groups and B should be all elements of groups except the "i"th one, concatenated.
Any ideas on implementing this efficiently?
val allData: Iterable[T] = ... // we get the data from somewhere
val groupSize = Math.ceil(allData.size / k).toInt
val groups = allData.grouped(groupSize).toList
val trainAndTestDataList = ... // fill out this part
One thing to keep in mind is that allData can be very long, however "k" is very small (say 5). So it is very crucial to keep all the data vectors as Iterator (and not List, Seq, etc).
Update: Here is how I did (and I am not happy about it):
val trainAndTestDataList = {
(0 until k).map{ fold =>
val (a,b) = groups.zipWithIndex.partition{case (g, idx) => idx == fold}
(a.unzip._1.flatten.toIterable, b.unzip._1.flatten.toIterable)
}
}
Reasons I don't like it:
much twisted especially after partition where I do an unzip, then ._1 and flatten. I think one should be able to do a better job.
Although a is a Iterable[T], the output of a.unzip._1.flatten. is a List[T], I think. This is no good, since the number of the element in this list might be very large.

Answer 1

You could try that operation
implicit class TeeSplitOp[T](data: Iterable[T]) {
def teeSplit(count: Int): Stream[(Iterable[T], Iterable[T])] = {
val size = data.size
def piece(i: Int) = i * size / count
Stream.range(0, size - 1) map { i =>
val (prefix, rest) = data.splitAt(piece(i))
val (test, postfix) = rest.splitAt(piece(i + 1) - piece(i))
val train = prefix ++ postfix
(test, train)
}
}
}
This split will be as lazy as splitAt and ++ are within your collection type.
You can try it with
1 to 10 teeSplit 3 force

Answer 2

I believe this should work. It also takes care of the randomization (don't neglect this!) in a reasonably efficient manner, i.e. O(n) instead of O(n log(n)) required for the more naive approach using a random shuffle/permutation of the data.
import scala.util.Random
def testTrainDataList[T](
data: Seq[T],
k: Int,
seed: Long = System.currentTimeMillis()
): Seq[(Iterable[T], Iterable[T])] = {
def createKeys(n: Int, k: Int) = {
val groupSize = n/k
val rem = n % k
val cumCounts = Array.tabulate(k){ i =>
if (i < rem) (i + 1)*(groupSize + 1) else (i + 1)*groupSize + rem
}
val rng = new Random(seed)
for (count <- n to 1 by -1) yield {
val j = rng.nextInt(count)
val i = cumCounts.iterator.zipWithIndex.find(_._1 > j).map(_._2).get
for (s <- i until k) cumCounts(s) -= 1
}
}
val keys = createKeys(data.length, k)
for (i <- 0 until k) yield {
val testIterable = new Iterable[T] {
def iterator = (keys.iterator zip data.iterator).filter(_._1 == i).map(_._2)
}
val trainIterable = new Iterable[T] {
def iterator = (keys.iterator zip data.iterator).filter(_._1 != i).map(_._2)
}
(testIterator, trainIterator)
}
}
Note the way I define testIterable and trainIterable. This makes your test/train sets lazy and non-memoized, which I gathered is what you wanted.
Example usage:
val data = 'a' to 'z'
for (((testData, trainData), index) <- testTrainDataList(data, 4).zipWithIndex) {
println(s"index = $index")
println("test: " + testData.mkString(", "))
println("train: " + trainData.mkString(", "))
}
//index = 0
//test: i, l, o, q, v, w, y
//train: a, b, c, d, e, f, g, h, j, k, m, n, p, r, s, t, u, x, z
//
//index = 1
//test: a, d, e, h, n, r, z
//train: b, c, f, g, i, j, k, l, m, o, p, q, s, t, u, v, w, x, y
//
//index = 2
//test: b, c, m, t, u, x
//train: a, d, e, f, g, h, i, j, k, l, n, o, p, q, r, s, v, w, y, z
//
//index = 3
//test: f, g, j, k, p, s
//train: a, b, c, d, e, h, i, l, m, n, o, q, r, t, u, v, w, x, y, z