Efficient way to select a subset of elements from a Seq in Scala - scala

Question

I have a sequence
val input = Seq(1,3,4,5,9,11...)
I want to randomly select a subset of it. What is the fastest way.
I currently implement it like this:
//ratio is the percentage of the subgroup from the whole group
def randomSelect(ratio:Double): Boolean = {
val rr=scala.util.Random
if (rr.nextFloat() < ratio) true else false
}
val ratio = 0.3
val result = input.map(x=>(x, randomSelect(ratio))).filter(x._2).map(x=>x._1)
So I first attach a true/false label for each element, and filter out those false elements, and get back the subset of the sequence.
Is there any faster/ advantage way?

Answer 1

So there are basically two approaches to this:
select n elements at random
include or exclude each element with probability p
Your solution is the latter and can be simplified to:
l.filter(_ => r.nextFloat < p)
(I'm calling the list, l, the instance of Random r and your ratio p from here on out.)
If you wanted to sample exactly n elements you could do:
r.shuffle(l).take(n)
I compared these selecting 200 elements from a 1000 element list:
scala> val first = time{
| l.map(x => (x, r.nextFloat < p)).filter(_._2).map(_._1)
| }
Elapsed time: 3249507ns
scala> val second = time {
| r.shuffle(l).take(200)
| }
Elapsed time: 10640432ns
scala> val third = time{
| l.filter(_ => r.nextFloat < p)}
Elapsed time: 1689009ns
Dropping your extra two mapss appears to speed things up by about a third (which makes complete sense). The shuffle-and-take method is significantly slower, but does guarantee you a fixed number of elements.
I borrowed the timing function from here if you want to do a more rigorous investigation (i.e. average over many trials, rather than 1).

Answer 2

If your list isn't big, a simple filter as suggested by others should suffice:
list.filter(_ => Random.nextDouble < p)
In case you have a big list, the per-element call of Random could become the bottleneck. One approach to minimize the calls is to generate random gaps (0, 1, 2, ...) by which the data sampling will hop over the elements. Below is a simple implementation in Scala:
import scala.util.Random
import scala.math._
def gapSampling(list: List[Double], p: Double): List[Double] = {
def randomGap(p: Double): Double = {
val epsilon: Double = 1e-10
val u = max(Random.nextDouble, epsilon)
floor( log(u) / log(1 - p) )
}
#scala.annotation.tailrec
def samplingFcn(acc: List[Double], list: List[Double], p: Double): List[Double] = list match {
case Nil => acc
case _ =>
val gap = randomGap(p).toInt
val l = list.drop(gap + 1)
val accNew = l.headOption match {
case Some(e) => e :: acc
case None => acc
}
samplingFcn(accNew, l, p)
}
samplingFcn(List[Double](), list, p).reverse
}
val list = (1 to 100).toList.map(_.toDouble)
gapSampling(list, 0.3)
// res1: List[Double] = List(
// 2.0, 5.0, 7.0, 14.0, 15.0, 18.0, 20.0, 25.0, 26.0, 28.0, 33.0,
// 35.0, 42.0, 43.0, 47.0, 48.0, 50.0, 55.0, 56.0, 59.0, 62.0,
// 69.0, 72.0, 75.0, 76.0, 79.0, 82.0, 93.0, 96.0, 97.0, 98.0
// )
More details about such gap sampling can be found here.