Scala Spark: How do I bootstrap sample from a column of a Spark Dataframe? - scala

Question

I am looking to sample values, with replacement, from a column of a Spark DataFrame, using the Scala programming language in a Jupyter Notebook setting in a cluster environment. How do I do this?
I tried the following function that I found online:
import scala.util
def bootstrapMean(originalData: Array[Double]): Double = {
val n = originalData.length
def draw: Double = originalData(util.Random.nextInt(n))
// a tail recursive loop to randomly draw and add a value to the accumulating sum
def drawAndSumValues(current: Int, acc: Double = 0D): Double = {
if (current == 0) acc
else drawAndSumValues(current - 1, acc + draw)
}
drawAndSumValues(n) / n
}
Like so:
val data = stack.select("column_with_values").collect.map(_.toSeq).flatten
val m = 10
val bootstraps = Vector.fill(m)(bootstrapMean(data))
But I get the error:
An error was encountered:
<console>:47: error: type mismatch;
found : Array[Any]
required: Array[Double]
Note: Any >: Double, but class Array is invariant in type T.
You may wish to investigate a wildcard type such as `_ >: Double`. (SLS 3.2.10)
val bootstraps = Vector.fill(m)(bootstrapMean(data))
Not sure how to debug this, and whether I should bother to or try another approach. I'm looking for ideas/documentation/code. Thanks.
Update:
How do I put the user mck's solution below, in a for loop? I tried the following:
var bootstrap_container = Seq()
var a = 1
for( a <- 1 until 3){
var sampled = stack_b.select("diff_hours").sample(withReplacement = true, fraction = 0.5, seed = a)
var smpl_average = sampled.select(avg("diff_hours")).collect()(0)(0)
var bootstrap_smpls = bootstrap_container.union(Seq(smpl_average)).collect()
}
bootstrap_smpls
but that gives an error:
<console>:49: error: not enough arguments for method collect: (pf: PartialFunction[Any,B])(implicit bf: scala.collection.generic.CanBuildFrom[Seq[Any],B,That])That.
Unspecified value parameter pf.
var bootstrap_smpls = bootstrap_container.union(Seq(smpl_average)).collect()

Answer 1

You can use the sample method of dataframes, for example, if you want to sample with replacement and with a fraction of 0.5:
val sampled = stack.select("column_with_values").sample(true, 0.5)
To get the mean, you can do:
val col_average = sampled.select(avg("column_with_values")).collect()(0)(0)
EDIT:
var bootstrap_container = List[Double]()
var a = 1
for( a <- 1 until 3){
var sampled = stack_b2.select("diff_hours").sample(withReplacement = true, fraction = 0.5, seed = a)
var smpl_average = sampled.select(avg("diff_hours")).collect()(0)(0)
bootstrap_container = bootstrap_container :+ smpl_average.asInstanceOf[Double]
}
var mean_bootstrap = bootstrap_container.reduce(_ + _) / bootstrap_container.length