Scala recursion depth differences and StackOverflowError below allowed depth - scala

Question

The following program was written to check how deep recursion in Scala can go deep on my machine. I assume that it mostly depends on stack's size assigned to this program. However, after calculating the maximal depth of recursion (catching exception when it appears) and trying to simulate a recursion of this depth I got odd outputs.
object RecursionTest {
def sumArrayRec(elems: Array[Int]) = {
def goFrom(i: Int, size: Int, elms: Array[Int]): Int = {
if (i < size) elms(i) + goFrom(i + 1, size, elms)
else 0
}
goFrom(0, elems.length, elems)
}
def main(args: Array[String]) {
val recDepth = recurseTest(0, 0, Array(1))
println("sumArrayRec = " + sumArrayRec((0 until recDepth).toArray))
}
def recurseTest(i: Int, j: Int, arr: Array[Int]): Int = {
try {
recurseTest(i + 1, j + 1, arr)
} catch { case e: java.lang.StackOverflowError =>
println("Recursion depth on this system is " + i + ".")
i
}
}
}
The results vary among executions of the program.
One of them is a desirable output:
/usr/lib/jvm/java-8-oracle/bin/java
Recursion depth on this system is 7166.
sumArrayRec = 25672195
Process finished with exit code 0
Nevertheless, the second possible output which I get indicates an error:
/usr/lib/jvm/java-8-oracle/bin/java
Recursion depth on this system is 8129.
Exception in thread "main" java.lang.StackOverflowError
at RecursionTest$.goFrom$1(lab44.scala:6)
at RecursionTest$.goFrom$1(lab44.scala:6)
at RecursionTest$.goFrom$1(lab44.scala:6)
(...)
at RecursionTest$.goFrom$1(lab44.scala:6)
at RecursionTest$.goFrom$1(lab44.scala:6)
Process finished with exit code 1
I didn't observe any dependence or relationship of them, I just once got the first, and other time the second
All of that lead me to the following questions:
Why do I get an error and what causes it?
Is this the stack overflow error and if yes...
Why the size of stack changes in the same time when the program is running? Is it even possible?
When I changed println("sumArrayRec = " + sumArrayRec((0 until recDepth).toArray)) into
println("sumArrayRec = " + sumArrayRec((0 until recDepth - 5000).toArray)), so much less, the behaviour remains the same.

Answer 1

You get a stack overflow because you cause a stack overflow
Yes
The max stack doesn't change, the JVM sets this. Your two methods recurseTest and sumArrayRec push different amounts to the stack. recurseTest is basically adding 2 ints the stack with each call while sumArrayRec is adding 3 ints since you have a call to elms(i), probably more with addition/ifelse (more instructions). It's tough to tell since the JVM is handling all of this and it's purpose is to obscure this from what you need to know. Also, who knows what optimizations the JVM is doing behind the scenes which can drastically effect the stack sizes created from each method call. On multiple runs you'll get different depths for stack due to system timing etc, maybe the jvm will optimize in the short time the program has run, maybe it won't, it's non-deterministic in this scenario. If you ran some warmup code before hand that might make your tests more deterministic so any optimizations will or will not take place.
You should look into using something like JMH for your tests, it will help with the determinism.
Side note: you can also manually change your stack size with -Xss2M, common for SBT usage.