Why is this tail recursion:
def navigate(myList : List[Int]) : (Int, List[Int]) = {
def navigate(step: Int, offset: Int, myList: List[Int]): (Int, scala.List[Int]) = {
if //some test and exit condition, then a definition of jump
else navigate(step + 1, offset + jump, myList)
}
navigate(0, 0, myList)
}
while this is not:
def navigate(myList : List[Int]) : (Int, List[Int]) = {
navigate(0, 0, myList)
}
def navigate(step: Int, offset: Int, myList: List[Int]): (Int, scala.List[Int]) = {
if //some test and exit condition, then a definition of jump
else navigate(step + 1, offset + jump, myList)
}
If myList is very long, the first case does not give any problem, when the second one causes a StackOverflowError.
Also, is there any way to say the compiler that the latter should be compiled so that the recursion does not increase the stack?
In order for a method to be eligible for tail-recursion optimization, it must:
be tail-recursive (duh!)
not use return
be final
Both of your examples conform to #1 and #2, but only the first example conforms to #3 (local methods are implicitly final).
The reason why a method is not tail-recursive if it is not final is that "tail-recursive" means "tail-calls itself", but if the method is virtual, then you cannot know whether it tail-calls itself or an overridden version of itself. Figuring out at compile time whether a method has been overridden requires Class Hierarchy Analysis, which is known to be equivalent to solving the Halting Problem … IOW is impossible.
Also, is there any way to say the compiler that the latter should be compiled so that the recursion does not increase the stack?
No. There is no way to turn tail-recursion optimization on or off. Methods that are tail-recursive (according to the Scala Language Specification's definition of "tail-recursive", of course) are always optimized. Any implementation of Scala that does not do this is in violation of the Scala Language Specification.
There is, however, the scala.annotation.tailrec annotation, which guarantees that the compiler will generate an error if a method that is annotated with this annotation does not comply with the SLS's definition of tail-recursion.
Related
Why won't the Scala compiler apply tail call optimization unless a method is final?
For example, this:
class C {
#tailrec def fact(n: Int, result: Int): Int =
if(n == 0)
result
else
fact(n - 1, n * result)
}
results in
error: could not optimize #tailrec annotated method: it is neither private nor final so can be overridden
What exactly would go wrong if the compiler applied TCO in a case such as this?
Consider the following interaction with the REPL. First we define a class with a factorial method:
scala> class C {
def fact(n: Int, result: Int): Int =
if(n == 0) result
else fact(n - 1, n * result)
}
defined class C
scala> (new C).fact(5, 1)
res11: Int = 120
Now let's override it in a subclass to double the superclass's answer:
scala> class C2 extends C {
override def fact(n: Int, result: Int): Int = 2 * super.fact(n, result)
}
defined class C2
scala> (new C).fact(5, 1)
res12: Int = 120
scala> (new C2).fact(5, 1)
What result do you expect for this last call? You might be expecting 240. But no:
scala> (new C2).fact(5, 1)
res13: Int = 7680
That's because when the superclass's method makes a recursive call, the recursive call goes through the subclass.
If overriding worked such that 240 was the right answer, then it would be safe for tail-call optimization to be performed in the superclass here. But that isn't how Scala (or Java) works.
Unless a method is marked final, it might not be calling itself when it makes a recursive call.
And that's why #tailrec doesn't work unless a method is final (or private).
UPDATE: I recommend reading the other two answers (John's and Rex's) as well.
Recursive calls might be to a subclass instead of to a superclass; final will prevent that. But why might you want that behavior? The Fibonacci series doesn't provide any clues. But this does:
class Pretty {
def recursivePrinter(a: Any): String = { a match {
case xs: List[_] => xs.map(recursivePrinter).mkString("L[",",","]")
case xs: Array[_] => xs.map(recursivePrinter).mkString("A[",",","]")
case _ => a.toString
}}
}
class Prettier extends Pretty {
override def recursivePrinter(a: Any): String = { a match {
case s: Set[_] => s.map(recursivePrinter).mkString("{",",","}")
case _ => super.recursivePrinter(a)
}}
}
scala> (new Prettier).recursivePrinter(Set(Set(0,1),1))
res8: String = {{0,1},1}
If the Pretty call was tail-recursive, we'd print out {Set(0, 1),1} instead since the extension wouldn't apply.
Since this sort of recursion is plausibly useful, and would be destroyed if tail calls on non-final methods were allowed, the compiler inserts a real call instead.
Let foo::fact(n, res) denote your routine. Let baz::fact(n, res) denote someone else's override of your routine.
The compiler is telling you that the semantics allow baz::fact() to be a wrapper, that MAY upcall (?) foo::fact() if it wants to. Under such a scenario, the rule is that foo::fact(), when it recurs, must activate baz::fact() rather than foo::fact(), and, while foo::fact() is tail-recursive, baz::fact() may not be. At that point, rather than looping on the tail-recursive call, foo::fact() must return to baz::fact(), so it can unwind itself.
What exactly would go wrong if the compiler applied TCO in a case such as this?
Nothing would go wrong. Any language with proper tail call elimination will do this (SML, OCaml, F#, Haskell etc.). The only reason Scala does not is that the JVM does not support tail recursion and Scala's usual hack of replacing self-recursive calls in tail position with goto does not work in this case. Scala on the CLR could do this as F# does.
The popular and accepted answer to this question is actually misleading, because the question itself is confusing. The OP does not make the distinction between tailrec and TCO, and the answer does not address this.
The key point is that the requirements for tailrec are more strict than the requirements for TCO.
The tailrec annotation requires that tail calls are made to the same function, whereas TCO can be used on tail calls to any function.
The compiler could use TCO on fact because there is a call in the tail position. Specifically, it could turn the call to fact into a jump to fact by adjusting the stack appropriately. It does not matter that this version of fact is not the same as the function making the call.
So the accepted answer correctly explains why a non-final function cannot be tailrec because you cannot guarantee that the tail calls are to the same function and not to an overloaded version of the function. But it incorrectly implies that it is not safe to use TCO on this method, when in fact this would be perfectly safe and a good optimisation.
[ Note that, as explained by Jon Harrop, you cannot implement TCO on the JVM, but that is a restriction of the compiler, not the language, and is unrelated to tailrec ]
And for reference, here is how you can avoid the problem without making the method final:
class C {
def fact(n: Int): Int = {
#tailrec
def loop(n: Int, result: Int): Int =
if (n == 0) {
result
} else {
loop(n - 1, n * result)
}
loop(n, 1)
}
}
This works because loop is a concrete function rather than a method and cannot be overridden. This version also has the advantage of removing the spurious result parameter to fact.
This is the pattern I use for all recursive algorithms.
In the Strategy pattern, implemented like this:
object StrategyPattern {
def add(a: Int, b: Int) = a + b
def subtract(a: Int, b: Int) = a - b
def multiply(a: Int, b: Int) = a * b
def execute(callback:(Int, Int) => Int, x: Int, y: Int) = callback(x, y)
def main(args: Array[String]) {
println("Add: " + execute(add, 3, 4))
println("Subtract: " + execute(subtract, 3, 4))
println("Multiply: " + execute(multiply, 3, 4))
}
}
I wanted to know (and find how to understand also the other cases, if there is a good reference for the types/forms of the binding times) if the binding time of methods add, substract, and multiply is "construction time" (if I can say so), or at runtime?
The simple answer is that (for concrete classes) method definitions are bound to method names at compile time, just as they are in Java. Your add method, for example, is completely equivalent to this Java definition:
public int add(int a, int b) {
return a + b;
}
Non-final methods
If you're analyzing the binding time of a non-final method from the perspective of a call-site where the concrete class is not known statically, then the method name would be considered to have runtime binding to its implementation (due to subclassing/overriding).
Dynamic Classloading
The simple answer is close to the truth, but dynamic classloading in the JVM complicates matters a bit. Because of dynamic classloading, method definitions are technically bound to fully qualified names (e.g., my.pkg.StrategyPattern.add) at runtime. It's certainly possible to have alternative implementations of a my.package.StrategyPattern module and to choose among them dynamically (by loading the corresponding class file(s)).
Of course, this distinction is only relevant to code outside of the compilation unit containing the StrategyPattern definition. Within the compilation unit, methods would always be considered bound at compile time.
Strategy
Since you're asking about the strategy pattern, I guess you have something else in mind? If you're asking whether you can select among the "strategies" at runtime, you can:
val op: (Int, Int) => Int =
if (args(0) == "+") add
else if (args(0) == "-") subtract
else multiply
execute(op, 3, 4)
In this case, op is bound to a "strategy" function at runtime, but add, subtract, and multiply are still bound to their definitions at compile time.
It's an implementation detail that each of these methods is also associated with an anonymous Function2 class at compile time, and that the appropriate class is instantiated at runtime based on the outcome of the conditional expressions. This detail really isn't relevant to the binding time analysis, since the meaning of the add, subtract, and multiply identifiers is fixed at compile time.
I don't know if I am doing something wrong or it is just a property of the Scala compiler - I get the mentioned compilation error when I try to compile this code:
#tailrec
def shiftDown2(x: Int, bound: Int) {
val childOfX = chooseChild(x, bound)
for (child <- childOfX) {
swap(x, child)
shiftDown2(child, bound)
}
}
whilst the following code compiles without problems:
#tailrec
def shiftDown(x: Int, bound: Int) {
val childOfX = chooseChild(x, bound)
if (childOfX.isDefined) {
swap(x, childOfX.get)
shiftDown(childOfX.get, bound)
}
}
I believe that above code fragments are semantically the same and both should work with tail recursion.
Tail recursion optimization will not work with recursive invocations inside for loop, because for loop here is just a syntactic sugar for calling foreach higher-order method. So, your code is equivalent to:
#tailrec
def shiftDown2(x: Int, bound: Int) {
val childOfX = chooseChild(x, bound)
childOfX.foreach { child =>
swap(x, child)
shiftDown2(child, bound)
}
}
scalac can optimize tail calls only if recursive method is tail-calling itself directly - by translating it into something similar to while loop in bytecode.
Unfortunately, this is not the case here - shiftDown2 calls childOfX.foreach passing it an anonymous function. Then, foreach (potentially) calls apply on that anonymous function and that anonymous function finally calls shiftDown2 again. So this is an indirect recursion and cannot be optimized by scalac. This limitation has its roots in the JVM, which doesn't have native tail call support.
I understand the difference between zero-parameter and parameterless methods, but what I don't really understand is the language design choice that made parameterless methods necessary.
Disadvantages I can think of:
It's confusing. Every week or two there are questions here or on the Scala mailing list about it.
It's complicated; we also have to distinguish between () => X and => X.
It's ambiguous: does x.toFoo(y) mean what it says, or x.toFoo.apply(y)? (Answer: it depends on what overloads there are x's toFoo method and the overloads on Foo's apply method, but if there's a clash you don't see an error until you try to call it.)
It messes up operator style method calling syntax: there is no symbol to use in place of the arguments, when chaining methods, or at the end to avoid semicolon interference. With zero-arg methods you can use the empty parameter list ().
Currently, you can't have both defined in a class: you get an error saying the method is already defined. They also both convert to a Function0.
Why not just make methods def foo and def foo() exactly the same thing, and allow them to be called with or without parentheses? What are the upsides of how it is?
Currying, That's Why
Daniel did a great job at explaining why parameterless methods are necessary. I'll explain why they are regarded distinctly from zero-parameter methods.
Many people view the distinction between parameterless and zero-parameter functions as some vague form of syntactic sugar. In truth it is purely an artifact of how Scala supports currying (for completeness, see below for a more thorough explanation of what currying is, and why we all like it so much).
Formally, a function may have zero or more parameter lists, with zero or more parameters each.
This means the following are valid: def a, def b(), but also the contrived def c()() and def d(x: Int)()()(y: Int) etc...
A function def foo = ??? has zero parameter lists. A function def bar() = ??? has precisely one parameter list, with zero parameters. Introducing additional rules that conflate the two forms would have undermined currying as a consistent language feature: def a would be equivalent in form to def b() and def c()() both; def d(x: Int)()()(y: Int) would be equivalent to def e()(x: Int)(y: Int)()().
One case where currying is irrelevant is when dealing with Java interop. Java does not support currying, so there's no problem with introducing syntactic sugar for zero-parameter methods like "test".length() (which directly invokes java.lang.String#length()) to also be invoked as "test".length.
A quick explanation of currying
Scala supports a language feature called 'currying', named after mathematician Haskell Curry.
Currying allows you to define functions with several parameter lists, e.g.:
def add(a: Int)(b: Int): Int = a + b
add(2)(3) // 5
This is useful, because you can now define inc in terms of a partial application of add:
def inc: Int => Int = add(1)
inc(2) // 3
Currying is most often seen as a way of introducing control structures via libraries, e.g.:
def repeat(n: Int)(thunk: => Any): Unit = (1 to n) foreach { _ => thunk }
repeat(2) {
println("Hello, world")
}
// Hello, world
// Hello, world
As a recap, see how repeat opens up another opportunity to use currying:
def twice: (=> Any) => Unit = repeat(2)
twice {
println("Hello, world")
}
// ... you get the picture :-)
One nice thing about an issue coming up periodically on the ML is that there are periodic answers.
Who can resist a thread called "What is wrong with us?"
https://groups.google.com/forum/#!topic/scala-debate/h2Rej7LlB2A
From: martin odersky Date: Fri, Mar 2, 2012 at
12:13 PM Subject: Re: [scala-debate] what is wrong with us...
What some people think is "wrong with us" is that we are trying bend
over backwards to make Java idioms work smoothly in Scala. The
principaled thing would have been to say def length() and def length
are different, and, sorry, String is a Java class so you have to write
s.length(), not s.length. We work really hard to paper over it by
admitting automatic conversions from s.length to s.length(). That's
problematic as it is. Generalizing that so that the two are identified
in the type system would be a sure way to doom. How then do you
disambiguate:
type Action = () => () def foo: Action
Is then foo of type Action or ()? What about foo()?
Martin
My favorite bit of paulp fiction from that thread:
On Fri, Mar 2, 2012 at 10:15 AM, Rex Kerr <ich...#gmail.com> wrote:
>This would leave you unable to distinguish between the two with
>structural types, but how often is the case when you desperately
>want to distinguish the two compared to the case where distinguishing
>between the two is a hassle?
/** Note to maintenance programmer: It is important that this method be
* callable by classes which have a 'def foo(): Int' but not by classes which
* merely have a 'def foo: Int'. The correctness of this application depends
* on maintaining this distinction.
*
* Additional note to maintenance programmer: I have moved to zambia.
* There is no forwarding address. You will never find me.
*/
def actOnFoo(...)
So the underlying motivation for the feature is to generate this sort of ML thread.
One more bit of googlology:
On Thu, Apr 1, 2010 at 8:04 PM, Rex Kerr <[hidden email]> wrote: On
Thu, Apr 1, 2010 at 1:00 PM, richard emberson <[hidden email]> wrote:
I assume "def getName: String" is the same as "def getName(): String"
No, actually, they are not. Even though they both call a method
without parameters, one is a "method with zero parameter lists" while
the other is a "method with one empty parameter list". If you want to
be even more perplexed, try def getName()(): String (and create a
class with that signature)!
Scala represents parameters as a list of lists, not just a list, and
List() != List(List())
It's kind of a quirky annoyance, especially since there are so few
distinctions between the two otherwise, and since both can be
automatically turned into the function signature () => String.
True. In fact, any conflation between parameterless methods and
methods with empty parameter lists is entirely due to Java interop.
They should be different but then dealing with Java methods would be
just too painful. Can you imagine having to write str.length() each
time you take the length of a string?
Cheers
First off, () => X and => X has absolutely nothing to do with parameterless methods.
Now, it looks pretty silly to write something like this:
var x() = 5
val y() = 2
x() = x() + y()
Now, if you don't follow what the above has to do with parameterless methods, then you should look up uniform access principle. All of the above are method declarations, and all of them can be replaced by def. That is, assuming you remove their parenthesis.
Besides the convention fact mentioned (side-effect versus non-side-effect), it helps with several cases:
Usefulness of having empty-paren
// short apply syntax
object A {
def apply() = 33
}
object B {
def apply = 33
}
A() // works
B() // does not work
// using in place of a curried function
object C {
def m()() = ()
}
val f: () => () => Unit = C.m
Usefulness of having no-paren
// val <=> def, var <=> two related defs
trait T { def a: Int; def a_=(v: Int): Unit }
trait U { def a(): Int; def a_=(v: Int): Unit }
def tt(t: T): Unit = t.a += 1 // works
def tu(u: U): Unit = u.a += 1 // does not work
// avoiding clutter with apply the other way round
object D {
def a = Vector(1, 2, 3)
def b() = Vector(1, 2, 3)
}
D.a(0) // works
D.b(0) // does not work
// object can stand for no-paren method
trait E
trait F { def f: E }
trait G { def f(): E }
object H extends F {
object f extends E // works
}
object I extends G {
object f extends E // does not work
}
Thus in terms of regularity of the language, it makes sense to have the distinction (especially for the last shown case).
I would say both are possible because you can access mutable state with a parameterless method:
class X(private var x: Int) {
def inc() { x += 1 }
def value = x
}
The method value does not have side effects (it only accesses mutable state). This behavior is explicitly mentioned in Programming in Scala:
Such parameterless methods are quite common in Scala. By contrast, methods defined with empty parentheses, such as def height(): Int, are called empty-paren methods. The recommended convention is to use a parameterless method whenever there are no parameters and the method accesses mutable state only by reading fields of the containing object (in particular, it does not change mutable state).
This convention supports the uniform access principle [...]
To summarize, it is encouraged style in Scala to define methods that take no parameters and have no side effects as parameterless methods, i.e., leaving off the empty parentheses. On the other hand, you should never define a method that has side-effects without parentheses, because then invocations of that method would look like a field selection.
Why won't the Scala compiler apply tail call optimization unless a method is final?
For example, this:
class C {
#tailrec def fact(n: Int, result: Int): Int =
if(n == 0)
result
else
fact(n - 1, n * result)
}
results in
error: could not optimize #tailrec annotated method: it is neither private nor final so can be overridden
What exactly would go wrong if the compiler applied TCO in a case such as this?
Consider the following interaction with the REPL. First we define a class with a factorial method:
scala> class C {
def fact(n: Int, result: Int): Int =
if(n == 0) result
else fact(n - 1, n * result)
}
defined class C
scala> (new C).fact(5, 1)
res11: Int = 120
Now let's override it in a subclass to double the superclass's answer:
scala> class C2 extends C {
override def fact(n: Int, result: Int): Int = 2 * super.fact(n, result)
}
defined class C2
scala> (new C).fact(5, 1)
res12: Int = 120
scala> (new C2).fact(5, 1)
What result do you expect for this last call? You might be expecting 240. But no:
scala> (new C2).fact(5, 1)
res13: Int = 7680
That's because when the superclass's method makes a recursive call, the recursive call goes through the subclass.
If overriding worked such that 240 was the right answer, then it would be safe for tail-call optimization to be performed in the superclass here. But that isn't how Scala (or Java) works.
Unless a method is marked final, it might not be calling itself when it makes a recursive call.
And that's why #tailrec doesn't work unless a method is final (or private).
UPDATE: I recommend reading the other two answers (John's and Rex's) as well.
Recursive calls might be to a subclass instead of to a superclass; final will prevent that. But why might you want that behavior? The Fibonacci series doesn't provide any clues. But this does:
class Pretty {
def recursivePrinter(a: Any): String = { a match {
case xs: List[_] => xs.map(recursivePrinter).mkString("L[",",","]")
case xs: Array[_] => xs.map(recursivePrinter).mkString("A[",",","]")
case _ => a.toString
}}
}
class Prettier extends Pretty {
override def recursivePrinter(a: Any): String = { a match {
case s: Set[_] => s.map(recursivePrinter).mkString("{",",","}")
case _ => super.recursivePrinter(a)
}}
}
scala> (new Prettier).recursivePrinter(Set(Set(0,1),1))
res8: String = {{0,1},1}
If the Pretty call was tail-recursive, we'd print out {Set(0, 1),1} instead since the extension wouldn't apply.
Since this sort of recursion is plausibly useful, and would be destroyed if tail calls on non-final methods were allowed, the compiler inserts a real call instead.
Let foo::fact(n, res) denote your routine. Let baz::fact(n, res) denote someone else's override of your routine.
The compiler is telling you that the semantics allow baz::fact() to be a wrapper, that MAY upcall (?) foo::fact() if it wants to. Under such a scenario, the rule is that foo::fact(), when it recurs, must activate baz::fact() rather than foo::fact(), and, while foo::fact() is tail-recursive, baz::fact() may not be. At that point, rather than looping on the tail-recursive call, foo::fact() must return to baz::fact(), so it can unwind itself.
What exactly would go wrong if the compiler applied TCO in a case such as this?
Nothing would go wrong. Any language with proper tail call elimination will do this (SML, OCaml, F#, Haskell etc.). The only reason Scala does not is that the JVM does not support tail recursion and Scala's usual hack of replacing self-recursive calls in tail position with goto does not work in this case. Scala on the CLR could do this as F# does.
The popular and accepted answer to this question is actually misleading, because the question itself is confusing. The OP does not make the distinction between tailrec and TCO, and the answer does not address this.
The key point is that the requirements for tailrec are more strict than the requirements for TCO.
The tailrec annotation requires that tail calls are made to the same function, whereas TCO can be used on tail calls to any function.
The compiler could use TCO on fact because there is a call in the tail position. Specifically, it could turn the call to fact into a jump to fact by adjusting the stack appropriately. It does not matter that this version of fact is not the same as the function making the call.
So the accepted answer correctly explains why a non-final function cannot be tailrec because you cannot guarantee that the tail calls are to the same function and not to an overloaded version of the function. But it incorrectly implies that it is not safe to use TCO on this method, when in fact this would be perfectly safe and a good optimisation.
[ Note that, as explained by Jon Harrop, you cannot implement TCO on the JVM, but that is a restriction of the compiler, not the language, and is unrelated to tailrec ]
And for reference, here is how you can avoid the problem without making the method final:
class C {
def fact(n: Int): Int = {
#tailrec
def loop(n: Int, result: Int): Int =
if (n == 0) {
result
} else {
loop(n - 1, n * result)
}
loop(n, 1)
}
}
This works because loop is a concrete function rather than a method and cannot be overridden. This version also has the advantage of removing the spurious result parameter to fact.
This is the pattern I use for all recursive algorithms.