Here is a minimal code that raise the compilation error 'Recursive call not in tail position'. However, I'm using an #inline and the recursive call is in tail position. The reason why I'm using this #inline is that I have the code pf the original reccall duplicated twice.
import scala.annotation._
object Test {
#tailrec private def test(i: Int): Int = {
#inline def reccall(i: Int): Int = test(i-1)
i match {
case 0 => 0
case i => reccall(i)
}
}
}
I've looked at the answers Recursive call not in tail position #tailrec why does this method not compile with 'contains a recursive call not in tail position'? but they do not apply to my case. Using Scala 2.12
It appears, the way #inline is implemented is that it still passes the parameter via stack. The jump is eliminated, by inserting the code inline, but the stack is still used for the arguments. This makes it impossible to be in a tail position, because the stack needs to be cleaned up after the call is completed.
Besides, annotating a function with #inline does not guarantee that the optimizer will inline it, just that it will "try especially hard".
Well, the mechanism of how tail recursion is actualized in JVM is explained in following way:
Scala, in the case of tail recursion, can eliminate the creation of a
new stack frame and just re-use the current stack frame. The stack
never gets any deeper, no matter how many times the recursive call is
made.
So in your case it cannot reuse the current stack frame belonging to the test method since it MUST create a new stack frame for the reccall method anyway.
Recursive call is implicit in this case, made from another method. So I believe you cannot really have tail recursion implemented for such case.
You may just remove the reccall method altogether and write case i => test(i-1) and then compiler will not complain.
NOTE: also I believe #inline has nothing to do here and is not essential in this example, since if I remove it - compiler still complains the same reason.
The issue here is that #inline is strictly advisory: it doesn't guarantee that the compiler will inline the function. Since #tailrec only works if it's absolutely guaranteed that the tail-calls can be eliminated, this means that using #tailrec has to assume no inlining.
Related
In scala, the following 2 functions serve exactly the same purpose:
#tailrec
final def fn(str: String): Option[String] = {
Option(str).filter(_.nonEmpty).flatMap { v =>
fn(v.drop(1))
}
}
#tailrec
final def fn2(str: String): Option[String] = {
Option(str).filter(_.nonEmpty) match {
case None => None
case Some(v) => fn2(v.drop(1))
}
}
However #tailrec only works in second case, in the first case it will generate the following error:
Error: could not optimize #tailrec annotated method fn: it contains a
recursive call not in tail position
Option(str).filter(_.nonEmpty).flatMap { v =>
Why this error was given? And why these 2 codes generate different kinds JVM bytecode
For fn to be tail-recursive, the recursive call must be the last action in the function. If you pass fn to another function such as flatMap then the other function is free to perform other actions after calling fn and therefore the compiler cannot be sure that it is tail recursive.
In some cases the compiler could detect that calling fn is the last action in the other function, but not in the general case. And this would rely on a specific implementation of that other function so the tailrec annotation might become invalid if that other function were changed, which is an undesirable dependency.
Specifically for the last question:
And why these 2 codes generate different kinds JVM bytecode
Because on JVM there's no guarantee that the JAR containing Option class at runtime is the same as was seen at compile-time. This is good, because otherwise even minor versions of libraries (including standard Java and Scala libraries) would be incompatible, and you'd need all dependencies to be using the same minor version of their common dependencies.
If that class doesn't have a suitable flatMap method, you'll get AbstractMethodError, but otherwise semantics of Scala require that its flatMap method must be called. So the compiler has to emit bytecode to actually call the method.
Kotlin works around this by using inline functions and Scala 3 will support them too, but I don't know if it'll use them for such cases.
Consider the following:
List('a', 'b').flatMap(List(_,'g')) //res0: List[Char] = List(a, g, b, g)
I seems pretty obvious that flatMap() is doing some internal post-processing in order to achieve that result. How else would List('a','g') get combined with List('b','g')?
A mutable Set's retain method is implemented as follows:
def retain(p: A => Boolean): Unit =
for (elem <- this.toList) // SI-7269 toList avoids ConcurrentModificationException
if (!p(elem)) this -= elem
But if I implement my own method that doesn't make a copy for iterating, nothing blows up.
def dumbRetain[A](self: mutable.Set[A], p: A => Boolean): Unit =
for (elem <- self)
if (!p(elem)) self -= elem
dumbRetain(mutable.HashSet(1,2,3,4,5,6), Set(2,4,6))
// everything is ok
I see that SI-7269's test case uses the JavaConversions wrapper around a java Set/Map, and it seems like the issue arises from the underlying java collection.
I know there will never be a java collection passed to my algorithm, so can I use dumbRetain without worrying about the ConcurrentModificationException? Or is this "coincidental behavior" that I shouldn't rely on?
edit to clarify, I would be using dumbRetain as an implementation detail in an algorithm which would be in full control of what it passes to dumbRetain. And this would be run in a single-threaded context.
This seems to rely on the specific implementation of mutable.HashSet, and there is nothing in the API that guarantees that it would work for all other implementations of mutable.Set, even if we exclude all wrappers for the Java collections.
The for-loop
for (elem <- self) {
...
}
is desugared into foreach, which for mutable.HashSet is implemented as follows:
override def foreach[U](f: A => U) {
var i = 0
val len = table.length
while (i < len) {
val curEntry = table(i)
if (curEntry ne null) f(entryToElem(curEntry))
i += 1
}
}
Essentially, it simply loops through the Array of the underlying FlatHashTable, and invokes the passed function f on every element. The whole foreach simply does not have any lines which could throw anything, it doesn't check for concurrent [footnote-1] modifications at all.
A ConcurrentModificationException seems to be the less troubling case: at least, your program fails fast, and even returns a detailed stack trace that points to the line in which the problem occurred. It would be actually much worse if it simply deteriorated into undefined behavior without throwing anything. This would be the worst case. However, this worst case shouldn't occur for collections from the standard library: Throw ConcurrentModificationException exception's in scala collections? #188
Quote:
In scala/scala#5295 (merged in to 2.12.x) I made sure that removing the element last returned from an iterator would not cause a problem for the iterator.
So, as long as you clearly state in the documentation that only the collections from standard library are supported, you will most likely not have any problems using it in your own code. But if you use it in a public interface, this would be an invitation for a bug analogous to "SI-7269" quoted in your question.
[footnote-1] "concurrent" as in "ConcurrentModificationException", not as in "concurrently executed threads".
EDIT: I've tried to choose less ambiguous formulations. Great Thanks #Dima for the feedback and the numerous suggestions.
Yeah, you can do it, as long as you are sure this is the scala's native HashSet implementation, not a wrapper around java ... and with understanding, that this is not thread-safe, and should never be used concurrently (the original HashSet.retain is that way too as well as the other mutators).
Better yet, just use immutable Set.filter, unless you actually have real hard evidence (not just intuition) demonstrating that your specific case absolutely requires mutable container.
The scala standard library seems to offer two methods for ensuring a recursive function does not cause a stack overflow.
One is the #tailrec annotation, which supposedly causes the compiler to do something differently, or try extra hard to attempt tail recursion.
The other method is to actually change your function signature to return TailRec[T], and then use the TailCalls.done and TailCalls.tailcall functions to wrap your return values.
What is the difference? Is there any reason at all to use TailRec when the compiler seems to be able to do it for me?
The #tailrec annotation is a tail recursion guarantee. If a function annotated with #tailrec is not actually tail recursive, then compilation will fail. (Since a tail recursive function will not result in a stack overflow, this guarantee is particularly useful in production code.)
scala.util.control.TailCalls is a mechanism for implementing tail recursion using trampolining - refer to this question for more information.
I am teaching myself Scala by trying to implement the operations on List[T]. I just implemented dropWhile and it made me wonder about how tail recursion optimization works when the recursive call appears in different cases.
def dropWhile[T](list: List[T])(predicate: T => Boolean): List[T] = list match {
case head :: tail if predicate(head) => dropWhile(tail)(predicate)
case _ => list
}
Does it matter that the recursive call appears in the first case?
As some one said in the comments, you can apply the #tailrec annotations to your function and it will give a compiler error if the recursion can not be optimized to a loop.
Where the recursive call is does not matter. The important part is that there is no need for a stack frame to stay allocated waiting for the return from the recursive calls.
I am trying to find the documentation for the Scala operator method #::. I believe that it is defined in the Stream class because of an example I found that uses it.
My question is not particular to this method (although I would like to know where the docs are), but how to search the Scala docs in general. I tried entering #:: in the search box in the upper left of the documentation page (2.8.1), but found nothing.
I suggest using the Reference Index - it's designed specifically to look for any kind of symbol (class, traits, methods, vals, vars) regardless of it's hierarchical position - contrasting with the Scaladoc's left index which doesn't show inner classes, traits or objects.
Unfortunately it's only available in the nightly. You can see the whole thing at nightly Scaladoc. Notice the upper box in the left frame, above the index.
Hope it will be bundled with Scala 2.9.0.
Edit As of 2.9.0, the reference index started to be bundle with Scaladoc. No need to go to the nightly docs now.
As others have already mentioned, #:: is defined on scala.collection.immutable.Stream.ConsWrapper. I just wanted to take a minute to elaborate on why that is.
In general, to call an operator on an object, that object needs to exist. However, the idea with a Stream is the tail of the stream is not evaluated until it needs to be. So consider the following stream:
def fibs(a:Int,b:Int):Stream[Int] = a #:: fibs(b,a+b)
Ordinarily, we would need to evaluate the recursive fibs call so that we could call the #:: operator on it. This would lead to runaway recursion. This is NOT what we want. What we want is for the reciever to be a by-name Stream. Hence the ConsWrapper:
The constructor for ConsWrapper is class ConsWrapper[T](tail: => Stream[T]) taking a by-name Stream, and it's created through an implicit conversion Stream.consWrapper[T](stream: => Stream[T]), which also takes a by-name Stream.
Hence, we have performed an implicit conversion on the result of a function that has not yet been called, and we have mimiced the effect of calling #:: with a by-name this reference.
The problem here is that the scaladoc search does not allow you to look for an inner class/object (i.e. whose parent is not a package). The declaration of #:: is either Stream.#:: or Stream.ConsWrapper.#:::
object Stream {
//STUFF
/** An extractor that allows to pattern match streams with `#::`.
*/
object #:: {
def unapply[A](xs: Stream[A]): Option[(A, Stream[A])] =
if (xs.isEmpty) None
else Some((xs.head, xs.tail))
}
class ConsWrapper[A](tl: => Stream[A]) {
def #::(hd: A): Stream[A] = new Stream.Cons(hd, tl)
def #:::(prefix: Stream[A]): Stream[A] = prefix append tl
}
//MORE STUFF
}
You could request this as an RFE to the scaladoc tool in trac.
In IntelliJ IDEA's scala plugin, you could have used symbol lookup (CTRL+ ALT+ SHIFT+ N) and typed #:: and this would have brought up both declarations of #:: immediately.
Well, normally, if we see
foo bar baz
then bar is a method, defined for foo, so we first look in the class/object - definition of foo, then the inheritance/trait tree upwards (+ in implicit conversions to and from foo, in the current file, and in (directly) included files).
Except 'bar' ends in a colon, which is the case here. Then it is to be read in reverse order -
foo bar: baz
is not
foo.bar: (baz)
, but
baz.bar: (foo)
So we have to look up in the way described above, but not for foo, but for baz.
That particular method is defined in a nested class inside of Stream, called scala.collection.immutable.Stream.ConsWrapper.
And no, I have absolutely no idea how one would go about finding it. I only stumbled across it by accident. And even though I knew where to find it now, when I wanted to post the link to the class here in my answer, I still couldn't find it on the first (and even second and third) try.