transformations of data structures with structural sharing - scala

Question

In functional programming, it's possible to save a large amount of memory by employing structural sharing. For instance, these two lists are the same, but the second is represented in memory in a much more efficient way:
val n = 4
// n: Int = 4
val l1 = List.tabulate(n)(x => (n-x until n).toList)
// l1: List[List[Int]] = List(List(), List(3), List(2, 3), List(1, 2, 3))
val l2 = List.unfold((n, List.empty[Int])) { case (i, l) =>
if (i > 0) Some((l, ((i - 1), i - 1 :: l)))
else None
}
// l2: List[List[Int]] = List(List(), List(3), List(2, 3), List(1, 2, 3))
But the moment you actually do something with l2, this advantage is quickly lost. Not only will the result of l2.map(_.map(_ + 1)) require just as much memory as l1, it's also inefficient, because it will perform n*(n-1)/2 additions, even though there's only n-1 different numbers in this data structure.
With a mutable data structure this is easy: you can update the values in place, along with a marker that tells you whether the operation was already performed on that node. This way, you only traverse the data structure once, structural sharing is preserved and you only perform n-1 additions.
But is there some elegant, functional way to achieve this without using mutable data structures?

Answer 1

As far as I know the mutable collections in the standard library don't take advantage of structural sharing, so you'd be implementing one on your own.
l2's structural sharing is somewhat coincidental: a side effect of how it was built up. It's not encoding the underlying structure (in this case that the nth element is the tail of the n+1th element with the last element being all the underlying elements). But you can encode that structure fairly easily (I suspect it may be easier than with the mutable version).
Using the 2.13 APIs
case class TailsFirst[A](val elements: List[A]) extends Seq[List[A]] {
def length: Int = 1 + elements.length
def apply(idx: Int): List[A] = {
require(idx < length)
underlying.drop(elements.size - idx)
}
// useful for iteration, which will happen a lot (e.g. in toString)
lazy val reverseElements: List[A] = elements.reverse
def iterator(): Iterator[List[A]] =
new Iterator[List[A]] {
var state: Option[List[A]] = Some(reverseElements)
var toEmit: List[A] = Nil
def hasNext: Boolean = state.nonEmpty
def next(): List[A] = {
if (hasNext) {
val ret = toEmit
state.flatMap(_.headOption) match {
case None =>
assume(state.contains(Nil)) // hint for a mythical static analyzer
state = None
ret
case Some(toPrepend) =>
state = state.map(_.tail)
toEmit = toPrepend :: toEmit
ret
}
} else throw new NoSuchElementException("exhausted iterator")
def flatMapElements[B](f: A => List[B]): TailsFirst[B] =
TailsFirst(elements.flatMap(f))
def mapElements[B](f: A => B): TailsFirst[B] =
TailsFirst(elements.map(f))
}