I am trying to find the documentation for the Scala operator method #::. I believe that it is defined in the Stream class because of an example I found that uses it.
My question is not particular to this method (although I would like to know where the docs are), but how to search the Scala docs in general. I tried entering #:: in the search box in the upper left of the documentation page (2.8.1), but found nothing.
I suggest using the Reference Index - it's designed specifically to look for any kind of symbol (class, traits, methods, vals, vars) regardless of it's hierarchical position - contrasting with the Scaladoc's left index which doesn't show inner classes, traits or objects.
Unfortunately it's only available in the nightly. You can see the whole thing at nightly Scaladoc. Notice the upper box in the left frame, above the index.
Hope it will be bundled with Scala 2.9.0.
Edit As of 2.9.0, the reference index started to be bundle with Scaladoc. No need to go to the nightly docs now.
As others have already mentioned, #:: is defined on scala.collection.immutable.Stream.ConsWrapper. I just wanted to take a minute to elaborate on why that is.
In general, to call an operator on an object, that object needs to exist. However, the idea with a Stream is the tail of the stream is not evaluated until it needs to be. So consider the following stream:
def fibs(a:Int,b:Int):Stream[Int] = a #:: fibs(b,a+b)
Ordinarily, we would need to evaluate the recursive fibs call so that we could call the #:: operator on it. This would lead to runaway recursion. This is NOT what we want. What we want is for the reciever to be a by-name Stream. Hence the ConsWrapper:
The constructor for ConsWrapper is class ConsWrapper[T](tail: => Stream[T]) taking a by-name Stream, and it's created through an implicit conversion Stream.consWrapper[T](stream: => Stream[T]), which also takes a by-name Stream.
Hence, we have performed an implicit conversion on the result of a function that has not yet been called, and we have mimiced the effect of calling #:: with a by-name this reference.
The problem here is that the scaladoc search does not allow you to look for an inner class/object (i.e. whose parent is not a package). The declaration of #:: is either Stream.#:: or Stream.ConsWrapper.#:::
object Stream {
//STUFF
/** An extractor that allows to pattern match streams with `#::`.
*/
object #:: {
def unapply[A](xs: Stream[A]): Option[(A, Stream[A])] =
if (xs.isEmpty) None
else Some((xs.head, xs.tail))
}
class ConsWrapper[A](tl: => Stream[A]) {
def #::(hd: A): Stream[A] = new Stream.Cons(hd, tl)
def #:::(prefix: Stream[A]): Stream[A] = prefix append tl
}
//MORE STUFF
}
You could request this as an RFE to the scaladoc tool in trac.
In IntelliJ IDEA's scala plugin, you could have used symbol lookup (CTRL+ ALT+ SHIFT+ N) and typed #:: and this would have brought up both declarations of #:: immediately.
Well, normally, if we see
foo bar baz
then bar is a method, defined for foo, so we first look in the class/object - definition of foo, then the inheritance/trait tree upwards (+ in implicit conversions to and from foo, in the current file, and in (directly) included files).
Except 'bar' ends in a colon, which is the case here. Then it is to be read in reverse order -
foo bar: baz
is not
foo.bar: (baz)
, but
baz.bar: (foo)
So we have to look up in the way described above, but not for foo, but for baz.
That particular method is defined in a nested class inside of Stream, called scala.collection.immutable.Stream.ConsWrapper.
And no, I have absolutely no idea how one would go about finding it. I only stumbled across it by accident. And even though I knew where to find it now, when I wanted to post the link to the class here in my answer, I still couldn't find it on the first (and even second and third) try.
Related
I'm trying to understand the scalaz tree structure and am having some difficulty!
First I've defined a tree:
val tree: Tree[Int] =
1.node(
2.leaf,
3.node(
4.leaf,
5.leaf))
So far using TreeLoc I've worked out how to find the first element that matches some predicate. E.g. to find the first node where the value is 3:
tree.loc.find(x => x.getLabel == 3)
My next challenge was to try and find all nodes that match some predicate. For example I would like to find all leaf nodes (which should be pretty easy using TreeLoc and isLeaf). Unfortunately I can't for the life of me work out how to walk the tree to do this.
Edit: Sorry I don't think I was clear enough in my original question. To be clear I want to walk the tree in such a way that I have information about the Node available to me. Flatten, foldRight etc just allow me to operate on [Int] whereas I want to be able to operate on Tree[Int] (or TreeLoc[Int]).
Having a look to how find is implemented in scalaz, my suggestion is to implement something like:
implicit class FilterTreeLoc[A](treeLoc: TreeLoc[A]){
def filter(p: TreeLoc[A] => Boolean): Stream[TreeLoc[A]] =
Cobind[TreeLoc].cojoin(treeLoc).tree.flatten.filter(p)
}
It behaves like the find but it gives you back instead a Stream[TreeLoc[A]] instead of an Option[TreeLoc[A]].
You can use this as tree.loc.filter(_.isLeaf) and tree.loc.filter(_.getLabel == 3).
Note: the use of an implicit class can be obviously avoided if you prefer to have this declared as a method instead.
I have been looking into FP languages (off and on) for some time and have played with Scala, Haskell, F#, and some others. I like what I see and understand some of the fundamental concepts of FP (with absolutely no background in Category Theory - so don't talk Math, please).
So, given a type M[A] we have map which takes a function A=>B and returns a M[B]. But we also have flatMap which takes a function A=>M[B] and returns a M[B]. We also have flatten which takes a M[M[A]] and returns a M[A].
In addition, many of the sources I have read describe flatMap as map followed by flatten.
So, given that flatMap seems to be equivalent to flatten compose map, what is its purpose? Please don't say it is to support 'for comprehensions' as this question really isn't Scala-specific. And I am less concerned with the syntactic sugar than I am in the concept behind it. The same question arises with Haskell's bind operator (>>=). I believe they both are related to some Category Theory concept but I don't speak that language.
I have watched Brian Beckman's great video Don't Fear the Monad more than once and I think I see that flatMap is the monadic composition operator but I have never really seen it used the way he describes this operator. Does it perform this function? If so, how do I map that concept to flatMap?
BTW, I had a long writeup on this question with lots of listings showing experiments I ran trying to get to the bottom of the meaning of flatMap and then ran into this question which answered some of my questions. Sometimes I hate Scala implicits. They can really muddy the waters. :)
FlatMap, known as "bind" in some other languages, is as you said yourself for function composition.
Imagine for a moment that you have some functions like these:
def foo(x: Int): Option[Int] = Some(x + 2)
def bar(x: Int): Option[Int] = Some(x * 3)
The functions work great, calling foo(3) returns Some(5), and calling bar(3) returns Some(9), and we're all happy.
But now you've run into the situation that requires you to do the operation more than once.
foo(3).map(x => foo(x)) // or just foo(3).map(foo) for short
Job done, right?
Except not really. The output of the expression above is Some(Some(7)), not Some(7), and if you now want to chain another map on the end you can't because foo and bar take an Int, and not an Option[Int].
Enter flatMap
foo(3).flatMap(foo)
Will return Some(7), and
foo(3).flatMap(foo).flatMap(bar)
Returns Some(15).
This is great! Using flatMap lets you chain functions of the shape A => M[B] to oblivion (in the previous example A and B are Int, and M is Option).
More technically speaking; flatMap and bind have the signature M[A] => (A => M[B]) => M[B], meaning they take a "wrapped" value, such as Some(3), Right('foo), or List(1,2,3) and shove it through a function that would normally take an unwrapped value, such as the aforementioned foo and bar. It does this by first "unwrapping" the value, and then passing it through the function.
I've seen the box analogy being used for this, so observe my expertly drawn MSPaint illustration:
This unwrapping and re-wrapping behavior means that if I were to introduce a third function that doesn't return an Option[Int] and tried to flatMap it to the sequence, it wouldn't work because flatMap expects you to return a monad (in this case an Option)
def baz(x: Int): String = x + " is a number"
foo(3).flatMap(foo).flatMap(bar).flatMap(baz) // <<< ERROR
To get around this, if your function doesn't return a monad, you'd just have to use the regular map function
foo(3).flatMap(foo).flatMap(bar).map(baz)
Which would then return Some("15 is a number")
It's the same reason you provide more than one way to do anything: it's a common enough operation that you may want to wrap it.
You could ask the opposite question: why have map and flatten when you already have flatMap and a way to store a single element inside your collection? That is,
x map f
x filter p
can be replaced by
x flatMap ( xi => x.take(0) :+ f(xi) )
x flatMap ( xi => if (p(xi)) x.take(0) :+ xi else x.take(0) )
so why bother with map and filter?
In fact, there are various minimal sets of operations you need to reconstruct many of the others (flatMap is a good choice because of its flexibility).
Pragmatically, it's better to have the tool you need. Same reason why there are non-adjustable wrenches.
The simplest reason is to compose an output set where each entry in the input set may produce more than one (or zero!) outputs.
For example, consider a program which outputs addresses for people to generate mailers. Most people have one address. Some have two or more. Some people, unfortunately, have none. Flatmap is a generalized algorithm to take a list of these people and return all of the addresses, regardless of how many come from each person.
The zero output case is particularly useful for monads, which often (always?) return exactly zero or one results (think Maybe- returns zero results if the computation fails, or one if it succeeds). In that case you want to perform an operation on "all of the results", which it just so happens may be one or many.
The "flatMap", or "bind", method, provides an invaluable way to chain together methods that provide their output wrapped in a Monadic construct (like List, Option, or Future). For example, suppose you have two methods that produce a Future of a result (eg. they make long-running calls to databases or web service calls or the like, and should be used asynchronously):
def fn1(input1: A): Future[B] // (for some types A and B)
def fn2(input2: B): Future[C] // (for some types B and C)
How to combine these? With flatMap, we can do this as simply as:
def fn3(input3: A): Future[C] = fn1(a).flatMap(b => fn2(b))
In this sense, we have "composed" a function fn3 out of fn1 and fn2 using flatMap, which has the same general structure (and so can be composed in turn with further similar functions).
The map method would give us a not-so-convenient - and not readily chainable - Future[Future[C]]. Certainly we can then use flatten to reduce this, but the flatMap method does it in one call, and can be chained as far as we wish.
This is so useful a way of working, in fact, that Scala provides the for-comprehension as essentially a short-cut for this (Haskell, too, provides a short-hand way of writing a chain of bind operations - I'm not a Haskell expert, though, and don't recall the details) - hence the talk you will have come across about for-comprehensions being "de-sugared" into a chain of flatMap calls (along with possible filter calls and a final map call for the yield).
Well, one could argue, you don't need .flatten either. Why not just do something like
#tailrec
def flatten[T](in: Seq[Seq[T], out: Seq[T] = Nil): Seq[T] = in match {
case Nil => out
case head ::tail => flatten(tail, out ++ head)
}
Same can be said about map:
#tailrec
def map[A,B](in: Seq[A], out: Seq[B] = Nil)(f: A => B): Seq[B] = in match {
case Nil => out
case head :: tail => map(tail, out :+ f(head))(f)
}
So, why are .flatten and .map provided by the library? Same reason .flatMap is: convenience.
There is also .collect, which is really just
list.filter(f.isDefinedAt _).map(f)
.reduce is actually nothing more then list.foldLeft(list.head)(f),
.headOption is
list match {
case Nil => None
case head :: _ => Some(head)
}
Etc ...
I was exchanging emails with an acquaintance that is a big Kotlin, Clojure and Java8 fan and asked him why not Scala. He provided many reasons (Scala is too academic, too many features, not the first time I hear this and I think this is very subjective)
but his biggest pain point was as an example, that he doesn't like a language where he can't understand the implementation of basic data structures, and he gave LinkedList as an example.
I took a look at scala.collection.LinkedList and counted the things I either understand or somewhat understand.
CanBuildFrom - after some effort, I get it, type classes, not the longest suicide note
in history [1]
LinkedListLike - I can't remember where I read it, but I got convinced this is there for a good reason
But then I started to stare at these
GenericTraversableTemplate - now I'm scratching my head as well...
SeqFactory, GenericCompanion - OK, now you lost me, I start to understand his point
Can someone who understand this well please explain GenericTraversableTemplate SeqFactory and GenericCompanion in the context of LinkedList? What they are for, what impact on the end user they have (e.g. I'm sure they are there for a good reason, what is that reason?)
Are they there for a practical reason? or is it a level of abstraction that could have been simplified?
I like Scala collections because I don't have to understand the internals to be able to effectively use them. I don't mind a complex implementation if it helps me to keep my usage simpler. e.g. I don't mind paying the price of a complex library if I get the ability to write cleaner more elegant code using it in return. but it will sure be nice to better understand it.
[1] - Is the Scala 2.8 collections library a case of "the longest suicide note in history"?
I will try to describe the concepts from the point of view of a random pedestrian (I've never contributed a single line to the Scala collection library, so don't hit me too hard if I'm wrong).
Since LinkedList is now deprecated, and because Maps provide a better example, I will use TreeMap as example.
CanBuildFrom
The motivation is this: If we take a TreeMap[Int, Int] and map it with
case (x, y) => (2 * x, y * y * 0.3d)
we get TreeMap[Int, Double]. This type safety alone would already explain the necessity for
simple genericBuilder[X] constructs.
However, if we map it with
case (x, y) => x
we obtain an Iterable[Int] (more precisely: a List[Int]), this is no longer a Map, the type of the container has changed. This is where CBF's come into play:
CanBuildFrom[This, X, That]
can be seen as a kind of "type-level function" that tells us: if we map a collection of type
This with a function that returns values of type X, we can build a That. The most specific
CBF is provided at compile time, in the first case it will be something like
CanBuildFrom[TreeMap[_,_], (X,Y), TreeMap[X,Y]]
in the second case it will be something like
CanBuildFrom[TreeMap[_,_], X, Iterable[X]]
and so we always get the right type of the container. The pattern is pretty general.
Every time you have a generic function
foo[X1, ..., Xn](x1: X1, ..., xn: Xn): Y
where the result type Y depends on X1, ..., Xn, you can introduce an implicit parameter as
follows:
foo[X1, ...., Xn, Y](x1: X1, ..., xn: Xn)(implicit CanFooFrom[X1, ..., Xn, Y]): Y
and then define the type-level function X1, ..., Xn -> Y piecewise by providing multiple
implicit CanFooFrom's.
LinkedListLike
In the class definition, we see something like this:
TreeMap[A, B] extends SortedMap[A, B] with SortedMapLike[A, B, TreeMap[A, B]]
This is Scala's way to express the so-called F-bounded polymorphism.
The motivation is as follows: Suppose we have a dozen (or at least two...) implementations of the trait SortedMap[A, B]. Now we want to implement a method withoutHead, it could look
somewhat like this:
def withoutHead = this.remove(this.head)
If we move the implementation into SortedMap[A, B] itself, the best we can do is this:
def withoutHead: SortedMap[A, B] = this.remove(this.head)
But is this the most specific result type we can get? No, that's too vague.
We would like to return TreeMap[A, B] if the original map is a TreeMap, and
CrazySortedLinkedHashMap (or whatever...) if the original was a CrazySortedLinkedHashMap.
This is why we move the implementation into SortedMapLike, and give the following signature to the withoutHead method:
trait SortedMapLike[A, B, Repr <: SortedMap[A, B]] {
...
def withoutHead: Repr = this.remove(this.head)
}
now because TreeMap[A, B] extends SortedMapLike[A, B, TreeMap[A, B]], the result type of
withoutHead is TreeMap[A,B]. The same holds for CrazySortedLinkedHashMap: we get the exact type back. In Java, you would either have to return SortedMap[A, B] or override the method in each subclass (which turned out to be a maintenance nightmare for the feature-rich traits in Scala)
GenericTraversableTemplate
The type is: GenericTraversableTemplate[+A, +CC[X] <: GenTraversable[X]]
As far as i can tell, this is just a trait that provides implementations of
methods that somehow return regular collections with same container type but
possibly different content type (stuff like flatten, transpose, unzip).
Stuff like foldLeft, reduce, exists are not here because these methods care only about content type, not container type.
Stuff like flatMap is not here, because the container type can change (again, CBF's).
Why is it a separate trait, is there a fundamental reason why it exists?
I don't think so... It probably would be possible to group the godzillion of methods somewhat differently. But this is just what happens naturally: you start to implement a trait, and it turns out that it has very many methods. So instead you group loosely related methods, and put them into 10 different traits with awkward names like "GenTraversableTemplate", and them mix them all into traits/classes where you need them...
GenericCompanion
This is just an abstract class that implements some basic functionality which is common
for companion objects of most collection classes (essentially, it just implements very
simple factory methods apply(varargs) and empty).
For example there is method apply that takes varargs of some type A and returns a collection of type CC[A]:
Array(1, 2, 3, 4) // calls Array.apply[A](elems: A*) on the companion object
List(1, 2, 3, 4) // same for List
The implementation is very simple, it's something like this:
def apply[A](varargs: A*): CC[A] = {
val builder = newBuilder[A]
for (arg <- varargs) builder += arg
builder.result()
}
This is obviously the same for Arrays and Lists and TreeMaps and almost everything else, except 'constrained irregular Collections' like Bitset. So this is just common functionality in a common ancestor class of most companion objects. Nothing special about that.
SeqFactory
Similar to GenericCompanion, but this time more specifically for Sequences.
Adds some common factory methods like fill() and iterate() and tabulate() etc.
Again, nothing particularly rocket-scientific here...
Few general remarks
In general: I don't think that one should attempt to understand every single trait in this library. Rather, one should try to look at the library as a whole. As a whole, it has a very interesting architecture. And in my personal opinion, it's actually a very aesthetic piece of software, but one has to stare at it for quite a while (and try to re-implement the whole architectural pattern several times) to grasp it. On the other hand: for example CBF's are kind of "design pattern" that clearly should be eliminated in successors of this language. The whole story with the scope of implicit CBF's still seems like a total nightmare to me. But many things seemed completely inscrutable at first, and almost always, it ended with an epiphany (which is very specific for Scala: for the majority of other languages, such struggles usually end with the thought "Author of this is a complete idiot").
So I'm trying to work through Norvig & Russell's "Artificial Intelligence, A Modern Approach" as a way to learn Scala. I have a pretty good grasp on the language basics at this point, but I still find myself often "fighting" the type system.
Long story short, breadth-first and depth-first search algorithms are the same aside from the mechanics of pushing/popping to their underlying collection. Depth-first would prepend new possibilities and use a Stack, while Breadth-first would append and use a Queue.
To keep my algorithm the same, I created a typeclass called "GiveGrab" (I know, horrible name) with the intention of pimping ... err ... enriching collections with these "default" push (give) and pop-like (grab) operations.For example, grab would result in a call to .dequeue() for queues, and .pop() for stacks.
Here's (a somewhat abbreviated version of) the code:
object Example extends App {
trait GiveGrab[A, M[A]] {
def give(x: A*): M[A]
def grab(): A
}
implicit class GiveGrabQueue[T](q: Queue[T]) extends GiveGrab[T,Queue[T]] {
override def give(x: T*) = q ++= x
override def grab() = q.dequeue()
}
class TestClass[T, X <% GiveGrab[T, Queue[T]]](var storage: X) {}
val test = new TestClass[Int, Queue[Int]](new Queue[Int]())
}
When trying to compile this, I get the following errors:
Error:(18, 39) scala.collection.mutable.Queue[T] takes no type parameters, expected: one
class TestClass[T, X <% GiveGrab[T, Queue[T]]](var storage: X) {}
^
Error:(13, 67) scala.collection.mutable.Queue[T] takes no type parameters, expected: one
implicit class GiveGrabQueue[T](q: Queue[T]) extends GiveGrab[T,Queue[T]] {
^
That said, it took me a lot of trial and error to even get to this point. I'm not sure if my trait is really supposed to be typed
trait GiveGrab[A, M[A]]
or
trait GiveGrab[A, M[_]]
or
trait GiveGrab[A, M]
The error "takes no type parameters, expected: one" doesn't make a whole lot of sense to me at this point, and there's only a handful of other posts about that message (some related to dependent types, and some related to the Play framework).
Somewhat related: is there a good article for understanding Scala type signatures? I read through Programming in Scala 2nd Ed, but it didn't really touch on this sort of type gymnastics (either that, or I just missed it.)
Edit: Typos
What #PatrykĆwiek proposed is not a workaround but actually what you are meant to be doing: M[A] in trait GiveGrab defines a type function. Roughly speaking this means: M is a type where you can apply a single type parameter to yield a concrete type. That the parameter is called A is pure coincidence. The following means the same:
trait GiveGrab[A,M[MyRandomName]] { ... }
In the definition of give, you actually use this type function to create a type, when saying M[A]. Therefore, as #PatrykĆwiek said, you should write Queue instead of Queue[T]. While Queue is precisely one of these type functions, Queue[T] is a concrete type and therefore doesn't apply to the definition of M.
The error message you get says exactly that: In the place of M, you are supposed to put a type that takes a parameter (like Queue), but you have put one which takes none (Queue[T] in your case, another example would be String or Int).
In Scala, if I define a method called apply in a class or a top-level object, that method will be called whenever I append a pair a parentheses to an instance of that class, and put the appropriate arguments for apply() in between them. For example:
class Foo(x: Int) {
def apply(y: Int) = {
x*x + y*y
}
}
val f = new Foo(3)
f(4) // returns 25
So basically, object(args) is just syntactic sugar for object.apply(args).
How does Scala do this conversion?
Is there a globally defined implicit conversion going on here, similar to the implicit type conversions in the Predef object (but different in kind)? Or is it some deeper magic? I ask because it seems like Scala strongly favors consistent application of a smaller set of rules, rather than many rules with many exceptions. This initially seems like an exception to me.
I don't think there's anything deeper going on than what you have originally said: it's just syntactic sugar whereby the compiler converts f(a) into f.apply(a) as a special syntax case.
This might seem like a specific rule, but only a few of these (for example, with update) allows for DSL-like constructs and libraries.
It is actually the other way around, an object or class with an apply method is the normal case and a function is way to construct implicitly an object of the same name with an apply method. Actually every function you define is an subobject of the Functionn trait (n is the number of arguments).
Refer to section 6.6:Function Applications of the Scala Language Specification for more information of the topic.
I ask because it seems like Scala strongly favors consistent application of a smaller set of rules, rather than many rules with many exceptions.
Yes. And this rule belongs to this smaller set.