There are a lot of places in the Scala API, particularly in collections, where method signatures are wrong.
For example, the scaladoc signature for Map.flatMap says
def flatMap[B](f: (A) ⇒ GenTraversableOnce[B]): Map[B]
But the actual signature is
flatMap[B, That](f: ((A, B)) ⇒ GenTraversableOnce[B])
(implicit bf: CanBuildFrom[Map[A, B], B, That]): That
This one especially makes no sense because the scaladoc signature includes Map[B], but Map has two type parameters, not one.
What's going on here? Is this a mistake?
The incorrect signatures you're see in the generated documentation are called "use cases". They're supposed to clarify the documentation by showing idealized API, similar to the real one but omitting tedious details (like the pervasive implicit CanBuildFrom parameter which bothers some people).
For methods with use cases, you can get to the real signature by clicking the method name to show the details for that method, and then clicking "Full Signature" to expand another section that shows the signature.
References
Scala issue SI-3448, created May 2010, deals specifically with the wrong number of type parameters being shown for Map. This issue as closed "Won't Fix" in July 2012.
Paul Phillips' talk Scala Collections: Why Not? from January 2014 scorns the use cases as "lies" in slide 1, slide 2, slide 3.
In GenTraversableLike.scala you can see an example of a directive which causes use case documentation to be generated:
#usecase def flatMap[B](f: A => TraversableOnce[B]): $Coll[B]
Similar questions
Why does the scaladoc say HashMap.toArray returns Array[A] instead of Array[(A,B)]?
The List :: method and covariance ... why do they hide the real signature?
scaladoc for map on Map ... is the documentation wrong, or am I missing something?
What's going on here with scala.collection.immutable.Stack.+: (prepend)? ...
Why do some method descriptions in Scaladoc start with [use case]?
How to explain Map.map result
Related
Following the awesomely enlightening question by #TravisBrown concerning the enumeration of ADTs using shapeless, I am left with the following code snippet:
implicitly[EnumerableAdt[Foo]].values
I would like to encapsulate this within a method so that I don't have to .values after each invocation (It seems a cleaner API to me, that way). But I can't seem to get it right. Whenever I try to encapsulate the implicitly[EnumerableAdt[Foo]] I get implicit resolution errors.
What I had tried, that made most sense to me, was, for example:
def imply[T](implicit ev: T):Set[T] = implicitly[EnumerableAdt[T]].values
certainly without the ev made even less sense to me.
I am no expert in type level programming.
If you look at the definition of implicitly[X], you can see that is requires an implicit argument of type X in scope. In your example, you have implicit ev: T in scope, which is not enough to call implicitly[EnumerableAdt[T]]! Try the following definition instead:
def imply[T](implicit ev: EnumerableAdt[T]):Set[T] = ev.values
Looking at the <> method in the following scala slick class, from http://slick.typesafe.com/doc/2.1.0/api/index.html#scala.slick.lifted.ToShapedValue, it reminds me of that iconic stackoverflow thread about scala prototypes.
def <>[R, U](f: (U) ⇒ R, g: (R) ⇒ Option[U])
(implicit arg0: ClassTag[R], shape: Shape[_ <: FlatShapeLevel, T, U, _]):
MappedProjection[R, U]
Can someone bold and knowledgeable provide an articulate walkthrough of that long prototype definition, carefully clarifying all of its type covariance/invariance, double parameter lists, and other advanced scala aspects?
This exercise will also greatly help dealing with similarly convoluted prototypes!
Ok, let's take a look:
class ToShapedValue[T](val value: T) extends AnyVal {
...
#inline def <>[R: ClassTag, U](f: (U) ⇒ R, g: (R) ⇒ Option[U])(implicit shape: Shape[_ <: FlatShapeLevel, T, U, _]): MappedProjection[R, U]
}
The class is an AnyVal wrapper; while I can't actually see the implicit conversion from a quick look, it smells like the "pimp my library" pattern. So I'm guessing this is meant to add <> as an "extension method" onto some (or maybe all) types.
#inline is an annotation, a way of putting metadata on, well, anything; this one is a hint to the compiler that this should be inlined. <> is the method name - plenty of things that look like "operators" are simply ordinary methods in scala.
The documentation you link has already expanded the R: ClassTag to ordinary R and an implicit ClassTag[R] - this is a "context bound" and it's simply syntactic sugar. ClassTag is a compiler-generated thing that exists for every (concrete) type and helps with reflection, so this is a hint that the method will probably do some reflection on an R at some point.
Now, the meat: this is a generic method, parameterized by two types: [R, U]. Its arguments are two functions, f: U => R and g: R => Option[U]. This looks a bit like the functional Prism concept - a conversion from U to R that always works, and a conversion from R to U that sometimes doesn't work.
The interesting part of the signature (sort of) is the implicit shape at the end. Shape is described as a "typeclass", so this is probably best thought of as a "constraint": it limits the possible types U and R that we can call this function with, to only those for which an appropriate Shape is available.
Looking at the documentation forShape, we see that the four types are Level, Mixed, Unpacked and Packed. So the constraint is: there must be a Shape, whose "level" is some subtype of FlatShapeLevel, where the Mixed type is T and the Unpacked type is R (the Packed type can be any type).
So, this is a type-level function that expresses that R is "the unpacked version of" T. To use the example from the Shape documentation again, if T is (Column[Int], Column[(Int, String)], (Int, Option[Double])) then R will be (Int, (Int, String), (Int, Option[Double]) (and it only works for FlatShapeLevel, but I'm going to make a judgement call that that's probably not important). U is, interestingly enough, completely unconstrained.
So this lets us create a MappedProjection[unpacked-version-of-T, U] from any T, by providing conversion functions in both directions. So in a simple version, maybe T is a Column[String] - a representation of a String column in a database - and we want to represent it as some application-specific type, e.g. EmailAddress. So R=String, U=EmailAddress, and we provide conversion functions in both directions: f: EmailAddress => String and g: String => Option[EmailAddress]. It makes sense that it's this way around: every EmailAddress can be represented as a String (at least, they'd better be, if we want to be able to store them in the database), but not every String is a valid EmailAddress. If our database somehow had e.g. "http://www.foo.com/" in the email address column, our g would return None, and Slick could handle this gracefully.
MappedProjection itself is, sadly, undocumented. But I'm guessing it's some kind of lazy representation of a thing we can query; where we had a Column[String], now we have a pseudo-column-thing whose (underlying) type is EmailAddress. So this might allow us to write pseudo-queries like 'select from users where emailAddress.domain = "gmail.com"', which would be impossible to do directly in the database (which doesn't know which part of an email address is the domain), but is easy to do with the help of code. At least, that's my best guess at what it might do.
Arguably the function could be made clearer by using a standard Prism type (e.g. the one from Monocle) rather than passing a pair of functions explicitly. Using the implicit to provide a type-level function is awkward but necessary; in a fully dependently typed language (e.g. Idris), we could write our type-level function as a function (something like def unpackedType(t: Type): Type = ...). So conceptually, this function looks something like:
def <>[U](p: Prism[U, unpackedType(T)]): MappedProjection[unpackedType(T), U]
Hopefully this explains some of the thought process of reading a new, unfamiliar function. I don't know Slick at all, so I have no idea how accurate I am as to what this <> is used for - did I get it right?
Heiko Seeberger wrote a great blog post on category theory here:
https://hseeberger.wordpress.com/2010/11/25/introduction-to-category-theory-in-scala/
In it, he defines a GenericFunctor like so:
trait GenericFunctor[->>[_, _], ->>>[_, _], F[_]] {
def fmap[A, B](f: A ->> B): F[A] ->>> F[B]
}
I did not have any luck finding documentation references to the ->> and ->>> symbols in documentation. Could someone please explain what they are doing?
The symbols themselves don't mean anything. They are arbitrary names Heiko picked:
> class Foo[A, B]
defined class Foo
> class Foo[M1[_], M2[_]]
defined class Foo
> class GenericFunctor[->>[_, _], ->>>[_, _], F[_]]
defined class GenericFunctor
They are parts of type parameters that they themselves are type constructors (higher-kinded types if you want to sound fancy).
Type applications can be written infix, so A ->> B is same as ->>[A, B].
As per what's going on... Heiko says
Looking at the ingredients, we find all that we need: Types A and B are mapped to types F[A] and F[B] and maps A ->> B are mapped to maps F[A] ->>> F[B].
Since we are talking category theory, we want to avoid the term function because that's implementation specific, but we want to describe something kind of like a function. Something-like-a-function in their lingo is an arrow. We need two of them since we don't want to assume the incoming and outgoing arrows to be the same. These two arrows are represented by ->> and ->>>. F[_] is a container like List and Option. I think..
So fmap (aka map method in Scala) takes an arrow of values and returns another arrow of containers. Except unlike map method, fmap returns an arrow that takes a container.
A specific application of the GenericFunctor using Function for both arrows is Functor. And specific application of Functor that uses List for the container is ListFunctor.
object ListFunctor extends Functor[List] {
def fmap[A, B](f: A => B): List[A] => List[B] = as => as map f
}
So that's taking a function from A to B, and returning a function from List[A] to List[B], calling map internally.
A clue is that they are within square brackets in the trait definition: they are just arbitrary symbols that have been picked by the blog author, just as [T] is often chosen for generic classes, traits, and methods. These here just happen to be higher-kinded types (i.e. parameters with parameters).
The arrow-like name was chosen because, as he says,
"A ->> B is just another way to write ->>[A, B], which nicely reflects
the fact that we are talking about maps here."
Answer copied from my answer to a duplicate question on request:
It's a higher-kinded type, described nicely in this introduction and in this research paper.
The reason you might find it confusing is that ->> is the name for the higher-kinded type -- it might have as well been called Arrow instead.
everyone . Please forgive me asking a stupid question on Scala.
Though I have been programming in Scala for about 2 years, I still find it hard to understand implicit usage. Let's take an example for discussion:
Array(1,2,3,4).map(x => x)
If you look up the scaladoc, you cant' find the method map on Array class. The reason that map can apply on Array(1,2,3,4) is that there is an implicit function implicit def intArrayOps (xs: Array[Int]): ArrayOps[Int] defined in scala.Predef.
However, there are two parameter lists, where the second one is written as implicit bf: CanBuildFrom[Array[T], B, That]). Now I wonder where the compiler finds a proper argument for type CanBuildFrom when applying map on Array(1,2,3,4).
The implicit resolution includes searching the companion object for the type of the implicit parameter as well as the companion objects for the type parameters of the implicit parameter. In the example above the signature of map is the following
def map[B, That](f: (Int) => B)(implicit bf: CanBuildFrom[Array[Int], B, That]): That
Since we have no type requirements for That we can ignore it for now. After we look in the local and container scopes and find no matching implicits, the next place to look for an implicit would be the companion object for CanBuildFrom. However it has no companion object. So we continue on and look in Array for an implicit. We find one in the form of
implicit def canBuildFrom[T](implicit m: ClassManifest[T]): CanBuildFrom[Array[_], T, Array[T]]
Since we have no type requirements and a matching implicit, "That" is forced to be of type Array[Int] and completes our typing.
This question is partially answered with other question on StackOverflow. Let me give a try to summarize them:
The first part you need to know is where the Scala compiler looks for implicits. You can find some more detail about CanBuildFrom here.
If you have understood what is mentioned in the answers about implicits ‣you should take a look to the construction of Scala Collections. Their inheritance-hierarchy is explained here and for List here. All of them are build up with Builders. This is explained in detail in a question about breakOut.
To round up your knowledge, you should know how to pimp the Collections. Also, this is explained on StackOverflow in this question.
Please note, the best answers on StackOverflow are summarized in the Scala-Tag-Wiki.
I'm going throught collection api and the returned type of operations which can modify the type of container and the type of contained element have all the header in its non-usecase form similar to the following:def map [B, That] (f: (A) ⇒ B)(implicit bf: CanBuildFrom[List[A], B, That]):That
It's clear that there is sure 'fallback' to other collection type with That (as in case of BitSet fallbacking to Set if the contained item is changed to other than Int). But how can one be sure of the type of the item? I would expect the return type to That[B], but it is not. Why? Educate me please :-)
You already gave the answer yourself: If you want to be able to return a BitSet in appropriate cases, you need to get rid of that type parameter in the return type. There is no BitSet[Int], obviously.
All the information you need is already handled by the appropriate CanBuildFrom implicits which have a fallback definition of the type CanBuildFrom[Coll, A, Traversable[A]] (or whatever may be over Traversable in the type hierarchy).
Of course, the CanBuildFrom must be sensible but I think it’s actually possible to abuse it and return a truly strange That. Nonetheless, the defaults are pretty sane.