Understanding List[+A] for Covariance - scala

Looking at the source for List.scala:
sealed abstract class List[+A] extends ...
...
def isEmpty: Boolean
def head: A
def tail: List[A]
List[+A] is covariant based on the +A. Does this mean that, it's possible to create a List[T] where T can be the type itself, or any of its sub-classes?
example:
scala> trait Kid
defined trait Kid
scala> case class Boy(name: String) extends Kid
defined class Boy
scala> case class Girl(name: String) extends Kid
defined class Girl
scala> val list: List[Kid] = List(Boy("kevin"), Girl("sally"))
list: List[Kid] = List(Boy(kevin), Girl(sally))
Observe that head and tail's types are A and List[A], respectively. Once we've defined List[+A], then head and tail's A is also covariant?
I've read this StackOverflow answer 3 or 4 times, but I don't understand yet.

Your example does not relate to variance. Moreover, head and tail have nothing to do with variance too.
scala> val list: List[Kid] = List(Boy("kevin"), Girl("sally"))
list: List[Kid] = List(Boy(kevin), Girl(sally))
This would work even if List weren't covariant, because Scala will automatically deduce common supertype of Boy and Girl, that is, Kid, and type of the expression on the right side will be List[Kid], exactly what you require on the left side.
The following, however, doesn't work because java.util.List is not covariant (it is invariant since it is Java type):
scala> import java.util.{List => JList, Arrays}
import java.util.{List=>JList, Arrays}
scala> trait Kid
defined trait Kid
scala> case class Boy(name: String) extends Kid
defined class Boy
scala> val list1 = Arrays.asList(Boy("kevin"), Boy("bob"))
list1: java.util.List[Boy] = [Boy(kevin), Boy(bob)]
scala> val list2: JList[Kid] = list1
<console>:12: error: type mismatch;
found : java.util.List[Boy]
required: java.util.List[Kid]
Note: Boy <: Kid, but Java-defined trait List is invariant in type E.
You may wish to investigate a wildcard type such as `_ <: Kid`. (SLS 3.2.10)
val list2: JList[Kid] = list1
^
Arrays.asList method has signature like this:
def asList[T](args: T*): java.util.List[T]
As java.util.List[T] is invariant, it is impossible to assign JList[Boy] (list1) to JList[Kid] (list2). And there is a reason: if you could, then because JList is mutable, you could also add anything extending Kid (not only Boy) into the same list, breaking type safety.
On the other hand, scala.List will work in exactly the same situation:
scala> val list1 = List(Boy("kevin"), Boy("bob"))
list1: List[Boy] = List(Boy(kevin), Boy(bob))
scala> val list2: List[Kid] = list1
list2: List[Kid] = List(Boy(kevin), Boy(bob))
That is because scala.List is covariant in its type parameter. Note that covariant List type works as if List[Boy] were subtype of List[Kid], very similar to the case when you can assign everything to a variable of type Any because every other type is a subtype of Any. This is very helpful analogy.
Contravariance works in a very similar way, but in other direction. Consider this trait:
trait Predicate[-T] {
def apply(obj: T): Boolean
}
object Predicate {
// convenience method to convert functions to predicates
def apply[T](f: (T) => Boolean) = new Predicate[T] {
def apply(obj: T) = f(obj)
}
}
Note the - before T parameter: it is a contravariance annotation, that is, Predicate[T] is defined to be contravariant in its only type parameter.
Recall that for covariant list List[Boy] was a subtype of List[Kid]. Well, for contravariant predicate it works in the opposite way: Predicate[Kid] is a subtype of Predicate[Boy], so you can assign a value of type Predicate[Kid] to a variable of type Predicate[Boy]:
scala> val pred1: Predicate[Kid] = Predicate { kid => kid.hashCode % 2 == 0 }
pred1: Predicate[Kid] = Predicate$$anon$1#3bccdcdd
scala> val pred2: Predicate[Boy] = pred1
pred2: Predicate[Boy] = Predicate$$anon$1#3bccdcdd
If Predicate[T] weren't contravariant, we wouldn't be able to assign pred1 to pred2, though it is completely legitimate and safe: obviously, predicates defined on supertypes can easily work on subtypes.
In short, variance affects type compatibility between parameterized types. List is covariant, so you can assign a value of type List[Boy] to a variable of type List[Kid] (in fact, for any T extending S, you can assign a value of type List[T] to a variable of type List[S]).
On the other hand, because, Predicate is contravariant, you can assign Predicate[Kid] to Predicate[Boy] (that is, for any T extending S, you can assign a value of type Predicate[S] to a variable of type Predicate[T]).
If a type is invariant in its type parameter, neither of the above can be done (as is demonstrated by JList).
Note the correspondence between parameterized types and their parameters:
T <: S ===> List [T] <: List [S] (covariance)
T <: S ===> Predicate[S] <: Predicate[T] (contravariance)
This is the reason why the first effect is called *co*variance (T <: S on the left, and
..T.. <: ..S.. on the right), and the second is *contra*variance (T <: S on the left, but ..S.. <: ..T.. on the right).
Whether to make your own parameterized types covariant or contravariant or invariant depends on your class responsibilities. If it may only return values of generic type, then it makes sense to use covariance. List[T], for example, only contains methods which return T, never accept T as a parameter, so it is safe to make it covariant in order to increase expressiveness. Such parameterized types can be called producers.
If your class only accepts values of the generic type as a parameter, not returns them (exactly like Predicate above which has single method def apply(obj: T): Boolean), then you can safely make it contravariant. Such parameterized types can be called consumers
If your class both accepts and returns values of the generic type, i.e. it is both a producer and a consumer, then you have no choice but to leave the class invariant in this generic type parameter.
This idiom is usually called "PECS" ("Producer extends, Consumer super") because variance annotations are written extends and super in Java.

Related

LUB of List of Invariant Class's Instances

Given a class with a covariant type parameter:
scala> class G[+A]
defined class G
The following list shows a Least Upper Bound of List[G[Any]].
scala> List(new G[Int], new G[String])
res1: List[G[Any]] = List(G#5aa360ea, G#6548bb7d)
Then, given a class with an invariant type parameter:
scala> class F[A]
defined class F
I see a Least Upper Bound (LUB) of List[F[_ >: String with Int]].
scala> List(new F[Int], new F[String])
res0: List[F[_ >: String with Int]] = List(F#6c4980d3, F#327bcebd)
A simpler example shows a LUB of List[Any]:
scala> List( (42 : Int), "foobar" )
res2: List[Any] = List(42, foobar)
Please explain the LUB of the List of F's.
G is covariant over its type parameter A. This means that given any types A and B, if A is a sub-type of G, then G[A] is a sub-type of G[B]. Let's write this in short-hand to using A <:< B to denote that A is a sub-type of B, then G[A] <:< G[B].
This means that when we have a List(new G[Int], new G[String]), the compiler is allowed to infer it as a List[Any], because List[G[Int]] <:< List[G[Any]] and List[G[String]] <:< List[G[Any]] (since List is also covariant over its type parameter).
You probably already know that, but it's worth explaining for those that don't yet.
F is invariant over its type parameter A, so we cannot make the same deductions. Okay, so then what is the type of List(new F[Int], new F[String]) ? Because F is invariant, we cannot say that F[Int] <:< F[Any] or that F[String] <:< F[Any] (because it's not true!). So it's not a List[F[Any]].
So what can the compiler infer? It's only real choice is some existential type, because it cannot be Any, or String with Int, or anything else without breaking covariance. Since it is looking for the least upper-bound, it infers an existential type that is bounded below by the compound of the contained types (in F). That is, _ >: String with Int is some nameless type that has a lower-bound of String with Int.
Or, in other words, we know we have a List of Fs, but we don't know what type is contained in each of them. Only that any given F in the List is an F[Int], or a F[String], but that's it.

Why method defined like "cons[B >: A](v: B)" accepts argument of type which is not supertype of A?

I am studying variance in scala right now, and I think I have a good understanding of contravariance. For example given trait List[-A], I know that List[Int] is a supertype of List[AnyVal].
But say that I have the following trait:
trait List[+A] {
def cons(hd: A): List[A]
}
Why is cons parameter type wrong?
Why it is necessary to have def cons[B >: A](v: B): List[B] ?
For example:
val animal_list: List[Animal] = List(tiger, dog)
if we call:
animal_list.cons(tiger)
since Tiger <: Animal, doesn't cons ran into problem? Since B is Tiger and A is Animal and B >: A is not true.
Why is cons's parameter type wrong?
trait List[+A] {
def cons(hd: A): List[A]
}
Compiler give you error:
covariant type A occurs in contravariant position in type A of value hd
because method parameters count as contravariant positions, but A is covariant.
Let's imagine that this method declaration would compile. Then we could do:
class ListImpl[A] extends List[A] {
override def cons(hd: A): List[A] = ???
}
val strings: List[String] = new ListImpl[String]
val values: List[Any] = strings // OK, since List[String] <: List[Any] (in List[A], A is covariant)
values.cons(13) // OK(??), since values's static type is List[Any], so argument of cons should be Any, and 13 conforms to type Any
Is the last line above really OK? We are calling cons on values. values is the same as strings, and strings is object of type ListImpl[String]. So cons invocation in the last line is expecting String argument, but we are passing Int, because values's static type is List[Any] and Int conforms to Any. Something is definitely wrong here - which line is to blame? The answer is: cons method declaration. To fix this issue, we have to remove covariant type parameter A from contravariant position (in cons declaration). Alternatively we can make A non-covariant.
See also these questions: #1, #2.
... doesn't cons ran into problem?
trait List[+A] {
def cons[B >: A](v: B): List[B]
}
val animal_list: List[Animal] = List(tiger, dog) // We are assuming that List.apply and concrete implementation of List is somewhere defined.
No, animal_list.cons(tiger) invocation is type-correct.
I assume that Animal is common supertype of Dog and Tiger, and that dog and tiger are instances of Dog and Tiger respectively.
In animal_list.cons(tiger) invocation, both A and B type parameters are instantiated to Animal, so cons method takes form of:
def cons[Animal >: Animal](v: Animal): List[Animal]
Animal >: Animal constraint is satisfied because:
Supertype and subtype relationships are reflexive, which means a type
is both a supertype and a subtype of itself. [source]
The argument to cons is Tiger which conforms to type Animal, so the method invocation is type-correct.
Notice that if you force B to be instantiated to Tiger, like animal_list.cons[Tiger](tiger), then this invocation won't be type-correct, and you'll get compiler error.
See similar example here.

What does `T {}` do in Scala

Browsing Shapeless code, I came across this seemingly extraneous {} here and here:
trait Witness extends Serializable {
type T
val value: T {}
}
trait SingletonOps {
import record._
type T
def narrow: T {} = witness.value
}
I almost ignored it as a typo since it does nothing but apparently it does something. See this commit: https://github.com/milessabin/shapeless/commit/56a3de48094e691d56a937ccf461d808de391961
I have no idea what it does. Can someone explain?
Any type can be followed by a {} enclosed sequence of type and abstract non-type member definitions. This is known as a "refinement" and is used to provide additional precision over the base type that is being refined. In practice refinements are most commonly used to express constraints on abstract type members of the type being refined.
It's a little known fact that this sequence is allowed to be empty, and in the form that you can see in the shapeless source code, T {} is the type T with an empty refinement. Any empty refinement is ... empty ... so doesn't add any additional constraints to the refined type and hence the types T and T {} are equivalent. We can get the Scala compiler to verify that for us like so,
scala> implicitly[Int =:= Int {}]
res0: =:=[Int,Int] = <function1>
So why would I do such an apparently pointless thing in shapeless? It's because of the interaction between the presence of refinements and type inference. If you look in the relevant section of the Scala Language Specification you will see that the type inference algorithm attempts to avoid inferring singleton types in at least some circumstances. Here is an example of it doing just that,
scala> class Foo ; val foo = new Foo
defined class Foo
foo: Foo = Foo#8bd1b6a
scala> val f1 = foo
f1: Foo = Foo#8bd1b6a
scala> val f2: foo.type = foo
f2: foo.type = Foo#8bd1b6a
As you can see from the definition of f2 the Scala compiler knows that the value foo has the more precise type foo.type (ie. the singleton type of val foo), however, unless explicitly requested it won't infer that more precise type. Instead it infers the non-singleton (ie. widened) type Foo as you can see in the case of f1.
But in the case of Witness in shapeless I explicitly want the singleton type to be inferred for uses of the value member (the whole point of Witness is enable us to pass between the type and value levels via singleton types), so is there any way the Scala compiler can be persuaded to do that?
It turns out that an empty refinement does exactly that,
scala> def narrow[T <: AnyRef](t: T): t.type = t
narrow: [T <: AnyRef](t: T)t.type
scala> val s1 = narrow("foo") // Widened
s1: String = foo
scala> def narrow[T <: AnyRef](t: T): t.type {} = t // Note empty refinement
narrow: [T <: AnyRef](t: T)t.type
scala> val s2 = narrow("foo") // Not widened
s2: String("foo") = foo
As you can see in the above REPL transcript, in the first case s1 has been typed as the widened type String whereas s2 has been assigned the singleton type String("foo").
Is this mandated by the SLS? No, but it is consistent with it, and it makes some sort of sense. Much of Scala's type inference mechanics are implementation defined rather than spec'ed and this is probably one of the least surprising and problematic instances of that.

Shapeless: Generic.Aux

I'm trying to understand how Generic works (and TypeClass too). The github wiki is very sparse on examples and documentation. Is there a canonical blog post / documentation page describing Generic and TypeClass in detail?
In concrete, what is the difference between these two methods?:
def find1[T](implicit gen: Generic[T]): Generic[T] = gen
def find2[T](implicit gen: Generic[T]): Generic[T] { type Repr = gen.Repr } = gen
given
object Generic {
type Aux[T, Repr0] = Generic[T] { type Repr = Repr0 }
def apply[T](implicit gen: Generic[T]): Aux[T, gen.Repr] = gen
implicit def materialize[T, R]: Aux[T, R] = macro GenericMacros.materialize[T, R]
}
The issues involved in how Generic and TypeClass are implemented and what they do are different enough that they probably deserve separate questions, so I'll stick to Generic here.
Generic provides a mapping from case classes (and potentially similar types) to heterogeneous lists. Any case class has a unique hlist representation, but any given hlist corresponds to a very, very large number of potential case classes. For example, if we have the following case classes:
case class Foo(i: Int, s: String)
case class Bar(x: Int, y: String)
The hlist representation provided by Generic for both Foo and Bar is Int :: String :: HNil, which is also the representation for (Int, String) and any other case classes we could define with these two types in this order.
(As a side note, LabelledGeneric allows us to distinguish between Foo and Bar, since it includes the member names in the representation as type-level strings.)
We generally want to be able to specify the case class and let Shapeless figure out the (unique) generic representation, and making Repr a type member (instead of a type parameter) allows us to do this pretty cleanly. If the hlist representation type were a type parameter, then your find methods would have to have a Repr type parameter as well, which means that you wouldn't be able to specify only the T and have the Repr inferred.
Making Repr a type member makes sense only because the Repr is uniquely determined by the first type parameter. Imagine a type class like Iso[A, B] that witnesses that A and B are isomorphic. This type class is very similar to Generic, but A doesn't uniquely dermine B—we can't just ask "what is the type that's isomorphic to A?"—so it wouldn't be useful to make B a type member (although we could if we really wanted to—Iso[A] just wouldn't really mean anything).
The problem with type members is that they're easy to forget, and once they're gone, they're gone forever. The fact that the return type of your find1 isn't refined (i.e. doesn't include the type member) means that the Generic instance it returns is pretty much useless. For example, the static type of res0 here might as well be Any:
scala> import shapeless._
import shapeless._
scala> def find1[T](implicit gen: Generic[T]): Generic[T] = gen
find1: [T](implicit gen: shapeless.Generic[T])shapeless.Generic[T]
scala> case class Foo(i: Int, s: String)
defined class Foo
scala> find1[Foo].to(Foo(1, "ABC"))
res0: shapeless.Generic[Foo]#Repr = 1 :: ABC :: HNil
scala> res0.head
<console>:15: error: value head is not a member of shapeless.Generic[Foo]#Repr
res0.head
^
When Shapeless's Generic.materialize macro creates the Generic[Foo] instance we're asking for, it's statically typed as a Generic[Foo] { type Repr = Int :: String :: HNil }, so the gen argument that the compiler hands to find1 has all the static information we need. The problem is that we then explicitly up-cast that type to a plain old unrefined Generic[Foo], and from that point on the compiler doesn't know what the Repr is for that instance.
Scala's path-dependent types give us a way not to forget the refinement without adding another type parameter to our method. In your find2, the compiler statically knows the Repr for the incoming gen, so when you say that the return type is Generic[T] { type Repr = gen.Repr }, it will be able to keep track of that information:
scala> find2[Foo].to(Foo(1, "ABC"))
res2: shapeless.::[Int,shapeless.::[String,shapeless.HNil]] = 1 :: ABC :: HNil
scala> res2.head
res3: Int = 1
To sum up: Generic has a type parameter T that uniquely determines its type member Repr, Repr is a type member instead of a type parameter so that we don't have to include it in all of our type signatures, and path-dependent types make this possible, allowing us to keep track of Repr even though it's not in our type signatures.

Scala: type inference of generic and it's type argument

Lets assume I have instance of arbitrary one-argument generic class (I'll use List in demonstration but this can me any other generic).
I'd like to write generic function that can take instances (c) and be able to understand what generic class (A) and what type argument (B) produced the class (C) of that instance.
I've come up with something like this (body of the function is not really relevant but demonstrates that C conforms to A[B]):
def foo[C <: A[B], A[_], B](c: C) {
val x: A[B] = c
}
... and it compiles if you invoke it like this:
foo[List[Int], List, Int](List.empty[Int])
... but compilation fails with error if I omit explicit type arguments and rely on inference:
foo(List.empty[Int])
The error I get is:
Error:Error:line (125)inferred kinds of the type arguments (List[Int],List[Int],Nothing) do not conform to the expected kinds of the type parameters (type C,type A,type B).
List[Int]'s type parameters do not match type A's expected parameters:
class List has one type parameter, but type A has one
foo(List.empty[Int])
^
Error:Error:line (125)type mismatch;
found : List[Int]
required: C
foo(List.empty[Int])
^
As you can see Scala's type inference cannot infer the types correctly in this case (seems like it's guess is List[Int] instead of List for 2nd argument and Nothing instead of Int for 3rd).
I assume that type bounds for foo I've come up with are not precise/correct enough, so my question is how could I implement it, so Scala could infer arguments?
Note: if it helps, the assumption that all potential generics (As) inherit/conform some common ancestor can be made. For example, that A can be any collection inherited from Seq.
Note: the example described in this question is synthetic and is a distilled part of the bigger problem I am trying to solve.
This is a known limitation of current Scala's type inference for type constructors. Defining the type of formal parameter c as C only collects type constraints for C (and indirectly to A) but not B. In other words List[Int] <: C => { List[Int] <: C <: Any, C <: A[_] <: Any }.
There is a pretty simple translation that allows to guide type inference for such cases. In your case it is:
def foo[C[_] <: A[_], A[_], B](c: A[B]) { val x: A[B] = c }
Same semantics, just slightly different type signature.
In addition to hubertp answer, you can fix you function by removing obsolete (in you example) type variable C, e.g:
def foo[A[_], B](c: A[B]) {
val x: A[B] = c
}
In this case scalac would infer A[_] as List and B as Int.
Update (according to the comment).
If you need an evidence that C is subtype of A[B], then use implicit:
def foo[A[_], B, C](c: C)(implicit ev: C <:< A[B]) = {
val x: A[B] = c
}
Then it won't compile this:
scala> foo[List, String, List[Int]](List.empty[Int])
<console>:9: error: Cannot prove that List[Int] <:< List[String].
foo[List, String, List[Int]](List.empty[Int])