About generics in Java and Scala - scala

I am confused by the generic subtyping.
In Java, if type A is a subtype of B, generic type C<A> and C<B> are invariant. For instance, ArrayList<Base> is not a subtype of ArrayList<Derived>.
However, in Scala, generic type C<A> and C<B> are covariant if type A is a subtype of B. So what's the property of generic class in Scala has but not in Java?

Firstly note that variance is a property of generic type parameters, not of the parameterized types themselves.
Secondly: you are wrong about scala - type parameters are invariant by default. Let us investigate!
Java
Java has use-site variance annotations. That is, you can declare methods like this:
boolean addAll(Collection<? extends T> c);
There is, however, one form of "parameterized type" (I use the term loosely) in which the type parameters are covariant: Java Arrays! (This is actually insane because Arrays are mutable and hence it is easy to circumvent the type system). Consider the following:
public static void subvert(Object[] arr) { arr[0] = "Oh Noes!"; }
And then:
Integer[] arr = new Integer[1];
subvert(arr); //this call is allowed as arrays are covariant
Integer i = arr[0];
A good interview question this one: what happens?
Scala
In Scala, you have declaration-site variance. That is, the variance of a type parameter is declared alongside the parameter (using the annotations + and -):
trait Function1[-I, +O]
This says that the trait Function1 has two type parameters, I and O which are contra- and co-variant respectively. If no +/- annotation is declared, then the type parameter is invariant. For example, Set is invariant in its type parameter:
scala> def foo(set: Set[Any]) = ()
foo: (set: Set[Any])Unit
scala> Set(1)
res4: scala.collection.immutable.Set[Int] = Set(1)
scala> foo(res4)
<console>:10: error: type mismatch;
found : scala.collection.immutable.Set[Int]
required: Set[Any]
Note: Int <: Any, but trait Set is invariant in type A.
You may wish to investigate a wildcard type such as `_ <: Any`. (SLS 3.2.10)
foo(res4)
^
List is however, declared as being covariant:
scala> def bar(list: List[Any]) = ()
bar: (list: List[Any])Unit
scala> List(1)
res6: List[Int] = List(1)
scala> bar(res6)
Why not ask the compiler?
Another way of demonstrating this is to ask the compiler directly for subtype-evidence:
scala> class Cov[+A]
defined class Cov
scala> implicitly[Cov[Int] <:< Cov[Any]]
res8: <:<[Cov[Int],Cov[Any]] = <function1>
But with an invariant type parameter
scala> class Inv[A]
defined class Inv
scala> implicitly[Inv[Int] <:< Inv[Any]]
<console>:9: error: Cannot prove that Inv[Int] <:< Inv[Any].
implicitly[Inv[Int] <:< Inv[Any]]
^
Lastly, contravariance:
scala> class Con[-A]
defined class Con
scala> implicitly[Con[Any] <:< Con[Int]]
res10: <:<[Con[Any],Con[Int]] = <function1>
See also identifier <:<

Related

Type alias parameter bounds not enforced in all cases

TL;DR: It appears that type parameters of type aliases (e.g. type T[X<:Serializable]) do not enforce their constraints when referenced as variables, parameters and perhaps other cases. Case classes, however, do enforce the bounds correctly for their parameters.
Consider a type alias designed to represent a subset of generic type. For example, let us say I want a type for lists of Serializable things:
scala> type SerializableList[T <: Serializable] = List[T]
defined type alias SerializableList
Now say that I want a case class with a parameter of these things:
scala> case class NetworkDataCC(things: SerializableList[_])
<console>:9: error: type arguments [_$1] do not conform to type SerializableList's type parameter bounds [T <: Serializable]
case class NetworkDataCC(things: SerializableList[_])
Well, that doesn't work. Scala (annoyingly) does not carry the parameter bounds with the type, but it's easy to fix:
scala> case class NetworkDataCC(things: SerializableList[_ <: Serializable])
defined class NetworkDataCC
Alright. Looks good. Now, what if I want just a regular class with those things, but I again forget to explicitly declare the type bounds. I expect an error:
scala> class NetworkData(val things: SerializableList[_])
defined class NetworkData
Oh, wait. No error... huh.
So, now I can do this?
scala> new NetworkData(List(1))
res3: NetworkData = NetworkData#e344ad3
Well, that seems quite broken. The case class, works fine of course (because the restrictions were declared):
scala> NetworkDataCC(List(1))
<console>:11: error: type mismatch;
found : Int(1)
required: Serializable
NetworkDataCC(List(1))
In my project, I am making use of reflection to generate some metadata about my classes. The metadata for the non-case-class shows a lack of bounds on things:
scala> classOf[NetworkData].getDeclaredFields()(0).getGenericType
res0: java.lang.reflect.Type = scala.collection.immutable.List<?>
Whereas the case class is correct:
scala> classOf[NetworkDataCC].getDeclaredFields()(0).getGenericType
res1: java.lang.reflect.Type = scala.collection.immutable.List<? extends scala.Serializable>
I wasn't able to find any bugs in the scala compiler bug tracker for this. Am I misunderstanding how these bounds should be used?
Scala's underscore is not equivalent to SerializableList[X forSome {type X}] by default:
scala> def aa(a: SerializableList[_]) = a
aa: (a: SerializableList[_])List[Any]
scala> def aa(a: SerializableList[X forSome {type X}]) = a
<console>:11: error: type arguments [X forSome { type X }] do not conform to type SerializableList's type parameter bounds [T <: Serializable]
def aa(a: SerializableList[X forSome {type X}]) = a
^
scala> class NetworkDataCC(things: SerializableList[X forSome {type X}])
<console>:11: error: type arguments [X forSome { type X }] do not conform to typ
e SerializableList's type parameter bounds [T <: Serializable]
class NetworkDataCC(things: SerializableList[X forSome {type X}])
It is equivalent to
scala> def aa(a: SerializableList[X] forSome {type X} ) = a
aa: (a: SerializableList[_])List[Any]
So such "unbounded" behavior is fine. Check out this answer: https://stackoverflow.com/a/15204140/1809978
Case classes seem to have additional type restrictions (due to this bug, which affects unapply method, automatically generated for case class).
If you want to have "unbounded" existential type in case class, just specify higher-order type explicitly:
scala> case class NetworkDataCC[SerializableList[_]](things: SerializableList[_])
warning: there were 2 feature warning(s); re-run with -feature for details
defined class NetworkDataCC
scala> NetworkDataCC(List(1))
res5: NetworkDataCC[List] = NetworkDataCC(List(1))
Or even:
scala> type SLL = SerializableList[_]
defined type alias SLL
scala> case class NetworkDataDD(things: SLL)
defined class NetworkDataDD
So this is definitely a bug regarding that you can workaround it with semantically equivalent type alias (see SI-8997)

Understanding List[+A] for Covariance

Looking at the source for List.scala:
sealed abstract class List[+A] extends ...
...
def isEmpty: Boolean
def head: A
def tail: List[A]
List[+A] is covariant based on the +A. Does this mean that, it's possible to create a List[T] where T can be the type itself, or any of its sub-classes?
example:
scala> trait Kid
defined trait Kid
scala> case class Boy(name: String) extends Kid
defined class Boy
scala> case class Girl(name: String) extends Kid
defined class Girl
scala> val list: List[Kid] = List(Boy("kevin"), Girl("sally"))
list: List[Kid] = List(Boy(kevin), Girl(sally))
Observe that head and tail's types are A and List[A], respectively. Once we've defined List[+A], then head and tail's A is also covariant?
I've read this StackOverflow answer 3 or 4 times, but I don't understand yet.
Your example does not relate to variance. Moreover, head and tail have nothing to do with variance too.
scala> val list: List[Kid] = List(Boy("kevin"), Girl("sally"))
list: List[Kid] = List(Boy(kevin), Girl(sally))
This would work even if List weren't covariant, because Scala will automatically deduce common supertype of Boy and Girl, that is, Kid, and type of the expression on the right side will be List[Kid], exactly what you require on the left side.
The following, however, doesn't work because java.util.List is not covariant (it is invariant since it is Java type):
scala> import java.util.{List => JList, Arrays}
import java.util.{List=>JList, Arrays}
scala> trait Kid
defined trait Kid
scala> case class Boy(name: String) extends Kid
defined class Boy
scala> val list1 = Arrays.asList(Boy("kevin"), Boy("bob"))
list1: java.util.List[Boy] = [Boy(kevin), Boy(bob)]
scala> val list2: JList[Kid] = list1
<console>:12: error: type mismatch;
found : java.util.List[Boy]
required: java.util.List[Kid]
Note: Boy <: Kid, but Java-defined trait List is invariant in type E.
You may wish to investigate a wildcard type such as `_ <: Kid`. (SLS 3.2.10)
val list2: JList[Kid] = list1
^
Arrays.asList method has signature like this:
def asList[T](args: T*): java.util.List[T]
As java.util.List[T] is invariant, it is impossible to assign JList[Boy] (list1) to JList[Kid] (list2). And there is a reason: if you could, then because JList is mutable, you could also add anything extending Kid (not only Boy) into the same list, breaking type safety.
On the other hand, scala.List will work in exactly the same situation:
scala> val list1 = List(Boy("kevin"), Boy("bob"))
list1: List[Boy] = List(Boy(kevin), Boy(bob))
scala> val list2: List[Kid] = list1
list2: List[Kid] = List(Boy(kevin), Boy(bob))
That is because scala.List is covariant in its type parameter. Note that covariant List type works as if List[Boy] were subtype of List[Kid], very similar to the case when you can assign everything to a variable of type Any because every other type is a subtype of Any. This is very helpful analogy.
Contravariance works in a very similar way, but in other direction. Consider this trait:
trait Predicate[-T] {
def apply(obj: T): Boolean
}
object Predicate {
// convenience method to convert functions to predicates
def apply[T](f: (T) => Boolean) = new Predicate[T] {
def apply(obj: T) = f(obj)
}
}
Note the - before T parameter: it is a contravariance annotation, that is, Predicate[T] is defined to be contravariant in its only type parameter.
Recall that for covariant list List[Boy] was a subtype of List[Kid]. Well, for contravariant predicate it works in the opposite way: Predicate[Kid] is a subtype of Predicate[Boy], so you can assign a value of type Predicate[Kid] to a variable of type Predicate[Boy]:
scala> val pred1: Predicate[Kid] = Predicate { kid => kid.hashCode % 2 == 0 }
pred1: Predicate[Kid] = Predicate$$anon$1#3bccdcdd
scala> val pred2: Predicate[Boy] = pred1
pred2: Predicate[Boy] = Predicate$$anon$1#3bccdcdd
If Predicate[T] weren't contravariant, we wouldn't be able to assign pred1 to pred2, though it is completely legitimate and safe: obviously, predicates defined on supertypes can easily work on subtypes.
In short, variance affects type compatibility between parameterized types. List is covariant, so you can assign a value of type List[Boy] to a variable of type List[Kid] (in fact, for any T extending S, you can assign a value of type List[T] to a variable of type List[S]).
On the other hand, because, Predicate is contravariant, you can assign Predicate[Kid] to Predicate[Boy] (that is, for any T extending S, you can assign a value of type Predicate[S] to a variable of type Predicate[T]).
If a type is invariant in its type parameter, neither of the above can be done (as is demonstrated by JList).
Note the correspondence between parameterized types and their parameters:
T <: S ===> List [T] <: List [S] (covariance)
T <: S ===> Predicate[S] <: Predicate[T] (contravariance)
This is the reason why the first effect is called *co*variance (T <: S on the left, and
..T.. <: ..S.. on the right), and the second is *contra*variance (T <: S on the left, but ..S.. <: ..T.. on the right).
Whether to make your own parameterized types covariant or contravariant or invariant depends on your class responsibilities. If it may only return values of generic type, then it makes sense to use covariance. List[T], for example, only contains methods which return T, never accept T as a parameter, so it is safe to make it covariant in order to increase expressiveness. Such parameterized types can be called producers.
If your class only accepts values of the generic type as a parameter, not returns them (exactly like Predicate above which has single method def apply(obj: T): Boolean), then you can safely make it contravariant. Such parameterized types can be called consumers
If your class both accepts and returns values of the generic type, i.e. it is both a producer and a consumer, then you have no choice but to leave the class invariant in this generic type parameter.
This idiom is usually called "PECS" ("Producer extends, Consumer super") because variance annotations are written extends and super in Java.

Higher kinded existentials without covariant annotation

When trying to use higher kinded existentials in Scala I run into the following problem:
trait A[H[_]]
trait Test {
val l: A[List]
// [error] type mismatch;
// [error] found : A[List]
// [error] required: A[_[_] <: Any]
// [error] Note: List <: Any, but trait A is invariant in type H.
// [error] You may wish to define H as +H instead. (SLS 4.5)
val x: A[B] forSome { type B[_] } = l
}
Adding a covariant annotation to H as the compiler suggests works. Is there a way to work around this if I don't want H to be covariant?
A slight variation of the example gives a more helpful error message:
scala> l: (A[B] forSome { type B[_] })
<console>:10: error: type mismatch;
found : A[List]
required: A[_[_] <: Any]
Note: List <: Any, but trait A is invariant in type H.
You may wish to define H as +H instead. (SLS 4.5)
l: (A[B] forSome { type B[_] })
^
<console>:10: error: can't existentially abstract over parameterized type B
l: (A[B] forSome { type B[_] })
^
Looking for this error brings us to a TODO in the compiler.
Since existential types are going to disappear, per Odersky's email, I don't think this limitation will be fixed. However, Martin Odersky's email also reminds us that existential types are equivalent to abstract types. Hence, the above example can be encoded as follows:
scala> trait A { type H[_] }
defined trait A
scala> val l: A {type H[X] = List[X]} = null
l: A{type H[X] = List[X]} = null
scala> l: A
res0: A = null
Type application is syntactically ugly, but turning a value into an existential becomes trivial (also to implement in a compiler, which is part of Odersky's point).
What's useful to encode existentials is that type members don't have to be instantiated ever. So, to encode A[_] we can write:
scala> class A { type T }
defined class A
scala> new A
res1: A = A#3a7049a6
What's confusing here is that this does not work for objects:
scala> object B { type T }
<console>:8: error: only classes can have declared but undefined members
object B { type T }
^
But I recently got that accepted as a bug — see here for a pull request clarifying the spec (approved by Adriaan Moors), and here for my bug report and one-line fix to the compiler (still waiting for review).

Why won't Scala use implicit conversion here?

I'm trying to call this set method documented here, in the Java library jOOQ, with signature:
<T> ... set(Field<T> field, T value)
This Scala line is a problem:
.set(table.MODIFIED_BY, userId)
MODIFIED_BY is a Field<Integer> representing the table column. userId is Int. Predef has an implicit conversion from Int to Integer, so why doesn't it use it? I get this:
type mismatch; found: org.jooq.TableField[gen.tables.records.DocRecord,Integer]
required: org.jooq.Field[Any]
Note: Integer <: Any
(and org.jooq.TableField[gen.tables.records.DocRecord,Integer] <:
org.jooq.Field[Integer]), but Java-defined trait Field is invariant in type T.
You may wish to investigate a wildcard type such as `_ <: Any`. (SLS 3.2.10)
Update - About Vinicius's Example
Rather than try to explain this in comments, here is a demonstration that there is no implicit conversion being called when you use a type with covariant parameter, like List[+T]. Let's say I put this code in a file, compile, and run it...
case class Foo(str: String)
object StackOver1 extends App {
implicit def str2Foo(s: String): Foo = {
println("In str2Foo.")
new Foo(s)
}
def test[T](xs: List[T], x: T): List[T] = {
println("test " + x.getClass)
xs
}
val foo1 = new Foo("foo1")
test(List(foo1), "abc")
}
You'll see that it calls test, but never the implicit conversion from String "abc" to Foo. Instead it's picking a T for test[T] that is a common base class between String and Foo. When you use Int and Integer it picks Any, but it's confusing because the runtime representation of the Int in the list is Integer. So it looks like it used the implicit conversion, but it didn't. You can verify by opening a Scala prompt...
scala> :type StackOver1.test(List(new java.lang.Integer(1)), 2)
List[Any]
I don't know anything aboutjOOQ, but I think the issue is that Scala does not understand java generics very well. Try:
scala> def test[T](a : java.util.ArrayList[T], b: T) = { println(a,b) }
scala> val a = new java.util.ArrayList[Integer]()
scala> val b = 12
scala> test(a,b)
<console>:11: error: type mismatch;
found : java.util.ArrayList[Integer]
required: java.util.ArrayList[Any]
Note: Integer <: Any, but Java-defined class ArrayList is invariant in type E.
You may wish to investigate a wildcard type such as `_ <: Any`. (SLS 3.2.10)
test(a,b)
Sounds familiar??
And to fix, just inform the type T to call the method: test[Integer](a,b) works fine.
EDIT:
There a few things involved here:
Erasure -> When compiled the type of the generic will disappear by erasure. The compiler will use Object which Scala, will treat as Any. However a ArrayList[Integer] is not an ArrayList[Any], even though Integer is any. The same way that TableField[gen.tables.records.DocRecord,Integer] is not a Field[Any].
Type inference mechanism -> it will figure out what type T should be and to do that it will use the intersection dominator of the types passed (in our case the first common ancestor). Page 36 of Scala Language Spec, which in our examples above will lead use to Any.
Implicit conversion -> it is the last step and would be called if there was some type to be converted to another one, but since the type of the arguments were determined to be the first common ancestor, there is no need to convert and we will never have a implicit conversion if we don't force the type T.
A example to show how the common ancestor is used to determine T:
scala> def test[T](a: T, b: T): T = a
scala> class Foo
scala> class Boo extends Foo
scala> test(new Boo,new Foo)
res2: Foo = Boo#139c2a6
scala> test(new Boo,new Boo)
res3: Boo = Boo#141c803
scala> class Coo extends Foo
scala> test(new Boo,new Coo)
res4: Foo = Boo#aafc83
scala> test(new Boo,"qsasad")
res5: Object = Boo#16989d8
Summing up, the implicit method does not get called, because the type inference mechanism, determines the types before getting the argument and since it uses the common ancestor, there is no need for a implicit conversion.
Your code produces an error due to erasure mechanism which disappear with the type information that would be important to determine the correct type of the argument.
#RobN, thanks for questioning my answer, I learned a lot with the process.

Context bounds shortcut with higher kinded-types

Is it possible to use the context bounds syntax shortcut with higher kinded-types?
trait One { def test[W : ClassManifest]: Unit } // first-order ok
trait Two { def test[W[_]: ClassManifest]: Unit } // not possible??
trait Six { def test[W[_]](implicit m: ClassManifest[W[_]]): Unit } // hmm...
Yes, it is, but your context bound type must have a higher kinded type parameter (which ClassManifest doesn't).
scala> trait HKTypeClass[CC[_]]
defined trait HKTypeClass
scala> implicit def listHKTC = new HKTypeClass[List] {}
listHKTC: java.lang.Object with HKTypeClass[List]
scala> def frob[CC[_] : HKTypeClass] = implicitly[HKTypeClass[CC]]
frob: [CC[_]](implicit evidence$1: HKTypeClass[CC])HKTypeClass[CC]
scala> frob[List]
res0: HKTypeClass[List] = $anon$1#13e02ed
Update
It's possible to use a type alias to allow a higher-kinded type parameter to be bounded by a first-order context bound type. We use the type alias as a type-level function to make a higher-kinded type out of the first-order type. For ClassManifest it could go like this,
scala> type HKClassManifest[CC[_]] = ClassManifest[CC[_]]
defined type alias HKClassManifest
scala> def frob[CC[_] : HKClassManifest] = implicitly[HKClassManifest[CC]]
test: [CC[_]](implicit evidence$1: HKClassManifest[CC])HKClassManifest[CC]
scala> frob[List]
res1: HKClassManifest[List] = scala.collection.immutable.List[Any]
Note that on the right hand side of the type alias CC[_] is a first-order type ... the underscore here is the wildcard. Consequently it can be used as the type argument for ClassManifest.
Update
For completeness I should note that the type alias can be inlined using a type lambda,
scala> def frob[CC[_] : ({ type λ[X[_]] = ClassManifest[X[_]] })#λ] = implicitly[ClassManifest[CC[_]]]
frob: [CC[_]](implicit evidence$1: scala.reflect.ClassManifest[CC[_]])scala.reflect.ClassManifest[CC[_]]
scala> frob[List]
res0: scala.reflect.ClassManifest[List[_]] = scala.collection.immutable.List[Any]
Note that implicitly[ClassManifest[List[_]]] is short for implicitly[ClassManifest[List[T] forSome {type T}]].
That's why it works: ClassManifest expects a proper type argument, and List[T] forSome {type T} is a proper type, but List is a type constructor. (Please see What is a higher kinded type in Scala? for a definition of "proper" etc.)
To make both ClassManifest[List[String]] and ClassManifest[List] work, we'd need to overload ClassManifest somehow with versions that take type parameters of varying kinds, something like:
class ClassManifest[T] // proper type
class ClassManifest[T[_]] // type constructor with one type parameter
class ClassManifest[T[_, _]] // type constructor with two type parameters
// ... ad nauseam
(On an academic note, the "proper" way to do this, would be to allow abstracting over kinds:
class ClassManifest[T : K][K]
implicitly[ClassManifest[String]] // --> compiler infers ClassManifest[String][*]
implicitly[ClassManifest[List]] // --> compiler infers ClassManifest[List][* -> *]
)