I'm trying to understand how Generic works (and TypeClass too). The github wiki is very sparse on examples and documentation. Is there a canonical blog post / documentation page describing Generic and TypeClass in detail?
In concrete, what is the difference between these two methods?:
def find1[T](implicit gen: Generic[T]): Generic[T] = gen
def find2[T](implicit gen: Generic[T]): Generic[T] { type Repr = gen.Repr } = gen
given
object Generic {
type Aux[T, Repr0] = Generic[T] { type Repr = Repr0 }
def apply[T](implicit gen: Generic[T]): Aux[T, gen.Repr] = gen
implicit def materialize[T, R]: Aux[T, R] = macro GenericMacros.materialize[T, R]
}
The issues involved in how Generic and TypeClass are implemented and what they do are different enough that they probably deserve separate questions, so I'll stick to Generic here.
Generic provides a mapping from case classes (and potentially similar types) to heterogeneous lists. Any case class has a unique hlist representation, but any given hlist corresponds to a very, very large number of potential case classes. For example, if we have the following case classes:
case class Foo(i: Int, s: String)
case class Bar(x: Int, y: String)
The hlist representation provided by Generic for both Foo and Bar is Int :: String :: HNil, which is also the representation for (Int, String) and any other case classes we could define with these two types in this order.
(As a side note, LabelledGeneric allows us to distinguish between Foo and Bar, since it includes the member names in the representation as type-level strings.)
We generally want to be able to specify the case class and let Shapeless figure out the (unique) generic representation, and making Repr a type member (instead of a type parameter) allows us to do this pretty cleanly. If the hlist representation type were a type parameter, then your find methods would have to have a Repr type parameter as well, which means that you wouldn't be able to specify only the T and have the Repr inferred.
Making Repr a type member makes sense only because the Repr is uniquely determined by the first type parameter. Imagine a type class like Iso[A, B] that witnesses that A and B are isomorphic. This type class is very similar to Generic, but A doesn't uniquely dermine B—we can't just ask "what is the type that's isomorphic to A?"—so it wouldn't be useful to make B a type member (although we could if we really wanted to—Iso[A] just wouldn't really mean anything).
The problem with type members is that they're easy to forget, and once they're gone, they're gone forever. The fact that the return type of your find1 isn't refined (i.e. doesn't include the type member) means that the Generic instance it returns is pretty much useless. For example, the static type of res0 here might as well be Any:
scala> import shapeless._
import shapeless._
scala> def find1[T](implicit gen: Generic[T]): Generic[T] = gen
find1: [T](implicit gen: shapeless.Generic[T])shapeless.Generic[T]
scala> case class Foo(i: Int, s: String)
defined class Foo
scala> find1[Foo].to(Foo(1, "ABC"))
res0: shapeless.Generic[Foo]#Repr = 1 :: ABC :: HNil
scala> res0.head
<console>:15: error: value head is not a member of shapeless.Generic[Foo]#Repr
res0.head
^
When Shapeless's Generic.materialize macro creates the Generic[Foo] instance we're asking for, it's statically typed as a Generic[Foo] { type Repr = Int :: String :: HNil }, so the gen argument that the compiler hands to find1 has all the static information we need. The problem is that we then explicitly up-cast that type to a plain old unrefined Generic[Foo], and from that point on the compiler doesn't know what the Repr is for that instance.
Scala's path-dependent types give us a way not to forget the refinement without adding another type parameter to our method. In your find2, the compiler statically knows the Repr for the incoming gen, so when you say that the return type is Generic[T] { type Repr = gen.Repr }, it will be able to keep track of that information:
scala> find2[Foo].to(Foo(1, "ABC"))
res2: shapeless.::[Int,shapeless.::[String,shapeless.HNil]] = 1 :: ABC :: HNil
scala> res2.head
res3: Int = 1
To sum up: Generic has a type parameter T that uniquely determines its type member Repr, Repr is a type member instead of a type parameter so that we don't have to include it in all of our type signatures, and path-dependent types make this possible, allowing us to keep track of Repr even though it's not in our type signatures.
Related
Consider the following example:
sealed trait ST
object ST{
case class ST1() extends ST
case class ST2() extends ST
}
trait TypeClass[A]{
def doSome(a: A): Unit
}
case class Test[T](t: T)
implicit val tp: TypeClass[Test[_ <: ST]] = ??? //the implicit
def foo[A: TypeClass](a: A) = implicitly[TypeClass[A]].doSome(a)
val v: Test[_ <: ST] = ???
foo(v) //error: implicit not found
SCASTIE
As can be seen the required implicit is in scope while it is not recognized by the compile.
Why does that happen and is there a workaround to call foo?
If you change foo(v) to foo(v)(tp) it'll explain (a little bit) better why it doesn't want to use tp.
In a nutshell, def foo[A : TypeClass] wants an implicit of type TypeClass[A].
When you do foo(v), A becomes Test[_ <: ST] which means "Test of some specific but unknown subtype of ST". So, foo wants an implicit for that specific type.
But tp is not that. It is a "TypeClass for Test or any subclass of ST" (apparently _ means slightly different things in these two contexts, because v is a concrete instance, that must have a specific type).
Long story short, Test[_ <: ST] is not the actual type of v, but a supertype of its type. So, to make it work, you just need to make the TypeClass contravariant (TypeClass[-A]) - that'll make foo accept tp as the implicit, because its type will be a subtype of what it expects.
Existential types are not inferred during implicit resolution so f(v) fails because it is looking for an implicit value with non-inferred type
TypeClass[Test[_$4]]]
|
existential not inferred
however whey you explicitly provide the type variable instance foo[Test[_ <: ST]](v) then implicit resolution should work because we are past inference stage.
It works in Scala 3 probably because internally it rewrites existential types to refined types.
This seems to be a classic question for developers used to Scala type-level programming, but I couldn't find (or I don't know how to search for) a solution or pattern for this. Suppose I have a class like this:
abstract class TypedTest[Args <: HList](implicit val optMapper: Mapped[Args, Option]) {
type OptArgs = optMapper.Out
def options: OptArgs // to be implemented by subclasses
}
I want users of this class to instantiate it with an HList type parameter (Args) and the class provides a method to retrieve an HList instance containing an instance of each specified type inside an Option (OptArgs). I'm using shapeless Mapped type class for this. Note that I don't have an instance of Args to provide at instantiation time.
This code doesn't work, as the compiler doesn't infer the concrete type of OptArgs and even an obviously correct implementation such as def options = HNil yields a compilation error. The same code using the Aux pattern:
abstract class TypedTest[Args <: HList, OptArgs <: HList](implicit val optMapper: Mapped.Aux[Args, Option, OptArgs]) {
def options: OptArgs
}
This forces me to specify both lists at instantiation time, which makes the external API needlessly verbose. Is there an workaround for this?
This is my understanding, but I'm not 100% sure and will be happy to stand corrected.
The type member TypedTest.OptArgs is not an abstract type but a type alias. It is the same type for all subclasses of TypedTest – an alias for Mapped[Args, Option].Out, which is an abstract type and cannot be unified with any type but itself. When a subclass is created, the type member OptArgs is not overridden.
It becomes more clear when using Mapped.Aux with an existential type for Out0, which is IIUC more or less equivalent to the above:
abstract class TypedTest[Args <: HList](
implicit val optMapper: Mapped.Aux[Args, Option, T] forSome { type T }) {
type OptArgs = optMapper.Out
def options: OptArgs // to be implemented by subclasses
}
val intTest = new TypedTest[Int :: HNil] {
def options = Some(1) :: HNil
}
Error:(18, 29) type mismatch;
found : shapeless.::[Some[Int],shapeless.HNil]
required: this.OptArgs
(which expands to) T
def options = Some(1) :: HNil
Unfortunately I'm not aware of any possible solution, except adding Out as a type parameter or defining OptArgs as an abstract type and specifying it explicitly in each subclass.
Browsing Shapeless code, I came across this seemingly extraneous {} here and here:
trait Witness extends Serializable {
type T
val value: T {}
}
trait SingletonOps {
import record._
type T
def narrow: T {} = witness.value
}
I almost ignored it as a typo since it does nothing but apparently it does something. See this commit: https://github.com/milessabin/shapeless/commit/56a3de48094e691d56a937ccf461d808de391961
I have no idea what it does. Can someone explain?
Any type can be followed by a {} enclosed sequence of type and abstract non-type member definitions. This is known as a "refinement" and is used to provide additional precision over the base type that is being refined. In practice refinements are most commonly used to express constraints on abstract type members of the type being refined.
It's a little known fact that this sequence is allowed to be empty, and in the form that you can see in the shapeless source code, T {} is the type T with an empty refinement. Any empty refinement is ... empty ... so doesn't add any additional constraints to the refined type and hence the types T and T {} are equivalent. We can get the Scala compiler to verify that for us like so,
scala> implicitly[Int =:= Int {}]
res0: =:=[Int,Int] = <function1>
So why would I do such an apparently pointless thing in shapeless? It's because of the interaction between the presence of refinements and type inference. If you look in the relevant section of the Scala Language Specification you will see that the type inference algorithm attempts to avoid inferring singleton types in at least some circumstances. Here is an example of it doing just that,
scala> class Foo ; val foo = new Foo
defined class Foo
foo: Foo = Foo#8bd1b6a
scala> val f1 = foo
f1: Foo = Foo#8bd1b6a
scala> val f2: foo.type = foo
f2: foo.type = Foo#8bd1b6a
As you can see from the definition of f2 the Scala compiler knows that the value foo has the more precise type foo.type (ie. the singleton type of val foo), however, unless explicitly requested it won't infer that more precise type. Instead it infers the non-singleton (ie. widened) type Foo as you can see in the case of f1.
But in the case of Witness in shapeless I explicitly want the singleton type to be inferred for uses of the value member (the whole point of Witness is enable us to pass between the type and value levels via singleton types), so is there any way the Scala compiler can be persuaded to do that?
It turns out that an empty refinement does exactly that,
scala> def narrow[T <: AnyRef](t: T): t.type = t
narrow: [T <: AnyRef](t: T)t.type
scala> val s1 = narrow("foo") // Widened
s1: String = foo
scala> def narrow[T <: AnyRef](t: T): t.type {} = t // Note empty refinement
narrow: [T <: AnyRef](t: T)t.type
scala> val s2 = narrow("foo") // Not widened
s2: String("foo") = foo
As you can see in the above REPL transcript, in the first case s1 has been typed as the widened type String whereas s2 has been assigned the singleton type String("foo").
Is this mandated by the SLS? No, but it is consistent with it, and it makes some sort of sense. Much of Scala's type inference mechanics are implementation defined rather than spec'ed and this is probably one of the least surprising and problematic instances of that.
Looking at the source for List.scala:
sealed abstract class List[+A] extends ...
...
def isEmpty: Boolean
def head: A
def tail: List[A]
List[+A] is covariant based on the +A. Does this mean that, it's possible to create a List[T] where T can be the type itself, or any of its sub-classes?
example:
scala> trait Kid
defined trait Kid
scala> case class Boy(name: String) extends Kid
defined class Boy
scala> case class Girl(name: String) extends Kid
defined class Girl
scala> val list: List[Kid] = List(Boy("kevin"), Girl("sally"))
list: List[Kid] = List(Boy(kevin), Girl(sally))
Observe that head and tail's types are A and List[A], respectively. Once we've defined List[+A], then head and tail's A is also covariant?
I've read this StackOverflow answer 3 or 4 times, but I don't understand yet.
Your example does not relate to variance. Moreover, head and tail have nothing to do with variance too.
scala> val list: List[Kid] = List(Boy("kevin"), Girl("sally"))
list: List[Kid] = List(Boy(kevin), Girl(sally))
This would work even if List weren't covariant, because Scala will automatically deduce common supertype of Boy and Girl, that is, Kid, and type of the expression on the right side will be List[Kid], exactly what you require on the left side.
The following, however, doesn't work because java.util.List is not covariant (it is invariant since it is Java type):
scala> import java.util.{List => JList, Arrays}
import java.util.{List=>JList, Arrays}
scala> trait Kid
defined trait Kid
scala> case class Boy(name: String) extends Kid
defined class Boy
scala> val list1 = Arrays.asList(Boy("kevin"), Boy("bob"))
list1: java.util.List[Boy] = [Boy(kevin), Boy(bob)]
scala> val list2: JList[Kid] = list1
<console>:12: error: type mismatch;
found : java.util.List[Boy]
required: java.util.List[Kid]
Note: Boy <: Kid, but Java-defined trait List is invariant in type E.
You may wish to investigate a wildcard type such as `_ <: Kid`. (SLS 3.2.10)
val list2: JList[Kid] = list1
^
Arrays.asList method has signature like this:
def asList[T](args: T*): java.util.List[T]
As java.util.List[T] is invariant, it is impossible to assign JList[Boy] (list1) to JList[Kid] (list2). And there is a reason: if you could, then because JList is mutable, you could also add anything extending Kid (not only Boy) into the same list, breaking type safety.
On the other hand, scala.List will work in exactly the same situation:
scala> val list1 = List(Boy("kevin"), Boy("bob"))
list1: List[Boy] = List(Boy(kevin), Boy(bob))
scala> val list2: List[Kid] = list1
list2: List[Kid] = List(Boy(kevin), Boy(bob))
That is because scala.List is covariant in its type parameter. Note that covariant List type works as if List[Boy] were subtype of List[Kid], very similar to the case when you can assign everything to a variable of type Any because every other type is a subtype of Any. This is very helpful analogy.
Contravariance works in a very similar way, but in other direction. Consider this trait:
trait Predicate[-T] {
def apply(obj: T): Boolean
}
object Predicate {
// convenience method to convert functions to predicates
def apply[T](f: (T) => Boolean) = new Predicate[T] {
def apply(obj: T) = f(obj)
}
}
Note the - before T parameter: it is a contravariance annotation, that is, Predicate[T] is defined to be contravariant in its only type parameter.
Recall that for covariant list List[Boy] was a subtype of List[Kid]. Well, for contravariant predicate it works in the opposite way: Predicate[Kid] is a subtype of Predicate[Boy], so you can assign a value of type Predicate[Kid] to a variable of type Predicate[Boy]:
scala> val pred1: Predicate[Kid] = Predicate { kid => kid.hashCode % 2 == 0 }
pred1: Predicate[Kid] = Predicate$$anon$1#3bccdcdd
scala> val pred2: Predicate[Boy] = pred1
pred2: Predicate[Boy] = Predicate$$anon$1#3bccdcdd
If Predicate[T] weren't contravariant, we wouldn't be able to assign pred1 to pred2, though it is completely legitimate and safe: obviously, predicates defined on supertypes can easily work on subtypes.
In short, variance affects type compatibility between parameterized types. List is covariant, so you can assign a value of type List[Boy] to a variable of type List[Kid] (in fact, for any T extending S, you can assign a value of type List[T] to a variable of type List[S]).
On the other hand, because, Predicate is contravariant, you can assign Predicate[Kid] to Predicate[Boy] (that is, for any T extending S, you can assign a value of type Predicate[S] to a variable of type Predicate[T]).
If a type is invariant in its type parameter, neither of the above can be done (as is demonstrated by JList).
Note the correspondence between parameterized types and their parameters:
T <: S ===> List [T] <: List [S] (covariance)
T <: S ===> Predicate[S] <: Predicate[T] (contravariance)
This is the reason why the first effect is called *co*variance (T <: S on the left, and
..T.. <: ..S.. on the right), and the second is *contra*variance (T <: S on the left, but ..S.. <: ..T.. on the right).
Whether to make your own parameterized types covariant or contravariant or invariant depends on your class responsibilities. If it may only return values of generic type, then it makes sense to use covariance. List[T], for example, only contains methods which return T, never accept T as a parameter, so it is safe to make it covariant in order to increase expressiveness. Such parameterized types can be called producers.
If your class only accepts values of the generic type as a parameter, not returns them (exactly like Predicate above which has single method def apply(obj: T): Boolean), then you can safely make it contravariant. Such parameterized types can be called consumers
If your class both accepts and returns values of the generic type, i.e. it is both a producer and a consumer, then you have no choice but to leave the class invariant in this generic type parameter.
This idiom is usually called "PECS" ("Producer extends, Consumer super") because variance annotations are written extends and super in Java.
I'm trying to call this set method documented here, in the Java library jOOQ, with signature:
<T> ... set(Field<T> field, T value)
This Scala line is a problem:
.set(table.MODIFIED_BY, userId)
MODIFIED_BY is a Field<Integer> representing the table column. userId is Int. Predef has an implicit conversion from Int to Integer, so why doesn't it use it? I get this:
type mismatch; found: org.jooq.TableField[gen.tables.records.DocRecord,Integer]
required: org.jooq.Field[Any]
Note: Integer <: Any
(and org.jooq.TableField[gen.tables.records.DocRecord,Integer] <:
org.jooq.Field[Integer]), but Java-defined trait Field is invariant in type T.
You may wish to investigate a wildcard type such as `_ <: Any`. (SLS 3.2.10)
Update - About Vinicius's Example
Rather than try to explain this in comments, here is a demonstration that there is no implicit conversion being called when you use a type with covariant parameter, like List[+T]. Let's say I put this code in a file, compile, and run it...
case class Foo(str: String)
object StackOver1 extends App {
implicit def str2Foo(s: String): Foo = {
println("In str2Foo.")
new Foo(s)
}
def test[T](xs: List[T], x: T): List[T] = {
println("test " + x.getClass)
xs
}
val foo1 = new Foo("foo1")
test(List(foo1), "abc")
}
You'll see that it calls test, but never the implicit conversion from String "abc" to Foo. Instead it's picking a T for test[T] that is a common base class between String and Foo. When you use Int and Integer it picks Any, but it's confusing because the runtime representation of the Int in the list is Integer. So it looks like it used the implicit conversion, but it didn't. You can verify by opening a Scala prompt...
scala> :type StackOver1.test(List(new java.lang.Integer(1)), 2)
List[Any]
I don't know anything aboutjOOQ, but I think the issue is that Scala does not understand java generics very well. Try:
scala> def test[T](a : java.util.ArrayList[T], b: T) = { println(a,b) }
scala> val a = new java.util.ArrayList[Integer]()
scala> val b = 12
scala> test(a,b)
<console>:11: error: type mismatch;
found : java.util.ArrayList[Integer]
required: java.util.ArrayList[Any]
Note: Integer <: Any, but Java-defined class ArrayList is invariant in type E.
You may wish to investigate a wildcard type such as `_ <: Any`. (SLS 3.2.10)
test(a,b)
Sounds familiar??
And to fix, just inform the type T to call the method: test[Integer](a,b) works fine.
EDIT:
There a few things involved here:
Erasure -> When compiled the type of the generic will disappear by erasure. The compiler will use Object which Scala, will treat as Any. However a ArrayList[Integer] is not an ArrayList[Any], even though Integer is any. The same way that TableField[gen.tables.records.DocRecord,Integer] is not a Field[Any].
Type inference mechanism -> it will figure out what type T should be and to do that it will use the intersection dominator of the types passed (in our case the first common ancestor). Page 36 of Scala Language Spec, which in our examples above will lead use to Any.
Implicit conversion -> it is the last step and would be called if there was some type to be converted to another one, but since the type of the arguments were determined to be the first common ancestor, there is no need to convert and we will never have a implicit conversion if we don't force the type T.
A example to show how the common ancestor is used to determine T:
scala> def test[T](a: T, b: T): T = a
scala> class Foo
scala> class Boo extends Foo
scala> test(new Boo,new Foo)
res2: Foo = Boo#139c2a6
scala> test(new Boo,new Boo)
res3: Boo = Boo#141c803
scala> class Coo extends Foo
scala> test(new Boo,new Coo)
res4: Foo = Boo#aafc83
scala> test(new Boo,"qsasad")
res5: Object = Boo#16989d8
Summing up, the implicit method does not get called, because the type inference mechanism, determines the types before getting the argument and since it uses the common ancestor, there is no need for a implicit conversion.
Your code produces an error due to erasure mechanism which disappear with the type information that would be important to determine the correct type of the argument.
#RobN, thanks for questioning my answer, I learned a lot with the process.