Browsing Shapeless code, I came across this seemingly extraneous {} here and here:
trait Witness extends Serializable {
type T
val value: T {}
}
trait SingletonOps {
import record._
type T
def narrow: T {} = witness.value
}
I almost ignored it as a typo since it does nothing but apparently it does something. See this commit: https://github.com/milessabin/shapeless/commit/56a3de48094e691d56a937ccf461d808de391961
I have no idea what it does. Can someone explain?
Any type can be followed by a {} enclosed sequence of type and abstract non-type member definitions. This is known as a "refinement" and is used to provide additional precision over the base type that is being refined. In practice refinements are most commonly used to express constraints on abstract type members of the type being refined.
It's a little known fact that this sequence is allowed to be empty, and in the form that you can see in the shapeless source code, T {} is the type T with an empty refinement. Any empty refinement is ... empty ... so doesn't add any additional constraints to the refined type and hence the types T and T {} are equivalent. We can get the Scala compiler to verify that for us like so,
scala> implicitly[Int =:= Int {}]
res0: =:=[Int,Int] = <function1>
So why would I do such an apparently pointless thing in shapeless? It's because of the interaction between the presence of refinements and type inference. If you look in the relevant section of the Scala Language Specification you will see that the type inference algorithm attempts to avoid inferring singleton types in at least some circumstances. Here is an example of it doing just that,
scala> class Foo ; val foo = new Foo
defined class Foo
foo: Foo = Foo#8bd1b6a
scala> val f1 = foo
f1: Foo = Foo#8bd1b6a
scala> val f2: foo.type = foo
f2: foo.type = Foo#8bd1b6a
As you can see from the definition of f2 the Scala compiler knows that the value foo has the more precise type foo.type (ie. the singleton type of val foo), however, unless explicitly requested it won't infer that more precise type. Instead it infers the non-singleton (ie. widened) type Foo as you can see in the case of f1.
But in the case of Witness in shapeless I explicitly want the singleton type to be inferred for uses of the value member (the whole point of Witness is enable us to pass between the type and value levels via singleton types), so is there any way the Scala compiler can be persuaded to do that?
It turns out that an empty refinement does exactly that,
scala> def narrow[T <: AnyRef](t: T): t.type = t
narrow: [T <: AnyRef](t: T)t.type
scala> val s1 = narrow("foo") // Widened
s1: String = foo
scala> def narrow[T <: AnyRef](t: T): t.type {} = t // Note empty refinement
narrow: [T <: AnyRef](t: T)t.type
scala> val s2 = narrow("foo") // Not widened
s2: String("foo") = foo
As you can see in the above REPL transcript, in the first case s1 has been typed as the widened type String whereas s2 has been assigned the singleton type String("foo").
Is this mandated by the SLS? No, but it is consistent with it, and it makes some sort of sense. Much of Scala's type inference mechanics are implementation defined rather than spec'ed and this is probably one of the least surprising and problematic instances of that.
Related
I have been looking into the details of shapeless singletons and have encountered a small example that doesn't work as expected. I thought that as long as we pass a singleton type to a method, there should be an implicit Witness.Aux[_] available in scope:
import shapeless._
import syntax.singleton._
object SingletonTest extends App {
def check[K](a: K)(implicit witness: Witness.Aux[K]): Unit = {
println(witness.value)
}
val b = 'key.narrow
val c: Witness.`'key`.T = b
check(c) // Works!
check(b) /* Fails: shapeless.this.Witness.apply is not a valid implicit value for shapeless.Witness.Aux[Symbol with shapeless.tag.Tagged[String("key")]] because:
hasMatchingSymbol reported error: Type argument Symbol with shapeless.tag.Tagged[String("key")] is not a singleton type */
}
I would expect the types of b and c in the example to be the same (and checking it with =:= succeeds). If I add an implicit Witness[Witness.'key.T] into scope manually, the code compiles.
Environment: Scala 2.11.8; Shapeless 2.3.0
The explanation is that the Scala compiler will widen the singleton type on the right hand side when inferring the type of the val on the left ... you'll find chapter and verse in the relevant section of the Scala Language Specification.
Here's an example of the same phenomenon independent from shapeless,
scala> class Foo ; val foo = new Foo
defined class Foo
foo: Foo = Foo#8bd1b6a
scala> val f1 = foo
f1: Foo = Foo#8bd1b6a
scala> val f2: foo.type = foo
f2: foo.type = Foo#8bd1b6a
As you can see from the definition of f2 the Scala compiler knows that the value foo has the more precise type foo.type (ie. the singleton type of val foo), however, unless explicitly requested it won't infer that more precise type. Instead it infers the non-singleton (ie. widened) type Foo as you can see in the case of f1.
See also this related answer.
I'm trying to understand how Generic works (and TypeClass too). The github wiki is very sparse on examples and documentation. Is there a canonical blog post / documentation page describing Generic and TypeClass in detail?
In concrete, what is the difference between these two methods?:
def find1[T](implicit gen: Generic[T]): Generic[T] = gen
def find2[T](implicit gen: Generic[T]): Generic[T] { type Repr = gen.Repr } = gen
given
object Generic {
type Aux[T, Repr0] = Generic[T] { type Repr = Repr0 }
def apply[T](implicit gen: Generic[T]): Aux[T, gen.Repr] = gen
implicit def materialize[T, R]: Aux[T, R] = macro GenericMacros.materialize[T, R]
}
The issues involved in how Generic and TypeClass are implemented and what they do are different enough that they probably deserve separate questions, so I'll stick to Generic here.
Generic provides a mapping from case classes (and potentially similar types) to heterogeneous lists. Any case class has a unique hlist representation, but any given hlist corresponds to a very, very large number of potential case classes. For example, if we have the following case classes:
case class Foo(i: Int, s: String)
case class Bar(x: Int, y: String)
The hlist representation provided by Generic for both Foo and Bar is Int :: String :: HNil, which is also the representation for (Int, String) and any other case classes we could define with these two types in this order.
(As a side note, LabelledGeneric allows us to distinguish between Foo and Bar, since it includes the member names in the representation as type-level strings.)
We generally want to be able to specify the case class and let Shapeless figure out the (unique) generic representation, and making Repr a type member (instead of a type parameter) allows us to do this pretty cleanly. If the hlist representation type were a type parameter, then your find methods would have to have a Repr type parameter as well, which means that you wouldn't be able to specify only the T and have the Repr inferred.
Making Repr a type member makes sense only because the Repr is uniquely determined by the first type parameter. Imagine a type class like Iso[A, B] that witnesses that A and B are isomorphic. This type class is very similar to Generic, but A doesn't uniquely dermine B—we can't just ask "what is the type that's isomorphic to A?"—so it wouldn't be useful to make B a type member (although we could if we really wanted to—Iso[A] just wouldn't really mean anything).
The problem with type members is that they're easy to forget, and once they're gone, they're gone forever. The fact that the return type of your find1 isn't refined (i.e. doesn't include the type member) means that the Generic instance it returns is pretty much useless. For example, the static type of res0 here might as well be Any:
scala> import shapeless._
import shapeless._
scala> def find1[T](implicit gen: Generic[T]): Generic[T] = gen
find1: [T](implicit gen: shapeless.Generic[T])shapeless.Generic[T]
scala> case class Foo(i: Int, s: String)
defined class Foo
scala> find1[Foo].to(Foo(1, "ABC"))
res0: shapeless.Generic[Foo]#Repr = 1 :: ABC :: HNil
scala> res0.head
<console>:15: error: value head is not a member of shapeless.Generic[Foo]#Repr
res0.head
^
When Shapeless's Generic.materialize macro creates the Generic[Foo] instance we're asking for, it's statically typed as a Generic[Foo] { type Repr = Int :: String :: HNil }, so the gen argument that the compiler hands to find1 has all the static information we need. The problem is that we then explicitly up-cast that type to a plain old unrefined Generic[Foo], and from that point on the compiler doesn't know what the Repr is for that instance.
Scala's path-dependent types give us a way not to forget the refinement without adding another type parameter to our method. In your find2, the compiler statically knows the Repr for the incoming gen, so when you say that the return type is Generic[T] { type Repr = gen.Repr }, it will be able to keep track of that information:
scala> find2[Foo].to(Foo(1, "ABC"))
res2: shapeless.::[Int,shapeless.::[String,shapeless.HNil]] = 1 :: ABC :: HNil
scala> res2.head
res3: Int = 1
To sum up: Generic has a type parameter T that uniquely determines its type member Repr, Repr is a type member instead of a type parameter so that we don't have to include it in all of our type signatures, and path-dependent types make this possible, allowing us to keep track of Repr even though it's not in our type signatures.
Lets assume I have instance of arbitrary one-argument generic class (I'll use List in demonstration but this can me any other generic).
I'd like to write generic function that can take instances (c) and be able to understand what generic class (A) and what type argument (B) produced the class (C) of that instance.
I've come up with something like this (body of the function is not really relevant but demonstrates that C conforms to A[B]):
def foo[C <: A[B], A[_], B](c: C) {
val x: A[B] = c
}
... and it compiles if you invoke it like this:
foo[List[Int], List, Int](List.empty[Int])
... but compilation fails with error if I omit explicit type arguments and rely on inference:
foo(List.empty[Int])
The error I get is:
Error:Error:line (125)inferred kinds of the type arguments (List[Int],List[Int],Nothing) do not conform to the expected kinds of the type parameters (type C,type A,type B).
List[Int]'s type parameters do not match type A's expected parameters:
class List has one type parameter, but type A has one
foo(List.empty[Int])
^
Error:Error:line (125)type mismatch;
found : List[Int]
required: C
foo(List.empty[Int])
^
As you can see Scala's type inference cannot infer the types correctly in this case (seems like it's guess is List[Int] instead of List for 2nd argument and Nothing instead of Int for 3rd).
I assume that type bounds for foo I've come up with are not precise/correct enough, so my question is how could I implement it, so Scala could infer arguments?
Note: if it helps, the assumption that all potential generics (As) inherit/conform some common ancestor can be made. For example, that A can be any collection inherited from Seq.
Note: the example described in this question is synthetic and is a distilled part of the bigger problem I am trying to solve.
This is a known limitation of current Scala's type inference for type constructors. Defining the type of formal parameter c as C only collects type constraints for C (and indirectly to A) but not B. In other words List[Int] <: C => { List[Int] <: C <: Any, C <: A[_] <: Any }.
There is a pretty simple translation that allows to guide type inference for such cases. In your case it is:
def foo[C[_] <: A[_], A[_], B](c: A[B]) { val x: A[B] = c }
Same semantics, just slightly different type signature.
In addition to hubertp answer, you can fix you function by removing obsolete (in you example) type variable C, e.g:
def foo[A[_], B](c: A[B]) {
val x: A[B] = c
}
In this case scalac would infer A[_] as List and B as Int.
Update (according to the comment).
If you need an evidence that C is subtype of A[B], then use implicit:
def foo[A[_], B, C](c: C)(implicit ev: C <:< A[B]) = {
val x: A[B] = c
}
Then it won't compile this:
scala> foo[List, String, List[Int]](List.empty[Int])
<console>:9: error: Cannot prove that List[Int] <:< List[String].
foo[List, String, List[Int]](List.empty[Int])
Looking at the source for List.scala:
sealed abstract class List[+A] extends ...
...
def isEmpty: Boolean
def head: A
def tail: List[A]
List[+A] is covariant based on the +A. Does this mean that, it's possible to create a List[T] where T can be the type itself, or any of its sub-classes?
example:
scala> trait Kid
defined trait Kid
scala> case class Boy(name: String) extends Kid
defined class Boy
scala> case class Girl(name: String) extends Kid
defined class Girl
scala> val list: List[Kid] = List(Boy("kevin"), Girl("sally"))
list: List[Kid] = List(Boy(kevin), Girl(sally))
Observe that head and tail's types are A and List[A], respectively. Once we've defined List[+A], then head and tail's A is also covariant?
I've read this StackOverflow answer 3 or 4 times, but I don't understand yet.
Your example does not relate to variance. Moreover, head and tail have nothing to do with variance too.
scala> val list: List[Kid] = List(Boy("kevin"), Girl("sally"))
list: List[Kid] = List(Boy(kevin), Girl(sally))
This would work even if List weren't covariant, because Scala will automatically deduce common supertype of Boy and Girl, that is, Kid, and type of the expression on the right side will be List[Kid], exactly what you require on the left side.
The following, however, doesn't work because java.util.List is not covariant (it is invariant since it is Java type):
scala> import java.util.{List => JList, Arrays}
import java.util.{List=>JList, Arrays}
scala> trait Kid
defined trait Kid
scala> case class Boy(name: String) extends Kid
defined class Boy
scala> val list1 = Arrays.asList(Boy("kevin"), Boy("bob"))
list1: java.util.List[Boy] = [Boy(kevin), Boy(bob)]
scala> val list2: JList[Kid] = list1
<console>:12: error: type mismatch;
found : java.util.List[Boy]
required: java.util.List[Kid]
Note: Boy <: Kid, but Java-defined trait List is invariant in type E.
You may wish to investigate a wildcard type such as `_ <: Kid`. (SLS 3.2.10)
val list2: JList[Kid] = list1
^
Arrays.asList method has signature like this:
def asList[T](args: T*): java.util.List[T]
As java.util.List[T] is invariant, it is impossible to assign JList[Boy] (list1) to JList[Kid] (list2). And there is a reason: if you could, then because JList is mutable, you could also add anything extending Kid (not only Boy) into the same list, breaking type safety.
On the other hand, scala.List will work in exactly the same situation:
scala> val list1 = List(Boy("kevin"), Boy("bob"))
list1: List[Boy] = List(Boy(kevin), Boy(bob))
scala> val list2: List[Kid] = list1
list2: List[Kid] = List(Boy(kevin), Boy(bob))
That is because scala.List is covariant in its type parameter. Note that covariant List type works as if List[Boy] were subtype of List[Kid], very similar to the case when you can assign everything to a variable of type Any because every other type is a subtype of Any. This is very helpful analogy.
Contravariance works in a very similar way, but in other direction. Consider this trait:
trait Predicate[-T] {
def apply(obj: T): Boolean
}
object Predicate {
// convenience method to convert functions to predicates
def apply[T](f: (T) => Boolean) = new Predicate[T] {
def apply(obj: T) = f(obj)
}
}
Note the - before T parameter: it is a contravariance annotation, that is, Predicate[T] is defined to be contravariant in its only type parameter.
Recall that for covariant list List[Boy] was a subtype of List[Kid]. Well, for contravariant predicate it works in the opposite way: Predicate[Kid] is a subtype of Predicate[Boy], so you can assign a value of type Predicate[Kid] to a variable of type Predicate[Boy]:
scala> val pred1: Predicate[Kid] = Predicate { kid => kid.hashCode % 2 == 0 }
pred1: Predicate[Kid] = Predicate$$anon$1#3bccdcdd
scala> val pred2: Predicate[Boy] = pred1
pred2: Predicate[Boy] = Predicate$$anon$1#3bccdcdd
If Predicate[T] weren't contravariant, we wouldn't be able to assign pred1 to pred2, though it is completely legitimate and safe: obviously, predicates defined on supertypes can easily work on subtypes.
In short, variance affects type compatibility between parameterized types. List is covariant, so you can assign a value of type List[Boy] to a variable of type List[Kid] (in fact, for any T extending S, you can assign a value of type List[T] to a variable of type List[S]).
On the other hand, because, Predicate is contravariant, you can assign Predicate[Kid] to Predicate[Boy] (that is, for any T extending S, you can assign a value of type Predicate[S] to a variable of type Predicate[T]).
If a type is invariant in its type parameter, neither of the above can be done (as is demonstrated by JList).
Note the correspondence between parameterized types and their parameters:
T <: S ===> List [T] <: List [S] (covariance)
T <: S ===> Predicate[S] <: Predicate[T] (contravariance)
This is the reason why the first effect is called *co*variance (T <: S on the left, and
..T.. <: ..S.. on the right), and the second is *contra*variance (T <: S on the left, but ..S.. <: ..T.. on the right).
Whether to make your own parameterized types covariant or contravariant or invariant depends on your class responsibilities. If it may only return values of generic type, then it makes sense to use covariance. List[T], for example, only contains methods which return T, never accept T as a parameter, so it is safe to make it covariant in order to increase expressiveness. Such parameterized types can be called producers.
If your class only accepts values of the generic type as a parameter, not returns them (exactly like Predicate above which has single method def apply(obj: T): Boolean), then you can safely make it contravariant. Such parameterized types can be called consumers
If your class both accepts and returns values of the generic type, i.e. it is both a producer and a consumer, then you have no choice but to leave the class invariant in this generic type parameter.
This idiom is usually called "PECS" ("Producer extends, Consumer super") because variance annotations are written extends and super in Java.
I'm trying to call this set method documented here, in the Java library jOOQ, with signature:
<T> ... set(Field<T> field, T value)
This Scala line is a problem:
.set(table.MODIFIED_BY, userId)
MODIFIED_BY is a Field<Integer> representing the table column. userId is Int. Predef has an implicit conversion from Int to Integer, so why doesn't it use it? I get this:
type mismatch; found: org.jooq.TableField[gen.tables.records.DocRecord,Integer]
required: org.jooq.Field[Any]
Note: Integer <: Any
(and org.jooq.TableField[gen.tables.records.DocRecord,Integer] <:
org.jooq.Field[Integer]), but Java-defined trait Field is invariant in type T.
You may wish to investigate a wildcard type such as `_ <: Any`. (SLS 3.2.10)
Update - About Vinicius's Example
Rather than try to explain this in comments, here is a demonstration that there is no implicit conversion being called when you use a type with covariant parameter, like List[+T]. Let's say I put this code in a file, compile, and run it...
case class Foo(str: String)
object StackOver1 extends App {
implicit def str2Foo(s: String): Foo = {
println("In str2Foo.")
new Foo(s)
}
def test[T](xs: List[T], x: T): List[T] = {
println("test " + x.getClass)
xs
}
val foo1 = new Foo("foo1")
test(List(foo1), "abc")
}
You'll see that it calls test, but never the implicit conversion from String "abc" to Foo. Instead it's picking a T for test[T] that is a common base class between String and Foo. When you use Int and Integer it picks Any, but it's confusing because the runtime representation of the Int in the list is Integer. So it looks like it used the implicit conversion, but it didn't. You can verify by opening a Scala prompt...
scala> :type StackOver1.test(List(new java.lang.Integer(1)), 2)
List[Any]
I don't know anything aboutjOOQ, but I think the issue is that Scala does not understand java generics very well. Try:
scala> def test[T](a : java.util.ArrayList[T], b: T) = { println(a,b) }
scala> val a = new java.util.ArrayList[Integer]()
scala> val b = 12
scala> test(a,b)
<console>:11: error: type mismatch;
found : java.util.ArrayList[Integer]
required: java.util.ArrayList[Any]
Note: Integer <: Any, but Java-defined class ArrayList is invariant in type E.
You may wish to investigate a wildcard type such as `_ <: Any`. (SLS 3.2.10)
test(a,b)
Sounds familiar??
And to fix, just inform the type T to call the method: test[Integer](a,b) works fine.
EDIT:
There a few things involved here:
Erasure -> When compiled the type of the generic will disappear by erasure. The compiler will use Object which Scala, will treat as Any. However a ArrayList[Integer] is not an ArrayList[Any], even though Integer is any. The same way that TableField[gen.tables.records.DocRecord,Integer] is not a Field[Any].
Type inference mechanism -> it will figure out what type T should be and to do that it will use the intersection dominator of the types passed (in our case the first common ancestor). Page 36 of Scala Language Spec, which in our examples above will lead use to Any.
Implicit conversion -> it is the last step and would be called if there was some type to be converted to another one, but since the type of the arguments were determined to be the first common ancestor, there is no need to convert and we will never have a implicit conversion if we don't force the type T.
A example to show how the common ancestor is used to determine T:
scala> def test[T](a: T, b: T): T = a
scala> class Foo
scala> class Boo extends Foo
scala> test(new Boo,new Foo)
res2: Foo = Boo#139c2a6
scala> test(new Boo,new Boo)
res3: Boo = Boo#141c803
scala> class Coo extends Foo
scala> test(new Boo,new Coo)
res4: Foo = Boo#aafc83
scala> test(new Boo,"qsasad")
res5: Object = Boo#16989d8
Summing up, the implicit method does not get called, because the type inference mechanism, determines the types before getting the argument and since it uses the common ancestor, there is no need for a implicit conversion.
Your code produces an error due to erasure mechanism which disappear with the type information that would be important to determine the correct type of the argument.
#RobN, thanks for questioning my answer, I learned a lot with the process.