Why won't Scala use implicit conversion here? - scala

I'm trying to call this set method documented here, in the Java library jOOQ, with signature:
<T> ... set(Field<T> field, T value)
This Scala line is a problem:
.set(table.MODIFIED_BY, userId)
MODIFIED_BY is a Field<Integer> representing the table column. userId is Int. Predef has an implicit conversion from Int to Integer, so why doesn't it use it? I get this:
type mismatch; found: org.jooq.TableField[gen.tables.records.DocRecord,Integer]
required: org.jooq.Field[Any]
Note: Integer <: Any
(and org.jooq.TableField[gen.tables.records.DocRecord,Integer] <:
org.jooq.Field[Integer]), but Java-defined trait Field is invariant in type T.
You may wish to investigate a wildcard type such as `_ <: Any`. (SLS 3.2.10)
Update - About Vinicius's Example
Rather than try to explain this in comments, here is a demonstration that there is no implicit conversion being called when you use a type with covariant parameter, like List[+T]. Let's say I put this code in a file, compile, and run it...
case class Foo(str: String)
object StackOver1 extends App {
implicit def str2Foo(s: String): Foo = {
println("In str2Foo.")
new Foo(s)
}
def test[T](xs: List[T], x: T): List[T] = {
println("test " + x.getClass)
xs
}
val foo1 = new Foo("foo1")
test(List(foo1), "abc")
}
You'll see that it calls test, but never the implicit conversion from String "abc" to Foo. Instead it's picking a T for test[T] that is a common base class between String and Foo. When you use Int and Integer it picks Any, but it's confusing because the runtime representation of the Int in the list is Integer. So it looks like it used the implicit conversion, but it didn't. You can verify by opening a Scala prompt...
scala> :type StackOver1.test(List(new java.lang.Integer(1)), 2)
List[Any]

I don't know anything aboutjOOQ, but I think the issue is that Scala does not understand java generics very well. Try:
scala> def test[T](a : java.util.ArrayList[T], b: T) = { println(a,b) }
scala> val a = new java.util.ArrayList[Integer]()
scala> val b = 12
scala> test(a,b)
<console>:11: error: type mismatch;
found : java.util.ArrayList[Integer]
required: java.util.ArrayList[Any]
Note: Integer <: Any, but Java-defined class ArrayList is invariant in type E.
You may wish to investigate a wildcard type such as `_ <: Any`. (SLS 3.2.10)
test(a,b)
Sounds familiar??
And to fix, just inform the type T to call the method: test[Integer](a,b) works fine.
EDIT:
There a few things involved here:
Erasure -> When compiled the type of the generic will disappear by erasure. The compiler will use Object which Scala, will treat as Any. However a ArrayList[Integer] is not an ArrayList[Any], even though Integer is any. The same way that TableField[gen.tables.records.DocRecord,Integer] is not a Field[Any].
Type inference mechanism -> it will figure out what type T should be and to do that it will use the intersection dominator of the types passed (in our case the first common ancestor). Page 36 of Scala Language Spec, which in our examples above will lead use to Any.
Implicit conversion -> it is the last step and would be called if there was some type to be converted to another one, but since the type of the arguments were determined to be the first common ancestor, there is no need to convert and we will never have a implicit conversion if we don't force the type T.
A example to show how the common ancestor is used to determine T:
scala> def test[T](a: T, b: T): T = a
scala> class Foo
scala> class Boo extends Foo
scala> test(new Boo,new Foo)
res2: Foo = Boo#139c2a6
scala> test(new Boo,new Boo)
res3: Boo = Boo#141c803
scala> class Coo extends Foo
scala> test(new Boo,new Coo)
res4: Foo = Boo#aafc83
scala> test(new Boo,"qsasad")
res5: Object = Boo#16989d8
Summing up, the implicit method does not get called, because the type inference mechanism, determines the types before getting the argument and since it uses the common ancestor, there is no need for a implicit conversion.
Your code produces an error due to erasure mechanism which disappear with the type information that would be important to determine the correct type of the argument.
#RobN, thanks for questioning my answer, I learned a lot with the process.

Related

What does `T {}` do in Scala

Browsing Shapeless code, I came across this seemingly extraneous {} here and here:
trait Witness extends Serializable {
type T
val value: T {}
}
trait SingletonOps {
import record._
type T
def narrow: T {} = witness.value
}
I almost ignored it as a typo since it does nothing but apparently it does something. See this commit: https://github.com/milessabin/shapeless/commit/56a3de48094e691d56a937ccf461d808de391961
I have no idea what it does. Can someone explain?
Any type can be followed by a {} enclosed sequence of type and abstract non-type member definitions. This is known as a "refinement" and is used to provide additional precision over the base type that is being refined. In practice refinements are most commonly used to express constraints on abstract type members of the type being refined.
It's a little known fact that this sequence is allowed to be empty, and in the form that you can see in the shapeless source code, T {} is the type T with an empty refinement. Any empty refinement is ... empty ... so doesn't add any additional constraints to the refined type and hence the types T and T {} are equivalent. We can get the Scala compiler to verify that for us like so,
scala> implicitly[Int =:= Int {}]
res0: =:=[Int,Int] = <function1>
So why would I do such an apparently pointless thing in shapeless? It's because of the interaction between the presence of refinements and type inference. If you look in the relevant section of the Scala Language Specification you will see that the type inference algorithm attempts to avoid inferring singleton types in at least some circumstances. Here is an example of it doing just that,
scala> class Foo ; val foo = new Foo
defined class Foo
foo: Foo = Foo#8bd1b6a
scala> val f1 = foo
f1: Foo = Foo#8bd1b6a
scala> val f2: foo.type = foo
f2: foo.type = Foo#8bd1b6a
As you can see from the definition of f2 the Scala compiler knows that the value foo has the more precise type foo.type (ie. the singleton type of val foo), however, unless explicitly requested it won't infer that more precise type. Instead it infers the non-singleton (ie. widened) type Foo as you can see in the case of f1.
But in the case of Witness in shapeless I explicitly want the singleton type to be inferred for uses of the value member (the whole point of Witness is enable us to pass between the type and value levels via singleton types), so is there any way the Scala compiler can be persuaded to do that?
It turns out that an empty refinement does exactly that,
scala> def narrow[T <: AnyRef](t: T): t.type = t
narrow: [T <: AnyRef](t: T)t.type
scala> val s1 = narrow("foo") // Widened
s1: String = foo
scala> def narrow[T <: AnyRef](t: T): t.type {} = t // Note empty refinement
narrow: [T <: AnyRef](t: T)t.type
scala> val s2 = narrow("foo") // Not widened
s2: String("foo") = foo
As you can see in the above REPL transcript, in the first case s1 has been typed as the widened type String whereas s2 has been assigned the singleton type String("foo").
Is this mandated by the SLS? No, but it is consistent with it, and it makes some sort of sense. Much of Scala's type inference mechanics are implementation defined rather than spec'ed and this is probably one of the least surprising and problematic instances of that.

Shapeless: Generic.Aux

I'm trying to understand how Generic works (and TypeClass too). The github wiki is very sparse on examples and documentation. Is there a canonical blog post / documentation page describing Generic and TypeClass in detail?
In concrete, what is the difference between these two methods?:
def find1[T](implicit gen: Generic[T]): Generic[T] = gen
def find2[T](implicit gen: Generic[T]): Generic[T] { type Repr = gen.Repr } = gen
given
object Generic {
type Aux[T, Repr0] = Generic[T] { type Repr = Repr0 }
def apply[T](implicit gen: Generic[T]): Aux[T, gen.Repr] = gen
implicit def materialize[T, R]: Aux[T, R] = macro GenericMacros.materialize[T, R]
}
The issues involved in how Generic and TypeClass are implemented and what they do are different enough that they probably deserve separate questions, so I'll stick to Generic here.
Generic provides a mapping from case classes (and potentially similar types) to heterogeneous lists. Any case class has a unique hlist representation, but any given hlist corresponds to a very, very large number of potential case classes. For example, if we have the following case classes:
case class Foo(i: Int, s: String)
case class Bar(x: Int, y: String)
The hlist representation provided by Generic for both Foo and Bar is Int :: String :: HNil, which is also the representation for (Int, String) and any other case classes we could define with these two types in this order.
(As a side note, LabelledGeneric allows us to distinguish between Foo and Bar, since it includes the member names in the representation as type-level strings.)
We generally want to be able to specify the case class and let Shapeless figure out the (unique) generic representation, and making Repr a type member (instead of a type parameter) allows us to do this pretty cleanly. If the hlist representation type were a type parameter, then your find methods would have to have a Repr type parameter as well, which means that you wouldn't be able to specify only the T and have the Repr inferred.
Making Repr a type member makes sense only because the Repr is uniquely determined by the first type parameter. Imagine a type class like Iso[A, B] that witnesses that A and B are isomorphic. This type class is very similar to Generic, but A doesn't uniquely dermine B—we can't just ask "what is the type that's isomorphic to A?"—so it wouldn't be useful to make B a type member (although we could if we really wanted to—Iso[A] just wouldn't really mean anything).
The problem with type members is that they're easy to forget, and once they're gone, they're gone forever. The fact that the return type of your find1 isn't refined (i.e. doesn't include the type member) means that the Generic instance it returns is pretty much useless. For example, the static type of res0 here might as well be Any:
scala> import shapeless._
import shapeless._
scala> def find1[T](implicit gen: Generic[T]): Generic[T] = gen
find1: [T](implicit gen: shapeless.Generic[T])shapeless.Generic[T]
scala> case class Foo(i: Int, s: String)
defined class Foo
scala> find1[Foo].to(Foo(1, "ABC"))
res0: shapeless.Generic[Foo]#Repr = 1 :: ABC :: HNil
scala> res0.head
<console>:15: error: value head is not a member of shapeless.Generic[Foo]#Repr
res0.head
^
When Shapeless's Generic.materialize macro creates the Generic[Foo] instance we're asking for, it's statically typed as a Generic[Foo] { type Repr = Int :: String :: HNil }, so the gen argument that the compiler hands to find1 has all the static information we need. The problem is that we then explicitly up-cast that type to a plain old unrefined Generic[Foo], and from that point on the compiler doesn't know what the Repr is for that instance.
Scala's path-dependent types give us a way not to forget the refinement without adding another type parameter to our method. In your find2, the compiler statically knows the Repr for the incoming gen, so when you say that the return type is Generic[T] { type Repr = gen.Repr }, it will be able to keep track of that information:
scala> find2[Foo].to(Foo(1, "ABC"))
res2: shapeless.::[Int,shapeless.::[String,shapeless.HNil]] = 1 :: ABC :: HNil
scala> res2.head
res3: Int = 1
To sum up: Generic has a type parameter T that uniquely determines its type member Repr, Repr is a type member instead of a type parameter so that we don't have to include it in all of our type signatures, and path-dependent types make this possible, allowing us to keep track of Repr even though it's not in our type signatures.

Scala generics type mismatch

In Scala, I'm trying:
import scala.reflect.runtime.{universe => ru}
def foo[T <: Any]: ru.WeakTypeTag[T] = ru.weakTypeTag[String]
But this yields me:
<console>:34: error: type mismatch;
found : reflect.runtime.universe.WeakTypeTag[String]
required: reflect.runtime.universe.WeakTypeTag[T]
def foo[T <: Any]: ru.WeakTypeTag[T] = ru.weakTypeTag[String]
What's up here? I'm relatively sure String should satisfy T's type constraint of deriving from Any...
I guess String failed to bind to the T type parameter. In my use case other types may be returned as well though, and I'm not sure how I could get that answer to the compiler up-front before executing the function, if that's what it's expecting.
Your method foo claims that, for any T <: Any, it returns a WeakTypeTag[T]. That is, if T is (for instance) Int, it should return a WeakTypeTag[Int]. However, your method always returns a WeakTypeTag[String], hence the type mistmatch.
As an alternative you can use wildcard, anyway your type is extended from Any.
def foo[T <: Any]: ru.WeakTypeTag[_] = ru.weakTypeTag[String]
In general the problem is with definition of WeakTypeTag[T] class. It is defined invariantly. So you can not use it in covariant case.
Let's go with an examples.
def bom[T >: String]: List[T] = List[String]() // works fine
def foo[T >: String]: WeakTypeTag[T] = ru.weakTypeTag[String] // compilation fails
I'm defining T as any subtype of String and it works good for Lists but fails for WeakTypeTag that is invariant.
You can define sub type of WeakTypeTag and make it covariant so it will perfectly works.
trait WeakTypeTag1[+X] extends ru.WeakTypeTag {
}
def weakTypeTag1[T](implicit attag: WeakTypeTag1[T]) = attag
def foo[T >: String]: WeakTypeTag1[T] = weakTypeTag1[String] // no it's good

Result type of an implicit conversion must be more specific than AnyRef

Let
def h(a: AnyRef*) = a.mkString(",")
h: (a: AnyRef*)String
and so
h("1","2")
res: String = 1,2
However, h(1,2)
error: the result type of an implicit conversion must be more specific than AnyRef
h(1,2)
^
error: the result type of an implicit conversion must be more specific than AnyRef
h(1,2)
^
This is at least in Scala 2.11.1 and 2.11.1.
To ask on a workaround.
The reason is that the numeric type of the literals 1 and 2 is Int which extends AnyVal which, in turn, extends Any. On the other hand String extends AnyRef which, in turn, extends Any. So as you can see AnyVal (Int's parent) does not extend AnyRef. You can solve this in one of two ways.
The first one is changing the type from AnyRef to Any as described by Nate.
The second one is using a type ascription for the literals 1 and 2 so that they are considered of type java.lang.Integer which extends java.lang.Object. Note also that AnyRef is just an alias for java.lang.Object. So, using your definition the following should work:
scala> h(1: java.lang.Integer, 2: java.lang.Integer)
res2: String = 1,2
More info on Scala Hierarchy
You can reproduce the issue simply with:
val x: AnyRef = 42
Here's the relevant pull request on github that introduced the change
The rationale is that for security reasons some implicit conversions are explicitly disabled, namely when the conversion goes from T to U is disabled if:
T <: Null
or
AnyRef <: U
In your specific case, this means that an Int (which is not an AnyRef) will never be converted to AnyRef.
If you need to accept both Int and String, you can consider accepting Any instead. Since every scala object inherits from Any, there's no implicit conversion needed.
def h(a: Any*) = a.mkString(",")
Cast your variable to AnyRef by doing something like this:
h(1.asInstanceOf[AnyRef], 2.asInstanceOf[AnyRef])
Why?
In scala not everything extends Object (aka AnyRef) in the way that it would in java. Specifically primitives extend AnyVal, so if your function requires an AnyRef you'll need to cast / convert / restrict your scala variables.
There's a good discussion here: What are the relationships between Any, AnyVal, AnyRef, Object and how do they map when used in Java code?
I don't think you want to use AnyRef here. I think you want Any.
scala> def h(a: Any*) = a.mkString(",")
h: (a: Any*)String
scala> h(1,2)
res0: String = 1,2
The reason is that the numeric value 5 is an Int, but AnyRef is java's Object equivalence. So to invoke that method it would need to be a java.util.Integer.

Context bounds shortcut with higher kinded-types

Is it possible to use the context bounds syntax shortcut with higher kinded-types?
trait One { def test[W : ClassManifest]: Unit } // first-order ok
trait Two { def test[W[_]: ClassManifest]: Unit } // not possible??
trait Six { def test[W[_]](implicit m: ClassManifest[W[_]]): Unit } // hmm...
Yes, it is, but your context bound type must have a higher kinded type parameter (which ClassManifest doesn't).
scala> trait HKTypeClass[CC[_]]
defined trait HKTypeClass
scala> implicit def listHKTC = new HKTypeClass[List] {}
listHKTC: java.lang.Object with HKTypeClass[List]
scala> def frob[CC[_] : HKTypeClass] = implicitly[HKTypeClass[CC]]
frob: [CC[_]](implicit evidence$1: HKTypeClass[CC])HKTypeClass[CC]
scala> frob[List]
res0: HKTypeClass[List] = $anon$1#13e02ed
Update
It's possible to use a type alias to allow a higher-kinded type parameter to be bounded by a first-order context bound type. We use the type alias as a type-level function to make a higher-kinded type out of the first-order type. For ClassManifest it could go like this,
scala> type HKClassManifest[CC[_]] = ClassManifest[CC[_]]
defined type alias HKClassManifest
scala> def frob[CC[_] : HKClassManifest] = implicitly[HKClassManifest[CC]]
test: [CC[_]](implicit evidence$1: HKClassManifest[CC])HKClassManifest[CC]
scala> frob[List]
res1: HKClassManifest[List] = scala.collection.immutable.List[Any]
Note that on the right hand side of the type alias CC[_] is a first-order type ... the underscore here is the wildcard. Consequently it can be used as the type argument for ClassManifest.
Update
For completeness I should note that the type alias can be inlined using a type lambda,
scala> def frob[CC[_] : ({ type λ[X[_]] = ClassManifest[X[_]] })#λ] = implicitly[ClassManifest[CC[_]]]
frob: [CC[_]](implicit evidence$1: scala.reflect.ClassManifest[CC[_]])scala.reflect.ClassManifest[CC[_]]
scala> frob[List]
res0: scala.reflect.ClassManifest[List[_]] = scala.collection.immutable.List[Any]
Note that implicitly[ClassManifest[List[_]]] is short for implicitly[ClassManifest[List[T] forSome {type T}]].
That's why it works: ClassManifest expects a proper type argument, and List[T] forSome {type T} is a proper type, but List is a type constructor. (Please see What is a higher kinded type in Scala? for a definition of "proper" etc.)
To make both ClassManifest[List[String]] and ClassManifest[List] work, we'd need to overload ClassManifest somehow with versions that take type parameters of varying kinds, something like:
class ClassManifest[T] // proper type
class ClassManifest[T[_]] // type constructor with one type parameter
class ClassManifest[T[_, _]] // type constructor with two type parameters
// ... ad nauseam
(On an academic note, the "proper" way to do this, would be to allow abstracting over kinds:
class ClassManifest[T : K][K]
implicitly[ClassManifest[String]] // --> compiler infers ClassManifest[String][*]
implicitly[ClassManifest[List]] // --> compiler infers ClassManifest[List][* -> *]
)