Related
I have a question that's been bugging me.
Lists in Scala are covariant (List[+A])
Let's say we have these classes:
class A
class B extends A
The map function of List[B] takes a function f: B => C
But I can also use a f: A => C
which is a subclass of f: B => C
and it totally makes sense.
What I am currently confused by is that
the map function should accept only functions that are superclasses of the original function (since functions are contravariant on their arguments), which does not apply in the example I've given.
I know there's something wrong with my logic and I would like to enlightened.
Your error lies in the assumption that map(f: A => C) should only accept functions that are superclasses of A => C.
While in reality, map will accept any function that is a subclass of A => C.
In Scala, a function parameter can always be a subclass of the required type.
The covariance of A in List[A] only tells you that, wherever a List[A] is required, you can provide a List[B], as long as B <: A.
Or, in simpler words: List[B] can be treated as if it was a subclass of List[A].
I have compiled a small example to explain these two behaviours:
class A
class B extends A
// this means: B <: A
val listA: List[A] = List()
val listB: List[B] = List()
// Example 1: List[B] <: List[A]
// Note: Here the List[+T] is our parameter! (Covariance!)
def printListA(list: List[A]): Unit = println(list)
printListA(listA)
printListA(listB)
// Example 2: Function[A, _] <: Function[B, _]
// Note: Here a Function1[-T, +R] is our parameter (Contravariance!)
class C
def fooA(a: A): C = ???
def fooB(b: B): C = ???
listB.map(fooB)
listB.map(fooA)
Try it out!
I hope this helps.
As you already suspected, you are mixing up things here.
On the one hand, you have a List[+A], which tells us something about the relationships between List[A] and List[B], given a relationship between A and B. The fact that List is covariant in A just means that List[B] <: List[A] when B <: A, as you know already know.
On the other hand, List exposes a method map to change its "contents". This method does not really care about List[A], but only about As, so the variance of List is irrelevant here. The bit that is confusing you here is that there is indeed some sub-typing to take into consideration: map accepts an argument (a A => C in this case, but it's not really relevant) and, as usual with methods and functions, you can always substitute its argument with anything that is a subtype of it. In your specific case, any AcceptedType will be ok, as long as AcceptedType <: Function1[A,C]. The variance that matters here is Function's, not List's.
I can see in the API docs for Predef that they're subclasses of a generic function type (From) => To, but that's all it says. Um, what? Maybe there's documentation somewhere, but search engines don't handle "names" like "<:<" very well, so I haven't been able to find it.
Follow-up question: when should I use these funky symbols/classes, and why?
These are called generalized type constraints. They allow you, from within a type-parameterized class or trait, to further constrain one of its type parameters. Here's an example:
case class Foo[A](a:A) { // 'A' can be substituted with any type
// getStringLength can only be used if this is a Foo[String]
def getStringLength(implicit evidence: A =:= String) = a.length
}
The implicit argument evidence is supplied by the compiler, iff A is String. You can think of it as a proof that A is String--the argument itself isn't important, only knowing that it exists. [edit: well, technically it actually is important because it represents an implicit conversion from A to String, which is what allows you to call a.length and not have the compiler yell at you]
Now I can use it like so:
scala> Foo("blah").getStringLength
res6: Int = 4
But if I tried use it with a Foo containing something other than a String:
scala> Foo(123).getStringLength
<console>:9: error: could not find implicit value for parameter evidence: =:=[Int,String]
You can read that error as "could not find evidence that Int == String"... that's as it should be! getStringLength is imposing further restrictions on the type of A than what Foo in general requires; namely, you can only invoke getStringLength on a Foo[String]. This constraint is enforced at compile-time, which is cool!
<:< and <%< work similarly, but with slight variations:
A =:= B means A must be exactly B
A <:< B means A must be a subtype of B (analogous to the simple type constraint <:)
A <%< B means A must be viewable as B, possibly via implicit conversion (analogous to the simple type constraint <%)
This snippet by #retronym is a good explanation of how this sort of thing used to be accomplished and how generalized type constraints make it easier now.
ADDENDUM
To answer your follow-up question, admittedly the example I gave is pretty contrived and not obviously useful. But imagine using it to define something like a List.sumInts method, which adds up a list of integers. You don't want to allow this method to be invoked on any old List, just a List[Int]. However the List type constructor can't be so constrainted; you still want to be able to have lists of strings, foos, bars, and whatnots. So by placing a generalized type constraint on sumInts, you can ensure that just that method has an additional constraint that it can only be used on a List[Int]. Essentially you're writing special-case code for certain kinds of lists.
Not a complete answer (others have already answered this), I just wanted to note the following, which maybe helps to understand the syntax better: The way you normally use these "operators", as for example in pelotom's example:
def getStringLength(implicit evidence: A =:= String)
makes use of Scala's alternative infix syntax for type operators.
So, A =:= String is the same as =:=[A, String] (and =:= is just a class or trait with a fancy-looking name). Note that this syntax also works with "regular" classes, for example you can write:
val a: Tuple2[Int, String] = (1, "one")
like this:
val a: Int Tuple2 String = (1, "one")
It's similar to the two syntaxes for method calls, the "normal" with . and () and the operator syntax.
Read the other answers to understand what these constructs are. Here is when you should use them. You use them when you need to constrain a method for specific types only.
Here is an example. Suppose you want to define a homogeneous Pair, like this:
class Pair[T](val first: T, val second: T)
Now you want to add a method smaller, like this:
def smaller = if (first < second) first else second
That only works if T is ordered. You could restrict the entire class:
class Pair[T <: Ordered[T]](val first: T, val second: T)
But that seems a shame--there could be uses for the class when T isn't ordered. With a type constraint, you can still define the smaller method:
def smaller(implicit ev: T <:< Ordered[T]) = if (first < second) first else second
It's ok to instantiate, say, a Pair[File], as long as you don't call smaller on it.
In the case of Option, the implementors wanted an orNull method, even though it doesn't make sense for Option[Int]. By using a type constraint, all is well. You can use orNull on an Option[String], and you can form an Option[Int] and use it, as long as you don't call orNull on it. If you try Some(42).orNull, you get the charming message
error: Cannot prove that Null <:< Int
It depends on where they are being used. Most often, when used while declaring types of implicit parameters, they are classes. They can be objects too in rare instances. Finally, they can be operators on Manifest objects. They are defined inside scala.Predef in the first two cases, though not particularly well documented.
They are meant to provide a way to test the relationship between the classes, just like <: and <% do, in situations when the latter cannot be used.
As for the question "when should I use them?", the answer is you shouldn't, unless you know you should. :-) EDIT: Ok, ok, here are some examples from the library. On Either, you have:
/**
* Joins an <code>Either</code> through <code>Right</code>.
*/
def joinRight[A1 >: A, B1 >: B, C](implicit ev: B1 <:< Either[A1, C]): Either[A1, C] = this match {
case Left(a) => Left(a)
case Right(b) => b
}
/**
* Joins an <code>Either</code> through <code>Left</code>.
*/
def joinLeft[A1 >: A, B1 >: B, C](implicit ev: A1 <:< Either[C, B1]): Either[C, B1] = this match {
case Left(a) => a
case Right(b) => Right(b)
}
On Option, you have:
def orNull[A1 >: A](implicit ev: Null <:< A1): A1 = this getOrElse null
You'll find some other examples on the collections.
Noticing that my code was essentially iterating over a list and updating a value in a Map, I first created a trivial helper method which took a function for the transformation of the map value and return an updated map. As the program evolved, it gained a few other Map-transformation functions, so it was natural to turn it into an implicit value class that adds methods to scala.collection.immutable.Map[A, B]. That version works fine.
However, there's nothing about the methods that require a specific map implementation and they would seem to apply to a scala.collection.Map[A, B] or even a MapLike. So I would like it to be generic in the map type as well as the key and value types. This is where it all goes pear-shaped.
My current iteration looks like this:
implicit class RichMap[A, B, MapType[A, B] <: collection.Map[A, B]](
val self: MapType[A, B]
) extends AnyVal {
def updatedWith(k: A, f: B => B): MapType[A, B] =
self updated (k, f(self(k)))
}
This code does not compile because self updated (k, f(self(k))) isa scala.collection.Map[A, B], which is not a MapType[A, B]. In other words, the return type of self.updated is as if self's type was the upper type bound rather than the actual declared type.
I can "fix" the code with a downcast:
def updatedWith(k: A, f: B => B): MapType[A, B] =
self.updated(k, f(self(k))).asInstanceOf[MapType[A, B]]
This does not feel satisfactory because downcasting is a code smell and indicates misuse of the type system. In this particular case it would seem that the value will always be of the cast-to type, and that the whole program compiles and runs correctly with this downcast supports this view, but it still smells.
So, is there a better way to write this code to have scalac correctly infer types without using a downcast, or is this a compiler limitation and a downcast is necessary?
[Edited to add the following.]
My code which uses this method is somewhat more complex and messy as I'm still exploring a few ideas, but an example minimum case is the computation of a frequency distribution as a side-effect with code roughly like this:
var counts = Map.empty[Int, Int] withDefaultValue 0
for (item <- items) {
// loads of other gnarly item-processing code
counts = counts updatedWith (count, 1 + _)
}
There are three answers to my question at the time of writing.
One boils down to just letting updatedWith return a scala.collection.Map[A, B] anyway. Essentially, it takes my original version that accepted and returned an immutable.Map[A, B], and makes the type less specific. In other words, it's still insufficiently generic and sets policy on which types the caller uses. I can certainly change the type on the counts declaration, but that is also a code smell to work around a library returning the wrong type, and all it really does is move the downcast into the caller's code. So I don't really like this answer at all.
The other two are variations on CanBuildFrom and builders in that they essentially iterate over the map to produce a modified copy. One inlines a modified updated method, whereas the other calls the original updated and appends it to the builder and thus appears to make an extra temporary copy. Both are good answers which solve the type correctness problem, although the one that avoids an extra copy is the better of the two from a performance standpoint and I prefer it for that reason. The other is however shorter and arguably more clearly shows intent.
In the case of a hypothetical immutable Map that shares large trees in a similar vein to List, this copying would break the sharing and reduce performance and so it would be preferable to use the existing modified without performing copies. However, Scala's immutable maps don't appear to do this and so copying (once) seems to be the pragmatic solution that is unlikely to make any difference in practice.
Yes! Use CanBuildFrom. This is how the Scala collections library infers the closest collection type to the one you want, using CanBuildFrom evidence. So long as you have implicit evidence of CanBuildFrom[From, Elem, To], where From is the type of collection you're starting with, Elem is the type contained within the collection, and To is the end result you want. The CanBuildFrom will supply a Builder to which you can add elements to, and when you're done, you can call Builder#result() to get the completed collection of the appropriate type.
In this case:
From = MapType[A, B]
Elem = (A, B) // The type actually contained in maps
To = MapType[A, B]
Implementation:
import scala.collection.generic.CanBuildFrom
implicit class RichMap[A, B, MapType[A, B] <: collection.Map[A, B]](
val self: MapType[A, B]
) extends AnyVal {
def updatedWith(k: A, f: B => B)(implicit cbf: CanBuildFrom[MapType[A, B], (A, B), MapType[A, B]]): MapType[A, B] = {
val builder = cbf()
builder ++= self.updated(k, f(self(k)))
builder.result()
}
}
scala> val m = collection.concurrent.TrieMap(1 -> 2, 5 -> 3)
m: scala.collection.concurrent.TrieMap[Int,Int] = TrieMap(1 -> 2, 5 -> 3)
scala> m.updatedWith(1, _ + 10)
res1: scala.collection.concurrent.TrieMap[Int,Int] = TrieMap(1 -> 12, 5 -> 3)
Please note that updated method returns Map class, rather than generic, so I would say you should be fine returning Map as well. But if you really want to return a proper type, you could have a look at implementation of updated in List.updated
I've wrote a small example. I'm not sure it covers all the cases, but it works on my tests. I also used mutable Map, because it was harder for me to test immutable, but I guess it can be easily converted.
implicit class RichMap[A, B, MapType[x, y] <: Map[x, y]](val self: MapType[A, B]) extends AnyVal {
import scala.collection.generic.CanBuildFrom
def updatedWith[R >: B](k: A, f: B => R)(implicit bf: CanBuildFrom[MapType[A, B], (A, R), MapType[A, R]]): MapType[A, R] = {
val b = bf(self)
for ((key, value) <- self) {
if (key != k) {
b += (key -> value)
} else {
b += (key -> f(value))
}
}
b.result()
}
}
import scala.collection.immutable.{TreeMap, HashMap}
val map1 = HashMap(1 -> "s", 2 -> "d").updatedWith(2, _.toUpperCase()) // map1 type is HashMap[Int, String]
val map2 = TreeMap(1 -> "s", 2 -> "d").updatedWith(2, _.toUpperCase()) // map2 type is TreeMap[Int, String]
val map3 = HashMap(1 -> "s", 2 -> "d").updatedWith(2, _.asInstanceOf[Any]) // map3 type is HashMap[Int, Any]
Please also note that CanBuildFrom pattern is much more powerfull and this example doesn't use all of it's power. Thanks to CanBuildFrom some operations can change the type of collection completely like BitSet(1, 3, 5, 7) map {_.toString } type is actually SortedSet[String].
I recently asked Map and reduce/fold over HList of scalaz.Validation and got a great answer as to how to transform a fixed sized tuple of Va[T] (which is an alias for scalaz.Validation[String, T]) into a scalaz.ValidationNel[String, T]. I've since then been studying Shapeless and type level programming in general to try to come up with a solution that works on tuples of any size.
This is what I'm starting out with:
import scalaz._, Scalaz._, shapeless._, contrib.scalaz._, syntax.std.tuple._
type Va[A] = Validation[String, A]
// only works on pairs of Va[_]
def validate[Ret, In1, In2](params: (Va[In1], Va[In2]))(fn: (In1, In2) => Ret) = {
object toValidationNel extends Poly1 {
implicit def apply[T] = at[Va[T]](_.toValidationNel)
}
traverse(params.productElements)(toValidationNel).map(_.tupled).map(fn.tupled)
}
so then validate is a helper I call like this:
val params = (
postal |> nonEmpty[String]("no postal"),
country |> nonEmpty[String]("no country") >=> isIso2Country("invalid country")
)
validate(params) { (postal, country) => ... }
I started out by taking any Product instead of a pair and constraining its contents to Va[T]:
// needs to work with a tuple of Va[_] of arbitrary size
def validateGen[P <: Product, F, L <: HList, R](params: P)(block: F)(
implicit
gen: Generic.Aux[P, L],
va: UnaryTCConstraint[L, Va],
fp: FnToProduct.Aux[F, L => R]
) = ???
I do have the feeling that simply adding the constraint only makes sure the input is valid but doesn't help at all with implementing the body of the function but I don't know how to go about correcting that.
traverse then started complaining about a missing evidence so I ended up with:
def validateGen[P <: Product, F, L <: HList, R](params: P)(block: F)(
implicit
gen: Generic.Aux[P, L],
va: UnaryTCConstraint[L, Va],
tr: Traverser[L, toValidationNel.type],
fp: FnToProduct.Aux[F, L => R]
) = {
traverse(gen.to(params): HList)(toValidationNel).map(_.tupled).map(block.toProduct)
}
The compiler however continued to complain about a missing Traverser[HList, toValidationNel.type] implicit parameter even though it's there.
Which additional evidence do I need to provide to the compiler in order for the traverse call to compile? Has it got to do with the UnaryTCConstraint not being declared in a way that is useful to the traverse call, i.e. it cannot apply toValidationNel to params because it cannot prove that params contains only Va[_]?
P.S. I also found leftReduce Shapeless HList of generic types and tried to use foldRight instead of traverse to no avail; the error messages weren't too helpful when trying to diagnose which evidence the compiler was really lacking.
UPDATE:
As per what lmm has pointed out, I've removed the cast to HList, however, the problem's now that, whereas in the non-generic solution I can call .map(_.tupled).map(block.toProduct) on the result of the traverse call, I'm now getting:
value map is not a member of shapeless.contrib.scalaz.Out
How come it's possible that it was possible on the result of the traverse(params.productElements)(toValidationNel) call and not the generic traverse?
UPDATE 2:
Changing the Traverser[...] bit to Traverser.Aux[..., Va[L]] helped the compiler figure out the expected result type of the traversal, however, this only makes the validateGen function compile successfully but yields another error at the call site:
[error] could not find implicit value for parameter tr: shapeless.contrib.scalaz.Traverser.Aux[L,toValidationNel.type,Va[L]]
[error] validateGen(params) { (x: String :: String :: HNil) => 3 }
[error] ^
I'm also getting the feeling here that the UnaryTCConstraint is completely unnecessary — but I'm still too new to Shapeless to know if that's the case.
UPDATE 3:
Having realized the type that comes out of the traverser cannot be Va[L] because L itself is already a hlist of Va[_], I've split the L type parameter to In and Out:
def validateGen[P <: Product, F, In <: HList, Out <: HList, R](params: P)(block: F)(
implicit
gen: Generic.Aux[P, In],
va: UnaryTCConstraint[In, Va], // just for clarity
tr: Traverser.Aux[In, toValidationNel.type, Va[Out]],
fn: FnToProduct.Aux[F, Out => R]
): Va[R] = {
traverse(gen.to(params))(toValidationNel).map(block.toProduct)
}
this compiles well — I'd be curious to find out how come the previous version with Va[L] being the return value (i.e. the 3rd param to Traverser.Aux) even compiled — however, at the call site, I now get:
Unspecified value parameters tr, fn
You have a Traverser[L, toValidationNel.type] which is not the same thing as Traverser[HList, toValidationNel.type] (which would have to work for any HList - no chance). I don't know why you've written gen.to(params): HList, but this is throwing away type information; shouldn't that be of type L?
This will probably only kick the problem one level higher; I doubt you'll be able to get the Traverser you need automatically. But you should be able to write an implicit method that supplies one based on the UnaryTCConstraint, and it's possible shapeless already includes that and it will Just Work.
Update:
In the first example, the compiler knew the specific Traverser instance it was using, so it knew what the Out type was. In validateGen you haven't constrained anything about tr.Out, so the compiler has no way of knowing that it's a type that supports .map. If you know what the output of the traverse needs to be then you can probably require an appropriate Traverser.Aux i.e.:
tr: Traverser.Aux[L, toValidationNel.type, Va[L]]
(Just don't ask me how to make sure the type inference still works).
I think you probably don't want the .map(_.tupled), because the _ there is already a HList (I suspect it's redundant in the original validate too), but I've never used .toProduct before so maybe you have it right.
Update 2:
Right, this is as I initially suspected. Looking at the implementation of Sequencer I suspect you're right and the UnaryTCConstraint will be subsumed by the Traverser. If you're not using it then no point requiring it.
The only advice I can give is to chase through the calls that should be providing your implicits. E.g. the Traverser should be coming from Traverser.mkTraverser. So if you try calling Traverser.mkTraverser[String :: String :: HNil, toValidationNel.type, Va[String] :: Va[String] :: HNil] then you should be able to see whether it's the Mapper or the Sequencer that can't be found. Then you can recurse through the implicit calls that should happen until you find a simpler case of something that should be working, but isn't.
After long hours of experimentation, frustration and dead brain cells, I've started from scratch without Traverser and instead gone with Mapper and Sequencer; I'll later try to see if I can make it use Traverser again (if not for practicality, at least for learning purposes):
def validate[P <: Product, F, L1 <: HList, L2 <: HList, L3 <: HList, R](params: P)(block: F)(
implicit
gen: Generic.Aux[P, L1],
mp: Mapper.Aux[toValidationNel.type, L1, L2],
seq: Sequencer.Aux[L2, VaNel[L3]],
fn: FnToProduct.Aux[F, L3 => R]
): VaNel[R] = {
sequence(gen.to(params).map(toValidationNel)).map(block.toProduct)
}
Here's proof — pun intended — that it runs http://www.scastie.org/7086.
I can see in the API docs for Predef that they're subclasses of a generic function type (From) => To, but that's all it says. Um, what? Maybe there's documentation somewhere, but search engines don't handle "names" like "<:<" very well, so I haven't been able to find it.
Follow-up question: when should I use these funky symbols/classes, and why?
These are called generalized type constraints. They allow you, from within a type-parameterized class or trait, to further constrain one of its type parameters. Here's an example:
case class Foo[A](a:A) { // 'A' can be substituted with any type
// getStringLength can only be used if this is a Foo[String]
def getStringLength(implicit evidence: A =:= String) = a.length
}
The implicit argument evidence is supplied by the compiler, iff A is String. You can think of it as a proof that A is String--the argument itself isn't important, only knowing that it exists. [edit: well, technically it actually is important because it represents an implicit conversion from A to String, which is what allows you to call a.length and not have the compiler yell at you]
Now I can use it like so:
scala> Foo("blah").getStringLength
res6: Int = 4
But if I tried use it with a Foo containing something other than a String:
scala> Foo(123).getStringLength
<console>:9: error: could not find implicit value for parameter evidence: =:=[Int,String]
You can read that error as "could not find evidence that Int == String"... that's as it should be! getStringLength is imposing further restrictions on the type of A than what Foo in general requires; namely, you can only invoke getStringLength on a Foo[String]. This constraint is enforced at compile-time, which is cool!
<:< and <%< work similarly, but with slight variations:
A =:= B means A must be exactly B
A <:< B means A must be a subtype of B (analogous to the simple type constraint <:)
A <%< B means A must be viewable as B, possibly via implicit conversion (analogous to the simple type constraint <%)
This snippet by #retronym is a good explanation of how this sort of thing used to be accomplished and how generalized type constraints make it easier now.
ADDENDUM
To answer your follow-up question, admittedly the example I gave is pretty contrived and not obviously useful. But imagine using it to define something like a List.sumInts method, which adds up a list of integers. You don't want to allow this method to be invoked on any old List, just a List[Int]. However the List type constructor can't be so constrainted; you still want to be able to have lists of strings, foos, bars, and whatnots. So by placing a generalized type constraint on sumInts, you can ensure that just that method has an additional constraint that it can only be used on a List[Int]. Essentially you're writing special-case code for certain kinds of lists.
Not a complete answer (others have already answered this), I just wanted to note the following, which maybe helps to understand the syntax better: The way you normally use these "operators", as for example in pelotom's example:
def getStringLength(implicit evidence: A =:= String)
makes use of Scala's alternative infix syntax for type operators.
So, A =:= String is the same as =:=[A, String] (and =:= is just a class or trait with a fancy-looking name). Note that this syntax also works with "regular" classes, for example you can write:
val a: Tuple2[Int, String] = (1, "one")
like this:
val a: Int Tuple2 String = (1, "one")
It's similar to the two syntaxes for method calls, the "normal" with . and () and the operator syntax.
Read the other answers to understand what these constructs are. Here is when you should use them. You use them when you need to constrain a method for specific types only.
Here is an example. Suppose you want to define a homogeneous Pair, like this:
class Pair[T](val first: T, val second: T)
Now you want to add a method smaller, like this:
def smaller = if (first < second) first else second
That only works if T is ordered. You could restrict the entire class:
class Pair[T <: Ordered[T]](val first: T, val second: T)
But that seems a shame--there could be uses for the class when T isn't ordered. With a type constraint, you can still define the smaller method:
def smaller(implicit ev: T <:< Ordered[T]) = if (first < second) first else second
It's ok to instantiate, say, a Pair[File], as long as you don't call smaller on it.
In the case of Option, the implementors wanted an orNull method, even though it doesn't make sense for Option[Int]. By using a type constraint, all is well. You can use orNull on an Option[String], and you can form an Option[Int] and use it, as long as you don't call orNull on it. If you try Some(42).orNull, you get the charming message
error: Cannot prove that Null <:< Int
It depends on where they are being used. Most often, when used while declaring types of implicit parameters, they are classes. They can be objects too in rare instances. Finally, they can be operators on Manifest objects. They are defined inside scala.Predef in the first two cases, though not particularly well documented.
They are meant to provide a way to test the relationship between the classes, just like <: and <% do, in situations when the latter cannot be used.
As for the question "when should I use them?", the answer is you shouldn't, unless you know you should. :-) EDIT: Ok, ok, here are some examples from the library. On Either, you have:
/**
* Joins an <code>Either</code> through <code>Right</code>.
*/
def joinRight[A1 >: A, B1 >: B, C](implicit ev: B1 <:< Either[A1, C]): Either[A1, C] = this match {
case Left(a) => Left(a)
case Right(b) => b
}
/**
* Joins an <code>Either</code> through <code>Left</code>.
*/
def joinLeft[A1 >: A, B1 >: B, C](implicit ev: A1 <:< Either[C, B1]): Either[C, B1] = this match {
case Left(a) => a
case Right(b) => Right(b)
}
On Option, you have:
def orNull[A1 >: A](implicit ev: Null <:< A1): A1 = this getOrElse null
You'll find some other examples on the collections.