Why do some Scala methods use polymorphic arguments instead of using Any/Nothing? - scala

For example, Exception.allCatch is defined as
def allCatch[T]: Catch[T]
Why not just
val allCatch: Catch[Nothing]
when Catch is covariant in its argument?
Or, why PartialFunction object defines
def empty[A, B]: PartialFunction[A, B]
instead of just
val empty: PartialFunction[Any,Nothing]
?
Update: So far it seems that the answers miss the point. So please include a specific examples in your answer that really target the question. For example: Show a piece of code that works with def empty[A, B]: PartialFunction[A, B] but doesn't work (or is less convenient) with val empty: PartialFunction[Any,Nothing].

This saves the need for casting later and allows to treat the args type as T instead of Any, which is usually more convenient.
Here is an example:
scala> def func1[T](arg : T) : T = { arg }
func1: [T](arg : T)T
scala> def func2(arg : Any) : Any = { arg }
func2: (arg: Any)Any
scala> func1(4)
res4: Int = 4
scala> func2(4)
res7: Any = 4

Looking at the source of PartialFunction available here we can see that it in fact calls a private method on the PartialFunction object:
private[this] val empty_pf: PartialFunction[Any, Nothing]
So the return type of empty will always be PartialFunction[Any, Nothing]. As for the reasoning behind I have no idea. Maybe someone else have better insight as to why. You could try the language mailing list as well...

If you hard-code the type, like PartialFunction[Any,Nothing], you cannot restrict your function to take a more specific parameter than Any.
By using a generic type parameter, you can end up with a more flexible satisfying all cases and especially making the function safe.
Let's assume you want a function aiming to take an Animal as parameter and returning an Integer.
Let's assume that function is declared as being:
def myFunction: PartialFunction[Any,Nothing]
Firstly, PartialFunction would not be specialized to Animal at parameter side but to Any. What about if I pass a Human as parameter...., it would pass.. What about safety?
Secondly, If this function is declared as returning Nothing, you can't return from it any value but Nothing! Indeed, Nothing subclasses all classes in Scala.
This leads to the known rule that return type parameter must always be covariant in order to make a function interesting, not the case with Nothing.
In fact, Nothing is interesting only when dealing with the empty method of PartialFunction. Logic since an empty PartialFunction by definition involves nothing to return and should be forced to do it :)
You would ask: "So why don't we change the return type to Any?"
Answer: Because you'd lose all the benefit of generic erasure time making compiler to add needed casts automatically => You wouldn't retrieve directly the Integer value, but Any. annoying..

Actually, it seems that Scala standard library has some more places where generic type parameter is redundant (because of type variance). For example, see my question about foreach. My guess, based on #Peter's answer, is that these redundant generics make the interface more clear. Thanks to them, we don't have to remember which types are covariant, contravariant and invariant. Also, this makes things way simpler for people who are not familiar with variance, which is rather an advanced feature of Scala.

Related

How does type class resolution in scala work?

I have a function with a type parameter and I want to find out whether the type parameter is an Option or not. I have read some blogposts, i.e. this one, about type classes in scala recently, so I came up with this solution:
case class OptionFinder[A](isOption: Boolean)
implicit def notOption[A]: OptionFinder[A] = OptionFinder(false)
implicit def hitOption[A]: OptionFinder[Option[A]] = OptionFinder(true)
def myFunction[A](value: A)(implicit optionFinder: OptionFinder[A]): String = {
if (optionFinder.isOption) {"Found Option!"} else {"Found something else."}
}
This works seemingly as desired:
scala> val x: Option[Int] = Some(3)
scala> myFunction(x)
res0: String = Found Option!
scala> val y: String = "abc"
scala> myFunction(y)
res1: String = Found something else.
In the case of Some(3) hitOption is the implicit parameter, even though notOption would match as well (with A = Option[Int]). Obviously the more specific is type chosen. But am I guaranteed that the compiler always chooses the more specific type? And how does that work in the compiler anyway? I did not find a documentation of this behavior yet.
Note: Maybe the title for this question is not best, I'll happily change it for a better one.
There is already a question about this: Scala: Implicit parameter resolution precedence. Which answers itself through a complicated blog post. I think the most important piece of information is in Martin Odersky's comment on the blog post:
Here's a more high-level explanation what goes on with implicit search
in Scala, and which corresponds to how the spec explains it, but in
slightly less formalistic language.
First, we look for implicits that are visible either as locals or as members of enclosing classes and packages or as imports - the
precise rule is that we should be able to access them using their name
only, without any prefix.
If no implicits are found in step 1, we look in the "implicit scope", which contains all sort of companion objects that bear some
relation to the type which we search for (i.e. companion object of the
type itself, of its parameters if any are given, and also of its
supertype and supertraits; the importance is to be as general as
possible without reverting to whole program analysis like Haskell
does).
If at either stage we find more than one implicit, disambiguation
kicks in. Disambiguation is exactly the same as for overloading
resolution. Static overloading resolution resolution rules are a bit
involved, and I won't repeat them here. If it's any consolation:
Java's rules and C#'s rules are considerably more complex than Scala's
in this area.
Now according to this explanation it are "the rules of static overloading resolution" which will disambiguate between notOption and hitOption. To be honest, I fail to see how.
This answer explains that indeed methods with more specific arguments have priority, but I don't know if or how that is related to the overloading rules.
If I were you I would not depend on this behavior too much, but use the easier to understand concept of implicit priority through inheritance. It's a good idea to put your implicits in the companion object anyway.
It boils down to the fact that implicits that are inherited have lower priority. So it's safe to put the implicit you fall back to if hitOption doesn't match in a trait that the companion object extends.
case class OptionFinder[A](isOption: Boolean)
object OptionFinder extends LowerPriority {
implicit def hitOption[A]: OptionFinder[Option[A]] = OptionFinder(true)
}
trait LowerPriority {
implicit def notOption[A]: OptionFinder[A] = OptionFinder(false)
}
def myFunction[A](value: A)(implicit optionFinder: OptionFinder[A]): String = {
if (optionFinder.isOption) {"Found Option!"} else {"Found something else."}
}
This should also work if you put your implicits in a non companion object MyImplicits and import them with import MyImplicits._.

Why do we have to explicitly specify the ClassTag typeclass

Now that scala has iterated towards a JVM type erasure fix with the ClassTag typeclass, why is it an opt-in, rather than having the compiler always capture the type signature for runtime inspection. Having it an implicit parametrized type constraint would make it possible to call classTag[T] regardless of the generic parameter declaration.
EDIT: I should clarify that I don't mean that scala should change the signature behind the scenes to always inlcude ClassTag. Rather I mean that since ClassTag shows that scala can capture runtime Type information and therefore avoid type erasure limitations, why can't that capture be implicit as part of the compiler so that that information is always available in scala code?
My suspicion is that it's backwards compatibility, java ecosystem compatibility, binary size or runtime overhead related, but those are just speculation.
Backwards compatibility would be completely destroyed, really. If you have a simple method like:
def foo[A](a: A)(implicit something: SomeType) = ???
Then suppose that in the next version of Scala, the compiler suddenly added implicit ClassTags to the signature of all methods with type parameters. This method would be broken. Anywhere it was being explicitly called like foo(a)(someTypeValue) wouldn't work anymore. Binary and source compatibility would be gone.
Java interoperability would be ugly. Supposing our method now looks like this:
def foo[A : ClassTag](a: A) = ???
Because ClassTags are generated by the Scala compiler, using this method from Java would prove more difficult. You'd have to create the ClassTag yourself.
ClassTag<MyClass> tag = scala.reflect.ClassTag$.MODULE$.apply(MyClass.class);
foo(a, tag);
My Java might not be 100% correct, but you get the idea. Anything parameterized would become very ugly. Well, it already is if it requires an implicit ClassTag, but the class of methods where this would be necessary would increase dramatically.
Moreover, type erasure isn't that much of a problem in most parameterized methods we (I, at least) use. I think automatically requiring a ClassTag for each type parameter would be far more trouble than it would help, for the above reasons.
Certainly this would add more compiler overhead, as it would need to generate more ClassTags than it usually would. I don't think it would add much more runtime overhead unless the ClassTag makes a difference. For example, in a simple method like below, the ClassTag doesn't really do anything:
def foo[A : ClassTag](a: A): A = a
We should also note that they're not perfect, either. So adding them isn't an end-all solution to erasure problems.
val list = List(1, "abc", List(1, 2, 3), List("a", "b"))
def find[A: ClassTag](l: List[Any]): Option[A] =
l collectFirst { case a: A => a }
scala> find[List[String]]
res2: Option[List[String]] = Some(List(1, 2, 3)) // Not quite! And no warnings, either.
Adding a ClassTag to every single class instance would add overhead, and surely also break compatibility. It's also in many places not possible. We can't just infuse java.lang.String with a ClassTag. Furthermore, we'd still be just as susceptible to erasure. Having a ClassTag field in each class is really no better than using getClass. We could make comparisons like
case a if(a.getClass == classOf[String]) => a.asInstanceOf[String]
But this is horribly ugly, requires a cast, and isn't necessarily what the ClassTag is meant to fix. If I tried something like this with my find method, it wouldn't work--at all.
// Can't compile
def find[A](l: List[Any]): Option[A] =
l collectFirst { case a if(a.getClass == classOf[A]) => a.asInstanceOf[A] }
Even if I were to fashion this to somehow work with ClassTag, where would it come from? I could not say a.classTag == classTag[A], because A has already been erased. I need the ClassTag at the method call site.
Yes, you would penalise any generic method or class for a use case that is quite seldom (e.g. requiring array construction or heterogeneous map value recovery). With this idea of "always class tags" you would also effectively destroy the possibility to call Scala code from Java. In summery, it simply doesn't make any sense to require a class tag to be always present, neither from compatibility, performance or class size point of view.

Deciphering one of the toughest scala method prototypes (slick)

Looking at the <> method in the following scala slick class, from http://slick.typesafe.com/doc/2.1.0/api/index.html#scala.slick.lifted.ToShapedValue, it reminds me of that iconic stackoverflow thread about scala prototypes.
def <>[R, U](f: (U) ⇒ R, g: (R) ⇒ Option[U])
(implicit arg0: ClassTag[R], shape: Shape[_ <: FlatShapeLevel, T, U, _]):
MappedProjection[R, U]
Can someone bold and knowledgeable provide an articulate walkthrough of that long prototype definition, carefully clarifying all of its type covariance/invariance, double parameter lists, and other advanced scala aspects?
This exercise will also greatly help dealing with similarly convoluted prototypes!
Ok, let's take a look:
class ToShapedValue[T](val value: T) extends AnyVal {
...
#inline def <>[R: ClassTag, U](f: (U) ⇒ R, g: (R) ⇒ Option[U])(implicit shape: Shape[_ <: FlatShapeLevel, T, U, _]): MappedProjection[R, U]
}
The class is an AnyVal wrapper; while I can't actually see the implicit conversion from a quick look, it smells like the "pimp my library" pattern. So I'm guessing this is meant to add <> as an "extension method" onto some (or maybe all) types.
#inline is an annotation, a way of putting metadata on, well, anything; this one is a hint to the compiler that this should be inlined. <> is the method name - plenty of things that look like "operators" are simply ordinary methods in scala.
The documentation you link has already expanded the R: ClassTag to ordinary R and an implicit ClassTag[R] - this is a "context bound" and it's simply syntactic sugar. ClassTag is a compiler-generated thing that exists for every (concrete) type and helps with reflection, so this is a hint that the method will probably do some reflection on an R at some point.
Now, the meat: this is a generic method, parameterized by two types: [R, U]. Its arguments are two functions, f: U => R and g: R => Option[U]. This looks a bit like the functional Prism concept - a conversion from U to R that always works, and a conversion from R to U that sometimes doesn't work.
The interesting part of the signature (sort of) is the implicit shape at the end. Shape is described as a "typeclass", so this is probably best thought of as a "constraint": it limits the possible types U and R that we can call this function with, to only those for which an appropriate Shape is available.
Looking at the documentation forShape, we see that the four types are Level, Mixed, Unpacked and Packed. So the constraint is: there must be a Shape, whose "level" is some subtype of FlatShapeLevel, where the Mixed type is T and the Unpacked type is R (the Packed type can be any type).
So, this is a type-level function that expresses that R is "the unpacked version of" T. To use the example from the Shape documentation again, if T is (Column[Int], Column[(Int, String)], (Int, Option[Double])) then R will be (Int, (Int, String), (Int, Option[Double]) (and it only works for FlatShapeLevel, but I'm going to make a judgement call that that's probably not important). U is, interestingly enough, completely unconstrained.
So this lets us create a MappedProjection[unpacked-version-of-T, U] from any T, by providing conversion functions in both directions. So in a simple version, maybe T is a Column[String] - a representation of a String column in a database - and we want to represent it as some application-specific type, e.g. EmailAddress. So R=String, U=EmailAddress, and we provide conversion functions in both directions: f: EmailAddress => String and g: String => Option[EmailAddress]. It makes sense that it's this way around: every EmailAddress can be represented as a String (at least, they'd better be, if we want to be able to store them in the database), but not every String is a valid EmailAddress. If our database somehow had e.g. "http://www.foo.com/" in the email address column, our g would return None, and Slick could handle this gracefully.
MappedProjection itself is, sadly, undocumented. But I'm guessing it's some kind of lazy representation of a thing we can query; where we had a Column[String], now we have a pseudo-column-thing whose (underlying) type is EmailAddress. So this might allow us to write pseudo-queries like 'select from users where emailAddress.domain = "gmail.com"', which would be impossible to do directly in the database (which doesn't know which part of an email address is the domain), but is easy to do with the help of code. At least, that's my best guess at what it might do.
Arguably the function could be made clearer by using a standard Prism type (e.g. the one from Monocle) rather than passing a pair of functions explicitly. Using the implicit to provide a type-level function is awkward but necessary; in a fully dependently typed language (e.g. Idris), we could write our type-level function as a function (something like def unpackedType(t: Type): Type = ...). So conceptually, this function looks something like:
def <>[U](p: Prism[U, unpackedType(T)]): MappedProjection[unpackedType(T), U]
Hopefully this explains some of the thought process of reading a new, unfamiliar function. I don't know Slick at all, so I have no idea how accurate I am as to what this <> is used for - did I get it right?

Inject methods into existing classes

I want to come out a way to define a new method in some existing class in scala.
For example, I think the asInstanceOf[T] method has too long a name, I want to replace it with as[T].
A straight forward approach can be:
class WrappedAny(val a: Any) {
def as[T] = a.asInstanceOf[T]
}
implicit def wrappingAny(a: Any): WrappedAny = new WrappedAny(a)
Is there a more natural way with less code?
Also, a strange thing happens when I try this:
scala> class A
defined class A
scala> implicit def toA(x: Any): A = x
toA: (x: Any)A
scala> toA(1)
And the console hang. It seems that toA(Any) should not pass the type checking phase, and it can't when it's not implicit. And putting all the code into a external source code can produce the same problem. How did this happen? Is it a bug of the compiler(version 2.8.0)?
There's nothing technically wrong with your approach to pimping Any, although I think it's generally ill-advised. Likewise, there's a reason asInstanceOf and isInstanceOf are so verbosely named; it's to discourage you from using them! There's almost certainly a better, statically type-safe way to do whatever you're trying to do.
Regarding the example which causes your console to hang: the declared type of toA is Any => A, yet you've defined its result as x, which has type Any, not A. How can this possibly compile? Well, remember that when an apparent type error occurs, the compiler looks around for any available implicit conversions to resolve the problem. In this case, it needs an implicit conversion Any => A... and finds one: toA! So the reason toA type checks is because the compiler is implicitly redefining it as:
implicit def toA(x: Any): A = toA(x)
... which of course results in infinite recursion when you try to use it.
In your second example you are passing Any to a function that must return A. However it never returns A but the same Any you passed in. The compiler then tries to apply the implicit conversion which in turn does not return an A but Any, and so on.
If you define toA as not being implicit you get:
scala> def toA(x: Any): A = x
<console>:6: error: type mismatch;
found : Any
required: A
def toA(x: Any): A = x
^
As it happens, this has been discussed on Scala lists before. The pimp my class pattern is indeed a bit verbose for what it does, and, perhaps, there might be a way to clean the syntax without introducing new keywords.
The bit about new keywords is that one of Scala goals is to make the language scalable through libraries, instead of turning the language into a giant quilt of ideas that passed someone's criteria for "useful enough to add to the language" and, at the same time, making other ideas impossible because they weren't deemed useful and/or common enough.
Anyway, nothing so far has come up, and I haven't heard that there is any work in progress towards that goal. You are welcome to join the community through its mailing lists and contribute to its development.

What is a Manifest in Scala and when do you need it?

Since Scala 2.7.2 there is something called Manifest which is a workaround for Java's type erasure. But how does Manifest work exactly and why / when do you need to use it?
The blog post Manifests: Reified Types by Jorge Ortiz explains some of it, but it doesn't explain how to use it together with context bounds.
Also, what is ClassManifest, what's the difference with Manifest?
I have some code (part of a larger program, can't easily include it here) that has some warnings with regard to type erasure; I suspect I can solve these by using manifests, but I'm not sure exactly how.
The compiler knows more information about types than the JVM runtime can easily represent. A Manifest is a way for the compiler to send an inter-dimensional message to the code at runtime about the type information that was lost.
It isn't clear if a Manifest would benefit the errors you are seeing without knowing more detail.
One common use of Manifests is to have your code behave differently based on the static type of a collection. For example, what if you wanted to treat a List[String] differently from other types of a List:
def foo[T](x: List[T])(implicit m: Manifest[T]) = {
if (m <:< manifest[String])
println("Hey, this list is full of strings")
else
println("Non-stringy list")
}
foo(List("one", "two")) // Hey, this list is full of strings
foo(List(1, 2)) // Non-stringy list
foo(List("one", 2)) // Non-stringy list
A reflection-based solution to this would probably involve inspecting each element of the list.
A context bound seems most suited to using type-classes in scala, and is well explained here by Debasish Ghosh:
http://debasishg.blogspot.com/2010/06/scala-implicits-type-classes-here-i.html
Context bounds can also just make the method signatures more readable. For example, the above function could be re-written using context bounds like so:
def foo[T: Manifest](x: List[T]) = {
if (manifest[T] <:< manifest[String])
println("Hey, this list is full of strings")
else
println("Non-stringy list")
}
A Manifest was intended to reify generic types that get type-erased to run on the JVM (which does not support generics). However, they had some serious issues: they were too simplistic, and were unable to fully support Scala's type system. They were thus deprecated in Scala 2.10, and are replaced with TypeTags (which are essentially what the Scala compiler itself uses to represent types, and therefore fully support Scala types). For more details on the difference, see:
Scala: What is a TypeTag and how do I use it?
How do the new Scala TypeTags improve the (deprecated) Manifests?
In other words
when do you need it?
Before 2013-01-04, when Scala 2.10 was released.
Not a complete answer, but regarding the difference between Manifest and ClassManifest, you can find an example in the Scala 2.8 Array paper:
The only remaining question is how to implement generic array creation. Unlike Java, Scala allows an instance creation new Array[T] where T is a type parameter. How can this be implemented, given the fact that there does not exist a uniform array representation in Java?
The only way to do this is to require additional runtime information which describes the type T. Scala 2.8 has a new mechanism for this, which is called a Manifest. An object of type Manifest[T] provides complete information about the type T.
Manifest values are typically passed in implicit parameters; and the compiler knows how to construct them for statically known types T.
There exists also a weaker form named ClassManifest which can be constructed from knowing just the top-level class of a type, without necessarily knowing all its argument types.
It is this type of runtime information that’s required for array creation.
Example:
One needs to provide this information by passing a ClassManifest[T] into the
method as an implicit parameter:
def tabulate[T](len:Int, f:Int=>T)(implicit m:ClassManifest[T]) = {
val xs = new Array[T](len)
for (i <- 0 until len) xs(i) = f(i)
xs
}
As a shorthand form, a context bound1 can be used on the type parameter T instead,
(See this SO question for illustration)
, giving:
def tabulate[T: ClassManifest](len:Int, f:Int=>T) = {
val xs = new Array[T](len)
for (i <- 0 until len) xs(i) = f(i)
xs
}
When calling tabulate on a type such as Int, or String, or List[T], the Scala compiler can create a class manifest to pass as implicit argument to tabulate.
Let's also chck out manifest in scala sources (Manifest.scala), we see:
Manifest.scala:
def manifest[T](implicit m: Manifest[T]) = m
So with regards to following example code:
def foo[A](somelist: List[A])(implicit m: Manifest[A]): String = {
if (m <:< manifest[String]) {
"its a string"
} else {
"its not a string"
}
}
we can see that the manifest function searches for an implicit m: Manifest[T] which satisfies the type parameter you provide in our example code it was manifest[String]. So when you call something like:
if (m <:< manifest[String]) {
you are checking if the current implicit m which you defined in your function is of type manifest[String] and as the manifest is a function of type manifest[T] it would search for a specific manifest[String] and it would find if there is such an implicit.