I work with Spark often, and it would save me a lot of time if the compiler could ensure that a type is serializable.
Perhaps with a type class?
def foo[T: IsSerializable](t: T) = {
// do stuff requiring T to be serializable
}
It's not enough to constrain T <: Serializable. It could still fail at runtime. Unit tests are a good substitute, but you can still forget them, especially when working with big teams.
I think this is probably impossible to do at compile time without the types being sealed.
Yes, it is possible, but not in the way that you're hoping. Your type class IsSerializable could provide a mechanism to convert your T to a value of a type which is guaranteed to be Serializable and back again,
trait IsSerializable[T] {
def toSerializable(t: T): String
def fromSerializable(s: String): Option[T]
}
But, of course, this is just an alternative type class based serialization mechanism in it's own right, making the use of JVM serialization redundant.
Your best course of action would be to lobby Spark to support type class based serialization directly.
I am new with scala. I am trying to figure out how the whole contravariance relationship works. I understand the concept of covariance and invariant and I also know how I would implement them in practise.
I also understand the concept of contravariance (the reverse of covariance) and how it is implemented in the Function1 trait in Scala. It gives you an abstraction without redefining Function1 implementations for different classes. But, I still don’t get it completely, strange? Now, I am almost there… how can I solve the following problem with contravariance:
class GarbageCan[-A] {
def doSomething(a: A): Unit ={
// do something with 'a' of subtype that is not possible with the supertype
}
}
def setGarbageCanForPlastic(gc: GarbageCan[PlasticItem]): Unit = {
}
The above example is extracted from http://blog.kamkor.me/Covariance-And-Contravariance-In-Scala/. A really good explanation concerning this subject. The hierarchy is as follows: Item (base class) -> PlasticItem (subclass) -> PlasticBottle (subclass of subclass)
The setGarbageCanForPlastic function accepts a GarbageCan with the type of PlasticItem. Because the Parameterized type is contravariant, the following statement is completely legal:
setGarbageCanForPlastic(new GarbageCan[Item])
Now, the doSomething function accepts a Type parameter which is contravariant. How can I work with this type, if I don’t know if the type is a Base class “Item” or subclass “PlasticItem”? I can do something which is permissible within the subclass and not in the base class. If this was an covariant parameter, this would be no problem, a subclass inherits everything from the base class.
I lost it right?... Hope somebody can help me out.
First of all, the doSomething method actually cannot do anything other than essentially discarding a since it does not know what A is. To make it more useful, you would need a bound like class GarbageCan[-A <: Item].
Now, suppose setGarbageCanForPlastic(gc) calls gc.doSomething(new PlasticItem). Since A in GarbageCan[A] is contravariant, we have GarbageCan[Item] <: GarbageCan[PlasticItem] <: GarbageCan[PlasticBottle]. Indeed, the function call setGarbageCanForPlastic(new GarbageCan[Item])) is safe as GarbageCan[Item]'s doSomething can process any Item including a PlasticItem, while the call setGarbageCanForPlastic(new GarbageCan[PlasticBottle])) is unsafe because GarbageCan[PlasticBottle]'s doSomething may not be able to take a PlasticItem which is not necessarily a PlasticBottle.
What is the different between the following Generics definitions in Scala:
class Foo[T <: List[_]]
and
class Bar[T <: List[Any]]
My gut tells me they are about the same but that the latter is more explicit. I am finding cases where the former compiles but the latter doesn't, but can't put my finger on the exact difference.
Thanks!
Edit:
Can I throw another into the mix?
class Baz[T <: List[_ <: Any]]
OK, I figured I should have my take on it, instead of just posting comments. Sorry, this is going to be long, if you want the TL;DR skip to the end.
As Randall Schulz said, here _ is a shorthand for an existential type. Namely,
class Foo[T <: List[_]]
is a shorthand for
class Foo[T <: List[Z] forSome { type Z }]
Note that contrary to what Randall Shulz's answer mentions (full disclosure: I got it wrong too in an earlier version of this post, thanks to Jesper Nordenberg for pointing it out) this not the same as:
class Foo[T <: List[Z]] forSome { type Z }
nor is it the same as:
class Foo[T <: List[Z forSome { type Z }]]
Beware, it is easy to get it wrong (as my earlier goof shows): the author of the article referenced by Randall Shulz's answer got it wrong himself (see comments), and fixed it later. My main problem with this article is that in the example shown, the use of existentials is supposed to save us from a typing problem, but it does not. Go check the code, and try to compile compileAndRun(helloWorldVM("Test")) or compileAndRun(intVM(42)). Yep, does not compile. Simply making compileAndRun generic in A would make the code compile, and it would be much simpler.
In short, that's probably not the best article to learn about existentials and what they are good for (the author himself acknowledge in a comment that the article "needs tidying up").
So I would rather recommend reading this article: http://www.artima.com/scalazine/articles/scalas_type_system.html, in particular the sections named "Existential types" and "Variance in Java and Scala".
The important point that you hould get from this article is that existentials are useful (apart from being able to deal with generic java classes) when dealing with non-covariant types.
Here is an example.
case class Greets[T]( private val name: T ) {
def hello() { println("Hello " + name) }
def getName: T = name
}
This class is generic (note also that is is invariant), but we can see that hello really doesn't make any use of the type parameter (unlike getName), so if I get an instance of Greets I should always be able to call it, whatever T is. If I want to define a method that takes a Greets instance and just calls its hello method, I could try this:
def sayHi1( g: Greets[T] ) { g.hello() } // Does not compile
Sure enough, this does not compile, as T comes out of nowhere here.
OK then, let's make the method generic:
def sayHi2[T]( g: Greets[T] ) { g.hello() }
sayHi2( Greets("John"))
sayHi2( Greets('Jack))
Great, this works. We could also use existentials here:
def sayHi3( g: Greets[_] ) { g.hello() }
sayHi3( Greets("John"))
sayHi3( Greets('Jack))
Works too. So all in all, there is no real benefit here from using an existential (as in sayHi3) over type parameter (as in sayHi2).
However, this changes if Greets appears itself as a type parameter to another generic class. Say by example that we want to store several instances of Greets (with different T) in a list. Let's try it:
val greets1: Greets[String] = Greets("John")
val greets2: Greets[Symbol] = Greets('Jack)
val greetsList1: List[Greets[Any]] = List( greets1, greets2 ) // Does not compile
The last line does not compile because Greets is invariant, so a Greets[String] and Greets[Symbol] cannot be treated as a Greets[Any] even though String and Symbol both extends Any.
OK, let's try with an existential, using the shorthand notation _:
val greetsList2: List[Greets[_]] = List( greets1, greets2 ) // Compiles fine, yeah
This compiles fine, and you can do, as expected:
greetsSet foreach (_.hello)
Now, remember that the reason we had a type checking problem in the first place was because Greets is invariant. If it was turned into a covariant class (class Greets[+T]) then everything would have worked out of the box and we would never have needed existentials.
So to sum up, existentials are useful to deal with generic invariant classes, but if the generic class does not need to appear itself as a type parameter to another generic class, chances are that you don't need existentials and simply adding a type parameter to your method will work
Now come back(at last, I know!) to your specific question, regarding
class Foo[T <: List[_]]
Because List is covariant, this is for all intents and purpose the same as just saying:
class Foo[T <: List[Any]]
So in this case, using either notation is really just a matter of style.
However, if you replace List with Set, things change:
class Foo[T <: Set[_]]
Set is invariant and thus we are in the same situation as with the Greets class from my example. Thus the above really is very different from
class Foo[T <: Set[Any]]
The former is a shorthand for an existential type when the code doesn't need to know what the type is or constrain it:
class Foo[T <: List[Z forSome { type Z }]]
This form says that the element type of List is unknown to class Foo rather than your second form, which says specifically that the List's element type is Any.
Check out this brief explanatory blog article on Existential Types in Scala (EDIT: this link is now dead, a snapshot is available at archive.org)
Because traits with representation types are self-referential, declaring that a variable holds an instance of that trait is a little difficult. In this example I simply declare that a variable holds an instance of the trait, declare that a function takes and returns and instance of that trait, and call that function with the variable:
trait Foo[+A <: Foo[A]]
case class Bar() extends Foo[Bar]
case class Grill() extends Foo[Grill]
// Store a generic instance of Foo
val b: Foo[_] = if(true) {
Bar()
} else {
Grill()
}
// Declare a function that take any Foo and returns a Foo of the same type
// that "in" has in the calling context
def echoFoo[A <: Foo[A]](in: A): A = in
// Call said function
val echo = echoFoo(b)
It fails with the error:
inferred type arguments [this.Foo[_$1]] do not conform to method
echoFoo's type parameter bounds [A <: this.Foo[A]]
val echo = echoFoo(b)
^
Now, this makes sense because [_] is like Any (in ways I don't fully understand). What it probably wants is something like Foo[Foo[_]], so that the type parameter conforms to the bounds of A <: Foo[A]. But now there's an inner Foo that has a non-conforming type parameter, suggesting that the solution is something like Foo[Foo[Foo[Foo[..., which is clearly not correct.
So my question can probably be distilled down to: What is the Scala syntax for "This variable holds any legal Foo"?
Self-referential type parameters like this are a bit problematic, because they're not sound. For example, it's possible to define a type like the following:
case class BeerGarden extends Foo[Grill]
As you can see, the A <: Foo[A] bound isn't sufficiently tight. What I prefer in situations like this is to use the cake pattern, and abstract type members:
trait FooModule {
type Foo <: FooLike
def apply(): Foo
trait FooLike {
def echo: Foo
}
}
Now you can use the Foo type recursively and safely:
object Foos {
def echo(foo: FooModule#Foo) = foo.echo
}
Obviously, this isn't an ideal solution to all the problems you might want to solve with such types, but the important observation is that FooLike is an extensible trait, so you can always continue to refine FooLike to add the members that you need, without violating the bound that the type member is intended to enforce. I've found that in every real-world case where the set of types I want to represent is not closed, this is about the best that one can do. The important thing to see is that FooModule abstracts over both the type and the instance constructor, while enforcing the "self-type." You can't abstract over one without abstracting over the other.
Some additional information on this sort of thing (and a bit of a record of my own early struggles with recursive types) is available here:
https://issues.scala-lang.org/browse/SI-2385
While I agree the problem of propagating generics exists, when you hit this problem you should see a big WARNING on your screen because its typically a sign of a bad design. These are general suggestions on the topic.
If you use generics, the type parameter is there for a reason. It lets you interact with a Foo[A] in a type-safe manner by passing in or receiving parameters of type A and allows to put you constraint on A. If you lose the type information, you lose the type-safety and in that case so there is no point of writing a generic class if you do not need the generic anymore: you can change all your signatures to Any and do pattern matching.
In most of the cases, recursive types can be avoided by implementing something like the CanBuildFrom approach for collections, using a "typeclass"
Finally,type-projection (FooModule#Foo) has little application and you might want to look to path-dependent types. However, these have little application as well.
As I understand from this blog post "type classes" in Scala is just a "pattern" implemented with traits and implicit adapters.
As the blog says if I have trait A and an adapter B -> A then I can invoke a function, which requires argument of type A, with an argument of type B without invoking this adapter explicitly.
I found it nice but not particularly useful. Could you give a use case/example, which shows what this feature is useful for ?
One use case, as requested...
Imagine you have a list of things, could be integers, floating point numbers, matrices, strings, waveforms, etc. Given this list, you want to add the contents.
One way to do this would be to have some Addable trait that must be inherited by every single type that can be added together, or an implicit conversion to an Addable if dealing with objects from a third party library that you can't retrofit interfaces to.
This approach becomes quickly overwhelming when you also want to begin adding other such operations that can be done to a list of objects. It also doesn't work well if you need alternatives (for example; does adding two waveforms concatenate them, or overlay them?) The solution is ad-hoc polymorphism, where you can pick and chose behaviour to be retrofitted to existing types.
For the original problem then, you could implement an Addable type class:
trait Addable[T] {
def zero: T
def append(a: T, b: T): T
}
//yup, it's our friend the monoid, with a different name!
You can then create implicit subclassed instances of this, corresponding to each type that you wish to make addable:
implicit object IntIsAddable extends Addable[Int] {
def zero = 0
def append(a: Int, b: Int) = a + b
}
implicit object StringIsAddable extends Addable[String] {
def zero = ""
def append(a: String, b: String) = a + b
}
//etc...
The method to sum a list then becomes trivial to write...
def sum[T](xs: List[T])(implicit addable: Addable[T]) =
xs.FoldLeft(addable.zero)(addable.append)
//or the same thing, using context bounds:
def sum[T : Addable](xs: List[T]) = {
val addable = implicitly[Addable[T]]
xs.FoldLeft(addable.zero)(addable.append)
}
The beauty of this approach is that you can supply an alternative definition of some typeclass, either controlling the implicit you want in scope via imports, or by explicitly providing the otherwise implicit argument. So it becomes possible to provide different ways of adding waveforms, or to specify modulo arithmetic for integer addition. It's also fairly painless to add a type from some 3rd-party library to your typeclass.
Incidentally, this is exactly the approach taken by the 2.8 collections API. Though the sum method is defined on TraversableLike instead of on List, and the type class is Numeric (it also contains a few more operations than just zero and append)
Reread the first comment there:
A crucial distinction between type classes and interfaces is that for class A to be a "member" of an interface it must declare so at the site of its own definition. By contrast, any type can be added to a type class at any time, provided you can provide the required definitions, and so the members of a type class at any given time are dependent on the current scope. Therefore we don't care if the creator of A anticipated the type class we want it to belong to; if not we can simply create our own definition showing that it does indeed belong, and then use it accordingly. So this not only provides a better solution than adapters, in some sense it obviates the whole problem adapters were meant to address.
I think this is the most important advantage of type classes.
Also, they handle properly the cases where the operations don't have the argument of the type we are dispatching on, or have more than one. E.g. consider this type class:
case class Default[T](val default: T)
object Default {
implicit def IntDefault: Default[Int] = Default(0)
implicit def OptionDefault[T]: Default[Option[T]] = Default(None)
...
}
I think of type classes as the ability to add type safe metadata to a class.
So you first define a class to model the problem domain and then think of metadata to add to it. Things like Equals, Hashable, Viewable, etc. This creates a separation of the problem domain and the mechanics to use the class and opens up subclassing because the class is leaner.
Except for that, you can add type classes anywhere in the scope, not just where the class is defined and you can change implementations. For example, if I calculate a hash code for a Point class by using Point#hashCode, then I'm limited to that specific implementation which may not create a good distribution of values for the specific set of Points I have. But if I use Hashable[Point], then I may provide my own implementation.
[Updated with example]
As an example, here's a use case I had last week. In our product there are several cases of Maps containing containers as values. E.g., Map[Int, List[String]] or Map[String, Set[Int]]. Adding to these collections can be verbose:
map += key -> (value :: map.getOrElse(key, List()))
So I wanted to have a function that wraps this so I could write
map +++= key -> value
The main issue is that the collections don't all have the same methods for adding elements. Some have '+' while others ':+'. I also wanted to retain the efficiency of adding elements to a list, so I didn't want to use fold/map which create new collections.
The solution is to use type classes:
trait Addable[C, CC] {
def add(c: C, cc: CC) : CC
def empty: CC
}
object Addable {
implicit def listAddable[A] = new Addable[A, List[A]] {
def empty = Nil
def add(c: A, cc: List[A]) = c :: cc
}
implicit def addableAddable[A, Add](implicit cbf: CanBuildFrom[Add, A, Add]) = new Addable[A, Add] {
def empty = cbf().result
def add(c: A, cc: Add) = (cbf(cc) += c).result
}
}
Here I defined a type class Addable that can add an element C to a collection CC. I have 2 default implementations: For Lists using :: and for other collections, using the builder framework.
Then using this type class is:
class RichCollectionMap[A, C, B[_], M[X, Y] <: collection.Map[X, Y]](map: M[A, B[C]])(implicit adder: Addable[C, B[C]]) {
def updateSeq[That](a: A, c: C)(implicit cbf: CanBuildFrom[M[A, B[C]], (A, B[C]), That]): That = {
val pair = (a -> adder.add(c, map.getOrElse(a, adder.empty) ))
(map + pair).asInstanceOf[That]
}
def +++[That](t: (A, C))(implicit cbf: CanBuildFrom[M[A, B[C]], (A, B[C]), That]): That = updateSeq(t._1, t._2)(cbf)
}
implicit def toRichCollectionMap[A, C, B[_], M[X, Y] <: col
The special bit is using adder.add to add the elements and adder.empty to create new collections for new keys.
To compare, without type classes I would have had 3 options:
1. to write a method per collection type. E.g., addElementToSubList and addElementToSet etc. This creates a lot of boilerplate in the implementation and pollutes the namespace
2. to use reflection to determine if the sub collection is a List / Set. This is tricky as the map is empty to begin with (of course scala helps here also with Manifests)
3. to have poor-man's type class by requiring the user to supply the adder. So something like addToMap(map, key, value, adder), which is plain ugly
Yet another way I find this blog post helpful is where it describes typeclasses: Monads Are Not Metaphors
Search the article for typeclass. It should be the first match. In this article, the author provides an example of a Monad typeclass.
The forum thread "What makes type classes better than traits?" makes some interesting points:
Typeclasses can very easily represent notions that are quite difficult to represent in the presence of subtyping, such as equality and ordering.
Exercise: create a small class/trait hierarchy and try to implement .equals on each class/trait in such a way that the operation over arbitrary instances from the hierarchy is properly reflexive, symmetric, and transitive.
Typeclasses allow you to provide evidence that a type outside of your "control" conforms with some behavior.
Someone else's type can be a member of your typeclass.
You cannot express "this method takes/returns a value of the same type as the method receiver" in terms of subtyping, but this (very useful) constraint is straightforward using typeclasses. This is the f-bounded types problem (where an F-bounded type is parameterized over its own subtypes).
All operations defined on a trait require an instance; there is always a this argument. So you cannot define for example a fromString(s:String): Foo method on trait Foo in such a way that you can call it without an instance of Foo.
In Scala this manifests as people desperately trying to abstract over companion objects.
But it is straightforward with a typeclass, as illustrated by the zero element in this monoid example.
Typeclasses can be defined inductively; for example, if you have a JsonCodec[Woozle] you can get a JsonCodec[List[Woozle]] for free.
The example above illustrates this for "things you can add together".
One way to look at type classes is that they enable retroactive extension or retroactive polymorphism. There are a couple of great posts by Casual Miracles and Daniel Westheide that show examples of using Type Classes in Scala to achieve this.
Here's a post on my blog
that explores various methods in scala of retroactive supertyping, a kind of retroactive extension, including a typeclass example.
I don't know of any other use case than Ad-hoc polymorhism which is explained here the best way possible.
Both implicits and typeclasses are used for Type-conversion. The major use-case for both of them is to provide ad-hoc polymorphism(i.e) on classes that you can't modify but expect inheritance kind of polymorphism. In case of implicits you could use both an implicit def or an implicit class (which is your wrapper class but hidden from the client). Typeclasses are more powerful as they can add functionality to an already existing inheritance chain(eg: Ordering[T] in scala's sort function).
For more detail you can see https://lakshmirajagopalan.github.io/diving-into-scala-typeclasses/
In scala type classes
Enables ad-hoc polymorphism
Statically typed (i.e. type-safe)
Borrowed from Haskell
Solves the expression problem
Behavior can be extended
- at compile-time
- after the fact
- without changing/recompiling existing code
Scala Implicits
The last parameter list of a method can be marked implicit
Implicit parameters are filled in by the compiler
In effect, you require evidence of the compiler
… such as the existence of a type class in scope
You can also specify parameters explicitly, if needed
Below Example extension on String class with type class implementation extends the class with a new methods even though string is final :)
/**
* Created by nihat.hosgur on 2/19/17.
*/
case class PrintTwiceString(val original: String) {
def printTwice = original + original
}
object TypeClassString extends App {
implicit def stringToString(s: String) = PrintTwiceString(s)
val name: String = "Nihat"
name.printTwice
}
This is an important difference (needed for functional programming):
consider inc:Num a=> a -> a:
a received is the same that is returned, this cannot be done with subtyping
I like to use type classes as a lightweight Scala idiomatic form of Dependency Injection that still works with circular dependencies yet doesn't add a lot of code complexity. I recently rewrote a Scala project from using the Cake Pattern to type classes for DI and achieved a 59% reduction in code size.