Here is my code example:
case class Person(name:String,tel:String){
def equals(that:Person):Boolean = that.name == this.name && this.tel == that.tel}
val persons = Array(Person("peter","139"),Person("peter","139"),Person("john","111"))
sc.parallelize(persons).distinct.collect
It returns
res34: Array[Person] = Array(Person(john,111), Person(peter,139), Person(peter,139))
Why distinct doesn't work?I want the result as Person("john",111),Person("peter",139)
Extending further from the observation of #aaronman, there is a workaround for this issue.
On the RDD, there're two definitions for distinct:
/**
* Return a new RDD containing the distinct elements in this RDD.
*/
def distinct(numPartitions: Int)(implicit ord: Ordering[T] = null): RDD[T] =
map(x => (x, null)).reduceByKey((x, y) => x, numPartitions).map(_._1)
/**
* Return a new RDD containing the distinct elements in this RDD.
*/
def distinct(): RDD[T] = distinct(partitions.size)
It's apparent from the signature of the first distinct that there must be an implicit ordering of the elements and it's assumed null if absent, which is what the short version .distinct() does.
There's no default implicit ordering for case classes, but it's easy to implement one:
case class Person(name:String,tel:String) extends Ordered[Person] {
def compare(that: Person): Int = this.name compare that.name
}
Now, trying the same example delivers the expected results (note that I'm comparing names):
val ps5 = Array(Person("peter","138"),Person("peter","55"),Person("john","138"))
sc.parallelize(ps5).distinct.collect
res: Array[P5] = Array(P5(john,111), P5(peter,139))
Note that case classes already implement equals and hashCode, so the impl on the provided example is unnecessary and also incorrect. The correct signature for equals is: equals(arg0: Any): Boolean -- BTW, I first thought that the issue had to do with the incorrect equals signature, which sent me looking in the wrong path.
For me the problem was related to object equality, as mentioned by Martin Odersky in Programming in Scala (chapter 30), although I have a normal class (not a case class). For a correct equality test, you must re-define (override) hashCode() if you have a custom equals(). Also you need to have a canEqual() method for 100% correctness. I haven't looked at the implementation details of an RDD, but since it is a collection, probably it uses some complex/parallel variation of a HashSet or other hash-based data structure for comparing objects and ensuring distinctness.
Declaring hashSet(), equals(), canEqual(), and compare() methods solved my problem:
override def hashCode(): Int = {
41 * (41 + name.hashCode) + tel.hashCode
}
override def equals(other: Any) = other match {
case other: Person =>
(other canEqual this) &&
(this.name == other.name) && (this.tel == other.tel)
case _ =>
false
}
def canEqual(other: Any) = other.isInstanceOf[Person]
def compare(that: Person): Int = {
this.name compare that.name
}
As others have pointed out this is a bug in spark 1.0.0. My theory as to where it is coming from is that if you look at the diff of 1.0.0 to 9.0 you see
- def repartition(numPartitions: Int): RDD[T] = {
+ def repartition(numPartitions: Int)(implicit ord: Ordering[T] = null): RDD[T] = {
And if you run
case class A(i:Int)
implicitly[Ordering[A]]
You get an error
<console>:13: error: No implicit Ordering defined for A.
implicitly[Ordering[A]]
So I think the workaround is define an implicit ordering for a the case class, unfortunately I'm not a scala expert but this answer seems to do it correctly
Related
I have a situation where I need a method that can take in types:
Array[Int]
Array[Array[Int]]
Array[Array[Array[Int]]]
Array[Array[Array[Array[Int]]]]
etc...
let's call this type RAI for "recursive array of ints"
def make(rai: RAI): ArrayPrinter = { ArrayPrinter(rai) }
Where ArrayPrinter is a class that is initialized with an RAI and iterates through the entire rai (let's say it prints all the values in this Array[Array[Int]])
val arrayOfArray: Array[Array[Int]] = Array(Array(1, 2), Array(3, 4))
val printer: ArrayPrinter[Array[Array[Int]]] = make(arrayOfArray)
printer.print_! // prints "1, 2, 3, 4"
It can also return the original Array[Array[Int]] without losing any type information.
val arr: Array[Array[Int]] = printer.getNestedArray()
How do you implement this in Scala?
Let's first focus on type. According to your definition, a type T should typecheck as an argument for ArrayPrinter is it accepted by the following type function:
def accept[T]: Boolean =
T match { // That's everyday business in agda
case Array[Int] => true
case Array[X] => accept[X]
case _ => false
}
In Scala, you can encode that type function using implicit resolution:
trait RAI[T]
object RAI {
implicit val e0: RAI[Array[Int]] = null
implicit def e1[T](implicit i: RAI[T]): RAI[Array[T]] = null
}
case class ArrayPrinter[T: RAI](getNestedArray: T) // Only compiles it T is a RAI
To print things the simplest solution is to treat the rai: T as a rai: Any:
def print_!: Unit = {
def print0(a: Any): Unit = a match {
case a: Int => println(a)
case a: Array[_] => a.foreach(print0)
case _ => ???
}
}
You could also be fancy and write print_! using type classes, but that would probably be less efficient and take more time to write than the above... Left as an exercise for the reader ;-)
The way this is typically done is by defining an abstract class that contains all the functionality that you would want related to this recursive type, but does not actually take any constructor arguments. Rather, all of its methods take (at least one of) the type as an argument. The canonical example would be Ordering. Define one or more implicit implementations of this class, and then any time you need to use it, accept it as an implicit parameter. The corresponding example would be List's sorted method.
In your case, this might look like:
abstract class ArrayPrinter[A] {
def mkString(a: A): String
}
implicit object BaseArrayPrinter extends ArrayPrinter[Int] {
override def mkString(x: Int) = x.toString
}
class WrappedArrayPrinter[A](wrapped: ArrayPrinter[A]) extends ArrayPrinter[Array[A]] {
override def mkString(xs: Array[A]) = xs.map(wrapped.mkString).mkString(", ")
}
implicit def makeWrappedAP[A](implicit wrapped: ArrayPrinter[A]): ArrayPrinter[Array[A]] = new WrappedArrayPrinter(wrapped)
def printHello[A](xs: A)(implicit printer: ArrayPrinter[A]): Unit = {
println("hello, array: " + printer.mkString(xs))
}
This tends to be a bit cleaner than having that RAIOps class (or ArrayPrinter) take in an object as part of its constructor. That usually leads to more "boxing" and "unboxing", complicated type signatures, strange pattern matching, etc.
It also has the added benefit of being easier to extend. If later someone else has a reason to want an implementation of ArrayPrinter for a Set[Int], they can define it locally to their code. I have many times defined a custom Ordering.
I have a simple trait that requires the implementation to have a method quality(x:A) which I want to return an Ordered[B]. In other words, quality transforms A to Ordered[B]. Such that I can compare to B's.
I have the following basic code:
trait Foo[A] {
def quality[B](x:A):Ordered[B]
def baz(x1:A, x2:A) = {
// some algorithm work here, and then
if (quality(x1) > quality(x2)) {
// some stuff
}
}
Which I want to implement like follows:
class FooImpl extends Foo[Bar] {
def quality(x:Bar):Double = {
someDifficultQualityCalculationWhichReturnsADouble
}
}
I figured this could work because Double is implicitly converted to RichDouble which implements Ordered[Double] if I am correct.
But at the > in the baz method of the trait it gives me an error of quality(x2) stating: Type mismatch, expected Nothing, actual Ordered[Nothing]
I do not understand this, because, coming from C#, I find it comparable to returning something like IEnumerable<A> and then using al the nice extension methods of an IEnumerable.
What am I missing here? What I want to to with the trait is define a complex algorithm inside the trait, but the key functions need to be defined by the class implementing the trait. On of these functions is needed to calculate a quality factor. This can be a Double, Int or whatnot but it can also be something more sophisticated. I could rewrite it that it always returns Double and that is certainly possible, but I want the trait to be as generic as possible because I want it to describe behavior and not implementation. I thought about the class A implementing Ordered[A] but that also seems weird, because it is not the 'purpose' of this class to be compared.
Using Ordering[A] you can compare As without requiring A to implement Ordered[A].
We can request that an Ordering[A] exists in baz by adding an implicit parameter :
trait Foo[A] {
def baz(x1:A, x2:A)(implicit ord: Ordering[A]) =
if (ord.gt(x1, x2)) "first is bigger"
else "first is smaller or equal"
}
Lets create a Person case class with an Ordering in its companion object.
case class Person(name: String, age: Int)
object Person {
implicit val orderByAge = Ordering.by[Person, Int](_.age)
}
We can now use Foo[Person].baz because an Ordering[Person] exists :
val (alice, bob) = (Person("Alice", 50), Person("Bob", 40))
val foo = new Foo[Person] {}
foo.baz(alice, bob)
// String = first is bigger
// using an explicit ordering
foor.baz(alice, bob)(Ordering.by[Person, String](_.name))
// String = first is smaller or equal
In the same manner as I compared Persons by age, you could create an Ordering[A] to compare your A by your quality function.
To complement Peter's answer: in Scala we have two traits: Ordering[T] and Ordered[A]. You should use them in different situations.
Ordered[A] is for cases when a class you implement can be ordered naturally and that order is the only one.
Example:
class Fraction(val numerator: Int, val denominator: Int) extends Ordered[Fraction]
{
def compare(that: Fraction) = {
(this.numerator * that.denominator) compare (this.denominator * that.numerator)
}
}
Ordering[T] is for cases when you want to have different ways to order things. This way the strategy of defining the order can be decoupled from the class being ordered.
For an example I will borrow Peter's Person:
case class Person(name: String, age: Int)
object PersonNameOrdering extends Ordering[Person]
{
def compare(x: Person, y: Person) = x.name compare y.name
}
Note, that since PersonNameOrdering doesn't have any instance fields, all it does is encapsulate the logic of defining an order of two Person's. Thus I made it an object rather than a class.
To cut down the boilerplate you can use Ordering.on to define an Ordering:
val personAgeOrdering: Ordering[Person] = Ordering.on[Person](_.age)
Now to the fun part: how to use all this stuff.
In your original code Foo[A].quantity was indirectly defining a way to order your A's. Now to make it idiomatic Scala let's use Ordering[A] instead, and rename quantity to ord:
trait Foo[A] {
def baz(x1: A, x2: A, ord: Ordering[A]) = {
import ord._
if (x1 > x2) "first is greater"
else "first is less or equal"
}
}
Several things to note here:
import ord._ allows to use infix notation for comparisons, i.e. x1 > x2 vs ord.gt(x1, x2)
baz is now parametrized by ordering, so you can dynamically choose how to order x1 and x2 on a case-by-case basis:
foo.baz(person1, person2, PersonNameOrdering)
foo.baz(person1, person2, personAgeOrdering)
The fact that ord is now an explicit parameter can sometimes be inconvenient: you may not want to pass it explicitly all the time, while there might be some cases when you want to do so. Implicits to the rescue!
def baz(x1: A, x2: A) = {
def inner(implicit ord: Ordering[A]) = {
import ord._
if (x1 > x2) "first is greater"
else "first is less or equal"
}
inner
}
Note the implicit keyword. It is used to tell the compiler to draw the parameter from the implicit scope in case you don't provide it explicitly:
// put an Int value to the implicit scope
implicit val myInt: Int = 5
def printAnInt(implicit x: Int) = { println(x) }
// explicitly pass the parameter
printAnInt(10) // would print "10"
// let the compiler infer the parameter from the implicit scope
printAnInt // would print "5"
You might want to learn where does Scala look for implicits.
Another thing to note is the need of a nested function. You cannot write def baz(x1: A, x2: A, implicit ord: Ordering[A]) - that would not compile, because the implicit keyword applies to the whole parameter list.
In order to cope with this little problem baz was rewritten in such a clunky way.
This form of rewritting turned out to be so common that a nice syntactic sugar was introduced for it - multiple parameter list:
def baz(x1: A, x2: A)(implicit ord: Ordering[A]) = {
import ord._
if (x1 > x2) "first is greater"
else "first is less or equal"
}
The need of an implicit parametrized by a type is also quite common so the code above can be rewritten with even more sugar - context bound:
def baz[A: Ordering](x1: A, x2: A) = {
val ord = implicitly[Ordering[A]]
import ord._
if (x1 > x2) "first is greater"
else "first is less or equal"
}
Please bear in mind that all these transformations of baz function are nothing but syntactic sugar application. So all the versions are exactly the same and compiler would desugarize each of the versions to the same bytecode.
To recap:
extract the A ordering logic from quantity function to the Ordering[A] class;
put an instance of Ordering[A] to the implicit scope or pass the ordering explicitly depending on your needs;
pick "your flavor" of syntactic sugar for baz: no sugar/nested functions, multiple parameter list or context bound.
UPD
To answer the original question "why doesn't it compile?" let me start from a little digression on how infix comparison operator works in Scala.
Given the following code:
val x: Int = 1
val y: Int = 2
val greater: Boolean = x > y
Here's what actually happens. Scala doesn't have infix operators per se, instead infix operators are just a syntactic sugar for single parameter method invocation. So internally the code above transforms to this:
val greater: Boolean = x.>(y)
Now the tricky part: Int doesn't have an > method on its own. Pick ordering by inheritance on the ScalaDoc page and check that this method is listed in a group titled "Inherited by implicit conversion intWrapper from Int to RichInt".
So internally compiler does this (well, except that for performance reasons that there is no actual instantiation of an extra object on heap):
val greater: Boolean = (new RichInt(x)).>(y)
If we proceed to ScalaDoc of RichInt and again order methods by inheritance it turns out that the > method actually comes from Ordered!
Let's rewrite the whole block to make it clearer what actually happens:
val x: Int = 1
val y: Int = 2
val richX: RichInt = new RichInt(x)
val xOrdered: Ordered[Int] = richX
val greater: Boolean = xOrdered.>(y)
The rewriting should have highlighted the types of variables involved in comparison: Ordered[Int] on the left and Int on the right. Refer > documentation for confirmation.
Now let's get back to the original code and rewrite it the same way to highlight the types:
trait Foo[A] {
def quality[B](x: A): Ordered[B]
def baz(x1: A, x2: A) = {
// some algorithm work here, and then
val x1Ordered: Ordered[B] = quality(x1)
val x2Ordered: Ordered[B] = quality(x2)
if (x1Ordered > x2Ordered) {
// some stuff
}
}
}
As you can see the types do not align: they are Ordered[B] and Ordered[B], while for > comparison to work they should have been Ordered[B] and B respectively.
The question is where do you get this B to put on the right? To me it seems that B is in fact the same as A in this context. Here's what I came up with:
trait Foo[A] {
def quality(x: A): Ordered[A]
def baz(x1: A, x2: A) = {
// some algorithm work here, and then
if (quality(x1) > x2) {
"x1 is greater"
} else {
"x1 is less or equal"
}
}
}
case class Cargo(weight: Int)
class CargoFooImpl extends Foo[Cargo] {
override def quality(x: Cargo): Ordered[Cargo] = new Ordered[Cargo] {
override def compare(that: Cargo): Int = x.weight compare that.weight
}
}
The downside of this approach is that it is not obvious: the implementation of quality is too verbose and quality(x1) > x2 is not symmetrical.
The bottom line:
if you want the code to be idiomatic Scala go for Ordering[T]
if you don't want to mess with implicits and other Scala magic implement quality as quality(x: A): Double for all As; Doubles are good and generic enough to be compared and ordered.
I've been experimenting with implicit conversions, and I have a decent understanding of the 'enrich-my-libray' pattern that uses these. I tried to combine my understanding of basic implicits with the use of implicit evidence... But I'm misunderstanding something crucial, as shown by the method below:
import scala.language.implicitConversions
object Moo extends App {
case class FooInt(i: Int)
implicit def cvtInt(i: Int) : FooInt = FooInt(i)
implicit def cvtFoo(f: FooInt) : Int = f.i
class Pair[T, S](var first: T, var second: S) {
def swap(implicit ev: T =:= S, ev2: S =:= T) {
val temp = first
first = second
second = temp
}
def dump() = {
println("first is " + first)
println("second is " + second)
}
}
val x = new Pair(FooInt(200), 100)
x.dump
x.swap
x.dump
}
When I run the above method I get this error:
Error:(31, 5) Cannot prove that nodescala.Moo.FooInt =:= Int.
x.swap
^
I am puzzled because I would have thought that my in-scope implict conversion would be sufficient 'evidence' that Int's can be converted to FooInt's and vice versa. Thanks in advance for setting me straight on this !
UPDATE:
After being unconfused by Peter's excellent answer, below, the light bulb went on for me one good reason you would want to use implicit evidence in your API. I detail that in my own answer to this question (also below).
=:= checks if the two types are equal and FooInt and Int are definitely not equal, although there exist implicit conversion for values of these two types.
I would create a CanConvert type class which can convert an A into a B :
trait CanConvert[A, B] {
def convert(a: A): B
}
We can create type class instances to transform Int into FooInt and vise versa :
implicit val Int2FooInt = new CanConvert[Int, FooInt] {
def convert(i: Int) = FooInt(i)
}
implicit val FooInt2Int = new CanConvert[FooInt, Int] {
def convert(f: FooInt) = f.i
}
Now we can use CanConvert in our Pair.swap function :
class Pair[A, B](var a: A, var b: B) {
def swap(implicit a2b: CanConvert[A, B], b2a: CanConvert[B, A]) {
val temp = a
a = b2a.convert(b)
b = a2b.convert(temp)
}
override def toString = s"($a, $b)"
def dump(): Unit = println(this)
}
Which we can use as :
scala> val x = new Pair(FooInt(200), 100)
x: Pair[FooInt,Int] = (FooInt(200), 100)
scala> x.swap
scala> x.dump
(FooInt(100), 200)
A =:= B is not evidence that A can be converted to B. It is evidence that A can be cast to B. And you have no implicit evidence anywhere that Int can be cast to FooInt vice versa (for good reason ;).
What you are looking for is:
def swap(implicit ev: T => S, ev2: S => T) {
After working through this excercise I think I have a better understanding of WHY you'd want to use implicit evidence serves in your API.
Implicit evidence can be very useful when:
you have a type parameterized class that provides various methods
that act on the types given by the parameters, and
when one or more of those methods only make sense when additional
constraints are placed on parameterized types.
So, in the case of the simple API given in my original question:
class Pair[T, S](var first: T, var second: S) {
def swap(implicit ev: T =:= S, ev2: S =:= T) = ???
def dump() = ???
}
We have a type Pair, which keeps two things together, and we can always call dump() to examine the two things. We can also, under certain conditions, swap the positions of the first and second items in the pair. And those conditions are given by the implicit evidence constraints.
The Programming in Scala book gives a nice example of how this technique
is used in Scala collections, specifically on the toMap method of Traversables.
The book points out that Map's constructor
wants key-value pairs, i.e., two-tuples, as arguments. If we have a
sequence [Traversable] of pairs, wouldn’t it be nice to create a Map
out of them in one step? That’s what toMap does, but we have a
dilemma. We can’t allow the user to call toMap if the sequence is not
a sequence of pairs.
So there's an example of a type [Traversable] that has a method [toMap] that can't be used in all situations... It can only be used when the compiler can 'prove' (via implicit evidence) that the items in the Traversable are pairs.
Specialization promises to provide high-efficiency implmentations for primitive types
with minimal extra boilerplate. But specialization seems to be too eager for its own good.
If I want to specialize a class or method,
def foo[#specialized(Byte) A](a: A): String = ???
class Bar[#specialized(Int) B] {
var b: B = ???
def baz: B = ???
}
then I am required to write a single implementation that covers both the specialized and the generic cases.
What if those cases are really different from each other, such that the implementations do not overlap?
For example, if I wanted to perform math on bytes, I would need to insert a bunch of & 0xFFs into the
logic.
I could possibly write a specialized type-class to do the math right, but doesn't that just push the same
problem back one level? How do I write my specialized + method for that type class in a way that doesn't
conflict with a more general implementation?
class Adder[#specialized(Byte) A] {
def +(a1: A, a2: A): A = ???
}
Also, once I do create a type-class this way, how do I make sure the correct type class is used for my specialized methods
instead of the general version (which, if it is truly general, should probably compile and certainly would run, except that it isn't what I want)?
Is there a way to do this without macros? Is it easier with macros?
This is my best attempt so far. It works but the implementation isn't pretty (even if the results are). Improvements are welcome!
There is a macro-free way to do this, both at the class and method level, and it does involve type classes--quite a lot of
them! And the answer is not exactly the same for classes and methods. So bear with me.
Manually Specialized Classes
You manually specialize classes the same way that you manually provide any kind of different implementation for classes:
your superclass is abstract (or is a trait), and the subclasses provide the implementation details.
abstract class Bippy[#specialized(Int) B] {
def b: B
def next: Bippy[B]
}
class BippyInt(initial: Int) extends Bippy[Int] {
private var myB: Int = initial
def b: Int = myB
def next = { myB += 1; this }
}
class BippyObject(initial: Object) extends Bippy[Object] {
private var myB: Object = initial
def b: B = myB
def next = { myB = myB.toString; this }
}
Now, if only we had a specialized method to pick out the right implementations, we'd be done:
object Bippy{
def apply[#specialized(Int) B](initial: B) = ??? // Now what?
}
So we've converted our problem of providing custom specialized classes and methods into just
needing to provide custom specialized methods.
Manually Specialized Methods
Manually specializing a method requires a way to write one implementation that can nonetheless
select which implementation you want (at compile time). Type classes are great at this. Suppose
we already had type classes that implemented all of our functionality, and that the compiler would
select the right one. Then we could just write
def foo[#specialized(Int) A: SpecializedFooImpl](a: A): String =
implicitly[SpecializedFooImpl[A]](a)
...or we could if implicitly was guaranteed to preserve specialization and if we only
ever wanted a single type parameter. In general these things are not true, so we'll write
our type class out as an implicit parameter rather than relying on the A: TC syntactic sugar.
def foo[#specialized(Int) A](a: A)(implicit impl: SpecializedFooImpl[A]): String =
impl(a)
(Actually, that's less boilerplate anyway.)
So we've converted our problem of providing custom specialized methods into just needing
to write specialized typeclasses and getting the compiler to fill in the correct ones.
Manually Specialized Type Classes
Type classes are just classes, and now we have to write specialized classes again, but
there's a critical difference. The user isn't the one asking for arbitrary instances.
This gives us just enough extra flexibility for it to work.
For foo, we need an Int version and a fully generic version.
trait SpecFooImpl[#specialized (Int), A] {
def apply(param: A): String
}
final class SpecFooImplAny[A] extends SpecFooImpl[A] {
def apply(param: A) = param.toString
}
final class SpecFooImplInt extends SpecFooImpl[Int] {
def apply(param: Int) = "!" * math.max(0, param)
}
Now we could create implicits to supply those type classes like so
implicit def specFooAsAny[A] = new SpecFooImplAny[A]
implicit val specFooAsInt = new SpecFooImplInt
except we have a problem: if we actually try to call foo: Int, both implicits will apply.
So if we just had a way to prioritize which type class we chose, we'd be all set.
Selection of type classes (and implicits in general)
One of the secret ingredients the compiler uses to determine the right implicit to use
is inheritance. If implicits come from A via B extends A, but B
declares its own that also could apply, those in B win if all else is equal.
So we put the ones we want to win deeper in the inheritance hierarchy.
Also, since you're free to define implicits in traits, you can mix them in anywhere.
So the last piece of our puzzle is to pop our type class implicits into a chain
of traits that extend each other, with the more generic ones appearing earlier.
trait LowPriorityFooSpecializers {
implicit def specializeFooAsAny[A] = new SpecializedFooImplAny[A]
}
trait FooSpecializers extends LowPriorityFooSpecializers {
implicit val specializeFooAsInt = new SpecializedFooImplInt
}
Mix in the highest-priority trait to wherever the implicits are needed, and the
type classes will be picked as desired.
Note that the type classes will be as specialized as you make them even if the
specialized annotation is not used. So you can do without specialized at all,
as long as you know the type precisely enough, unless you want to use specialized
functions or interoperate with other specialized classes. (And you probably do.)
A complete example
Let's suppose we want to make a two-parameter specialized bippy function that
will do apply the following transformation:
bippy(a, b) -> b
bippy(a, b: Int) -> b+1
bippy(a: Int, b) -> b
bippy(a: Int, b: Int) -> a+b
We should be able to achieve this with three type classes and a single specialized
method. Let's try, first the method:
def bippy[#specialized(Int) A, #specialized(Int) B](a: A, b: B)(implicit impl: SpecBippy[A, B]) =
impl(a, b)
Then the type classes:
trait SpecBippy[#specialized(Int) A, #specialized(Int) B] {
def apply(a: A, b: B): B
}
final class SpecBippyAny[A, B] extends SpecBippy[A, B] {
def apply(a: A, b: B) = b
}
final class SpecBippyAnyInt[A] extends SpecBippy[A, Int] {
def apply(a: A, b: Int) = b + 1
}
final class SpecBippyIntInt extends SpecBippy[Int, Int] {
def apply(a: Int, b: Int) = a + b
}
Then the implicits in chained traits:
trait LowerPriorityBippySpeccer {
// Trick to avoid allocation since generic case is erased anyway!
private val mySpecBippyAny = new SpecBippyAny[AnyRef, AnyRef]
implicit def specBippyAny[A, B] = mySpecBippyAny.asInstanceOf[SpecBippyAny[A, B]]
}
trait LowPriorityBippySpeccer extends LowerPriorityBippySpeccer {
private val mySpecBippyAnyInt = new SpecBippyAnyInt[AnyRef]
implicit def specBippyAnyInt[A] = mySpecBippyAnyInt.asInstanceOf[SpecBippyAnyInt[A]]
}
// Make this last one an object so we can import the contents
object BippySpeccer extends LowPriorityBippySpeccer {
implicit val specBippyIntInt = new SpecBippyIntInt
}
and finally we'll try it out (after pasting everything in together in :paste in the REPL):
scala> import Speccer._
import Speccer._
scala> bippy(Some(true), "cod")
res0: String = cod
scala> bippy(1, "salmon")
res1: String = salmon
scala> bippy(None, 3)
res2: Int = 4
scala> bippy(4, 5)
res3: Int = 9
It works--our custom implementations are enabled. Just to check that we can use
any type, but we don't leak into the wrong implementation:
scala> bippy(4, 5: Short)
res4: Short = 5
scala> bippy(4, 5: Double)
res5: Double = 5.0
scala> bippy(3: Byte, 2)
res6: Int = 3
And finally, to verify that we have actually avoided boxing, we'll time bippy at
summing a bunch of integers:
scala> val th = new ichi.bench.Thyme
th: ichi.bench.Thyme = ichi.bench.Thyme#1130520d
scala> val adder = (i: Int, j: Int) => i + j
adder: (Int, Int) => Int = <function2>
scala> var a = Array.fill(1024)(util.Random.nextInt)
a: Array[Int] = Array(-698116967, 2090538085, -266092213, ...
scala> th.pbenchOff(){
var i, s = 0
while (i < 1024) { s = adder(a(i), s); i += 1 }
s
}{
var i, s = 0
while (i < 1024) { s = bippy(a(i), s); i += 1 }
s
}
Benchmark comparison (in 1.026 s)
Not significantly different (p ~= 0.2795)
Time ratio: 0.99424 95% CI 0.98375 - 1.00473 (n=30)
First 330.7 ns 95% CI 328.2 ns - 333.1 ns
Second 328.8 ns 95% CI 326.3 ns - 331.2 ns
So we can see that our specialized bippy-adder achieves the same kind of performance
as specialized Function2 does (about 3 adds per ns, which is about right for a modern
machine).
Summary
To write custom specialized code using the #specialized annotation,
Make the specialized class abstract and manually supply concrete implementations
Make specialized methods (including generators for a specialized class) take typeclasses that do the real work
Make the base typeclass trait #specialized and provide concrete implementations
Provide implicit vals or defs in an inheritance-hierarchy of traits so the correct one is selected
It's a lot of boilerplate, but at the end of it all you get a seamless custom-specialized experience.
This is an answer from the scala internals mailing list:
With miniboxing specialization, you can use the reflection feature:
import MbReflection._
import MbReflection.SimpleType._
import MbReflection.SimpleConv._
object Test {
def bippy[#miniboxed A, #miniboxed B](a: A, b: B): B =
(reifiedType[A], reifiedType[B]) match {
case (`int`, `int`) => (a.as[Int] + b.as[Int]).as[B]
case ( _ , `int`) => (b.as[Int] + 1).as[B]
case (`int`, _ ) => b
case ( _ , _ ) => b
}
def main(args: Array[String]): Unit = {
def x = 1.0
assert(bippy(3,4) == 7)
assert(bippy(x,4) == 5)
assert(bippy(3,x) == x)
assert(bippy(x,x) == x)
}
}
This way, you can choose the exact behavior of the bippy method based on the type arguments without defining any implicit classes.
I know it's quite old, but I bumped at it looking for something else and maybe it'll prove useful. I had a similar motivation, and answered it in how to check I'm inside a specialized function or class
I used a reverse lookup table - SpecializedKey is a specialized class which equals all other instances with the same specialization, so I can perform a check like this
def onlyBytes[#specialized E](arg :E) :Option[E] =
if (specializationFor[E]==specializationFor[Byte]) Some(arg)
else None
Of course, there's no performance benefit when working with individual primitive values, but with collections, especially iterators, it becomes useful.
final val AllButUnit = new Specializable.Group((Byte, Short, Int, Long, Char, Float, Double, Boolean, AnyRef))
def specializationFor[#specialized(AllButUnit) E] :ResolvedSpecialization[E] =
Specializations(new SpecializedKey[E]).asInstanceOf[ResolvedSpecialization[E]]
private val Specializations = Seq(
resolve[Byte],
resolve[Short],
resolve[Int],
resolve[Long],
resolve[Char],
resolve[Float],
resolve[Double],
resolve[Boolean],
resolve[Unit],
resolve[AnyRef]
).map(
spec => spec.key -> spec :(SpecializedKey[_], ResolvedSpecialization[_])
).toMap.withDefaultValue(resolve[AnyRef])
private def resolve[#specialized(AllButUnit) E :ClassTag] :ResolvedSpecialization[E] =
new ResolvedSpecialization[E](new SpecializedKey[E], new Array[E](0))
class ResolvedSpecialization[#specialized(AllButUnit) E] private[SpecializedCompanion]
(val array :Array[E], val elementType :Class[E], val classTag :ClassTag[E], private[SpecializedCompanion] val key :SpecializedKey[E]) {
private[SpecializedCompanion] def this(key :SpecializedKey[E], array :Array[E]) =
this(array, array.getClass.getComponentType.asInstanceOf[Class[E]], ClassTag(array.getClass.getComponentType.asInstanceOf[Class[E]]), key)
override def toString = s"#specialized($elementType)"
override def equals(that :Any) = that match {
case r :ResolvedSpecialization[_] => r.elementType==elementType
case _ => false
}
override def hashCode = elementType.hashCode
}
private class SpecializedKey[#specialized(AllButUnit) E] {
override def equals(that :Any) = that.getClass==getClass
override def hashCode = getClass.hashCode
def className = getClass.getName
override def toString = className.substring(className.indexOf("$")+1)
}
I would like to have the automatic companion class apply constructors of a case class to perform implicit conversions for me, but cannot figure out how to do so. I've searched all over and the closest answer I could find was for this question (I'll explain why it isn't what I'm looking for below).
I have a case class that looks something like this:
case class Container(a: Long, b: Long, c: Long)
I'm using the container to count instances where certain conditions apply, so I'd like to be able to have the constructor automatically convert boolean parameters to longs (if (boolean) 1L else 0L).
The real case class, of course, has many parameters, so it would be tedious and terribly repetitive to make my own companion object and overload apply to accept Boolean parameters. Additionally, something like the code below isn't ideal (if it were implemented correctly somehow) as it only accepts boolean arguments:
object Container {
def apply(args: Boolean*) = {
// doesn't REALLY work since number of arguments not enforced
Container(args map { if (_) 1L else 0L } toArray: _*)
}
}
val c1 = Container(1, 0, 1) // works
val c2 = Container(true, false, true) // might be workable if done correctly
val c3 = Container(true, 0, 1) // won't work
I tried adding an implicit conversion in the companion object (below), hoping that it would automatically be used in Container.apply, but it appears that this does not actually put the implicit conversion into the namespace of the code that calls apply.
object Container {
implicit def booleanToLong(x: Boolean): Long = if (x) 1L else 0L
}
I'm able to get things working using this hackish workaround:
{
import Container.booleanToLong
// all of these now work
val c1 = Container(1, 0, 1)
val c2 = Container(true, false, true)
val c3 = Container(true, 0, 1) // works!!!
}
The biggest problem is that I have to import booleanToLong into the code that wants to create a Container and thus must put it in its own block for safety (booleanToLong is generally undesirable).
Finally, the solution of using an implicit parameter that itself includes an implicit conversion doesn't work because it would require an explicit overriding of apply, defeating the goal of not repeating a long parameter list and marshaling types.
Is there a way to do this such that I get implicit conversions for free every time I make a Container, but not otherwise? Or is this impossible due to some sort of technical constraint?
You can use a kind of variant of the magnet pattern to make this a little safer. First for a type class:
trait ToLong[A] {
def apply(a: A): Long
}
implicit object longToLong extends ToLong[Long] {
def apply(l: Long) = l
}
implicit object booleanToLong extends ToLong[Boolean] {
def apply(b: Boolean) = if (b) 1L else 0L
}
And now we just need one extra constructor:
case class Container(a: Long, b: Long, c: Long)
object Container {
def apply[A: ToLong, B: ToLong, C: ToLong](a: A, b: B, c: C) = new Container(
implicitly[ToLong[A]].apply(a),
implicitly[ToLong[B]].apply(b),
implicitly[ToLong[C]].apply(c)
)
}
And we can write the following:
val c1 = Container(1, 0, 1)
val c2 = Container(true, false, true)
val c3 = Container(true, 0L, 1L)
Without having to introduce the rather scary general conversion from Boolean to Long.