I am trying to create a customRDD in Java.
RDD converts RDD[(K,V)] to PairRDDFunctions[K,V] using Scala implicit function rddToPairRDDFunctions() defined in object RDD.
I am trying to do the same with my CustomJavaRDD which extends CustomRDD which extends RDD.
Now it should call implicit function rddToCustomJavaRDDFunctions() whenever it encounters CustomJavaRDD[(K,V)], but for some reason it still goes to rddToPairRDDFunctions().
What am I doing wrong?
RDD.scala
class RDD[T]
object RDD {
implicit def rddToPairRDDFunctions[K, V](rdd: RDD[(K, V)])
(implicit kt: ClassTag[K], vt: ClassTag[V], ord: Ordering[K] = null):
PairRDDFunctions[K, V] = {
new PairRDDFunctions(rdd)
}
}
CustomRDD.scala
abstract class CustomRDD[T] extends RDD[T]
object CustomRDD {
implicit def rddToCustomJavaRDDFunctions[K,V](rdd: CustomJavaRDD[(K,V)]):
PairCustomJavaRDDFunction[K,V] = {
new PairCustomJavaRDDFunctions[K,V](rdd)
}
}
PairCustomJavaRDDFunctions.scala
class PairCustomJavaRDDFunctions[K: ClassTag, V: ClassTag](self: CustomRDD[(K, V)])
(implicit ord: Ordering[K] = null) {
def collectAsMap() = ???
}
There is no error; the program compiles successfully,
but let's say I have data: RDD which is an instance of CustomJavaRDD.
data.collectAsMap()
At the runtime it converts data into PairRDDFunctions; i.e. it makes implicit call to rddToPairRDDFunctions defined in RDD.scala.
But it should make call to rddToCustomJavaRDDFunctions defined in CustomRDD.scala and convert it into PairCustomJavaRDDFunctions.
But it should make call to rddToCustomJavaRDDFunctions defined in CustomRDD.scala and convert it into PairCustomJavaRDDFunctions
No, Scala simply does not work this way. What you want, overriding an implicit conversion depending on the runtime type of an object, is simply not possible (without pre-existing machinery on both the library's part and yours).
Implicits are a strictly compile-time feature. When the compiler sees you using an RDD as if it were a PairRDDFunctions, it splices in a call to RDD.rddToPairRDDFunctions, as if you wrote it yourself. Then, when the code is translated to bytecode, that call has already been baked in and nothing can change it. There is no dynamic dispatch for this, it's all static. The only situation where rddToCustomJavaRDDFunctions will be called is when the static type of the expression in question is already CustomJavaRDD.
Really, this should not be necessary. Implicit conversions are really no more than glorified helper methods that save you keystrokes. (Implicit parameters, now those are interesting. ;) ) There should be no need to override them because the helper methods should already be polymorphic and work whether you have RDD, CustomRDD, or `RDD that travels through time to compute things faster`.
Of course, you can still do it, but it will only actually do anything under the above conditions, and that is probably not very likely, making the whole thing rather pointless.
Related
When implementing Typeclasses for our types, we can use different syntaxes (an implicit val or an implicit object, for example). As an example:
A Typeclass definition:
trait Increment[A] {
def increment(value: A): A
}
And, as far as I know, we could implement it for Int in the two following ways:
implicit val fooInstance: Increment[Int] = new Increment[Int] {
override def increment(value: Int): Int = value + 1
}
// or
implicit object fooInstance extends Increment[Int] {
override def increment(value: Int): Int = value + 1
}
I always use the first one as for Scala 2.13 it has an abbreviation syntax that looks like this:
implicit val fooInstance: Increment[Int] = (value: Int) => value + 1
But, is there any real difference between them? or is there any recommendation or standard to do this?
There is a related question about implicit defs and implicit classes for conversions, but I'm going more to the point of how to create (best practices) instances of Typeclasses, not about implicit conversions
As far as I know the differences would be:
objects have different initialization rules - quite often they will be lazily initialized (it doesn't matter if you don't perform side effects in constructor)
it would also be seen differently from Java (but again, you probably won't notice that difference in Scala)
object X will have a type X.type which is a subtype of whatever X extends or implements (but implicit resolution would find that it extends your typeclass, though perhaps with a bit more effort)
So, I wouldn't seen any difference in the actual usage, BUT the implicit val version could generate less JVM garbage (I say garbage as you wouldn't use any of that extra compiler's effort in this particular case).
I have an interface defined using a structural type like this:
trait Foo {
def collection: {
def apply(a: Int) : String
def values() : collection.Iterable[String]
}
}
}
I wanted to have one of the implementers of this interface do so using a standard mutable HashMap:
class Bar {
val collection: HashMap[Int, String] = HashMap[Int, String]()
}
It compiles, but at runtime I get a NoSuchMethod exception when referring a Bar instance through a Foo typed variable. Dumping out the object's methods via reflection I see that the HashMap's apply method takes an Object due to type erasure, and there's some crazily renamed generated apply method that does take an int. Is there a way to make generics work with structural types? Note in this particular case I was able to solve my problem using an actual trait instead of a structural type and that is overall much cleaner.
Short answer is that the apply method parameter is causing you grief because it requires some implicit conversions of the parameter (Int => Integer). Implicits are resolved at compile time, the NoSuchMethodException is likely a result of these missing implicits.
Attempt to use the values method and it should work since there are no implicits being used.
I've attempted to find a way to make this example work but have had no success so far.
I would like to have a higher order function that takes in parameter a function that accepts a specific implicit parameter.
To be more precise, I am trying to make a function that takes a Future creation method that depends on an implicit context and returns a method that doesn't depend on the context.
To be more concrete, let's say that I have something like this:
def foo(a: Int)(implicit ctx: ExecutionContext): Future[Float] = future { somelongBar... }
I would like to do have a method like this:
def provideCtx[A](func: ExecutionContext => A): A = {
val ctx = setupCtx
func(ctx)
}
but if I call provideCtx(foo), the compiler complains about the implicit execution context missing.
The fact that I am dealing with an ExecutionContext is not very important. What I would like to find is how to write the parameter type to accept a function with an implicit argument of a specific type. I understand that the implicit part is a curryed argument, so that in fact I have a function like so: ExecutionContext => Int => Future[Float], and I am pretty sure that at runtime, the jvm doesn't know that that ExecutionContext is implicit, but I can't make the compiler understand that.
The problem is that foo is a method, not a function, and eta-expansion (which converts methods to functions) is not attempted until after implicit application. See section 6.26.2 of the language specification for the details, and this issue for additional discussion.
One workaround would be to write something like this:
provideCtx((ctx: ExecutionContext) => (a: Int) => foo(a)(ctx))
I'm not sure a more generic solution is possible (at least without some kind of reflection, etc.), since we can't even refer to foo (except in a method call, of course) without an implicit in scope.
It seems I don't understand something important, maybe about erasure (damn it).
I have a method, which I wanted to create array of size n filled with values from gen:
def testArray[T](n: Int, gen: =>T) {
val arr = Array.fill(n)(gen)
...
}
And use it, for example as:
testArray(10, util.Random.nextInt(10))
But I get error:
scala: could not find implicit value for evidence parameter of type scala.reflect.ClassManifest[T]
val arr = Array.fill(n)(gen)
^
Please, explain what I did wrong, why this error, and what kind of code it makes impossible?
That is because in testArray the concrete type of T is not known at compile time. Your signature has to look like def testArray[T : ClassManifest](n: Int, gen: =>T), this will add an implicit parameter of type ClassManifest[T] to your method, that is automatically passed to the call of testArray and then further passed to the Array.fill call. This is called a context bound.
The Array.fill method has the following signature:
def fill[T](n: Int)(elem: => T)(implicit arg0: ClassManifest[T]): Array[T]
In order to get an instance of ClassManifest[T] you need to know the concrete type. A ClassManifest can be obtained like this:
implicitly[ClassManifest[String]]
A ClassManifest is implicitly available for every concrete type.
For any implicit error, you can add the implicits you require to the method with the type parameter:
def wrap[T](n:Int)(elem: => T)(implicit c:ClassManifest[T], o:Ordering[T])
If you did not yourself introduce ClassManifest or Ordering, the writers of the library have (most likely) provided sensible defaults for you.
If you would call the wrap method:
wrap(2)(3)
It's expanded like this:
wrap[Int](2)(3)(implicitly[ClassManifest[Int]], implicitly[Ordering[Int]])
If you introduced a custom class Person here, you would get an error for not finding an implicit instance of Ordering[Person]. The writers of the library could not have known how to order Person. You could solve that like this:
class Person
implicit val o = new Ordering[Person] { // implement required methods }
wrap(2)(new Person)
The Scala compiler looks in different scopes for implicits, an Ordering would usually not be specified like this. I suggest you look up implicit resolution on the internet to learn more about it.
I am using the Azavea Numeric Scala library for generic maths operations. However, I cannot use these with the Scala Collections API, as they require a scala Numeric and it appears as though the two Numerics are mutually exclusive. Is there any way I can avoid re-implementing all mathematical operations on Scala Collections for Azavea Numeric, apart from requiring all types to have context bounds for both Numerics?
import Predef.{any2stringadd => _, _}
class Numeric {
def addOne[T: com.azavea.math.Numeric](x: T) {
import com.azavea.math.EasyImplicits._
val y = x + 1 // Compiles
val seq = Seq(x)
val z = seq.sum // Could not find implicit value for parameter num: Numeric[T]
}
}
Where Azavea Numeric is defined as
trait Numeric[#scala.specialized A] extends java.lang.Object with
com.azavea.math.ConvertableFrom[A] with com.azavea.math.ConvertableTo[A] with scala.ScalaObject {
def abs(a:A):A
...remaining methods redacted...
}
object Numeric {
implicit object IntIsNumeric extends IntIsNumeric
implicit object LongIsNumeric extends LongIsNumeric
implicit object FloatIsNumeric extends FloatIsNumeric
implicit object DoubleIsNumeric extends DoubleIsNumeric
implicit object BigIntIsNumeric extends BigIntIsNumeric
implicit object BigDecimalIsNumeric extends BigDecimalIsNumeric
def numeric[#specialized(Int, Long, Float, Double) A:Numeric]:Numeric[A] = implicitly[Numeric[A]]
}
You can use Régis Jean-Gilles solution, which is a good one, and wrap Azavea's Numeric. You can also try recreating the methods yourself, but using Azavea's Numeric. Aside from NumericRange, most should be pretty straightforward to implement.
You may be interested in Spire though, which succeeds Azavea's Numeric library. It has all the same features, but some new ones as well (more operations, new number types, sorting & selection, etc.). If you are using 2.10 (most of our work is being directed at 2.10), then using Spire's Numeric eliminates virtually all overhead of a generic approach and often runs as fast as a direct (non-generic) implementation.
That said, I think your question is a good suggestion; we should really add a toScalaNumeric method on Numeric. Which Scala collection methods were you planning on using? Spire adds several new methods to Arrays, such as qsum, qproduct, qnorm(p), qsort, qselect(k), etc.
The most general solution would be to write a class that wraps com.azavea.math.Numeric and implements scala.math.Numeric in terms of it:
class AzaveaNumericWrapper[T]( implicit val n: com.azavea.math.Numeric[T] ) extends scala.math.Numeric {
def compare (x: T, y: T): Int = n.compare(x, y)
def minus (x: T, y: T): T = n.minus(x, y)
// and so on
}
Then implement an implicit conversion:
// NOTE: in scala 2.10, we could directly declare AzaveaNumericWrapper as an implicit class
implicit def toAzaveaNumericWrapper[T]( implicit n: com.azavea.math.Numeric[T] ) = new AzaveaNumericWrapper( n )
The fact that n is itself an implicit is key here: it allows for implicit values of type com.azavea.math.Numeric to be automatically used where na implicit value of
type scala.math.Numeric is expected.
Note that to be complete, you'll probably want to do the reverse too (write a class ScalaNumericWrapper that implements com.azavea.math.Numeric in terms of scala.math.Numeric).
Now, there is a disadvantage to the above solution: you get a conversion (and thus an instanciation) on each call (to a method that has a context bound of type scala.math.Numeric, and where you only an instance of com.azavea.math.Numeric is in scope).
So you will actually want to define an implicit singleton instance of AzaveaNumericWrapper for each of your numeric type. Assuming that you have types MyType and MyOtherType for which you defined instances of com.azavea.math.Numeric:
implicit object MyTypeIsNumeric extends AzaveaNumericWrapper[MyType]
implicit object MyOtherTypeIsNumeric extends AzaveaNumericWrapper[MyOtherType]
//...
Also, keep in mind that the apparent main purpose of azavea's Numeric class is to greatly enhance execution speed (mostly due to type parameter specialization).
Using the wrapper as above, you lose the specialization and hence the speed that comes out of it. Specialization has to be used all the way down,
and as soon as you call a generic method that is not specialized, you enter in the world of unspecialized generics (even if that method then calls back a specialized method).
So in cases where speed matters, try to use azavea's Numeric directly instead of scala's Numeric (just because AzaveaNumericWrapper uses it internally
does not mean that you will get any speed increase, as specialization won't happen here).
You may have noticed that I avoided in my examples to define instances of AzaveaNumericWrapper for types Int, Long and so on.
This is because there are already (in the standard library) implicit values of scala.math.Numeric for these types.
You might be tempted to just hide them (via something like import scala.math.Numeric.{ShortIsIntegral => _}), so as to be sure that your own (azavea backed) version is used,
but there is no point. The only reason I can think of would be to make it run faster, but as explained above, it wont.