Why cannot I load the file with the following code in spark-shell
import org.apache.spark.sql.types._
import org.apache.spark.sql.Encoder import org.apache.spark.sql.Encoders
import org.apache.spark.sql.expressions.Aggregator
case class Data(i: Int)
val customSummer = new Aggregator[Data, Int, Int] {
def zero: Int = 0
def reduce(b: Int, a: Data): Int = b + a.i
def merge(b1: Int, b2: Int): Int = b1 + b2
def finish(r: Int): Int = r
}.toColumn()
The error:
<console>:47: error: object creation impossible, since:
it has 2 unimplemented members.
/** As seen from <$anon: org.apache.spark.sql.expressions.Aggregator[Data,Int,Int]>, the missing signatures are as follows.
* For convenience, these are usable as stub implementations.
*/
def bufferEncoder: org.apache.spark.sql.Encoder[Int] = ???
def outputEncoder: org.apache.spark.sql.Encoder[Int] = ???
val customSummer = new Aggregator[Data, Int, Int] {
Update: #user8371915's solution works. But the following script cannot be loaded with a different error. I used :load script.sc in the spark-shell.
import org.apache.spark.sql.expressions.Aggregator
class MyClass extends Aggregator
Error:
loading ./script.sc...
import org.apache.spark.sql.expressions.Aggregator
<console>:11: error: not found: type Aggregator
class MyClass extends Aggregator
Update(2017-12-03): it doesn't seem to work within Zeppelin, either.
As per error message you didn't implement bufferEncoder and outputEncoder. Please check API docs for the list of abstract methods that have to be implemented.
These two should suffice:
def bufferEncoder: Encoder[Int] = Encoders.scalaInt
def outputEncoder: Encoder[Int] = Encoders.scalaInt
Related
In the following example, I want to use tagged types for the ids of my classes. I've created an utility trait to reduce some boilerplate (of tags/reads/writes declaration):
import java.util.UUID
import play.api.libs.json.{Format, Json, Reads, Writes}
trait Opaque[A] {
protected type Tagged[U] = { type Tag = U }
type ##[U, T] = U with Tagged[T]
trait Tag
def tag(a: A): A ## Tag = a.asInstanceOf[A ## Tag]
def untag(a: A ## Tag): A = a
implicit def reads(implicit r: Reads[A]): Reads[A ## Tag] =
r.map(tag)
implicit def writes(implicit w: Writes[A]): Writes[A ## Tag] =
w.contramap(untag)
implicit def format(implicit r: Reads[A], w: Writes[A]): Format[A ## Tag] =
Format(reads(r), writes(w))
}
final case class Foo(id: Foo.FooId.T, f1: Boolean)
object Foo {
object FooId extends Opaque[UUID] {
type T = UUID ## Tag
}
import FooId._
implicit val fmt: Format[Foo] = Json.format[Foo]
}
final case class Bar(id: Bar.BarId.T, fooId: Foo.FooId.T, b1: String)
object Bar {
object BarId extends Opaque[UUID] {
type T = UUID ## Tag
}
import Foo.FooId._
import BarId._
implicit val format: Format[Bar] = Json.format[Bar]
}
I have the following error from the compiler:
implicit val format: Format[Bar] = Json.format[Bar]
^
<pastie>:43: error: No instance of play.api.libs.json.Format is available for Opaque.<refinement>, Opaque.<refinement> in the implicit scope (Hint: if declared in the same file, make sure it's declared before)
I'm not able to explain why I'm having this behaviour, the error message is not explicit. I'm importing the Format for FooId and BarId needed for deriving a format for Bar class.
The thing is that names of implicits are significant.
Very simple example of that is following:
object MyObject {
implicit val i: Int = ???
}
import MyObject._
implicit val i: String = ???
// implicitly[Int] // doesn't compile
// implicitly[String] // doesn't compile
but
object MyObject {
implicit val i: Int = ???
}
import MyObject._
implicit val i1: String = ???
implicitly[Int] // compiles
implicitly[String] // compiles
If you want derivation Json.format[Bar] to work, there should be implicits Format[Bar.BarId.T], Format[Foo.FooId.T] in scope i.e. Format instances for fields of Bar. If you make the only import
import Foo.FooId._
implicitly[Format[Foo.FooId.T]] // compiles
and
import BarId._
implicitly[Format[Bar.BarId.T]] // compiles
but if you import both, since names of implicits collide
import Foo.FooId._
import BarId._
// implicitly[Format[Foo.FooId.T]] // doesn't compiles
// implicitly[Format[Bar.BarId.T]] // doesn't compiles
For example you can move trait Tag outside trait Opaque and make the only import. Then
implicitly[Format[Foo.FooId.T]]
implicitly[Format[Bar.BarId.T]]
Json.format[Bar]
will compile.
https://youtu.be/1h8xNBykZqM?t=681 Some Mistakes We Made When Designing Implicits, Mistake #1
NullPointerException on implicit resolution
I'm testing my own home-brewed Monoid classes in scala using the ScalaCheck library and ScalaTest
when attempting to implement DRY tests, I get the implicit error in the title:
Error:(16, 12) could not find implicit value for parameter arbA: org.scalacheck.Arbitrary[A]
forAll { (a: A) =>
^
here is the implementation of intAddition Monoid:
trait Monoid[A] {
def op(a1: A, a2: A): A
def zero: A
}
object Monoid {
...
val intAddition: Monoid[Int] = new Monoid[Int] {
override def op(a1: Int, a2: Int): Int = a1 + a2
override def zero: Int = 0
}
...
}
And the test suite:
import org.fpinscala.monoids.Monoid._
import org.fpinscala.testutils.UnitSpec
import org.scalatest.prop.PropertyChecks
import org.scalacheck.Arbitrary._
import scala.language.implicitConversions
class MonoidSpec extends UnitSpec with PropertyChecks {
def assertIdentityBehaviour[A](M: Monoid[A]): Unit = {
import M._
forAll { (a: A) =>
op(zero, a) should be(a)
op(a, zero) should be(a)
}
}
behavior of "intAdditionMonoid"
it should "obey identity laws" in {
assertIdentityBehaviour(intAddition)
}
}
This code compiles but fails at runtime (runtime type erasure?).
Is what I'm trying to achieve possible in Scala?
This code compiles
It doesn't; the error you give is a compilation error. It should be fixed by adding the implicit parameter it complains about:
def assertIdentityBehaviour[A](M: Monoid[A])(implicit arbA: Arbitrary[A]) = ...
// or equivalently, def assertIdentityBehaviour[A: Arbitrary](M: Monoid[A]) = ...
You are calling assertIdentityBehaviour only with A for which the parameter is available, but the error is in its definition.
I am trying to extend spark ml pipeline model with a filter transformer, after
abstract class RuleFilter[IN, T <: RuleFilter[IN, T]]
extends RuleTransformer with HasInputCol {
// def filterFuntion: String
/** #group setParam */
def setInputCol(value: String): T = set(inputCol, value).asInstanceOf[T]
protected def createFilterFunc: IN => Boolean
override def transform(df: DataFrame): DataFrame = {
transformSchema(df.schema, logging = true)
val transformUDF = udf[Boolean, IN](this.createFilterFunc)
df.filter(transformUDF(df($(inputCol))))
}
}
this code did not compile with an error:
No TypeTag available for IN
[error] val transformUDF = udf[Boolean, IN](this.createFilterFunc)
how do I let this work?
I need it to work with some explicit defined type in inherit class such as
class PriceFilter extends RuleFilter {
def createFilterFunc(val: Double) = val > 500
}
You need to tell the compiler explicitly that you want a TypeTag for type In:
import scala.reflect.runtime.universe._
abstract class RuleFilter[In: TypeTag, T <: RuleFilter[In, T]]
Given the following object:
scala> object P2pClient {
| type Num = Double
| type Weights = Array[Array[Num]]
| }
defined object P2pClient
and the following import:
import P2pClient._
The Weights type seems to be properly understood:
val w: Weights = new Weights(3)
w: P2pClient.Weights = Array(null, null, null)
But then why is it not working in the following construct:
case class SendWeightsReq(W: Weights) extends P2pReq[Weights] {
| override def value() = W
| }
<console>:12: error: not found: type Weights
case class SendWeightsReq(W: Weights) extends P2pReq[Weights] {
^
<console>:12: error: not found: type Weights
case class SendWeightsReq(W: Weights) extends P2pReq[Weights] {
^
Any ideas on what were happening here (/workaround) ?
Update There appear to be significant limitations on wildcard imports int he REPL. Here is another simpler illustration:
scala> import reflect.runtime.universe._
import reflect.runtime.universe._
scala> trait TT { def x[T <: java.io.Serializable : TypeTag]: T }
<console>:10: error: not found: type TypeTag
trait TT { def x[T <: java.io.Serializable : TypeTag]: T }
^
So we see that the wildcard import did not work. Here is the same code with the explicit package:
scala> trait TT { def x[T <: java.io.Serializable : reflect.runtime.universe.TypeTag]: T }
defined trait TT
I see the issue is due to my using the Spark REPL spark-shell. The issue is not happening in the normal Scala REPL.
Good news: your code works! Bad news: I have no idea why it doesn't work for you.
Here is my REPL session; I was going to build up to your example and see what broke, but nothing did.
scala> object P2pClient { type Num = Int; type Weights = Array[Array[Num]] }
defined object P2pClient
scala> import P2pClient._
import P2pClient._
scala> val x = new Weights(3)
x: Array[Array[P2pClient.Num]] = Array(null, null, null)
scala> case class SendWeights(W: Weights)
defined class SendWeights
scala> val s = new SendWeights(new Weights(3))
s: SendWeights = SendWeights([[I#1cab0bfb)
Hmmm. Trivial example works.
scala> case class SendWeights2(w: Weights) { def dubs = w }
defined class SendWeights2
Works when it has a body.
scala> trait P2pReq[+T] { def value(): Unit }
defined trait P2pReq
scala> case class SendWeights3(W: Weights) extends P2pReq[Weights] {
override def value() = W }
defined class SendWeights3
scala> val foo = new SendWeights3(new Weights(3))
foo: SendWeights3 = SendWeights3([[I#51dcb805)
res2: P2pClient.Weights = Array(null, null, null)
Aaaaand works when it extends a polymorphic trait and overrides one of its members. The only thing that may be different is your definition of P2pReq, but I don't see how that could cause one of your types to go unrecognized by the interpreter.
I tried to simplify the creation of ASTs, but got a weird error message:
case class Box(i: Int)
object M {
import language.experimental.macros
import scala.reflect.makro.Context
case class meth(obj: String, method: String)(implicit val c: Context) {
import c.universe._
def apply(xs: Tree*) =
Apply(Select(Ident(obj), newTermName(method)), xs.toList)
}
def box(n: Int): Box = macro boxImpl
def boxImpl(c: Context)(n: c.Expr[Int]): c.Expr[Box] = {
import c.universe._
implicit val cc: c.type = c
n.tree match {
case arg # Literal(Constant(_)) =>
meth("Box", "apply").apply(arg)
}
}
}
Error:
<console>:26: error: type mismatch;
found : c.universe.Literal
required: _2.c.universe.Tree where val _2: M.meth
possible cause: missing arguments for method or constructor
meth("Box", "apply").apply(arg)
^
Is it possible to infer the correct types into class meth? Or is there a workaround for the problem?
EDIT: Based on #retronyms answer I got this to work:
object M {
import language.experimental.macros
import scala.reflect.makro.Context
def meth(implicit c: Context) = new Meth[c.type](c)
class Meth[C <: Context](val c: C) {
import c.universe._
def apply(obj: String, method: String, xs: Tree*) =
Apply(Select(Ident(obj), newTermName(method)), xs.toList)
}
def box(n: Int): Box = macro boxImpl
def boxImpl(c: Context)(n: c.Expr[Int]): c.Expr[Box] = {
import c.universe._
implicit val cc: c.type = c
n.tree match {
case arg # Literal(Constant(_)) =>
c.Expr(meth.apply("Box", "apply", arg))
}
}
}
Constructors are not currently allowed to have dependent method types (SI-5712). It's on the radar to be fixed, hopefully for 2.10.1 or 2.11.
In the meantime, you can follow the pattern I used in macrocosm to reuse code within macro implementations.