Suppose I have an interface for a Thing:
abstract class Thing[A](a_thing: A) {
def thingA = a_thing
}
and I implement that Thing as follows:
class SpecificThing(a: String) extends Thing[String](a)
Furthermore, suppose I have a function that takes a Thing and a lambda that does something to that Thing as parameters:
def doSomething[A](fn: Thing[A] => A, t: Thing[A]) : A = fn(t)
Now, let's use this stuff:
val st = new SpecificThing("hi")
val fn1: (Thing[String]) => String = (t: Thing[String]) => { t.thingA }
println(doSomething(fn1, st))
This prints hi. So far, so good. But I'm lazy, and I don't like typing so much, so I change my program to the following:
type MyThing = Thing[String]
val st = new SpecificThing("hi")
val fn2: (MyThing) => String = (t: MyThing) => { t.thingA }
println(doSomething(fn2, st))
and this also prints hi. Fabulous! The compiler can tell that a SpecificThing is both a Thing[String] and a MyThing. But what about this case?
val st = new SpecificThing("hi")
val fn3: (SpecificThing) => String = (t: SpecificThing) => { t.thingA }
println(doSomething(fn3, st))
Now I get:
Error:(14, 23) type mismatch;
found : SpecificThing => String
required: Thing[?] => ?
println(doSomething(fn3, st))
^
What's going on? What's a Thing[?]?
f3 isn't a Thing[String] => String, it's a SpecificThing => String. For an example of why they couldn't be compatible:
class SpecificThing2 extends Thing[String] {
def thingB = 2.0
}
val f4: SpecificThing2 => String = {
st: SpecificThing2 => f"%f${st.thingB / 3.0}"
}
val t = new Thing[String]("haha"){}
f4(t) // would be an error when f4 tried to access t.thingB
More formally, Function1 is contravariant in its first type parameter, Function1[-T, +R].
A Thing[?] is what it looks like; it's a Thing[X] for some unknown type X. The compiler is gamely trying to infer what the type A should be, but it can't make it work: it needs a Thing[A] => A for some (unknown-to-it) type A, and you're passing it a SpecificThing => String; that's the error.
Related
i am trying to make some scala functions that would help making flink map and filter operations that redirect their error to a dead letter queue.
However, i'm struggling with scala's type erasure which prevents me from making them generic. The implementation of mapWithDeadLetterQueue below does not compile.
sealed trait ProcessingResult[T]
case class ProcessingSuccess[T,U](result: U) extends ProcessingResult[T]
case class ProcessingError[T: TypeInformation](errorMessage: String, exceptionClass: String, stackTrace: String, sourceMessage: T) extends ProcessingResult[T]
object FlinkUtils {
// https://stackoverflow.com/questions/1803036/how-to-write-asinstanceofoption-in-scala
implicit class Castable(val obj: AnyRef) extends AnyVal {
def asInstanceOfOpt[T <: AnyRef : ClassTag] = {
obj match {
case t: T => Some(t)
case _ => None
}
}
}
def mapWithDeadLetterQueue[T: TypeInformation,U: TypeInformation](source: DataStream[T], func: (T => U)): (DataStream[U], DataStream[ProcessingError[T]]) = {
val mapped = source.map(x => {
val result = Try(func(x))
result match {
case Success(value) => ProcessingSuccess(value)
case Failure(exception) => ProcessingError(exception.getMessage, exception.getClass.getName, exception.getStackTrace.mkString("\n"), x)
}
} )
val mappedSuccess = mapped.flatMap((x: ProcessingResult[T]) => x.asInstanceOfOpt[ProcessingSuccess[T,U]]).map(x => x.result)
val mappedFailure = mapped.flatMap((x: ProcessingResult[T]) => x.asInstanceOfOpt[ProcessingError[T]])
(mappedSuccess, mappedFailure)
}
}
I get:
[error] FlinkUtils.scala:35:36: overloaded method value flatMap with alternatives:
[error] [R](x$1: org.apache.flink.api.common.functions.FlatMapFunction[Product with Serializable with ProcessingResult[_ <: T],R], x$2: org.apache.flink.api.common.typeinfo.TypeInformation[R])org.apache.flink.streaming.api.datastream.SingleOutputStreamOperator[R] <and>
[error] [R](x$1: org.apache.flink.api.common.functions.FlatMapFunction[Product with Serializable with ProcessingResult[_ <: T],R])org.apache.flink.streaming.api.datastream.SingleOutputStreamOperator[R]
[error] cannot be applied to (ProcessingResult[T] => Option[ProcessingSuccess[T,U]])
[error] val mappedSuccess = mapped.flatMap((x: ProcessingResult[T]) => x.asInstanceOfOpt[ProcessingSuccess[T,U]]).map(x => x.result)
Is there a way to make this work ?
Ok, i'm going to answer my own question. I made a couple of mistakes:
First of all, i accidentically included the java DataStream class instead of the scala DataStream class (this happens all the time). The java variant obviously doesn't accept a scala lambda for map/filter/flatmap
Second, sealed traits are not supported by flinks serialisation. There is a project that should solve it but I didn't try it yet.
Solution: first i didn't use sealed trait but a simple case class with two Options (bit less expressive, but still works):
case class ProcessingError[T](errorMessage: String, exceptionClass: String, stackTrace: String, sourceMessage: T)
case class ProcessingResult[T: TypeInformation, U: TypeInformation](result: Option[U], error: Option[ProcessingError[T]])
Then, i could have everything working like so:
object FlinkUtils {
def mapWithDeadLetterQueue[T: TypeInformation: ClassTag,U: TypeInformation: ClassTag]
(source: DataStream[T], func: (T => U)):
(DataStream[U], DataStream[ProcessingError[T]]) = {
implicit val typeInfo = TypeInformation.of(classOf[ProcessingResult[T,U]])
val mapped = source.map((x: T) => {
val result = Try(func(x))
result match {
case Success(value) => ProcessingResult[T, U](Some(value), None)
case Failure(exception) => ProcessingResult[T, U](None, Some(
ProcessingError(exception.getMessage, exception.getClass.getName,
exception.getStackTrace.mkString("\n"), x)))
}
} )
val mappedSuccess = mapped.flatMap((x: ProcessingResult[T,U]) => x.result)
val mappedFailure = mapped.flatMap((x: ProcessingResult[T,U]) => x.error)
(mappedSuccess, mappedFailure)
}
}
the flatMap and filter functions look very similar, but they use a ProcessingResult[T,List[T]] and a ProcessingResult[T,T] respectively.
I use the functions like this:
val (result, errors) = FlinkUtils.filterWithDeadLetterQueue(input, (x: MyMessage) => {
x.`type` match {
case "something" => throw new Exception("how how how")
case "something else" => false
case _ => true
}
})
I'm puzzled by the compile error for the line that defines endState. Here is the code:
import java.util.UUID
object Evaluation {
def default: Evaluation[Unit, Unit] = new Evaluation[Unit, Unit](identity)
}
case class Evaluation[From, +To](evaluate: (From) => To)
object FSMLike {
val uuidEmpty: UUID = new UUID(0L, 0L)
val endState: Evaluation[Unit, FSMLike[Nothing]] = new Evaluation(() => FSMEmpty)
lazy val stop: FSMEntry[Unit, Unit, Nothing] = FSMEntry(uuidEmpty, Evaluation.default, endState)
def apply[From1, From2, To](
action: Evaluation[From1, Unit],
nextState: Evaluation[From2, FSMLike[To]]
): (UUID, FSMLike[To]) = {
val uuid = UUID.randomUUID
uuid -> FSMEntry(uuid, action, nextState)
}
}
sealed trait FSMLike[+A]
case object FSMEmpty extends FSMLike[Nothing]
case class FSMEntry[From1, From2, +To](
id: UUID,
action: Evaluation[From1, Unit],
nextState: Evaluation[From2, FSMLike[To]]
) extends FSMLike[To] {
def transition(arg1: From1, arg2: From2): FSMLike[To] = {
action.evaluate(arg1)
nextState.evaluate(arg2)
}
override def toString: String = s"^$id^"
}
Here is the error:
Error:(14, 72) type mismatch;
found : () => FSMEmpty.type (with underlying type () => FSMEmpty.type)
required: Unit => FSMLike[Nothing]
val endState: Evaluation[Unit, FSMLike[Nothing]] = new Evaluation(() => FSMEmpty)
You are trying to pass () => FSMEmpty, a function with no arguments, where a function with one argument of type Unit is expected. Sure, when you use () as an expression, it's the only value of type Unit, but the left-hand side of => isn't an expression.
You should write _ => FSMEmpty instead. { case () => FSMEmpty } would work as well, I think, but there isn't much point to using it.
I have a templated class, and depending on the type of T, I want to print something different.
class Clazz[T](fn: T => String) {}
Ideally, I would like to do something like pattern match T (which I can't do):
T match {
case x:Int => println("function from int to string")
case _ => //... and so forth
}
I tried:
class Clazz[T](fn: T => String) {
def resolve():String = fn match {
case f:(Int => String) => println("function from int to string")
case _ => //...etc.
}
}
and the error message I got was that:
non-variable type argument Int in type pattern Int => String is unchecked since it is eliminated by erasure
Is there a way to work around this?
I would use TypeTag so I can compare T to other types without losing any type info:
import scala.reflect.runtime.universe._
class Clazz[T: TypeTag](fn: T => String) {
val resolve = typeOf[T] match {
case t if t =:= typeOf[Int] => "function from Int to String"
case t if t =:= typeOf[Byte] => ..
}
}
But if I just wanted to print out something different from toString, I would go with a type class:
trait Print[T] {
def toText(t: T): String
def apply(t: T) = print(toText(t))
}
def print[T: Print](t: T) = implicitly[Print[T]].apply(t)
implicit object PrintInt extends Print[Int] {
def toText(t: Int) = s"Int: $t"
}
implicit object PrintByte extends Print[Byte] {
def toText(t: Byte) = s"Byte: $t"
}
PrintInt(3) //=> Int: 3
print(3) //=> Int: 3
3.print is even possible if you add this:
implicit class Printable[T:Print](t: T) {
def print = implicitly[Print[T]].apply(t)
}
3.print //=> Int: 3
I am having trouble getting my code with parameterized types to pass the scala compiler. My goal is to be able to express (Predicate, Action) pairs as shown in the MyProg object.
trait ProgBase {
type Predicate[T] = T => Boolean
type Action[T] = T => Unit
private var prog = List[(Predicate[Any], Action[Any])]()
final def on[T <: Any](pred: Predicate[T])(action: Action[T]) = {
prog = (pred, action) :: prog // gives type mismatch
}
// remainder of trait elided
}
object MyProg extends ProgBase {
on[String](s => !s.isEmpty) { s =>
println(s + " is not empty")
}
on[Int](i => i.isValidByte) { i =>
println(i + " can fit in a byte")
}
}
By specifying that T has an upper bound of Any, I hoped this would appease the compiler, but clearly I am missing something:
[error] ......ProgBase.scala:8 type mismatch;
[error] found : (T => Boolean, T => Unit)
[error] required: (Any => Boolean, Any => Unit)
[error] prog = (pred, action) :: prog
[error] ^
First of all, answer to your question, if you write:
private var prog = List[(Predicate[_ <: Any], Action[_ <: Any])]()
It all compiles OK. We should use wildcards, 'cause the type of elements are unknown.
Second, maybe you mistyped, you can't use your on function as you used it, use it something like:
on[String](s => !s.isEmpty)(s => !s.isEmpty)
It caused by type mismatch: type Action[T] = T => Unit but println has type Unit,
so as a variant u can simply write: type Action = Unit. Obviously, u can avoid using this type alias at all.
Third, maybe you already know, and I shoudn't tell you, that in fact, you lose all information about predicate types - let's check it using Scala reflection:
import scala.reflect.runtime.{universe => ru}
def getTypeTag[T: ru.TypeTag](obj: T) = ru.typeTag[T]
val s: String = "123"
val i: Int = 123
on[String](s => !s.isEmpty)(s => !s.isEmpty)
on[Int](i => i.isValidByte)(i => i.isValidByte)
getTypeTag((MyProg.prog.head._1)).tpe =:= ru.typeOf[(String) => Boolean] //>false!
So you see the problem.
To deal with it you can use heterogeneous lists. Lists and other various powerful structures you can find in shapeless: https://github.com/milessabin/shapeless
MWE (obs.: I am avoiding to have to instantiate a class every call of c, this is why functions are desired):
object Main extends App {
def a(s:String, i:Int) ={
s + i * i //some complex op that yields String
}
def b(i:Int) ={
i / 3 //another complex op that yields Int
}
def c(f: Any => Any) = {
val L = List(1,2,3,4) //list of complex elements
L map f //apply f within some complex loop
}
println(c(a))
/*
scala: type mismatch;
found : (String, Int) => String
required: Any => Any
println(c(a))
^
*/
println(c(b))
/*
scala: type mismatch;
found : Int => Int
required: Any => Any
println(c(b))
^
*/
}
Maybe an equivalent question would be "Is there some kind of function inheritance?",
like
def f
def fa(i: Int):String extends f
def fb(s: String):Int extends f
What you're trying to do isn't type-safe, since if it were you could pass a String to a function which takes an Int parameter:
e.g.
def c(f: Any => Any) = {
val L = List("a", "b", "c")
L map f
}
c(a)
However you can take a function of type Int => Any, since it is safe to assign a more derived type to Any.
def c(f: Int => Any) = {
val l = List(1,2,3,4)
l.map(f)
}
This is now safe:
val bList: List[Any] = c(b)
You still can't pass a to c however, since it requires two arguments instead of one. You can partially apply the first String argument and pass that:
val cList = c(a("SomeString", _:Int))
If you find yourself using Any,
you are probably doing something wrong, and
you most likely need generics.
In your case
def c[X,Y](f: X => Y) = { ... }
would probably do the trick, depending on what you have inside that complex loop.