I'm puzzled by the compile error for the line that defines endState. Here is the code:
import java.util.UUID
object Evaluation {
def default: Evaluation[Unit, Unit] = new Evaluation[Unit, Unit](identity)
}
case class Evaluation[From, +To](evaluate: (From) => To)
object FSMLike {
val uuidEmpty: UUID = new UUID(0L, 0L)
val endState: Evaluation[Unit, FSMLike[Nothing]] = new Evaluation(() => FSMEmpty)
lazy val stop: FSMEntry[Unit, Unit, Nothing] = FSMEntry(uuidEmpty, Evaluation.default, endState)
def apply[From1, From2, To](
action: Evaluation[From1, Unit],
nextState: Evaluation[From2, FSMLike[To]]
): (UUID, FSMLike[To]) = {
val uuid = UUID.randomUUID
uuid -> FSMEntry(uuid, action, nextState)
}
}
sealed trait FSMLike[+A]
case object FSMEmpty extends FSMLike[Nothing]
case class FSMEntry[From1, From2, +To](
id: UUID,
action: Evaluation[From1, Unit],
nextState: Evaluation[From2, FSMLike[To]]
) extends FSMLike[To] {
def transition(arg1: From1, arg2: From2): FSMLike[To] = {
action.evaluate(arg1)
nextState.evaluate(arg2)
}
override def toString: String = s"^$id^"
}
Here is the error:
Error:(14, 72) type mismatch;
found : () => FSMEmpty.type (with underlying type () => FSMEmpty.type)
required: Unit => FSMLike[Nothing]
val endState: Evaluation[Unit, FSMLike[Nothing]] = new Evaluation(() => FSMEmpty)
You are trying to pass () => FSMEmpty, a function with no arguments, where a function with one argument of type Unit is expected. Sure, when you use () as an expression, it's the only value of type Unit, but the left-hand side of => isn't an expression.
You should write _ => FSMEmpty instead. { case () => FSMEmpty } would work as well, I think, but there isn't much point to using it.
Related
i am trying to make some scala functions that would help making flink map and filter operations that redirect their error to a dead letter queue.
However, i'm struggling with scala's type erasure which prevents me from making them generic. The implementation of mapWithDeadLetterQueue below does not compile.
sealed trait ProcessingResult[T]
case class ProcessingSuccess[T,U](result: U) extends ProcessingResult[T]
case class ProcessingError[T: TypeInformation](errorMessage: String, exceptionClass: String, stackTrace: String, sourceMessage: T) extends ProcessingResult[T]
object FlinkUtils {
// https://stackoverflow.com/questions/1803036/how-to-write-asinstanceofoption-in-scala
implicit class Castable(val obj: AnyRef) extends AnyVal {
def asInstanceOfOpt[T <: AnyRef : ClassTag] = {
obj match {
case t: T => Some(t)
case _ => None
}
}
}
def mapWithDeadLetterQueue[T: TypeInformation,U: TypeInformation](source: DataStream[T], func: (T => U)): (DataStream[U], DataStream[ProcessingError[T]]) = {
val mapped = source.map(x => {
val result = Try(func(x))
result match {
case Success(value) => ProcessingSuccess(value)
case Failure(exception) => ProcessingError(exception.getMessage, exception.getClass.getName, exception.getStackTrace.mkString("\n"), x)
}
} )
val mappedSuccess = mapped.flatMap((x: ProcessingResult[T]) => x.asInstanceOfOpt[ProcessingSuccess[T,U]]).map(x => x.result)
val mappedFailure = mapped.flatMap((x: ProcessingResult[T]) => x.asInstanceOfOpt[ProcessingError[T]])
(mappedSuccess, mappedFailure)
}
}
I get:
[error] FlinkUtils.scala:35:36: overloaded method value flatMap with alternatives:
[error] [R](x$1: org.apache.flink.api.common.functions.FlatMapFunction[Product with Serializable with ProcessingResult[_ <: T],R], x$2: org.apache.flink.api.common.typeinfo.TypeInformation[R])org.apache.flink.streaming.api.datastream.SingleOutputStreamOperator[R] <and>
[error] [R](x$1: org.apache.flink.api.common.functions.FlatMapFunction[Product with Serializable with ProcessingResult[_ <: T],R])org.apache.flink.streaming.api.datastream.SingleOutputStreamOperator[R]
[error] cannot be applied to (ProcessingResult[T] => Option[ProcessingSuccess[T,U]])
[error] val mappedSuccess = mapped.flatMap((x: ProcessingResult[T]) => x.asInstanceOfOpt[ProcessingSuccess[T,U]]).map(x => x.result)
Is there a way to make this work ?
Ok, i'm going to answer my own question. I made a couple of mistakes:
First of all, i accidentically included the java DataStream class instead of the scala DataStream class (this happens all the time). The java variant obviously doesn't accept a scala lambda for map/filter/flatmap
Second, sealed traits are not supported by flinks serialisation. There is a project that should solve it but I didn't try it yet.
Solution: first i didn't use sealed trait but a simple case class with two Options (bit less expressive, but still works):
case class ProcessingError[T](errorMessage: String, exceptionClass: String, stackTrace: String, sourceMessage: T)
case class ProcessingResult[T: TypeInformation, U: TypeInformation](result: Option[U], error: Option[ProcessingError[T]])
Then, i could have everything working like so:
object FlinkUtils {
def mapWithDeadLetterQueue[T: TypeInformation: ClassTag,U: TypeInformation: ClassTag]
(source: DataStream[T], func: (T => U)):
(DataStream[U], DataStream[ProcessingError[T]]) = {
implicit val typeInfo = TypeInformation.of(classOf[ProcessingResult[T,U]])
val mapped = source.map((x: T) => {
val result = Try(func(x))
result match {
case Success(value) => ProcessingResult[T, U](Some(value), None)
case Failure(exception) => ProcessingResult[T, U](None, Some(
ProcessingError(exception.getMessage, exception.getClass.getName,
exception.getStackTrace.mkString("\n"), x)))
}
} )
val mappedSuccess = mapped.flatMap((x: ProcessingResult[T,U]) => x.result)
val mappedFailure = mapped.flatMap((x: ProcessingResult[T,U]) => x.error)
(mappedSuccess, mappedFailure)
}
}
the flatMap and filter functions look very similar, but they use a ProcessingResult[T,List[T]] and a ProcessingResult[T,T] respectively.
I use the functions like this:
val (result, errors) = FlinkUtils.filterWithDeadLetterQueue(input, (x: MyMessage) => {
x.`type` match {
case "something" => throw new Exception("how how how")
case "something else" => false
case _ => true
}
})
I want to generate a bunch of objects at compile time that follow a simple pattern, so I wrote the following macro:
object MyMacro {
def readWrite[T](taName: String, readParse: String => T, label: String, format: T => String): Any = macro readWriteImpl[T]
def readWriteImpl[T: c.WeakTypeTag](c: Context)(taName: c.Expr[String], readParse: c.Expr[String => T], label: c.Expr[String], format: c.Expr[T => String]): c.Expr[Any] = {
import c.universe._
def termName(s: c.Expr[String]): TermName = s.tree match {
case Literal(Constant(s: String)) => TermName(s)
case _ => c.abort(c.enclosingPosition, "Not a string literal")
}
c.Expr[Any](q"""
object ${termName(taName)} extends TypeAdapter.=:=[${implicitly[c.WeakTypeTag[T]].tpe}] {
def read[WIRE](path: Path, reader: Transceiver[WIRE], isMapKey: Boolean = false): ${implicitly[c.WeakTypeTag[T]].tpe} =
reader.readString(path) match {
case null => null.asInstanceOf[${implicitly[c.WeakTypeTag[T]].tpe}]
case s => Try( $readParse(s) ) match {
case Success(d) => d
case Failure(u) => throw new ReadMalformedError(path, "Failed to parse "+${termName(label)}+" from input '"+s+"'", List.empty[String], u)
}
}
def write[WIRE](t: ${implicitly[c.WeakTypeTag[T]].tpe}, writer: Transceiver[WIRE], out: Builder[Any, WIRE]): Unit =
t match {
case null => writer.writeNull(out)
case _ => writer.writeString($format(t), out)
}
}
""")
}
}
I'm not sure I have the return value for readWrite and readWriteImpl correct--the compiler complains mightily about some assertion failure!
I'm also not sure how to actually consume this macro. First I tried (in a separate compilation unit):
object TimeFactories {
MyMacro.readWrite[Duration](
"DurationTypeAdapterFactory",
(s: String) => Duration.parse(s),
"Duration",
(t: Duration) => t.toString)
}
Didn't work. If I tried to reference TimeFactories.DurationTypeAdapterFactory I got an error saying it wasn't found. Next I thought I'd try assigning it to a val...didn't work either:
object Foo {
val duration = MyMacro.readWrite[Duration](
"DurationTypeAdapterFactory",
(s: String) => Duration.parse(s),
"Duration",
(t: Duration) => t.toString).asInstanceOf[TypeAdapterFactory]
}
How can I wire this up so I get generated code compiled like this:
object TimeFactories{
object DurationTypeAdapterFactory extends TypeAdapter.=:=[Duration] {
def read[WIRE](path: Path, reader: Transceiver[WIRE], isMapKey: Boolean = false): Duration =
reader.readString(path) match {
case null => null.asInstanceOf[Duration]
case s => Try( Duration.parse(s) ) match {
case Success(d) => d
case Failure(u) => throw new ReadMalformedError(path, "Failed to parse Duration from input 'Duration'", List.empty[String], u)
}
}
def write[WIRE](t: Duration, writer: Transceiver[WIRE], out: Builder[Any, WIRE]): Unit =
t match {
case null => writer.writeNull(out)
case _ => writer.writeString(t.toString, out)
}
}
// ... More invocations of the readWrite macro with other types for T
}
I don't think, that you can generate new identifiers using macros and than use them publicly.
Instead, try to replace object ${termName(taName)} extends TypeAdapter simply with new TypeAdapter and assign invocation of the macro to a val (as in your second example). You will then reference an anonymous (and generated) class stored in a val. Parameter taName becomes redundant.
I have the following:
object T {
abstract class First {
def doSomething= (s:String) => Unit
}
class Second extends First {
override def doSomething = {
(s:String) => ()
}
}
def main(args: Array[String]): Unit = {
new Second().doSomething
}
}
but this fails to compile with the error:
Error:(8, 21) type mismatch;
found : Unit
required: Unit.type
(s:String) => ()
Why isn't the override from class Second valid? How could I make it work?
The problem is that (s:String) => Unit is returning Unit.type - instead change it to (s:String) => () (or to a method if you didn't mean to be returning Function1[String, Unit] from your method.)
To put it another way:
def doSomething = (s:String) => Unit
is really:
def doSomething: (String) => Unit.type =
// A function that takes a string and returns the companion type of Unit
(s: String) => Unit
While what you probably wanted was:
def doSomething: (String) => Unit =
// A function that takes a string and returns Unit
(s: String) => ()
Or maybe:
def doSomething(s: String): Unit
// An abstract method that takes a string and returns nothing.
(s:String) => Unit returns Unit.type ie it returns the type rather than a value of that type. You want to do (s:String) => (), which will return a value, whose type is Unit.
' trait Processor0 {
def process = (x:String) => ()
}
case class Processor2() extends Processor0 {
override def process = (x:String) => println("Processor2 x :" + x)
}
case class Processor3() extends Processor0 {
override def process = (x:String) => println("Processor3 x :" + x)
}
object DemoProc2 {
def main( args : Array[String]):Unit ={
val s:String = "yes"
val myFuncs: Map[String,(String) => () ]= Map( //Error 1. '=>' expected but ']' found. 2.identifier expected but ';' found.
"string2" -> (() => ProcessorTwo().process(s)),
"string3" -> (() => ProcessorThree().process(s))
)
myFuncs.values.foreach(v => v());
}
} '
Suppose I have an interface for a Thing:
abstract class Thing[A](a_thing: A) {
def thingA = a_thing
}
and I implement that Thing as follows:
class SpecificThing(a: String) extends Thing[String](a)
Furthermore, suppose I have a function that takes a Thing and a lambda that does something to that Thing as parameters:
def doSomething[A](fn: Thing[A] => A, t: Thing[A]) : A = fn(t)
Now, let's use this stuff:
val st = new SpecificThing("hi")
val fn1: (Thing[String]) => String = (t: Thing[String]) => { t.thingA }
println(doSomething(fn1, st))
This prints hi. So far, so good. But I'm lazy, and I don't like typing so much, so I change my program to the following:
type MyThing = Thing[String]
val st = new SpecificThing("hi")
val fn2: (MyThing) => String = (t: MyThing) => { t.thingA }
println(doSomething(fn2, st))
and this also prints hi. Fabulous! The compiler can tell that a SpecificThing is both a Thing[String] and a MyThing. But what about this case?
val st = new SpecificThing("hi")
val fn3: (SpecificThing) => String = (t: SpecificThing) => { t.thingA }
println(doSomething(fn3, st))
Now I get:
Error:(14, 23) type mismatch;
found : SpecificThing => String
required: Thing[?] => ?
println(doSomething(fn3, st))
^
What's going on? What's a Thing[?]?
f3 isn't a Thing[String] => String, it's a SpecificThing => String. For an example of why they couldn't be compatible:
class SpecificThing2 extends Thing[String] {
def thingB = 2.0
}
val f4: SpecificThing2 => String = {
st: SpecificThing2 => f"%f${st.thingB / 3.0}"
}
val t = new Thing[String]("haha"){}
f4(t) // would be an error when f4 tried to access t.thingB
More formally, Function1 is contravariant in its first type parameter, Function1[-T, +R].
A Thing[?] is what it looks like; it's a Thing[X] for some unknown type X. The compiler is gamely trying to infer what the type A should be, but it can't make it work: it needs a Thing[A] => A for some (unknown-to-it) type A, and you're passing it a SpecificThing => String; that's the error.
In Scala 2.9.1, this does not compile, failing with not found: value b:
case class CaseClass(field: String)
object SomeObject {
//val kludge = field
def x(input: (CaseClass, String) => CaseClass): Unit = ()
val field = x((a, b) => a.copy(field = b))
}
However, this does:
case class CaseClass(field: String)
object SomeObject {
val kludge = field
def x(input: (CaseClass, String) => CaseClass): Unit = ()
val field = x((a, b) => a.copy(field = b))
}
The only difference is the commented line. If this isn't a bug, why is this the intended behavior?