Just when I thought I understood the basics of Scala's type system... :/
I'm trying to implement a class that reads the contents of a file and outputs a set of records. A record might be a single line, but it could also be a block of bytes, or anything. So what I'm after is a structure that allows the type of Reader to imply the type of the Record, which in turn will imply the correct Parser to use.
This structure works as long as MainApp.records(f) only returns one type of Reader. As soon as it can return more, I get this error:
could not find implicit value for parameter parser
I think the problem lies with the typed trait definitions at the top, but I cannot figure out how to fix the issue...
// Core traits
trait Record[T]
trait Reader[T] extends Iterable[Record[T]]
trait Parser[T] {
def parse(r: Record[T]): Option[Int]
}
// Concrete implementations
class LineRecord[T] extends Record[T]
class FileReader[T](f:File) extends Reader[T] {
val lines = Source.fromFile(f).getLines()
def iterator: Iterator[LineRecord[T]] =
new Iterator[LineRecord[T]] {
def next() = new LineRecord[T]
def hasNext = lines.hasNext
}
}
trait TypeA
object TypeA {
implicit object TypeAParser extends Parser[TypeA] {
def parse(r: Record[TypeA]): Option[Int] = ???
}
}
trait TypeB
object TypeB {
implicit object TypeBParser extends Parser[TypeB] {
def parse(r: Record[TypeB]): Option[Int] = ???
}
}
// The "app"
object MainApp {
def process(f: File) =
records(f) foreach { r => parse(r) }
def records(f: File) = {
if(true)
new FileReader[TypeA](f)
else
new FileReader[TypeB](f)
}
def parse[T](r: Record[T])(implicit parser: Parser[T]): Option[Int] =
parser.parse(r)
}
First off you must import the implicit object in order to use them:
import TypeA._
import TypeB._
That's not enough though. It seems like you're trying to apply implicits dynamically. That's not possible; they have to be found compile time.
If you import the objects as above and change the records so that the compiler finds the correct generic it will run fine:
def records(f: File) = new FileReader[TypeA](f)
But then it may not be what you were looking for ;)
The problem is that the return type of your records method is basically FileReader[_] (since it can return either FileReader[TypeA] or FileReader[TypeB]), and you don't provide an implicit argument of type Parser[Any]. If you remove the if-expression the return type is inferred to FileReader[TypeA], which works fine. I'm not sure what you're trying to do, but obviously the compiler can't select implicit argument based upon a type that is only known at runtime.
1) Using type with implicit inside as type parameter - doesn't bind this implicit to the host type, to do this change objects to the traits and mix them instead of generalizing (type-parametrizing):
def records(f: File) = {
if(true)
new FileReader(f) with TypeA
else
new FileReader(f) with TypeB
}
2) The parser should be in scope of function that calls parse. So you may try smthg like that:
def process(f: File) = {
val reader = records(f);
import reader._
reader foreach { r => parse(r) }
}
PlanB) Simpler alternative is to define type-parameter specific implicit methods inside the AppMain (or some trait mixed in), but it will work only if TypeA/TypeB is known on compile time, so records method can return concrete type:
implicit class TypeAParser(r: Record[TypeA]) {
def parse: Option[Int] = ???
}
implicit class TypeBParser(r: Record[TypeB]) {
def parse: Option[Int] = ???
}
def process[T <: TypeAorB](f: File) =
records[T](f).foreach(_.parse)
def recordsA[T <: TypeAorB](f: File) = new FileReader[T](f)
Here is, I think, the full set of modifications you need to do to get where I think you want to go.
import scala.io.Source
import java.io.File
import reflect.runtime.universe._
// Core traits
trait Record[+T]
trait Reader[+T] extends Iterable[Record[T]]
trait Parser[-T] {
def parse(r: Record[T]): Option[Int]
}
// Concrete implementations [unmodified]
class LineRecord[T] extends Record[T]
class FileReader[T](f:File) extends Reader[T] {
val lines = Source.fromFile(f).getLines()
def iterator: Iterator[LineRecord[T]] =
new Iterator[LineRecord[T]] {
def next() = new LineRecord[T]
def hasNext = lines.hasNext
}
}
sealed trait Alternatives
case class TypeA() extends Alternatives
object TypeA {
implicit object TypeAParser extends Parser[TypeA] {
def parse(r: Record[TypeA]): Option[Int] = ???
}
}
case class TypeB() extends Alternatives
object TypeB {
implicit object TypeBParser extends Parser[TypeB] {
def parse(r: Record[TypeB]): Option[Int] = ???
}
}
class ParseAlternator(parserA: Parser[TypeA], parserB: Parser[TypeB]) extends Parser[Alternatives] {
def parse(r: Record[Alternatives]): Option[Int] = r match {
case x: Record[TypeA #unchecked] if typeOf[Alternatives] =:= typeOf[TypeA] => parserA.parse(x)
case x: Record[TypeB #unchecked] if typeOf[Alternatives] =:= typeOf[TypeB] => parserB.parse(x)
}
}
object ParseAlternator {
implicit def parseAlternator(implicit parserA: Parser[TypeA], parserB: Parser[TypeB]): Parser[Alternatives] = new ParseAlternator(parserA, parserB)
}
// The "app"
object MainApp {
import ParseAlternator._
def process(f: File) =
records(f) foreach { r => parse(r) }
def records(f: File): Reader[Alternatives] = {
if(true)
new FileReader[TypeA](f)
else
new FileReader[TypeB](f)
}
def parse[T](r: Record[T])(implicit parser: Parser[T]): Option[Int] =
parser.parse(r)
}
The gist of it is: all of this would be completely classsical if only your parse instance did not have to pattern-match on a generic type but dealt directly with an Alternative instead.
It's this limitation (inherited from the JVM) that scala can't properly pattern-match on an object of a parametric type that requires the reflection & typeOf usage. Without it, you would just have type alternatives for your content (TypeA, TypeB), which you would add to a sealed trait, and which you would dispatch on, in an implicit that produces a Parser for their supertype.
Of course this isn't the only solution, it's just what I think is the meeting point of what's closest to what you're trying to do, with what's most idiomatic.
Related
I have a lot of classes such as DataFrameFlow, TextFlow, RDDFlow. They all derive from base class Flow.
Now I want to write a function judgeFlow which can read from a path: String and return something representing exact Flow type from which I can create corresponding instance. The whole code seems like the following
def judgeFlow(path:String) = /*1*/ {
Flow.getStoreType(path) match {
case StoreType.tdw =>
DataFrameFlow
case StoreType.hdfs =>
TextFlow
}
}
def createFlow(typeInfo:/*2*/) = /*3*/{
new typeInfo()
}
However, I don't know how to write in place 1, 2 and 3.
EDIT
Knowing how to construct them is not enough here, because I also want the following:
pattern matching through typeInfo
some ways to do asInstanceOf
EDIT 2
Definition of Flow
abstract class Flow(var outputName: String) extends Serializable{
def this() = this("")
...
}
Definition of DataFrameFlow
class DataFrameFlow(d: DataFrame, path: String) extends Flow {
var data: DataFrame = d
def this(data: DataFrame) = this(data, "")
def this(path: String) = this(null, path)
def this() = this(null, "")
...
}
Pattern matching can't return different types from different cases. The type returned by pattern matching is the least upper bound of types returned in cases.
When someone wants to return different types, most probably he/she wants a type class.
sealed abstract class Flow
class DataFrameFlow extends Flow
class TextFlow extends Flow
class RDDFlow extends Flow
trait JudgeFlow[In] {
type Out <: Flow
def judgeFlow(in: In): Out
}
object JudgeFlow {
implicit val `case1`: JudgeFlow[???] { type Out = DataFrameFlow } = ???
implicit val `case2`: JudgeFlow[???] { type Out = TextFlow } = ???
implicit val `case3`: JudgeFlow[???] { type Out = RDDFlow } = ???
}
def judgeFlow[In](in: In)(implicit jf: JudgeFlow[In]): jf.Out = jf.judgeFlow(in)
But the trouble is that types are resolved at compile time. You seem to want to choose a case based on a value of string i.e. at runtime. So you can't return more specific types than just Flow at compile time.
flatMap with Shapeless yield FlatMapper not found
It's hard to guess your use case completely.
But using Scala reflection you can try
import scala.reflect.runtime.universe._
import scala.reflect.runtime.currentMirror
def judgeFlow(path:String): Type = {
Flow.getStoreType(path) match {
case StoreType.tdw =>
typeOf[DataFrameFlow]
case StoreType.hdfs =>
typeOf[TextFlow]
}
}
def createFlow(typeInfo: Type): Flow = {
val constructorSymbol = typeInfo.decl(termNames.CONSTRUCTOR).asMethod
val classSymbol = typeInfo.typeSymbol.asClass
val classMirror = currentMirror.reflectClass(classSymbol)
val constructorMirror = classMirror.reflectConstructor(constructorSymbol)
constructorMirror().asInstanceOf[Flow]
}
Consider a trait that performs "encoding" of arbitrary objects:
trait Encoder[R] {
def encode(r: R): Array[Byte]
}
Assuming encoding of primitive types is known and custom types can be encoded by defining up a "serializer":
trait Serializer[T] {
def serialize(r: T): Array[Byte]
}
we can implement a macro for case class encoding by simply looping the fields and looking up type serializers implicitly. Here's a dummy implementation that looks up the serializer for R itself (in reality we look up serializers for case class field types):
object Encoder {
implicit def apply[R <: Product]: Encoder[R] = macro applyImpl[R]
def applyImpl[R: c.WeakTypeTag](c: blackbox.Context): c.Expr[Encoder[R]] = {
import c.universe._
c.Expr[Encoder[R]](q"""
new ${weakTypeOf[Encoder[R]]} {
override def encode(r: ${weakTypeOf[R]}): Array[Byte] =
implicitly[_root_.Serializer[${weakTypeOf[R]}]].serialize(r)
}
""")
}
}
Now define a base "processor":
abstract class BaseProcessor[R: Encoder] {
def process(r: R): Unit = {
println(implicitly[Encoder[R]].encode(r).length)
}
}
And try to use it:
case class Record(i: Int)
object Serializers {
implicit def recordSerializer: Serializer[Record] =
(r: Record) => Array.emptyByteArray
}
import Serializers._
class Processor extends BaseProcessor[Record]
This fails to compile with:
// [error] Loader.scala:10:22: could not find implicit value for parameter e: Serializer[Record]
// [error] class Processor extends BaseProcessor[Record]
// [error] ^
// [error] one error found
However the following do compile:
class Processor extends BaseProcessor[Record]()(Encoder[Record]) // Compiles!
object x { class Processor extends BaseProcessor[Record] } // Compiles!
I can't really understand why this happens, looks like it has something to do with the Processor definition being a top level definition, since as soon as I move it inside a class/object everything works as expected. Here's one more example that fails to compile:
object x {
import Serializers._ // moving the import here also makes it NOT compile
class Processor extends BaseProcessor[Record]
}
Why does this happen and is there any way to make it work?
To make it work you can try one of the following:
1) add context bound
implicit def apply[R <: Product : Serializer]: Encoder[R] = macro applyImpl[R]
def applyImpl[R: c.WeakTypeTag](c: blackbox.Context)(serializer: c.Expr[Serializer[R]]): c.Expr[Encoder[R]] = {
import c.universe._
c.Expr[Encoder[R]](q"""
new Encoder[${weakTypeOf[R]}] {
override def encode(r: ${weakTypeOf[R]}): Array[Byte] =
$serializer.serialize(r)
}
""")
}
2) make macro whitebox
3) put implicit instance recordSerializer: Serializer[Record] to the companion object of Record rather than some object Serializers to be imported
4) maybe your example is easier than actual use case but now it seems you don't need macros
implicit def apply[R <: Product : Serializer]: Encoder[R] = new Encoder[R] {
override def encode(r: R): Array[Byte] = implicitly[Serializer[R]].serialize(r)
}
Let's say we have the following traits:
trait MyValue
object MyValue {
case class MyBoolean(record: Boolean) extends MyValue
case class MyLong(record: Long) extends MyValue
}
trait MyValueExtractor[T] {
def apply(record: T): Option[MyValue]
}
trait MyThing[T] {
def name: String
def myValueExtractor: MyValueExtractor[T]
def myValue(record: T): Option[MyValue] = myValueExtractor(record)
}
What I want is something like this but without the second type parameter.
Note: I can't actually update the MyThing trait; I'm just using this as an illustration of the intended functionality.
trait MyThing[T, U] {
def name: String
def myValueExtractor: MyValueExtractor[T]
def myValue(record: T): Option[MyValue] = myValueExtractor(record)
def myRelatedValue(record: T): Option[U]
}
I'm wondering if I could use the type class pattern to help solve this (i.e., import some rich class that implicitly gives me a myRelatedValue method)?
Here's the rub. Every time T (above) is MyValue.MyBoolean, U must be a String. Every time T is MyValue.MyLong, U must be a Double. In other words, there's a sort of underlying mapping between T and U.
Is there a good way to do this using type class?
Sure. You just need to define some Mapping typeclass with implementations for your desired pairs of types. Then MyThing can have a method that takes an implicit typeclass instance and simply invokes its method.
Here's the code (I removed the unneeded details)
// types
case class MyBoolean(record: Boolean)
case class MyLong(record: Long)
// trait which uses the Mapping typeclass
trait MyThing[T] {
def myRelatedValue[U](record: T)(implicit ev: Mapping[T, U]): Option[U] = ev.relatedValue(record)
}
// typeclass itself
trait Mapping[T, U] {
def relatedValue(record: T): Option[U]
}
object Mapping {
implicit val boolStringMapping = new Mapping[MyBoolean, String] {
def relatedValue(record: MyBoolean) = Some(record.record.toString)
}
implicit val longDoubleMapping = new Mapping[MyLong, Double] {
def relatedValue(record: MyLong) = Some(record.record)
}
}
// usage
val myBoolThing = new MyThing[MyBoolean] {}
val myLongThing = new MyThing[MyLong] {}
val myStringThing = new MyThing[String] {}
myBoolThing.myRelatedValue(MyBoolean(true)) // Some(true)
myLongThing.myRelatedValue(MyLong(42L)) // Some(42.0)
myStringThing.myRelatedValue("someString") // error: could not find implicit value
Note that e.g. myBoolThing.myRelatedValue(MyBoolean(true)) will yield a type Option[U]. However, since myRelatedValue is parameterized, you can help the compiler and invoke it as myBoolThing.myRelatedValue[String](MyBoolean(true)), in which case you will obtain an Option[String]. If you try something other than String for MyBoolean, you will get an error.
I'm trying to access an implicit parameter for a generic type. Scala is able to find the implicit just fine in the straightforward case by calling a method with an explicit generic type, such as with printGenericType[Person] below.
However, if I create a TypeTag[Person] and pass it to printGenericTypeGivenTypeTag, Scala is unable to find the implicit parameter.
case class Person(name: String)
case class Animal(age: Int)
implicit val p = Person("cory")
implicit val a = Animal(2)
def main(args: Array[String]): Unit = {
// Can find the implicit Person, prints "Hello Person(cory)"
printGenericType[Person]
// Can find the implicit Animal, prints "Hello Animal(2)"
printGenericType[Animal]
// See comment below
printNamedType("Person")
printNamedType("Animal")
}
def printGenericType[T](implicit t: T) = {
println(s"Hello $t")
}
def printGenericTypeGivenTypeTag[T](typeTag: TypeTag[T])(implicit t: T) = {
println(s"Hello $t")
}
def printNamedType[T](name: String) = {
val typetag: TypeTag[T] = getTypeTag[T](name)
// Cannot find the implicit of type T, compiler error
printGenericTypeGivenTypeTag(typetag)
}
def getTypeTag[T](name:String): TypeTag[T] = ... //Implementation irrelevant
If I understand correctly, Scala locates implicit parameters at compile time, so it makes sense that it can't find an implicit parameter for the generic type T at compile time.
However, I know that an implicit instance of T does exist. Is it possible to rewrite printGenericTypeGivenTypeTag in such a way as to find the implicit value for T? At runtime, the method has access to the actual type of T, so it seems it should be able to locate an implicit parameter of the same type that is in scope.
For the curious, the reasoning behind this is to avoid this:
name match {
case "Person" => printGenericType[Person]
case "Animal" => printGenericType[Animal]
}
To answer the question
You're not really wanting to pass the T implicitly, but rather the TypeTag, and not the T. Here's what I mean, and you're probably better off with an implicit value class.
implicit class GenericPrinter[T](val obj: T) extends AnyVal {
def printGenericType()(implicit tag: TypeTag[T]) = {
// do stuff with the tag
Console.println("Hello $obj")
}
}
val x: Person = Person(...)
x.printGenericType
Now solving the real problem
If you are trying to print case classes, I'd probably go down the implicit macro route for added convenience. It's really trivial to write up a macro that does this for us, e.g output a debug string based on all the constructor params of an arbitrary case class.
trait DeepPrinter[T <: Product with Serializable] {
/**
* Prints a deeply nested debug string for a given case class.
* This uses implicit macros to materialise the printer type class.
* In English, when we request for an implicit printer: DeepPrinter[T],
* the pre-defined compile time macro will generate this method for us
* based on the fields of the given case class.
*
* #param sep A separator to use to delimit the rows in a case class.
* #return A fully traced debug string so we can see how a case class looks like.
*/
def debugString(sep: String = "\n"): String
}
object DeepPrinter {
implicit def deepPrinter[T <: Product with Serializable] = macro DeepPrinterImpl.deepPrinterImpl[T]
}
And the macro is pretty trivial, looks kind of like this.
import language.experimental.macros
import scala.reflect.macros.blackbox
#macrocompat.bundle
class DeepPrinterImpl(val c: blackbox.Context) {
import c.universe._
object CaseField {
def unapply(symbol: TermSymbol): Option[(Name, Type)] = {
if (symbol.isVal && symbol.isCaseAccessor) {
Some(symbol.name -> symbol.typeSignature)
} else {
None
}
}
}
def fields(tpe: Type): Iterable[(Name, Type)] = {
tpe.decls.collect { case CaseField(nm, tpeSn) => nm -> tpeSn }
}
def materialize[T : c.WeakTypeTag]: c.Expr[DeepPrinter[T]] = {
val tpe = weakTypeOf[T]
val (names, types) = fields(tpe).unzip
// change the package name to the correct one here!
val tree = q"""
new com.bla.bla.DeepPrinter[$tpe] {
def debugString(sep: String = "\n") = Seq(..$names).mkString(sep)
}
"""
c.Expr[DeepPrinter[T]](tree)
}
}
Using scala 2.10.3, my goal is to make the following work:
object A {
implicit class Imp(i: Int) {
def myPrint() {
println(i)
}
}
}
object B {
implicit class Imp(i: String) {
def myPrint() {
println(i)
}
}
}
import A._
import B._
object MyApp extends App {
3.myPrint()
}
This fails with
value myPrint is not a member of Int
If I give A.Imp and B.Imp different names (for example A.Imp1 and B.Imp2), it works.
Diving a bit deeper into it, there seems to be the same problem with implicit conversions.
This works:
object A {
implicit def Imp(i: Int) = new {
def myPrint() {
println(i)
}
}
implicit def Imp(i: String) = new {
def myPrint() {
println(i)
}
}
}
import A._
object MyApp extends App {
3.myPrint()
}
Whereas this doesn't:
object A {
implicit def Imp(i: Int) = new {
def myPrint() {
println(i)
}
}
}
object B {
implicit def Imp(i: String) = new {
def myPrint() {
println(i)
}
}
}
import A._
import B._
object MyApp extends App {
3.myPrint()
}
Why? Is this a bug in the scala compiler? I need this scenario, since my objects A and B derive from the same trait (with a type parameter) which then defines the implicit conversion for its type parameter. In this trait, I can only give one name for the implicit conversion. I want to be able to import more of these objects into my scope. Is there a way to do that?
edit: I can't give the implicit classes different names, since the examples above are only breaking down the problem. My actual code looks more like
trait P[T] {
implicit class Imp(i: T) {
def myPrint() {
...
}
}
}
object A extends P[Int]
object B extends P[String]
import A._
import B._
The implicits just have to be available as a simple name, so you can rename on import.
Just to verify:
scala> import A._ ; import B.{ Imp => BImp, _ }
import A._
import B.{Imp=>BImp, _}
scala> 3.myPrint
3
Actually, it works if you replace
import A._
import B._
with
import B._
import A._
What happens, I think, is that A.Imp is shadowed by B.Imp because it has the same name. Apparently shadowing applies on the function's name and do not take the signature into account.
So if you import A then B, then only B.Imp(i: String) will be available, and if you import B then A, then only A.Imp(i: Int) will be available.
If you need to use both A.Imp and B.Imp, you must rename one of them.
In case anyone else runs into this issue, there is a partial workaround, which I found here:
https://github.com/lihaoyi/scalatags/blob/3dea48c42c5581329e363d8c3f587c2c50d92f85/scalatags/shared/src/main/scala/scalatags/generic/Bundle.scala#L120
That code was written by Li Haoyi, so you can be pretty confident that no better solution exists...
Essentially, one can still use traits to define methods in terms of each other, but it will require boilerplate to copy those implicits into unique names. This example might be easier to follow:
trait Chainable
object Chainable {
implicit val _chainableFromInt = IntChainable.chainable _
implicit val _chainableFromIntTrav = IntChainable.traversable _
implicit val _chainableFromIntOpt = IntChainable.optional _
implicit val _chainableFromIntTry = IntChainable.tried _
implicit val _chainableFromDom = DomChainable.chainable _
implicit val _chainableFromDomTrav = DomChainable.traversable _
implicit val _chainableFromDomOpt = DomChainable.optional _
implicit val _chainableFromDomTry = DomChainable.tried _
private object IntChainable extends ImplChainables[Int] {
def chainable(n:Int) = Constant(n)
}
private object DomChainable extends ImplChainables[dom.Element]{
def chainable(e:Element) = Insertion(e)
}
private trait ImplChainables[T] {
def chainable(t:T):Chainable
def traversable(trav:TraversableOnce[T]):Chainable =
SeqChainable(trav.map(chainable).toList)
def optional(opt:Option[T]):Chainable =
opt match {
case Some(t) => chainable(t)
case None => NoneChainable
}
def tried(tried:Try[T]):Chainable =
optional(tried.toOption)
}
}
In other words, never write:
trait P[T] {
implicit def foo(i: T) = ...
}
object A extends P[X]
Because defining implicits in type parameterized traits will lead to these naming conflicts. Although, incidentally, the trait in mentioned in the link above is parameterized over many types, but idea there is that none of the implementations of that trait are ever needed in the same scope. (JSDom vs Text, and all._ vs short._ for those familiar with Scalatags)
I also recommend reading: http://www.lihaoyi.com/post/ImplicitDesignPatternsinScala.html
It does not address this issue specifically but is an excellent summary of how to use implicits.
However, putting all these pieces together, this still seems to be an issue:
trait ImplChainables[AnotherTypeClass]{
type F[A] = A=>AnotherTypeClass
implicit def transitiveImplicit[A:F](t: A):Chainable
implicit def traversable[A:F](trav: TraversableOnce[A]):Chainable = ...
}
What this trait would allow is:
object anotherImpl extends ImplChainables[AnotherTypeClass] {...}
import anotherImpl._
implicit val string2another: String=>AnotherTypeClass = ...
Seq("x"):Chainable
Because of the type parameter and the context binding (implicit parameter) those cannot be eta-expanded (i.e: Foo.bar _ ) into function values. The implicit parameter part has been fixed in Dotty: http://dotty.epfl.ch/blog/2016/12/05/implicit-function-types.html
I do not know if a complete solution would be possible, needless to say this is a complex problem. So it would be nice if same name implicits just worked and the whole issue could be avoided. And in any case, adding an unimport keyword would make so much more sense than turning off implicits by shadowing them.