Path-dependent typing over match/case - scala

Question

sealed trait Desc {
type T
}
trait Dataset[A] {
def toDS[A] = new Dataset[A] {}
}
trait DataFrame {}
sealed trait DFDesc extends Desc {
type T = Dummy
}
sealed trait DSDesc[A] extends Desc {
type T = A
}
trait JobConstruction {
def apply(desc: Desc): Job[desc.T]
}
sealed trait Job[DescType] {
def description: Desc { type T = DescType }
}
abstract class DSJob[V] extends Job[V] {
def result(con: JobConstruction): Dataset[V]
}
abstract class DFJob extends Job[Dummy] {
def result(con: JobConstruction): DataFrame
}
trait Dummy {}
case class SampleDFDesc() extends DFDesc
case class SampleDFJob(description: SampleDFDesc) extends DFJob {
override def result(con: JobConstruction) = new DataFrame {}
}
case class SampleDSDesc() extends DSDesc[Int]
case class SampleDSJob(description: SampleDSDesc) extends DSJob[Int] {
override def result(con: JobConstruction) = new Dataset[Int] {}
}
object Main {
val sampleConst = new JobConstruction {
override def apply(desc: Desc): Job[desc.T] = desc match {
case desc2: SampleDFDesc => SampleDFJob(desc2)
case desc2: SampleDSDesc => SampleDSJob(desc2)
}
}
}
Fails to compile with
/tmp/sample.scala:73: error: type mismatch;
found : this.SampleDFJob
required: this.Job[desc.T]
case desc2: SampleDFDesc => SampleDFJob(desc2)
^
/tmp/sample.scala:74: error: type mismatch;
found : this.SampleDSJob
required: this.Job[desc.T]
case desc2: SampleDSDesc => SampleDSJob(desc2)
EDIT:
I would like to get this to work in some kind:
case class SampleDepDesc(df: SampleDFDesc) extends DSDesc[Int]
case class SampleDepJob(description: SampleDepDesc) extends DSJob[Int] {
override def result(con: JobConstruction): Dataset[Int] = con(description.df).result(con).toDS[Int]
}

Answer 1

Analysis of the problem
The error gets formulated in a more interesting way if you write sampleConst like so:
object Main {
val sampleConst = new JobConstruction {
override def apply(desc: Desc): Job[desc.T] = {
val result = desc match {
case desc2: SampleDFDesc => SampleDFJob(desc2)
case desc2: SampleDSDesc => SampleDSJob(desc2)
}
result
}
}
The error message becomes:
type mismatch;
found : Product with Serializable with main.Job[_ >: main.Dummy with Int]{def description: Product with Serializable with main.Desc{type T >: main.Dummy with Int}}
required: main.Job[desc.T]
Note: Any >: desc.T (and Product with Serializable with main.Job[_ >: main.Dummy with Int]{def description: Product with Serializable with main.Desc{type T >: main.Dummy with Int}} <: main.Job[_ >: main.Dummy with Int]), but trait Job is invariant in type DescType. You may wish to define DescType as -DescType instead. (SLS 4.5)
This message is hard to read. The cause seems to be a problem with contravariance, as stated in the fourth line, but let's try to make this error message readable first.
The reason why this message is so long is that Scala is doing a lot of gymnastics in order to make sense of all those type castings and inheritances. We are going to (temporarily) flatten the type hierarchy a bit to see a bit clearer in all this.
Here, the intermediary classes SampleDFJob, SampleDSJob, SampleDFDesc and SampleDSDesc have been removed:
sealed trait Desc {
type T
}
sealed trait DFDesc extends Desc {
type T = Dummy
}
sealed trait DSDesc[A] extends Desc {
type T = A
}
trait JobConstruction {
def apply(desc: Desc): Job[desc.T]
}
sealed trait Job[DescType] {
def description: Desc { type T = DescType }
}
class DSJob[V] extends Job[V]
class DFJob extends Job[Dummy]
trait Dummy {}
object Main {
val sampleConst = new JobConstruction {
override def apply(desc: Desc): Job[desc.T] = {
val result = desc match {
case desc2: DFDesc => new DFJob
case desc2: DSDesc[Int] => new DSJob[Int]
}
result
}
}
}
And the error message is now :
type mismatch;
found : main.Job[_1] where type _1 >: main.Dummy with Int
required: main.Job[desc.T]
The problem seems to be that Scala cannot cast main.Job[_ >: main.Dummy with Int] into desc.T.
Note: why this weird type? Well, the generic type of result is different depending on the case of the pattern matching (in the first case, we have a Dummy, and in the second case, we have an Int). Since Scala is statically typed (at least during compilation), it will try to make up a return type which is a "common denominator" (or rather, a parent type) of all the possible types. The best thing it finds is _ >: main.Dummy with Int, which is "any type that is a parent of any of the types it found in the pattern matching" (main.Dummy and Int).
Why it does not work
I think the reason why this type cannot be cast into a desc.T is that Scala cannot confirm, at compilation time, that the returned type is always the same (since DescType is invariant) as Job[desc.T]. Indeed, desc.T comes from SampleDFDesc.T or SampleDSDesc.T, whereas the return type will be DescType, and nothing guarantees that those two types are the same (what if SampleDSJob extended DSJob[String] instead?)
Solution
I do not think that it is possible to code exactly in the way you're trying to do, but you can try to... dodge the issue:
If you are certain that the return type of each case will always be of the same type as desc.T, then you can specify an explicit cast with asInstanceOf, like so:
object Main {
val sampleConst = new JobConstruction {
override def apply(desc: Desc): Job[desc.T] = (desc match {
case desc2: SampleDFDesc => SampleDFJob(desc2)
case desc2: SampleDSDesc => SampleDSJob(desc2)
}).asInstanceOf[Job[desc.T]]
}
}
Of course, this is not type-safe.
Alternatively, if you can manage to write the Job class so that DescType can be contravariant (-DescType), you can write the Job.apply method to have the following signature instead:
def apply(desc: Desc): Job[_ <: desc.T]

Answer 2

It is not a real solution, but you could replace type inside trait, with type parameter, this code compiles:
sealed trait Desc[T]
trait Dataset[A]
trait DataFrame
sealed trait DFDesc extends Desc[Dummy]
sealed trait DSDesc[A] extends Desc[A]
trait JobConstruction {
def apply[A](desc: Desc[A]): Job[A]
}
sealed trait Job[A] {
def description: Desc[A]
}
abstract class DSJob[V] extends Job[V] {
def result: Dataset[V]
}
abstract class DFJob extends Job[Dummy] {
def result: DataFrame
}
trait Dummy
case class SampleDFDesc() extends DFDesc
case class SampleDFJob(description: SampleDFDesc) extends DFJob {
def result = new DataFrame {}
}
case class SampleDSDesc() extends DSDesc[Int]
case class SampleDSJob(description: SampleDSDesc) extends DSJob[Int] {
def result = new Dataset[Int] {}
}
val sampleConst = new JobConstruction {
override def apply[A](desc: Desc[A]): Job[A] = desc match {
case desc2: SampleDFDesc => SampleDFJob(desc2)
case desc2: SampleDSDesc => SampleDSJob(desc2)
}
}
As for how to make path dependent types work, I am curious myself.