Trouble with type variance - scala

Question

So, let's say I have a class with a contravariant type parameter:
trait Storage[-T] {
def list(path: String): Seq[String]
def getObject[R <: T](path: String): Future[R]
}
The idea of the type parameter is to constrain the implementation to the upper boundary of types that it can return. So, Storage[Any] can read anything, while Storage[avro.SpecificRecord] can read avro records, but not other classes:
def storage: Storage[avro.SpecificRecord]
storage.getObject[MyAvroDoc]("foo") // works
storage.getObject[String]("bar") // fails
Now, I have a utility class that can be used to iterate through objects in a given location:
class StorageIterator[+T](
val storage: Storage[_ >: T],
location: String
)(filter: String => Boolean) extends AbstractIterator[Future[T]] {
val it = storage.list(location).filter(filter)
def hasNext = it.hasNext
def next = storage.getObject[T](it.next)
}
This works, but sometimes I need to access the underlying storage from the iterator downstream, to read another type of object from an aux location:
def storage: Storage[avro.SpecificRecord]
val iter = new StorageIterator[MyAvroDoc]("foo")
iter.storage.getObject[AuxAvroDoc](aux)
This does not work, of course, because storage type parameter is a wildcard, and there is no proof that it can be used to read AuxAvroDoc
I try to fix it like this:
class StorageIterator2[P, +T <: P](storage: Storage[P])
extends StorageIterator[T](storage)
This works, but now I have to specify two type params when creating it, and that sucks :(
I tried to work around it by adding a method to the Storage itself:
trait Storage[-T] {
...
def iterate[R <: T](path: String) =
new StorageIterator2[T, R](this, path)
}
But this doesn't compile because it puts T into an invariant position :(
And if I make P contravariant, then StorageIterator2[-P, +T <: P] fails, because it thinks that P occurs in covariant position in type P of value T.
This last error I don't understand. Why exactly cannot P be contravariant here? If this position is really covariant (why is it?) then why does it allow me to specify an invariant parameter there?
Finally, does anyone have an idea how I can work around this?
Basically, the idea is to be able to
Do storage.iterate[MyAvroDoc] without having to give it the upper boundary again, and
Do iterator.storage.getObject[AnotherAvroDoc] without having to cast the storage to prove that it can read this type of object.
Any ideas are appreciated.

Answer 1

StorageIterator2[-P, +T <: P] fails because it is nonsensical. If you have a StorageIterator2[Foo, Bar], and Bar <: Foo, then because it is contravariant in the first parameter, it is also a StorageIterator[Nothing, Bar], but Nothing has no subtypes, so it is logically impossible that Bar <: Nothing, yet this is what must be true to have a StorageIterator2. Therefore, StorageIterator2 cannot exist.
The root problem is that Storage should not be contravariant. Think of what a Storage[T] is, in terms of the contract it gives to its users. A Storage[T] is an object that you give paths to, and will output Ts. It makes perfect sense for Storage to be covariant: something that knows how to output Strings, for example, is also outputting Anys, so it makes logical sense that Storage[String] <: Storage[Any]. You say that it should be the other way around, that a Storage[T] should know how to output any subtype of T, but how would that work? What if someone adds a subtype to T after the fact? T can even be final and still have this problem, because of singleton types. That is unnecessarily complicated, and is reflected in your problem. That is, Storage should be
trait Storage[+T] {
def list(path: String]: Seq[String]
def get(path: String): T
}
This does not open you up to the example mistake you gave in your question:
val x: Storage[avro.SpecificRecord] = ???
x.get(???): avro.SpecificRecord // ok
x.get(???): String // nope
(x: Storage[String]).get(???) // nope
Now, your issue is that you can't do something like storage.getObject[T] and have the cast be implicit. You can instead a match:
storage.getObject(path) match {
case value: CorrectType => ...
case _ => // Oops, something else. Error?
}
a plain asInstanceOf (undesirable), or you can add a helper method to Storage, like the one you had before:
def getCast[U <: T](path: String)(implicit tag: ClassTag[U]): Option[U] = tag.unapply(get(path))