I'd like to define a generic type. Something like:
from typing import TypeVar, Sequence, Union, Generic
T = TypeVar('T')
RecurSeqOf = Sequence[Union[Generic[T], Sequence[T]]]
# mypy error: Variable "typing.Generic" is not valid as a type
Is there a way to do it?
The whole background
Actually, I need a recursive generic type like
T = TypeVar('T')
RecurSeqOf = Sequence[Union[T, 'RecurSeqOf']]]
But the definition of recursive types is not yet supported by mypy.
That's why I work around this issue by making nested type definitions up to a limited depth (say, usually 5-6 levels, but in the example below two levels only for the sake of conciseness).
Hence, all the more the need to shorten the pattern because I need to use it for different parameter types:
from typing import Sequence, Union, TypeVar, Generic
class A:
pass
class B:
pass
# RecurSeqOfA = Sequence[Union[A, 'RecurSeqOfA']] # mypy error: Cannot resolve name "RecurSeqOfA" (possible cyclic definition)
RecurSeqOfA = Sequence[Union[A, Sequence[Union[A, Sequence[A]]]]]
# RecurSeqOfA = Sequence[Union[A, 'RecurSeqOfA']] # mypy error: Cannot resolve name "RecurSeqOfA" (possible cyclic definition)
RecurSeqOfB = Sequence[Union[B, Sequence[Union[B, Sequence[B]]]]]
T = TypeVar('T')
# RecurSeqOf = Sequence[Union[Generic[T], 'RecurSeqOf']] # error: Cannot resolve name "RecurSeqOf" (possible cyclic definition)
# additionally: error: Variable "typing.Generic" is not valid as a type
RecurSeqOf = Sequence[Union[Generic[T], Sequence[Generic[T]]]] # error: Variable "typing.Generic" is not valid as a type
As suggested by the comment of MisterMiyagi:
from typing import TypeVar, MutableSequence
T = TypeVar('T', bound='RecurSeqOf')
RecurSeqOf = MutableSequence[T]
a: RecurSeqOf[str] = []
a.append("abc")
a.append([]) # mypy error: error: Argument 1 to "append" of "MutableSequence" has incompatible type "List[<nothing>]"; expected "str"
b: RecurSeqOf[str] = []
a.append(b) # mypy error: Argument 1 to "append" of "MutableSequence" has incompatible type "MutableSequence[str]"; expected "str"
a.append(["cde"]) # mypy error: Argument 1 to "append" of "MutableSequence" has incompatible type "List[str]"; expected "str"
The definition itself is accepted by mypy. But it does not have the desired effect.
Since Sequence is already generic, one can directly use a type variable:
from typing import TypeVar, Sequence, Union
T = TypeVar('T')
# [T, ...] | [[T, ...], ...]
RecurSeqOf = Sequence[Union[T, Sequence[T]]]
# T | [T, ...] | [[T, ...], ...]
RecurSeqOfUnion = Union[RecurSeqOf[T], T]
This is what the documentation calls a "User defined generic type alias". RecurSeqOf = ... defines an alias, and Sequence[Union[T, Sequence[T]]] is generic.
This allows to define recursive types of fixed but arbitrary depth:
a0: RecurSeqOf[int]
a1: RecurSeqOf[RecurSeqOfUnion[int]]
a2: RecurSeqOf[RecurSeqOfUnion[RecurSeqOfUnion[int]]]
reveal_type(a0) # typing.Sequence[Union[builtins.int, typing.Sequence[builtins.int]]]
reveal_type(a1) # typing.Sequence[Union[typing.Sequence[Union[builtins.int, typing.Sequence[builtins.int]]], builtins.int, typing.Sequence[Union[typing.Sequence[Union[builtins.int, typing.Sequence[builtins.int]]], builtins.int]]]]
reveal_type(a2) # typing.Sequence[Union[typing.Sequence[Union[typing.Sequence[Union[builtins.int, typing.Sequence[builtins.int]]], builtins.int, typing.Sequence[Union[typing.Sequence[Union[builtins.int, typing.Sequence[builtins.int]]], builtins.int]]]], typing.Sequence[Union[builtins.int, typing.Sequence[builtins.int]]], builtins.int, typing.Sequence[Union[typing.Sequence[Union[typing.Sequence[Union[builtins.int, typing.Sequence[builtins.int]]], builtins.int, typing.Sequence[Union[typing.Sequence[Union[builtins.int, typing.Sequence[builtins.int]]], builtins.int]]]], typing.Sequence[Union[builtins.int, typing.Sequence[builtins.int]]], builtins.int]]]]
Related
I am trying to define a recursive Iterable type Iterable_of_TOIs:
# from __future__ import annotations
from typing import overload, Optional, Iterable, Union
class ToI:
"""
ToI = Token or Iterable of ToIs
"""
Iterable_of_ToIs = Iterable[Union[ToI, 'Iterable_of_ToIs']]
#overload
def __init__(self, *, token: str) -> None:
...
#overload
def __init__(self, *, iterable: Iterable_of_ToIs) -> None:
...
# actual implementation
def __init__(
self,
token: Optional[str] = None,
iterable: Optional[Iterable_of_ToIs] = None
) -> None:
self.token: Optional[str] = token
self.iterable: Optional[Iterable_of_ToIs] = iterable
But mypy complains
with error: Name 'Iterable_of_ToIs' is not defined or
with error: Cannot resolve name "Iterable_of_ToIs" (possible cyclic definition)
if I move the definition of Iterable_of_ToIs out of the class scope.
What am I doing wrong?
Apparently, I am doing wrong not that much.
Regarding error: Name 'Iterable_of_ToIs' is not defined:
in the section Motivation of PEP 613 (Status Accepted) I found:
Type aliases are declared as top level variable assignments.
Yikes! Couldn't find this rule for Python 3.9 or prior in any other doc, especially not in PEP 484.
But anyway... it explains the error. So, ok, let's move the line to the top level.
Regarding error: Cannot resolve name "Iterable_of_ToIs" (possible cyclic definition):
seems to be not yet supported: https://github.com/python/mypy/issues/731
For obscure reasons due to a specific library I'm using involving macros and serialization, I have chosen to try to use type members instead of type parameters for a class (background: Is there a way to use ClassTags with the Autowire library in Scala?)
sealed trait Command {
type M <: CommandMetaData
val meta: M
}
final case class SysCmdMetaData(
// ...
) extends CommandMetaData
// How to guarantee cmd.M <: SysCmdMetaData
def runInSystem(cmd: Command){
//...
}
Is there a way to do this statically? I think using type classes and context bounds is one option, but I feel it is a bit heavy handed since I'm already using subclassing here. Perhaps the better option is just to make metadata a trait, and embrace object oriented more fully and create all kinds of variations of Command based on which traits are mixed in.
You can define the function argument with a refined type:
def runInSystem(cmd: Command { type M <: SysCmdMetaData }) = { }
Note that unlike type parameters, type members are not inferred by default. The following code will fail:
scala> val cmd: Command = new Command {
type M = SysCmdMetaData
val meta = SysCmdMetaData()
}
cmd: Command = $anon$1#830b8a5
scala> runInSystem(cmd)
<console>:15: error: type mismatch;
found : cmd.type (with underlying type Command)
required: Command{type M <: SysCmdMetaData}
runInSystem(cmd)
^
You'd have to either:
Use some named subclass of Command, where M is statically fixed to an appropriate type.
Pass an anonymous subclass instance directly to runInSystem:
runInSystem(new Command {
type M = SysCmdMetaData
val meta = SysCmdMetaData()
})
Manually prescribe the type member of the variable you pass to runInSystem:
val cmd: Command { type M = SysCmdMetaData } = new Command {
type M = SysCmdMetaData
val meta = SysCmdMetaData()
}
runInSystem(cmd)
You can also define a type alias with a type parameter:
object Command {
type Aux[M0 <: CommandMetaData] = Command { type M = M0 }
}
val cmd: Command.Aux[SysCmdMetaData] = ???
runInSystem(cmd)
Scala 11.2 is giving me this error:
error: type mismatch;
found : Seq[Some[V]]
required: Seq[Option[?V8]] where type ?V8 <: V (this is a GADT skolem)
val output = f(ivs.map(iv => Some(iv.get._1)))
^
First off, this seems like a strange error message: Doesn't Seq[Some[V]] conform to Seq[Option[V]]?
Here are the parts of the surrounding code that seem relevant:
def evalDependencyTree[V]
(linkRelevance: LinkInfo => Option[LinkStrength])
(dtree: DependencyTree[V, LinkInfo], strengthSoFar: LinkStrength = 1.0)
: Option[(V, LinkStrength)] = dtree match {
. . .
case DFunction(f, inputs) => {
val ivs = inputs.map { input =>
evalDependencyTree(linkRelevance)(input, strengthSoFar) // <-- Recursive call
}
val output = f(ivs.map(iv => Some(iv.get._1))) // <-- The line with the error
. . .
}
}
trait DependencyTree[+V, +L]
case class DFunction[V, L](
f: Seq[Option[V]] => Option[V], inputs: Seq[DependencyTree[V, L]])
extends DependencyTree[V, L]
My (very limited) understanding of GADT skolems is that they're types defined by the compiler during type inference, which copy an existing type argument in order to prevent that type from "escaping" its scope, as in a recursive call—that is, to prevent its being referred to from a wider scope that has no access to the type.
I don't see how V could refer to different types in different scopes here. The recursive call to evalDependencyTree has the same type argument, V, as the current call to evalDependencyTree. I tried explicitly writing evalDependencyTree[V] for the recursive call, but the compiler returned the same error message. This code did work when evalDependencyTree did not have a type argument; in that version, dtree was hard-coded to DependencyTree[Int, LinkInfo].
What type is trying to escape? Or rather, what am I doing wrong?
I found a workaround myself: explicitly spell out the full type of f in the pattern-match, like this:
case DFunction(f: Seq[Option[V]] => Option[V], inputs) => . . .
This works, but I'm not accepting it as an answer because I don't have an explanation of why it's necessary. I still don't know when to expect this kind of error or what causes it. If you know, please post an answer!
Also, I would have thought that most of the type explicitly provided for f would have been lost by type erasure. So, there are two important things I can't explain about this workaround.
Could someone explain how the type keyword and # operator works in scala and how to use it?
Please look at examples.
//Example1
scala> type t1 = Option.type
defined type alias t1
//Shouldn't this work since previous example simply works?
scala> type t2 = String.type
<console>:7: error: type mismatch;
found : String.type
required: AnyRef
type t2 = String.type
^
//lets define custom trait T
scala> trait T
defined trait T
//... and obtain it's type like in Example1.
//Shouldn't this work since previous Example1 simply works?
scala> type t3 = T.type
<console>:7: error: not found: value T
type t3 = T.type
^
//Lets define some value of type T
scala> val v4 = new T{}
v4: T = $anon$1#5c3e8c76
//and obtain it's type (works)
scala> type t4 = v4.type
defined type alias t4
//this doesn't work
scala> type t4_1 = (new T{}).type
<console>:1: error: identifier expected but 'new' found.
type t4_1 = (new T{}).type
//as well as this (doesn't work)
scala> type t5 = "abc".type
<console>:1: error: identifier expected but string literal found.
type t5 = "abc".type
^
//but this compiles well
scala> val v6 = "abc"
v6: String = abc
scala> type t6 = v6.type
defined type alias t6
//lets create some values of created types:
scala> type t1 = Option.type
defined type alias t1
scala> val v1_1 = Some(10)
v1_1: Some[Int] = Some(10)
scala> type t7 = v1_1.type
defined type alias t7
scala> val v7:t7 = null
v7: t7 = null
scala> val v7_1:t7 = v1_1
v7_1: t7 = Some(10)
scala> val v7_2:t7 = Some(10)
<console>:9: error: type mismatch;
found : Some[Int]
required: t7
(which expands to) v1_1.type
val v7_2:t7 = Some(10)
^
//next let's try # operator
scala> class X[A,B](a:A,b:B)
defined class X
//doesn't work
scala> type xa = X[A,B]#A
<console>:8: error: not found: type A
type xa = X[A,B]#A
^
<console>:8: error: not found: type B
type xa = X[A,B]#A
^
//but such approach works:
scala> trait X2[C]{
type A
type B
val c:C
}
defined trait X2
scala> type xa2_1 = X2[String]#A
defined type alias xa2_1
scala> type xa2_2[M] = X2[M]#A
defined type alias xa2_2
First, your questions about type:
The right hand side of a type declaration has to be the name of a type with a stable path. So taking your examples one by one:
type t1 = Option.type
t1 is an alias for the type of the Option object, not the Option class.
type t2 = String.type
This is an error because there is no String object. The error's a little weird because String's a Java class and so operates under different rules (since Java classes never have companions).
type t3 = T.type
ditto. This time the error's clearer, because T is a Scala class and so the compiler can unambiguously say "T does not name an object with a type"
type t4 = v4.type
This is the singleton type of the object named by the val v4. It doesn't refer to any instance of type T, or even any instance of the anonymous class created by your new T{} expression. It refers to a type that is only represented by v4 and null, i.e. they are the only allowed values of that type.
type t4_1 = (new T{}).type
This is illegal because the thing you're taking the type of has to be a stable identifier (roughly, an identifier whose referant can never change -- if the full path to the idenfier consists of only the names of packages, objects, and vals, it's stable).
type t5 = "abc".type
Ditto.
type t6 = v6.type
v6 is a stable identifier. t6 is the type inhabited solely by that particular instance of String which is referred to by the name v6 (and null).
type v6 = v1_1.type
Again, a singleton type.
val v7: t7 = null
null is a valid value of type t7
val v7_1:t7 = v1_1
So is this particular object.
val v7_2:t7 = Some(10)
But this is a different object (even though it's == to v7, it's not eq to it) and therefore is not a member of this type.
Now about #:
class X[A,B](a:A,b:B)
A and B are type parameters. They can't be referred to outside the class. You can think of them like abstract type aliases with private[this] visibility, though that's not quite accurate.
type xa = X[A,B]#A
So yeah, not visible.
type xa2_1 = X2[String]#A
Since this A refers to a public type alias, it can be referred to by name outside the class. Note that this particular case is pretty useless, because you know absolutely nothing about this type. if your trait X2 had a method that returned values of type A, you could do something like
val aFromX2: xa2_1 = x2instance.methodThatReturnsAnA
..but then you couldn't do anything else with it, even pass it back to an instance of X2[String] because there's no guarantee that that the two As would refer to the same type! On the other hand, if you have a concrete instance, you could do this:
def passAroundA(x2instance: X2[String]) {
type x2a = x2instance.A // note dot, not #
val a: x2a = x2instance.methodThatReturnsAnA
x2instance.methodThatTakesAnA(a)
}
In this case it works because even though we have no idea what A actually is, we know that the two methods use the same type -- whatever was fixed at x2instance's construction.
Practically everything regarding your question is explained in this imho absolutely essential talk on use cases like pattern matching etc. I suggest you to take the time to watch it and the effort to thing about it. A lot of scala Type system related stuff was actually a magic to me until I watched this video. I might spare a lot of my man days trying to resolve weird behavior of the type system and type inference if I was aware of these things.
Recently I stumbled across a strange (to me) compiler error message. Consider the following code:
trait Foo {
type Res <: Foo
type Bar[X <: Res]
}
class MyFoo extends Foo {
override type Res = MyFoo
override type Bar[X <: Res] = List[X]
}
type FOO[F <: Foo, R <: Foo, B[_ <: R]] = F { type Res = R;
type Bar[X <: R] = B[X] }
def process[F <: Foo, R <: Foo, B[_ <: R]](f: FOO[F, R, B]) {}
Now, if I want to call the process method I have to explicitly write the type parameters:
process[MyFoo, MyFoo, List](new MyFoo) // fine
If I write:
process(new MyFoo)
or
process((new MyFoo): FOO[MyFoo, MyFoo, List])
I get the following error message:
inferred kinds of the type arguments (MyFoo,MyFoo,List[X]) do not conform to the expected kinds of the type parameters (type F,type R,type B). List[X]'s type parameters do not match type B's expected parameters: class List has one type parameter, but type B has one
Why isn´t the compiler able to infer the types (although I explicitly stated them at call parameter)? And what does that class List has one type parameter, but type B has one mean? Something has one, but the other has also one, and that´s why they don´t fit together???
If we look to the Scala compiler, the sources could help us understanding what the problem is. I have never contributed to the Scala compiler, but I found the sources very readable and I have already investigated on that.
The class responsible for type inference is scala.tools.nsctypechecker.Infer which you can find simply by looking in the Scala compiler sources for a part of your error. You'll find out the following fragment:
/** error if arguments not within bounds. */
def checkBounds(pos: Position, pre: Type, owner: Symbol,
tparams: List[Symbol], targs: List[Type], prefix: String) = {
//#M validate variances & bounds of targs wrt variances & bounds of tparams
//#M TODO: better place to check this?
//#M TODO: errors for getters & setters are reported separately
val kindErrors = checkKindBounds(tparams, targs, pre, owner)
if(!kindErrors.isEmpty) {
error(pos,
prefix + "kinds of the type arguments " + targs.mkString("(", ",", ")") +
" do not conform to the expected kinds of the type parameters "+ tparams.mkString("(", ",", ")") + tparams.head.locationString+ "." +
kindErrors.toList.mkString("\n", ", ", ""))
}
So now the point is understanding why checkKindBounds(tparams, targs, pre, owner) returns those errors. If you go down the method call chain, you will see that the checkKindBounds call another method
val errors = checkKindBounds0(tparams, targs, pre, owner, true)
You'll see the problem is connected to checking bounds of higher-kinded type, at line 5784, inside checkKindBoundsHK :
if (!sameLength(hkargs, hkparams)) {
if (arg == AnyClass || arg == NothingClass) (Nil, Nil, Nil) // Any and Nothing are kind-overloaded
else {error = true; (List((arg, param)), Nil, Nil) } // shortcut: always set error, whether explainTypesOrNot
}
The test is not passed, it appears that in my debugger:
hkargs$1 = {scala.collection.immutable.Nil$#2541}"List()"
arg$1 = {scala.tools.nsc.symtab.Symbols$ClassSymbol#2689}"class List"
param$1 = {scala.tools.nsc.symtab.Symbols$TypeSymbol#2557}"type B"
paramowner$1 = {scala.tools.nsc.symtab.Symbols$MethodSymbol#2692}"method process"
underHKParams$1 = {scala.collection.immutable.$colon$colon#2688}"List(type R)"
withHKArgs$1 = {scala.collection.immutable.Nil$#2541}"List()"
exceptionResult12 = null
hkparams$1 = {scala.collection.immutable.$colon$colon#2688}"List(type R)"
So it appears like there is one higher kinded param, type R, but there is no provided value for that.
If you actually go back to the to checkKindBounds, you see that after the snippet:
val (arityMismatches, varianceMismatches, stricterBounds) = (
// NOTE: *not* targ.typeSymbol, which normalizes
checkKindBoundsHK(tparamsHO, targ.typeSymbolDirect, tparam, tparam.owner, tparam.typeParams, tparamsHO)
)
the arityMismatches contains a tuple List, B. And now you can also see that the error message is wrong:
inferred kinds of the type arguments (MyFoo,MyFoo,List[X]) do not
conform to the expected kinds of the type parameters (type F,type
R,type B). List[X]'s type parameters do not match type B's expected
parameters: class List has one type parameter, but type B has ZERO
In fact if you put a breakpoint at line 5859 on the following call
checkKindBoundsHK(tparamsHO, targ.typeSymbolDirect, tparam, tparam.owner, tparam.typeParams, tparamsHO)
you can see that
tparam = {scala.tools.nsc.symtab.Symbols$TypeSymbol#2472}"type B"
targ = {scala.tools.nsc.symtab.Types$UniqueTypeRef#2473}"List[X]"
Conclusion:
For some reason, when dealing with complex higher-kinded types such as yours, Scala compiler inference is limited. I don't know where it does come from, maybe you want to send a bug to the compiler team
I only have a vague understanding of the exact workings of the type inferrer in Scala so consider this ideas not definitive answers.
Type inferring has problems with inferring more then one type at once.
You use an existential type in the definition of FOO, which translates to: there exists a type such, not sure if this is compatible with the specific type given in MyFoo