... or mishaps of a Haskell programmer that has to code Scala, part 5.
I have the following structure in Scala:
case class ResourceTree(
resources: Map[String, ResourceTree]
)
And, using Cats, I would like to define a Semigroup instance of it.
object ResourceTreeInstances {
implicit val semigroupInstance = new Semigroup[ResourceTree] {
override def combine(x: ResourceTree, y: ResourceTree): ResourceTree = {
ResourceTree(
x.resources |+| y.resources
)
}
}
This will result in the following error:
value |+| is not a member of Map[String, ResourceTree]
[error] Note: implicit value semigroupInstance is not applicable here because it comes after the application point and it lacks an explicit result type
[error] x.resources |+| y.resource
So, my guess was that since I'm defining the instance for Semigroup the Scala compiler cannot derive an instance for Semigroup of Map[String, ResourceTree]. This seems to be confirmed, since the following instance is compiles:
implicit val semigroupInstance = new Semigroup[ResourceTree] {
override def combine(x: ResourceTree, y: ResourceTree): ResourceTree = {
dummyCombine(x, y)
}
}
// FIXME: see if there's a better way to avoid the "no instance of Semigroup" problem
def dummyCombine(x: ResourceTree, y: ResourceTree): ResourceTree = {
ResourceTree(
x.resources |+| y.resources
)
}
I'm really hoping I'm wrong because if this is the right way of defining an instance for a Semigroup in Scala I'll start considering the idea of giving up doing FP in this language.
Is there a better way?
The following should work just fine:
import cats.Semigroup
import cats.instances.map._
import cats.syntax.semigroup._
case class ResourceTree(resources: Map[String, ResourceTree])
implicit val resourceTreeSemigroup: Semigroup[ResourceTree] =
new Semigroup[ResourceTree] {
def combine(x: ResourceTree, y: ResourceTree): ResourceTree =
ResourceTree(
x.resources |+| y.resources
)
}
The key is this part of the error message: "and it lacks an explicit result type". Recursive methods in Scala must have explicit return types, and similarly type class instances that depend on themselves (either directly or indirectly through something like the Map instance and |+| syntax in this case) also need them.
In general it's a good idea to put explicit return types on all implicit definitions—not doing so can lead to unexpected behavior, some of which makes sense if you think about it and read the spec (as in this case), and some of which just seems to be bugginess in the compiler.
Related
I have an implicit class that decodes server's response into a JSON and latter in the right case class to avoid repeating calls to .as and .getOrElse all around the tests:
implicit class RouteTestResultBody(testResult: RouteTestResult) {
def body: String = bodyOf(testResult)
def decodedBody[T](implicit d: Decoder[T]): T =
decode[Json](body)
.fold(err => throw new Exception(s"Body is not a valid JSON: $body"), identity)
.as[T]
.getOrElse(throw new Exception(s"JSON doesn't have the right shape: $body"))
}
Of course, it relies on us passing a decoder:
import io.circe.generic.semiauto.deriveDecoder
val result: RouteTestResult = ...
result.decodedBody(deriveDecoder[SomeType[AnotherType])
It works most of the time, but fails when the response is a list:
result.dedoceBody(deriveDecoder[List[SomeType]])
// throws "JSON doesn't have the right shape"
How can I semiautomatically derive a decoder for a list with specific types inside?
The terminology here is unfortunately overloaded, in that we use "deriving" in two senses:
Providing an instance for e.g. List[A] given an instance for A.
Providing an instance for a case class or sealed trait hierarchy given instances for all member types.
This problem isn't specific to Circe, or even Scala. In writing about Circe I generally try to avoid referring to the first kind of instance generation as "derivation" at all, and to refer to the second kind as "generic derivation" to emphasize that we're generating instances via a generic representation of the algebraic data type.
The fact that we sometimes use the same word to refer to both kinds of type class instance generation is a problem because they're typically very distinct mechanisms in Scala. In Circe the thing that provides an encoder or decoder instance for List[A] given one for A is a method in the type class companion object. For example, in the object Decoder in circe-core we have a method like this:
implicit def decodeList[A](implicit decodeA: Decoder[A]): Decoder[List[A]] = ...
Because this method definition is in the Decoder companion object, if you ask for an implicit Decoder[List[A]] in a context where you have an implicit Decoder[A], the compiler will find and use decodeList. You don't need any imports or extra definitions. For example:
scala> case class Foo(i: Int)
class Foo
scala> import io.circe.Decoder, io.circe.parser
import io.circe.Decoder
import io.circe.parser
scala> implicit val decodeFoo: Decoder[Foo] = Decoder[Int].map(Foo(_))
val decodeFoo: io.circe.Decoder[Foo] = io.circe.Decoder$$anon$1#6e992c05
scala> parser.decode[List[Foo]]("[1, 2, 3]")
val res0: Either[io.circe.Error,List[Foo]] = Right(List(Foo(1), Foo(2), Foo(3)))
If we desugared the implicit machinery here, it would look like this:
scala> parser.decode[List[Foo]]("[1, 2, 3]")(Decoder.decodeList(decodeFoo))
val res1: Either[io.circe.Error,List[Foo]] = Right(List(Foo(1), Foo(2), Foo(3)))
Note that we could replace first kind of derivation with the second, and it would still compile:
scala> import io.circe.generic.semiauto.deriveDecoder
import io.circe.generic.semiauto.deriveDecoder
scala> parser.decode[List[Foo]]("[1, 2, 3]")(deriveDecoder[List[Foo]])
val res2: Either[io.circe.Error,List[Foo]] = Left(DecodingFailure(CNil, List()))
This compiles because Scala's List is an algebraic data type that has a generic representation that circe-generic can create an instance for. The decoding fails for this input, though, since this representation doesn't result in the encoding we expect. We can derive the corresponding encoder to see what this encoding looks like:
scala> import io.circe.Encoder, io.circe.generic.semiauto.deriveEncoder
import io.circe.Encoder
import io.circe.generic.semiauto.deriveEncoder
scala> implicit val encodeFoo: Encoder[Foo] = Encoder[Int].contramap(_.i)
val encodeFoo: io.circe.Encoder[Foo] = io.circe.Encoder$$anon$1#2717857a
scala> deriveEncoder[List[Foo]].apply(List(Foo(1), Foo(2)))
val res3: io.circe.Json =
{
"::" : [
1,
2
]
}
So we're actually seeing the :: case class for List, which is basically never what we want.
If you need to provide a Decoder[List[Foo]] explicitly, the solution is to use either the Decoder.apply "summoner" method, or to call Decoder.decodeList explicitly:
scala> Decoder[List[Foo]]
val res4: io.circe.Decoder[List[Foo]] = io.circe.Decoder$$anon$44#5d40f590
scala> Decoder.decodeList[Foo]
val res5: io.circe.Decoder[List[Foo]] = io.circe.Decoder$$anon$44#2f936a01
scala> Decoder.decodeList(decodeFoo)
val res6: io.circe.Decoder[List[Foo]] = io.circe.Decoder$$anon$44#7f525e05
These all provide exactly the same instance, and which you should choose is a matter of taste.
As a footnote, I've thought about special-casing List in circe-generic so that deriveDecoder[List[X]] doesn't compile, since it's approximately never what you want (but seems like it might be, especially because of the confusing way we talk about instance derivation). I typically don't like the idea of having special cases like that, but I think in this case it might be the right thing to do, since this question comes up a lot.
I'd like to enrich a 'graph for scala' graph. For this purpose i've created an implicit value class:
import scalax.collection.mutable
import scalax.collection.edge.DiEdge
...
type Graph = mutable.Graph[Int, DiEdge]
implicit class EnrichGraph(val G: Graph) extends AnyVal {
def roots = G.nodes.filter(!_.hasPredecessors)
...
}
...
The problem lies with the return type of its methods, e.g.:
import ....EnrichGraph
val H: Graph = mutable.Graph[Int,DiEdge]()
val roots1 = H.nodes.filter(!_.hasPredecessors) // type Iterable[H.NodeT]
val roots2 = H.roots // type Iterable[RichGraph#G.NodeT] !!
val subgraph1 = H.filter(H.having(roots1)) // works!
val subgraph2 = H.filter(H.having(roots2)) // type mismatch!
Does the cause lie with fact that 'Graph' has dependent subtypes, e.g. NodeT? Is there a way to make this enrichment work?
What usually works is propagating the singleton type as a type parameter to EnrichGraph. That means a little bit of extra boilerplate since you have to split the implicit class into a class and an implicit def.
class EnrichGraph[G <: Graph](val G: G) extends AnyVal {
def roots: Iterable[G#NodeT] = G.nodes.filter(!_.hasPredecessors)
//...
}
implicit def EnrichGraph(g: Graph): EnrichGraph[g.type] = new EnrichGraph[g.type](g)
The gist here being that G#NodeT =:= H.NodeT if G =:= H.type, or in other words (H.type)#NodeT =:= H.NodeT. (=:= is the type equality operator)
The reason you got that weird type, is that roots has a path type dependent type. And that path contains the value G. So then the type of val roots2 in your program would need to contain a path to G. But since G is bound to an instance of EnrichGraph which is not referenced by any variable, the compiler cannot construct such a path. The "best" thing the compiler can do is construct a type with that part of the path left out: Set[_1.G.NodeT] forSome { val _1: EnrichGraph }. This is the type I actually got with your code; I assume you're using Intellij which is printing this type differently.
As pointed out by #DmytroMitin a version which might work better for you is:
import scala.collection.mutable.Set
class EnrichGraph[G <: Graph](val G: G) extends AnyVal {
def roots: Set[G.NodeT] = G.nodes.filter(!_.hasPredecessors)
//...
}
implicit def EnrichGraph(g: Graph): EnrichGraph[g.type] = new EnrichGraph[g.type](g)
Since the rest of your code actually requires a Set instead of an Iterable.
The reason why this still works despite reintroducing the path dependent type is quite tricky. Actually now roots2 will receive the type Set[_1.G.NodeT] forSome { val _1: EnrichGraph[H.type] } which looks pretty complex. But the important part is that this type still contains the knowledge that the G in _1.G.NodeT has type H.type because that information is stored in val _1: EnrichGraph[H.type].
With Set you can't use G#NodeT to give you the simpler type signatures, because G.NodeT is a subtype of G#NodeT and Set is unfortunately invariant. In our usage those type will actually always be equivalent (as I explained above), but the compiler cannot know that.
I want to reify a ValDef into runtime, but i does not work directly. If i encapsulate the ValDef into a Block, everything works perfectly, like in the following example:
case class Container(expr: Expr[Any])
def lift(expr: Any): Container = macro reifyValDef
def reifyValDef(c: Context)(expr: c.Expr[Any]): c.Expr[Container] = {
import c.universe._
expr.tree match {
case Block(List(v: ValDef), _) =>
val asBlock = q"{$v}"
val toRuntime = q"scala.reflect.runtime.universe.reify($asBlock)"
c.Expr[Container](q"Container($toRuntime)")
}
}
lift {
val x: Int = 10
}
If i would use v directly, instead of wrapping it into a block, I get the error:
Error:(10, 11) type mismatch;
found :
required: Any
Note that extends Any, not AnyRef.
Such types can participate in value classes, but instances
cannot appear in singleton types or in reference comparisons.
val x: Int = 10
^
Is it just not working directly with ValDefs or is something wrong with my code?
That's one of the known issues in the reflection API. Definitions are technically not expressions, so you can't e.g. pass them directly as arguments to functions. Wrapping the definition in a block is a correct way of addressing the block.
The error message is of course confusing, but it does make some twisted sense. To signify the fact that a definition by itself doesn't have a type, the tpe field of the corresponding Tree is set to NoType. Then the type of the argument of a macro is checked against Any and the check fails (because NoType is a special type, which isn't compatible with anything), so a standard error message is printed. The awkward printout is an artifact of how the prettyprinter behaves in this weird situation.
I am trying to test whether two "containers" use the same higher-kinded type. Look at the following code:
import scala.reflect.runtime.universe._
class Funct[A[_],B]
class Foo[A : TypeTag](x: A) {
def test[B[_]](implicit wt: WeakTypeTag[B[_]]) =
println(typeOf[A] <:< weakTypeOf[Funct[B,_]])
def print[B[_]](implicit wt: WeakTypeTag[B[_]]) = {
println(typeOf[A])
println(weakTypeOf[B[_]])
}
}
val x = new Foo(new Funct[Option,Int])
x.test[Option]
x.print[Option]
The output is:
false
Test.Funct[Option,Int]
scala.Option[_]
However, I expect the conformance test to succeed. What am I doing wrong? How can I test for higher-kinded types?
Clarification
In my case, the values I am testing (the x: A in the example) come in a List[c.Expr[Any]] in a Macro. So any solution relying on static resolution (as the one I have given), will not solve my problem.
It's the mixup between underscores used in type parameter definitions and elsewhere. The underscore in TypeTag[B[_]] means an existential type, hence you get a tag not for B, but for an existential wrapper over it, which is pretty much useless without manual postprocessing.
Consequently typeOf[Funct[B, _]] that needs a tag for raw B can't make use of the tag for the wrapper and gets upset. By getting upset I mean it refuses to splice the tag in scope and fails with a compilation error. If you use weakTypeOf instead, then that one will succeed, but it will generate stubs for everything it couldn't splice, making the result useless for subtyping checks.
Looks like in this case we really hit the limits of Scala in the sense that there's no way for us to refer to raw B in WeakTypeTag[B], because we don't have kind polymorphism in Scala. Hopefully something like DOT will save us from this inconvenience, but in the meanwhile you can use this workaround (it's not pretty, but I haven't been able to come up with a simpler approach).
import scala.reflect.runtime.universe._
object Test extends App {
class Foo[B[_], T]
// NOTE: ideally we'd be able to write this, but since it's not valid Scala
// we have to work around by using an existential type
// def test[B[_]](implicit tt: WeakTypeTag[B]) = weakTypeOf[Foo[B, _]]
def test[B[_]](implicit tt: WeakTypeTag[B[_]]) = {
val ExistentialType(_, TypeRef(pre, sym, _)) = tt.tpe
// attempt #1: just compose the type manually
// but what do we put there instead of question marks?!
// appliedType(typeOf[Foo], List(TypeRef(pre, sym, Nil), ???))
// attempt #2: reify a template and then manually replace the stubs
val template = typeOf[Foo[Hack, _]]
val result = template.substituteSymbols(List(typeOf[Hack[_]].typeSymbol), List(sym))
println(result)
}
test[Option]
}
// has to be top-level, otherwise the substituion magic won't work
class Hack[T]
An astute reader will notice that I used WeakTypeTag in the signature of foo, even though I should be able to use TypeTag. After all, we call foo on an Option which is a well-behaved type, in the sense that it doesn't involve unresolved type parameters or local classes that pose problems for TypeTags. Unfortunately, it's not that simple because of https://issues.scala-lang.org/browse/SI-7686, so we're forced to use a weak tag even though we shouldn't need to.
The following is an answer that works for the example I have given (and might help others), but does not apply to my (non-simplified) case.
Stealing from #pedrofurla's hint, and using type-classes:
trait ConfTest[A,B] {
def conform: Boolean
}
trait LowPrioConfTest {
implicit def ctF[A,B] = new ConfTest[A,B] { val conform = false }
}
object ConfTest extends LowPrioConfTest {
implicit def ctT[A,B](implicit ev: A <:< B) =
new ConfTest[A,B] { val conform = true }
}
And add this to Foo:
def imp[B[_]](implicit ct: ConfTest[A,Funct[B,_]]) =
println(ct.conform)
Now:
x.imp[Option] // --> true
x.imp[List] // --> false
tl;dr: How do I do something like the made up code below:
def notFunctor[M[_] : Not[Functor]](m: M[_]) = s"$m is not a functor"
The 'Not[Functor]', being the made up part here.
I want it to succeed when the 'm' provided is not a Functor, and fail the compiler otherwise.
Solved: skip the rest of the question and go right ahead to the answer below.
What I'm trying to accomplish is, roughly speaking, "negative evidence".
Pseudo code would look something like so:
// type class for obtaining serialization size in bytes.
trait SizeOf[A] { def sizeOf(a: A): Long }
// type class specialized for types whose size may vary between instances
trait VarSizeOf[A] extends SizeOf[A]
// type class specialized for types whose elements share the same size (e.g. Int)
trait FixedSizeOf[A] extends SizeOf[A] {
def fixedSize: Long
def sizeOf(a: A) = fixedSize
}
// SizeOf for container with fixed-sized elements and Length (using scalaz.Length)
implicit def fixedSizeOf[T[_] : Length, A : FixedSizeOf] = new VarSizeOf[T[A]] {
def sizeOf(as: T[A]) = ... // length(as) * sizeOf[A]
}
// SizeOf for container with scalaz.Foldable, and elements with VarSizeOf
implicit def foldSizeOf[T[_] : Foldable, A : SizeOf] = new VarSizeOf[T[A]] {
def sizeOf(as: T[A]) = ... // foldMap(a => sizeOf(a))
}
Keep in mind that fixedSizeOf() is preferable where relevant, since it saves us the traversal over the collection.
This way, for container types where only Length is defined (but not Foldable), and for elements where a FixedSizeOf is defined, we get improved performance.
For the rest of the cases, we go over the collection and sum individual sizes.
My problem is in the cases where both Length and Foldable are defined for the container, and FixedSizeOf is defined for the elements. This is a very common case here (e.g.,: List[Int] has both defined).
Example:
scala> implicitly[SizeOf[List[Int]]].sizeOf(List(1,2,3))
<console>:24: error: ambiguous implicit values:
both method foldSizeOf of type [T[_], A](implicit evidence$1: scalaz.Foldable[T], implicit evidence$2: SizeOf[A])VarSizeOf[T[A]]
and method fixedSizeOf of type [T[_], A](implicit evidence$1: scalaz.Length[T], implicit evidence$2: FixedSizeOf[A])VarSizeOf[T[A]]
match expected type SizeOf[List[Int]]
implicitly[SizeOf[List[Int]]].sizeOf(List(1,2,3))
What I would like is to be able to rely on the Foldable type class only when the Length+FixedSizeOf combination does not apply.
For that purpose, I can change the definition of foldSizeOf() to accept VarSizeOf elements:
implicit def foldSizeOfVar[T[_] : Foldable, A : VarSizeOf] = // ...
And now we have to fill in the problematic part that covers Foldable containers with FixedSizeOf elements and no Length defined. I'm not sure how to approach this, but pseudo-code would look something like:
implicit def foldSizeOfFixed[T[_] : Foldable : Not[Length], A : FixedSizeOf] = // ...
The 'Not[Length]', obviously, being the made up part here.
Partial solutions I am aware of
1) Define a class for low priority implicits and extend it, as seen in 'object Predef extends LowPriorityImplicits'.
The last implicit (foldSizeOfFixed()) can be defined in the parent class, and will be overridden by alternative from the descendant class.
I am not interested in this option because I'd like to eventually be able to support recursive usage of SizeOf, and this will prevent the implicit in the low priority base class from relying on those in the sub class (is my understanding here correct? EDIT: wrong! implicit lookup works from the context of the sub class, this is a viable solution!)
2) A rougher approach is relying on Option[TypeClass] (e.g.,: Option[Length[List]]. A few of those and I can just write one big ol' implicit that picks Foldable and SizeOf as mandatory and Length and FixedSizeOf as optional, and relies on the latter if they are available. (source: here)
The two problems here are lack of modularity and falling back to runtime exceptions when no relevant type class instances can be located (this example can probably be made to work with this solution, but that's not always possible)
EDIT: This is the best I was able to get with optional implicits. It's not there yet:
implicit def optionalTypeClass[TC](implicit tc: TC = null) = Option(tc)
type OptionalLength[T[_]] = Option[Length[T]]
type OptionalFixedSizeOf[T[_]] = Option[FixedSizeOf[T]]
implicit def sizeOfContainer[
T[_] : Foldable : OptionalLength,
A : SizeOf : OptionalFixedSizeOf]: SizeOf[T[A]] = new SizeOf[T[A]] {
def sizeOf(as: T[A]) = {
// optionally calculate using Length + FixedSizeOf is possible
val fixedLength = for {
lengthOf <- implicitly[OptionalLength[T]]
sizeOf <- implicitly[OptionalFixedSizeOf[A]]
} yield lengthOf.length(as) * sizeOf.fixedSize
// otherwise fall back to Foldable
fixedLength.getOrElse {
val foldable = implicitly[Foldable[T]]
val sizeOf = implicitly[SizeOf[A]]
foldable.foldMap(as)(a => sizeOf.sizeOf(a))
}
}
}
Except this collides with fixedSizeOf() from earlier, which is still necessary.
Thanks for any help or perspective :-)
I eventually solved this using an ambiguity-based solution that doesn't require prioritizing using inheritance.
Here is my attempt at generalizing this.
We use the type Not[A] to construct negative type classes:
import scala.language.higherKinds
trait Not[A]
trait Monoid[_] // or import scalaz._, Scalaz._
type NotMonoid[A] = Not[Monoid[A]]
trait Functor[_[_]] // or import scalaz._, Scalaz._
type NotFunctor[M[_]] = Not[Functor[M]]
...which can then be used as context bounds:
def foo[T: NotMonoid] = ...
We proceed by ensuring that every valid expression of Not[A] will gain at least one implicit instance.
implicit def notA[A, TC[_]] = new Not[TC[A]] {}
The instance is called 'notA' -- 'not' because if it is the only instance found for 'Not[TC[A]]' then the negative type class is found to apply; the 'A' is commonly appended for methods that deal with flat-shaped types (e.g. Int).
We now introduce an ambiguity to turn away cases where the undesired type class is applied:
implicit def notNotA[A : TC, TC[_]] = new Not[TC[A]] {}
This is almost exactly the same as 'NotA', except here we are only interested in types for which an instance of the type class specified by 'TC' exists in implicit scope. The instance is named 'notNotA', since by merely matching the implicit being looked up, it will create an ambiguity with 'notA', failing the implicit search (which is our goal).
Let's go over a usage example. We'll use the 'NotMonoid' negative type class from above:
implicitly[NotMonoid[java.io.File]] // succeeds
implicitly[NotMonoid[Int]] // fails
def showIfNotMonoid[A: NotMonoid](a: A) = a.toString
showIfNotMonoid(3) // fails, good!
showIfNotMonoid(scala.Console) // succeeds for anything that isn't a Monoid
So far so good! However, types shaped M[_] and type classes shaped TC[_[_]] aren't supported yet by the scheme above. Let's add implicits for them as well:
implicit def notM[M[_], TC[_[_]]] = new Not[TC[M]] {}
implicit def notNotM[M[_] : TC, TC[_[_]]] = new Not[TC[M]] {}
implicitly[NotFunctor[List]] // fails
implicitly[NotFunctor[Class]] // succeeds
Simple enough. Note that Scalaz has a workaround for the boilerplate resulting from dealing with several type shapes -- look for 'Unapply'. I haven't been able to make use of it for the basic case (type class of shape TC[_], such as Monoid), even though it worked on TC[_[_]] (e.g. Functor) like a charm, so this answer doesn't cover that.
If anybody's interested, here's everything needed in a single snippet:
import scala.language.higherKinds
trait Not[A]
object Not {
implicit def notA[A, TC[_]] = new Not[TC[A]] {}
implicit def notNotA[A : TC, TC[_]] = new Not[TC[A]] {}
implicit def notM[M[_], TC[_[_]]] = new Not[TC[M]] {}
implicit def notNotM[M[_] : TC, TC[_[_]]] = new Not[TC[M]] {}
}
import Not._
type NotNumeric[A] = Not[Numeric[A]]
implicitly[NotNumeric[String]] // succeeds
implicitly[NotNumeric[Int]] // fails
and the pseudo code I asked for in the question would look like so (actual code):
// NotFunctor[M[_]] declared above
def notFunctor[M[_] : NotFunctor](m: M[_]) = s"$m is not a functor"
Update: Similar technique applied to implicit conversions:
import scala.language.higherKinds
trait Not[A]
object Not {
implicit def not[V[_], A](a: A) = new Not[V[A]] {}
implicit def notNot[V[_], A <% V[A]](a: A) = new Not[V[A]] {}
}
We can now (e.g.) define a function that will only admit values if their types aren't viewable as Ordered:
def unordered[A <% Not[Ordered[A]]](a: A) = a
In Scala 3 (aka Dotty), the aforementioned tricks no longer work.
The negation of givens is built-in with NotGiven:
def f[T](value: T)(using ev: NotGiven[MyTypeclass[T]])
Examples:
f("ok") // no given instance of MyTypeclass[T] in scope
given MyTypeclass[String] = ... // provide the typeclass
f("bad") // compile error