I am reading this code for a long time. I typed it into REPL and it works as well.
but I don't have any idea of what's going on here. Why and how does this even work!!!
import shapeless._
case class Size[L <: HList](get : Int)
object Size {
implicit val hnilSize = Size[HNil](0)
implicit def hconsSize[H, T <: HList](implicit tailSize: Size[T]) =
Size[H :: T](1 + tailSize.get)
def apply[L <: HList](l : L)(implicit size: Size[L]) : Int = size.get
}
Size(1 :: "Foo" :: true :: HNil)
Can someone explain this step by step and help me understand what is going on here.
Yeah, that's pretty thick stuff.
The mind-bender here is that hconsSize is recursive without actually being self referential.
Both apply and hconsSize pull in an implicit of type Size[X]. There are only two implicits that could fit that bill:
hnilSize, but only if X is type HNil
hconsSize itself
So apply pulls in the hconsSize implicit, which adds 1 to the stack and pulls in another hconsSize implicit (not necessarily in that order). This continues until we encounter an element of type HNil. Then the hnilSize implicit is pulled in, the get is zero, the stack is unrolled and all those 1's are added up.
Result: number of elements in the shapeless HList.
Related
One of my projects uses a mix of scala features that appear to not mix well together:
type classes, and shapeless automated type class instance derivation
implicit conversion (to add useful syntax to types that have type class instances)
default parameters, because even though they're usually a bad thing they're just too convenient here
The problem I'm running into is type class instance derivation fails if either:
default parameters are not explicitly specified
shapeless derivation uses Lazy
Here's the smallest possible amount of code I could write to reproduce the issue:
Show.scala
import shapeless._
trait Show[A] {
def show(a: A): String
}
object Show {
def from[A](f: A => String): Show[A] = new Show[A] {
override def show(a: A) = f(a)
}
implicit val intShow: Show[Int] = Show.from(_.toString)
implicit def singletonShow[A](implicit
sa: Show[A]
): Show[A :: HNil] = Show.from {
case (a :: HNil) => sa.show(a)
}
implicit def singletonCaseClassShow[A, H <: HList](implicit
gen: Generic.Aux[A, H],
sh: Lazy[Show[H]]
): Show[A] = Show.from {
a => sh.value.show(gen.to(a))
}
}
Run.scala
object Run extends App {
implicit class ShowOps[A](val a: A) extends AnyVal {
def show(header: String = "> ")(implicit sa: Show[A]): String =
header + sa.show(a)
}
case class Foo(i: Int)
println(Foo(12).show())
}
This fails to compile with the following error message:
Run.scala:10: could not find implicit value for parameter sa: Show[Run.Foo]
[error] println(Foo(12).show())
The compilation error is fixed by either:
explicitly passing a header parameter to show in Run.scala
removing the Lazy wrapper to the implicit Show[H] in Show.scala
I must admit I'm at a complete loss here. I'd love to understand what happens, and I'd love to know of a workaround if one exists.
Short answer:
If you move the context bound to the implicit class, it also works fine. You have to sacrifice the value class to do it, but I think it's also cleaner to tell the compiler up front that only As which have a Show will get enriched by it:
implicit class Show2Ops[A : Show](a: A) {
def show2(header: String = "> ") = header + implicitly[Show[A]].show(a)
}
println(Foo(12).show2())
Long theory:
Lazy does some interesting tricks, which are hard to follow. You didn't specifically ask about what Lazy is doing, but I was curious about it, since I use it all the time without being sure how it works. So I took a look at it. As near as I can tell, it goes something like this.
You have a case class with a recursive field:
case class A(first: Int, next: Option[A])
And assume you had another case in Show's companion for Option:
implicit def opt[A](implicit showA: Show[A]): Show[Option[A]] = Show.from {
case Some(a) => s"Some(${showA.show(a)})"
case None => "None"
}
And instead of singletonShow you had a real HNil case and an inductive case, as is typical:
implicit val hnil: Show[HNil] = Show.from(_ => "")
implicit def hcons[H, T <: HList](implicit
showH: Show[H],
showT: Show[T]
): Show[H :: T] = Show.from {
case h :: t => showH(h) + ", " + showT(t) // for example
}
And let's rename singletonCaseClassShow to genericShow because it's not just for singletons anymore.
Now let's say you didn't have the Lazy there in genericShow. When you try to summon a Show[A], the compiler goes to:
genericShow[A] with open implicit search for Show[A]
hcons[Int :: Option[A] :: HNil] with open implicit search for Show[A] and Show[Int :: Option[A] :: HNil
intShow with open implicit search for Show[A] and Show[Int] and Show[Option[A] :: HNil]
hcons[Option[A] :: HNil] with open implicit search for Show[A] and Show[Option[A] :: HNil]
opt[A] with open implicit search for Show[A] and Show[Option[A]] and Show[Option[A] :: HNil]
genericShow[A] with open implicit search for Show[A] and Show[Option[A]] and Show[Option[A] :: HNil]
Now it's pretty clear that there's a problem, because it's going to go back to #2 and happen all over again, never making any progress.
How Lazy overcomes this is by going into a macro at the time the compiler attempts to materialize an implicit instance of it. So when you use implicit showH: Lazy[Show[H]] in hcons instead of just Show[H], the compiler goes to that macro to find Lazy[Show[H]] instead of staying in your implicit Show cases.
The macro checks the open implicits (which macros helpfully have access to) and goes into its own implicit resolution algorithm that always fully resolves open implicits before continuing with finding the implicit instance of T (for Lazy[T]). If it comes to resolving an implicit that's already open, it substitutes a dummy tree (essentially telling the compiler "I got this, don't worry about it") that tracks the knotted dependencies so that the rest of the resolution can finish. And at the end, it cleans up the dummy trees (I can't quite figure out how this works; there's a surprising amount of code there and it's pretty complicated!)
So why does Lazy seem to mess up your default parameter situation? I think it's the confluence of a few things (only a hypothesis):
With your original ShowOps, calling .show on a value causes a it to be implicitly wrapped in ShowOps[A]. What is A going to be? Is it going to be Foo, AnyRef, Any? Is it going to be a unique single type? It's not exactly clear, because at that time there is no constraint on A and Scala doesn't know that your call to .show will actually constraint it (due to the context bound).
Without Lazy, this works out OK, because if Scala chooses the wrong A and .show doesn't typecheck, it will realize its mistake and back out of the A it chose.
With Lazy, there is a bunch of other logic going on, and it kind of tricks Scala into thinking that whatever A it chose is fine. But when it comes time to close the loop, it doesn't work out, and by that time it's too late to back out.
Somehow, the default parameter being unspecified influences Scala's initial choice for choosing A in ShowOps[A].
I'm using libraryDependencies += "com.chuusai" %% "shapeless" % "2.2.4"
Currently i have model HList types like
sealed trait Section
case class Header(...) extends Section
case class Customer(...) extends Section
case class Supplier(...) extends Section
case class Tech(...) extends Section
type ContractView = Header :: (Customer :: Supplier :: HNil) :: Tech :: HNil
In my user code, i'd like to filter technical sections that are not supposed to view using foldRight as proposed in this answer:
trait collectAllRF extends Poly2 {
implicit def atAny[L <: HList, X] = at[X, L](_ :: _)
}
object collectVisRF extends collectAllRF {
implicit def atInvis[L <: HList, S <: Section : InvisibleSection] = at[S, L]((_, l) => l)
}
For example there are defined:
trait InvisibleSection[S <: Section]
implicit object _techisInvisible extends InvisibleSection[Tech]
Fold is working correctly, but suddenly i could not use following filter or map on this object, so for example this code:
val filtered = view.foldRight(HNil)(collectVisRF)
view.filter[Header]
produces compile error:
error: could not find implicit value for parameter partition:
shapeless.ops.hlist.Partition[shapeless.::[Header,shapeless.::[shapeless.::[Customer,shapeless.::[Supplier,shapeless.HNil]],shapeless.HNil.type]],Header]
while this
view.filter[Header]
and this
val h = view.select[Header]
val l = view.select[Customer::Supplier::HNil]
val c = l.select[Customer]
val s = l.select[Supplier]
val manual = h :: (c :: s :: HNil) :: HNil
manual.filter[Header]
compiles ok
Lately i've found little HNil.type at end of foldRight's result type and changed my filter definition to
view.foldRight(HNil.asInstanceOf[HNil])(collectVisRF)
And all worked properly
Is this an expected behaviour, and if yes why there is no some
val hNil: HNil = HNil
In the library?
Your eventual fix is almost, but not quite, right. Rather than asInstanceOf you should use a type ascription,
view.foldRight(HNil: HNil)(collectVisRF)
Your question about why there is no definition of an hnil value typed as HNil rather than as HNil.type is a good one. shapeless is different from typical Scala libraries in that it makes heavy use of singleton types, HNil.type included, so the current situation isn't as obviously wrong as the corresponding situation in the Scala standard library where None.type and Nil.type are almost never desired.
Nevertheless the situation you describe in your question comes up more often than I would like, so it's clearly a real problem. I think it would be too confusing to have two hnil values, one with a more precise type than the other, so the question boils down to: how much existing stuff would break if HNil (the type) was inferred as the type of HNil (the value) rather than HNil.type as it is now.
Feel free to open a ticket in the shapeless issue tracker on Github to investigate this, and if you'd like to give it a try, please do :-)
I had a neat idea (well, that's debatable, but let's say I had an idea) for making implicit dependency injection easier in Scala. The problem I have is that if you call any methods which require an implicit dependency, you must also decorate the calling method with the same dependency, all the way through until that concrete dependency is finally in scope. My goal was to be able to encode a trait as requiring a group of implicits at the time it's mixed in to a concrete class, so it could go about calling methods that require the implicits, but defer their definition to the implementor.
The obvious way to do this is with some kind of selftype a la this psuedo-scala:
object ThingDoer {
def getSomething(implicit foo: Foo): Int = ???
}
trait MyTrait { self: Requires[Foo and Bar and Bubba] =>
//this normally fails to compile unless doThing takes an implicit Foo
def doThing = ThingDoer.getSomething
}
After a few valiant attempts to actually implement a trait and[A,B] in order to get that nice syntax, I thought it would be smarter to start with shapeless and see if I could even get anywhere with that. I landed on something like this:
import shapeless._, ops.hlist._
trait Requires[L <: HList] {
def required: L
implicit def provide[T]:T = required.select[T]
}
object ThingDoer {
def needsInt(implicit i: Int) = i + 1
}
trait MyTrait { self: Requires[Int :: String :: HNil] =>
val foo = ThingDoer.needsInt
}
class MyImpl extends MyTrait with Requires[Int :: String :: HNil] {
def required = 10 :: "Hello" :: HNil
def showMe = println(foo)
}
I have to say, I was pretty excited when this actually compiled. But, it turns out that when you actually instantiate MyImpl, you get an infinite mutual recursion between MyImpl.provide and Required.provide.
The reason that I think it's due to some mistake I've made with shapeless is that when I step through, it's getting to that select[T] and then steps into HListOps (makes sense, since HListOps is what has the select[T] method) and then seems to bounce back into another call to Requires.provide.
My first thought was that it's attempting to get an implicit Selector[L,T] from provide, since provide doesn't explicitly guard against that. But,
The compiler should have realized that it wasn't going to get a Selector out of provide, and either chosen another candidate or failed to compile.
If I guard provide by requiring that it receive an implicit Selector[L,T] (in which case I could just apply the Selector to get the T) then it doesn't compile anymore due to diverging implicit expansion for type shapeless.ops.hlist.Selector[Int :: String :: HNil], which I don't really know how to go about addressing.
Aside from the fact that my idea is probably misguided to begin with, I'm curious to know how people typically go about debugging these kinds of mysterious, nitty-gritty things. Any pointers?
When I get confused about something related to implicits / type-level behaviour, I tend to find the reify technique useful:
scala> import scala.reflect.runtime.universe._
import scala.reflect.runtime.universe._
scala> val required: HList = HNil
required: shapeless.HList = HNil
scala> reify { implicit def provide[T]:T = required.select[T] }
res3: reflect.runtime.universe.Expr[Unit] =
Expr[Unit]({
implicit def provide[T]: T = HList.hlistOps($read.required).select[T](provide);
()
})
At this point it's easy to see what's gone wrong - the compiler thinks provide can provide any arbitrary T (because that's what you've told it), so it just calls provide to get the required Selector[L, T]. At compile time it only resolves this once, so there is no diverging implicit, no confusion at compile time - only at run-time.
The diverging implicit expansion happens because the compiler looks for a Selector[Int :: String :: HNil], it thinks provide could give it one if given a Selector[Int :: String :: HNil, Selector[Int :: String :: HNil]], it thinks provide could give it one if given a Selector[Int :: String :: HNil, Selector[Int :: String :: HNil, Selector[Int :: String :: HNil]] and at some point it realises this is an infinite loop. Where/how are you expecting it to get the Selector it needs? I think your provide is misguided because it's too general. Try making the call to ThingDoer.needsInt with an explicit int work first before trying to make it all implicit.
This general approach does work - I've written applications that use it as a DI mechanism -though beware of quadratic compile times.
I recently asked Map and reduce/fold over HList of scalaz.Validation and got a great answer as to how to transform a fixed sized tuple of Va[T] (which is an alias for scalaz.Validation[String, T]) into a scalaz.ValidationNel[String, T]. I've since then been studying Shapeless and type level programming in general to try to come up with a solution that works on tuples of any size.
This is what I'm starting out with:
import scalaz._, Scalaz._, shapeless._, contrib.scalaz._, syntax.std.tuple._
type Va[A] = Validation[String, A]
// only works on pairs of Va[_]
def validate[Ret, In1, In2](params: (Va[In1], Va[In2]))(fn: (In1, In2) => Ret) = {
object toValidationNel extends Poly1 {
implicit def apply[T] = at[Va[T]](_.toValidationNel)
}
traverse(params.productElements)(toValidationNel).map(_.tupled).map(fn.tupled)
}
so then validate is a helper I call like this:
val params = (
postal |> nonEmpty[String]("no postal"),
country |> nonEmpty[String]("no country") >=> isIso2Country("invalid country")
)
validate(params) { (postal, country) => ... }
I started out by taking any Product instead of a pair and constraining its contents to Va[T]:
// needs to work with a tuple of Va[_] of arbitrary size
def validateGen[P <: Product, F, L <: HList, R](params: P)(block: F)(
implicit
gen: Generic.Aux[P, L],
va: UnaryTCConstraint[L, Va],
fp: FnToProduct.Aux[F, L => R]
) = ???
I do have the feeling that simply adding the constraint only makes sure the input is valid but doesn't help at all with implementing the body of the function but I don't know how to go about correcting that.
traverse then started complaining about a missing evidence so I ended up with:
def validateGen[P <: Product, F, L <: HList, R](params: P)(block: F)(
implicit
gen: Generic.Aux[P, L],
va: UnaryTCConstraint[L, Va],
tr: Traverser[L, toValidationNel.type],
fp: FnToProduct.Aux[F, L => R]
) = {
traverse(gen.to(params): HList)(toValidationNel).map(_.tupled).map(block.toProduct)
}
The compiler however continued to complain about a missing Traverser[HList, toValidationNel.type] implicit parameter even though it's there.
Which additional evidence do I need to provide to the compiler in order for the traverse call to compile? Has it got to do with the UnaryTCConstraint not being declared in a way that is useful to the traverse call, i.e. it cannot apply toValidationNel to params because it cannot prove that params contains only Va[_]?
P.S. I also found leftReduce Shapeless HList of generic types and tried to use foldRight instead of traverse to no avail; the error messages weren't too helpful when trying to diagnose which evidence the compiler was really lacking.
UPDATE:
As per what lmm has pointed out, I've removed the cast to HList, however, the problem's now that, whereas in the non-generic solution I can call .map(_.tupled).map(block.toProduct) on the result of the traverse call, I'm now getting:
value map is not a member of shapeless.contrib.scalaz.Out
How come it's possible that it was possible on the result of the traverse(params.productElements)(toValidationNel) call and not the generic traverse?
UPDATE 2:
Changing the Traverser[...] bit to Traverser.Aux[..., Va[L]] helped the compiler figure out the expected result type of the traversal, however, this only makes the validateGen function compile successfully but yields another error at the call site:
[error] could not find implicit value for parameter tr: shapeless.contrib.scalaz.Traverser.Aux[L,toValidationNel.type,Va[L]]
[error] validateGen(params) { (x: String :: String :: HNil) => 3 }
[error] ^
I'm also getting the feeling here that the UnaryTCConstraint is completely unnecessary — but I'm still too new to Shapeless to know if that's the case.
UPDATE 3:
Having realized the type that comes out of the traverser cannot be Va[L] because L itself is already a hlist of Va[_], I've split the L type parameter to In and Out:
def validateGen[P <: Product, F, In <: HList, Out <: HList, R](params: P)(block: F)(
implicit
gen: Generic.Aux[P, In],
va: UnaryTCConstraint[In, Va], // just for clarity
tr: Traverser.Aux[In, toValidationNel.type, Va[Out]],
fn: FnToProduct.Aux[F, Out => R]
): Va[R] = {
traverse(gen.to(params))(toValidationNel).map(block.toProduct)
}
this compiles well — I'd be curious to find out how come the previous version with Va[L] being the return value (i.e. the 3rd param to Traverser.Aux) even compiled — however, at the call site, I now get:
Unspecified value parameters tr, fn
You have a Traverser[L, toValidationNel.type] which is not the same thing as Traverser[HList, toValidationNel.type] (which would have to work for any HList - no chance). I don't know why you've written gen.to(params): HList, but this is throwing away type information; shouldn't that be of type L?
This will probably only kick the problem one level higher; I doubt you'll be able to get the Traverser you need automatically. But you should be able to write an implicit method that supplies one based on the UnaryTCConstraint, and it's possible shapeless already includes that and it will Just Work.
Update:
In the first example, the compiler knew the specific Traverser instance it was using, so it knew what the Out type was. In validateGen you haven't constrained anything about tr.Out, so the compiler has no way of knowing that it's a type that supports .map. If you know what the output of the traverse needs to be then you can probably require an appropriate Traverser.Aux i.e.:
tr: Traverser.Aux[L, toValidationNel.type, Va[L]]
(Just don't ask me how to make sure the type inference still works).
I think you probably don't want the .map(_.tupled), because the _ there is already a HList (I suspect it's redundant in the original validate too), but I've never used .toProduct before so maybe you have it right.
Update 2:
Right, this is as I initially suspected. Looking at the implementation of Sequencer I suspect you're right and the UnaryTCConstraint will be subsumed by the Traverser. If you're not using it then no point requiring it.
The only advice I can give is to chase through the calls that should be providing your implicits. E.g. the Traverser should be coming from Traverser.mkTraverser. So if you try calling Traverser.mkTraverser[String :: String :: HNil, toValidationNel.type, Va[String] :: Va[String] :: HNil] then you should be able to see whether it's the Mapper or the Sequencer that can't be found. Then you can recurse through the implicit calls that should happen until you find a simpler case of something that should be working, but isn't.
After long hours of experimentation, frustration and dead brain cells, I've started from scratch without Traverser and instead gone with Mapper and Sequencer; I'll later try to see if I can make it use Traverser again (if not for practicality, at least for learning purposes):
def validate[P <: Product, F, L1 <: HList, L2 <: HList, L3 <: HList, R](params: P)(block: F)(
implicit
gen: Generic.Aux[P, L1],
mp: Mapper.Aux[toValidationNel.type, L1, L2],
seq: Sequencer.Aux[L2, VaNel[L3]],
fn: FnToProduct.Aux[F, L3 => R]
): VaNel[R] = {
sequence(gen.to(params).map(toValidationNel)).map(block.toProduct)
}
Here's proof — pun intended — that it runs http://www.scastie.org/7086.
This is essentially what I want:
case class Foo[T](x: T)
object combine extends Poly {
implicit def caseFoo[A, B] = use((f1: Foo[A], f2: Foo[B]) => Foo((f1.x, f2.x)))
}
def combineHLatest[L <: HList](l: L) = l.reduceLeft(combine)
So combineHLatest(Foo(1) :: Foo("hello") :: HNil) should yield Foo( (1, "hello") )
The above doesn't compile as it cannot find an implicit LeftReducer but I'm at a loss as to how to implement one.
It was pretty late last night when I answered this, and while the information in the original answer below is correct, it's not necessarily the most helpful presentation.
There's a good reason you should be at a loss as to how to implement the LeftReducer, since that's not your job. The compiler will create any valid instance of the type class you need—you just have to make sure it has all the information it needs.
For example, the following works just fine with your implementation:
scala> (Foo(1) :: Foo("hello") :: HNil).reduceLeft(combine)
res0: Foo[(Int, String)] = Foo((1,hello))
Here the compiler can see the type of the HList you want to reduce, and can create the appropriate LeftReducer instance.
When you wrap the call to leftReduce up in a method, on the other hand, the compiler doesn't know anything about the list you're calling it on except what you explicitly tell it. In your implementation of combineHLatest, the compiler knows that L is an HList, but that's it—it doesn't have any evidence that it can perform the reduction. Fortunately it's pretty easy to give it this evidence via an implicit parameter (see the original answer below).
I'd originally posted a kind of clunky solution to the flattened-tuple problem here, but the clunkiness was only because of a small typo in my original attempt. It's actually possible to write a fairly elegant implementation:
def combineHLatest[L <: HList, R <: HList](l: L)(implicit
r: RightFolder.Aux[L, Foo[HNil], combine.type, Foo[R]],
t: Tupler[R]
) = Foo(l.foldRight(Foo(HNil: HNil))(combine).x.tupled)
(My mistake was writing R instead of Foo[R] as the last type parameter on the Aux.)
Original answer
This will work as expected if you make sure your method has evidence that it can perform the reduction on the input:
import shapeless._, ops.hlist.LeftReducer
def combineHLatest[L <: HList](l: L)(implicit r: LeftReducer[L, combine.type]) =
l.reduceLeft(combine)
Note, though, that this approach will just build up a nested tuple if you have more than two arguments, so you may want something more like this:
object combine extends Poly {
implicit def caseFoo[A, B <: HList] = use(
(f1: Foo[A], f2: Foo[B]) => Foo(f1.x :: f2.x)
)
}
def combineHLatest[L <: HList](l: L)(implicit
r: RightFolder[L, Foo[HNil], combine.type]
) = l.foldRight(Foo(HNil: HNil))(combine)
And then for example:
scala> println(combineHLatest(Foo(1) :: Foo("hello") :: Foo('a) :: HNil))
Foo(1 :: hello :: 'a :: HNil)
If you wanted a (flattened) tuple instead, that'd also be pretty straightforward.