I'm chaining transformations, and I'd like to accumulate the result of each transformation so that it can potentially be used in any subsequent step, and also so that all the results are available at the end (mostly for debugging purposes). There are several steps and from time to time I need to add a new step or change the inputs for a step.
HList seems to offer a convenient way to collect the results in a flexible but still type-safe way. But I'd rather not complicate the actual steps by making them deal with the HList and the accompanying business.
Here's a simplified version of the combinator I'd like to write, which isn't working. The idea is that given an HList containing an A, and the index of A, and a function from A -> B, mapNth will extract the A, run the function, and cons the result onto the list. The resulting extended list captures the type of the new result, so several of these mapNth-ified steps can be composed to produce a list containing the result from each step:
def mapNth[L <: HList, A, B]
(l: L, index: Nat, f: A => B)
(implicit at: shapeless.ops.hlist.At[L, index.N]):
B :: L =
f(l(index)) :: l
Incidentally, I'll also need map2Nth taking two indices and f: (A, B) => C, but I believe the issues are the same.
However, mapNth does not compile, saying l(index) has type at.Out, but f's argument should be A. That's correct, of course, so what I suppose I need is a way to provide evidence that at.Out is in fact A (or, at.Out <: A).
Is there a way to express that constraint? I believe it will have to take the form of an implicit, because of course the constraint can only be checked when mapNth is applied to a particular list and function.
You're exactly right about needing evidence that at.Out is A, and you can provide that evidence by including the value of the type member in at's type:
def mapNth[L <: HList, A, B]
(l: L, index: Nat, f: A => B)
(implicit at: shapeless.ops.hlist.At[L, index.N] { type Out = A }):
B :: L =
f(l(index)) :: l
The companion objects for type classes like At in Shapeless also define an Aux type that includes the output type as a final type parameter.
def mapNth[L <: HList, A, B]
(l: L, index: Nat, f: A => B)
(implicit at: shapeless.ops.hlist.At.Aux[L, index.N, A]):
B :: L =
f(l(index)) :: l
This is pretty much equivalent but more idiomatic (and it looks a little nicer).
Related
I have the following function which takes a list of K-dimensional points of the shape Point(d0, d1,...,dK) and returns the index of the dimension at which the range is largest. The Index is returned as a Nat:
def findMaxRangeDim[T, H <: HList, L <: HList, K<: HList](data: List[T])(
implicit gen: Generic.Aux[T, H],
zipper: Zip.Aux[H::H::HNil, L],
maxMapper: Mapper.Aux[mergeMaxMap.type, L, H],
minMapper: Mapper.Aux[mergeMinMap.type, L, H],
diffMapper: Mapper.Aux[absDiffMap.type, L, H],
indexZipper: ZipWithIndex.Aux[H, K],
folder: LeftFolder.Aux[K, (BigDecimal, Nat), maxIndexFinder.type,
(BigDecimal, Nat)]
): Nat = {/*implementation*/}
Let's say that I have another function that takes a data point and a Nat, and returns the element located at the dimension denoted by the Nat:
def getAt[T, H<:HList, N<:Nat](p: T, n: N)(
implicit gen: Generic.Aux[T, H],
at: At.Aux[H, N, BigDecimal]
) = {/*implementation*/}
In can confirm that:
// This compiles:
getAt(data.head, Nat._1)
// This doesn't:
getAt(data.head, findMaxRangeDim(data))
The error is could not find implicit value for parameter at: shapeless.ops.hlist.At.Aux[H,shapeless.Nat,BigDecimal].
I'm guessing that this happens because shapeless is looking for an implicit of the exact type of the result of findMaxRangeDim(data), but this is known only at runtime. Is there any way of passing the right implicit and using the resulting Nat in another function?
I tried the approach by Dmytro Mitin but the compiler would simply complain that it couldn't find an implicit for the right type.
Then I asked in Shapeless' gitter channel, and realized that the compiler needs to know the specific type of the Nat at compile time. Because my code finds the right Nat at runtime, the compiler can't know beforehand which type to use.
The lesson learned being, Nats are not analogous to indexes in typical Iterables.
You should modify findMaxRangeDim so that it returns specific N <: Nat (e.g. add it as a type parameter) rather than just Nat (it's too rough).
Now getAt(data.head, findMaxRangeDim(data)) doesn't work since there is no At.Aux[H, Nat, BigDecimal] (but there are At.Aux[H, N, BigDecimal] for specific N <: Nat).
Implicits are resolved at compile time, not at runtime.
Given:
scala> import shapeless.nat.
_0 _10 _12 _14 _16 _18 _2 _21 _3 _5 _7 _9 natOps
_1 _11 _13 _15 _17 _19 _20 _22 _4 _6 _8 apply toInt
scala> import shapeless.ops.nat._
import shapeless.ops.nat._
After > 3 minutes, the following code has not compiled/run. Why's that?
scala> Sum[_22, _22]
Also, looking at the above REPL auto-completion, does _44 even exist in shapeless?
Why is it so slow?
Let's start with a smaller number. When you ask for Sum[_4, _4], the compiler is going to go looking for an instance, and it'll find these two methods:
implicit def sum1[B <: Nat]: Aux[_0, B, B] = new Sum[_0, B] { type Out = B }
implicit def sum2[A <: Nat, B <: Nat](implicit
sum: Sum[A, Succ[B]]
): Aux[Succ[A], B, sum.Out] = new Sum[Succ[A], B] { type Out = sum.Out }
The first one is clearly out since _4 is not _0. It knows that _4 is the same as Succ[_3] (more on that in a second), so it'll try sum2 with A as _3 and B as _4.
This means we need to find a Sum[_3, _5] instance. sum1 is out for similar reasons as before, so we try sum2 again, this time with A = _2 and B = _5, which means we need a Sum[_2, _6], which gets us back to sum2, with A = _1 and B = _6, which sends us looking for a Sum[_1, _7]. This is the last time we'll use sum2, with A = _0 and B = _7. This time when we go looking for a Sum[_0, _8] we'll hit sum1 and we're done.
So it's clear that for n + n we're going to have to do n + 1 implicit searches, and during each one the compiler is going to be doing type equality checks and other stuff (update: see Miles's answer for an explanation of what the biggest problem here is) that requires traversing the structure of the Nat types, so we're in exponential land. The compiler really, really isn't designed to work efficiently with types like this, which means that even for small numbers, this operation is going to take a long time.
Side note 1: implementation in Shapeless
Off the top of my head I'm not entirely sure why sum2 isn't defined like this:
implicit def sum2[A <: Nat, B <: Nat](implicit
sum: Sum[A, B]
): Aux[Succ[A], B, Succ[sum.Out]] = new Sum[Succ[A], B] { type Out = Succ[sum.Out] }
This is much faster, at least on my machine, where Sum[_18, _18] compiles in four seconds as opposed to seven minutes and counting.
Side note 2: induction heuristics
This doesn't seem to be a case where Typelevel Scala's -Yinduction-heuristics helps—I just tried compiling Shapeless with the #inductive annotation on Sum and it's still seems pretty much exactly as horribly slow as without it.
What about 44?
The _1, _2, _3 type aliases are defined in code produced by this boilerplate generator in Shapeless, which is configured only to produce values up to 22. In this case specifically, this is an entirely arbitrary limit. We can write the following, for example:
type _23 = Succ[_22]
And we've done exactly the same thing the code generator does, but going one step further.
It doesn't really matter much that Shapeless's _N aliases stop at 22, though, since they're just aliases. The important thing about a Nat is its structure, and that's independent of any nice names we might have for it. Even if Shapeless didn't provide any _N aliases at all, we could still write code like this:
import shapeless.Succ, shapeless.nat._0, shapeless.ops.nat.Sum
Sum[Succ[Succ[_0]], Succ[Succ[_0]]]
And it would be exactly the same as writing Sum[_2, _2], except that it's a lot more annoying to type.
So when you write Sum[_22, _22] the compiler isn't going to have any trouble representing the result type (i.e. 44 Succs around a _0), even though it doesn't have a _44 type alias.
Following on from Travis's excellent answer, it appears that it's the use of the member type in the definition of sum2 which is the root of the problem. With the following definition of Sum and its instances,
trait Sum[A <: Nat, B <: Nat] extends Serializable { type Out <: Nat }
object Sum {
def apply[A <: Nat, B <: Nat](implicit sum: Sum[A, B]): Aux[A, B, sum.Out] = sum
type Aux[A <: Nat, B <: Nat, C <: Nat] = Sum[A, B] { type Out = C }
implicit def sum1[B <: Nat]: Aux[_0, B, B] = new Sum[_0, B] { type Out = B }
implicit def sum2[A <: Nat, B <: Nat, C <: Nat]
(implicit sum : Sum.Aux[A, Succ[B], C]): Aux[Succ[A], B, C] =
new Sum[Succ[A], B] { type Out = C }
}
which replaces the use of the member type with an additional type variable, the compile time is 0+noise on my machine both with and without -Yinduction-heurisitics.
I think that the issue we're seeing is a pathological case for subtyping with member types.
Aside from that, the induction is so small that I wouldn't actually expect -Yinduction-heurisitics to make much of an improvement.
Update now fixed in shapeless.
Using shapeless, I'm trying to define a function:
import shapeless._
import ops.nat._
import nat._
def threeNatsLessThan3[N <: Nat](xs: Sized[List[N], N])
(implicit ev: LTEq[N, _3]) = ???
where it will only compile if the input xs is a List (of sized 3) of Nat where each element is <= 3.
But that fails to compile:
scala> threeNatsLessThan3[_3](List(_1,_2,_3))
<console>:22: error: type mismatch;
found : List[shapeless.Succ[_ >: shapeless.Succ[shapeless.Succ[shapeless._0]] with shapeless.Succ[shapeless._0] with shapeless._0 <: Serializable with shapeless.Nat]]
required: shapeless.Sized[List[shapeless.nat._3],shapeless.nat._3]
(which expands to) shapeless.Sized[List[shapeless.Succ[shapeless.Succ[shapeless.Succ[shapeless._0]]]],shapeless.Succ[shapeless.Succ[shapeless.Succ[shapeless._0]]]]a>
twoNatsFirstLtEqSecond[_3](List(_1,_2,_3))
^
How can I implement the above function correctly?
Also I would appreciate a solution using an HList too, where the HList consists only of Nat elements (if possible).
What you're typing as the signature is a List of size N which contains only elements of type N. To whit, Sized[List[N], N] denotes one of the following: List(_1), List(_2, _2), or finally List(_3, _3, _3), taking into consideration your type level constraint. That's almost what you want and explains the error the compiler is giving you:
required: shapeless.Sized[List[shapeless.nat._3],shapeless.nat._3]
To begin breaking down what you want to accomplish we need to note that you can't have a List[Nat] and also preserve the individual types. The abstractness of Nat would obscure them. So if you want to do things at compile time, you're going to have three choices: work with an HList, choose to fix the type of Nat within the list so that you have List[N] or choose to fix the size of the List[Nat] with Sized.
If you want to say that the List has size less than 3, then
def lessThanThree[N <: Nat](sz: Sized[List[Nat], N])(implicit ev: LTEq[N, _3]) = sz
If you want to say that the List has a Nat of less than three, again with a fixed N within the List:
def lessThanThree[N <: Nat, M <: Nat](sz: Sized[List[N], M])(implicit ev: LTEq[N, _3]) = sz
If you're looking to perhaps work with a Poly where you could define an at for any Nat such that it preserves the LTEq constraint, you'll need to understand that Sized does make working with map conform closer to the standard package map found on most collections, i.e. it requires a CanBuildFrom. That combined with the erasure of the individual Nat in List means that you'll have a lot of difficulty coming up with a solution which gives you the type of flexibility you're looking for.
If you were to work with an HList, you could do the following:
object LT3Identity extends Poly{
implicit def only[N <: Nat](implicit ev: LTEq[N, _3]) = at[N]{ i => i}
}
def lt3[L <: HList, M <: Nat](ls: L)(implicit lg: Length.Aux[L, M], lt: LTEq[M, _3]) = ls.map(LT3Identity)
which does both constrain your size of the Hlist to less than 3 while also only allowing HList that contain Nat of less than or equal to 3.
Could start value just be a parameter in op argument list ?
Food is defined on List as
def fold[A1 >: A](z: A1)(op: (A1, A1) ⇒ A1): A1 Folds the elements
of this traversable or iterator using the specified associative binary
operator.
What would the implications of defining fold as
def fold[A1 >: A](op: (z:A1,A1, A1) ⇒ A1): A1
So in this version the initial value is passed as a value to the function instead of being curried in a separate parameter list.
If you're looking to motivate that particular signature of foldLeft, it may be worthwhile to first examine reduceLeft.
// Slightly simplified to remove the supertype constraint
def reduceLeft(f: (A, A) => A): A
reduceLeft squishes the entire collection into a single element and it takes as an argument a function that tells it how to squish each new element in the collection onto what it's got so far.
There's, however, a problem. reduceLeft is partial. In particular if the collection is empty, reduceLeft has nowhere to begin squishing things. So we can make it total, by telling reduceLeft where to begin. So we give reduceLeft an additional parameter.
def reduceLeftTotal(initial: A, f: (A, A) => A): A
Note that if we just glommed initial as another argument to f, we wouldn't fix the partiality of reduceLeft. If this is an empty collection, we still blow up.
// This doesn't get us what we want. Where does the initial `A` come from?
def reduceLeftNotWhatWeWant(f: (A, A, A) => A): A
Okay, now that we've got reduceLeftTotal, there's an immediate new avenue for generalization. Why does the thing that we're squishing all the elements of our collection onto have to have the same type as the elements? The answer is it doesn't!
def generalReduceLeftTotal[B](initial: B, f: (B, A) => A): B
Finally because type information in previous argument lists, but not previous arguments in the same list, can be used to help Scala's type inference, we can reduce the amount of explicit type annotations we need by currying.
// And we're back to foldLeft!
def foldLeft[B](initial: B)(f: (B, A) => A): B
I was trying to do some stuff last night around accepting and calling a generic function (i.e. the type is known at the call site, but potentially varies across call sites, so the definition should be generic across arities).
For example, suppose I have a function f: (A, B, C, ...) => Z. (There are actually many such fs, which I do not know in advance, and so I cannot fix the types nor count of A, B, C, ..., Z.)
I'm trying to achieve the following.
How do I call f generically with an instance of (A, B, C, ...)? If the signature of f were known in advance, then I could do something involving Function.tupled f or equivalent.
How do I define another function or method (for example, some object's apply method) with the same signature as f? That is to say, how do I define a g for which g(a, b, c, ...) type checks if and only if f(a, b, c, ...) type checks? I was looking into Shapeless's HList for this. From what I can tell so far, HList at least solves the "representing an arbitrary arity args list" issue, and also, Shapeless would solve the conversion to and from tuple issue. However, I'm still not sure I understand how this would fit in with a function of generic arity, if at all.
How do I define another function or method with a related type signature to f? The biggest example that comes to mind now is some h: (A, B, C, ...) => SomeErrorThing[Z] \/ Z.
I remember watching a conference presentation on Shapeless some time ago. While the presenter did not explicitly demonstrate these things, what they did demonstrate (various techniques around abstracting/genericizing tuples vs HLists) would lead me to believe that similar things as the above are possible with the same tools.
Thanks in advance!
Yes, Shapeless can absolutely help you here. Suppose for example that we want to take a function of arbitrary arity and turn it into a function of the same arity but with the return type wrapped in Option (I think this will hit all three points of your question).
To keep things simple I'll just say the Option is always Some. This takes a pretty dense four lines:
import shapeless._, ops.function._
def wrap[F, I <: HList, O](f: F)(implicit
ftp: FnToProduct.Aux[F, I => O],
ffp: FnFromProduct[I => Option[O]]
): ffp.Out = ffp(i => Some(ftp(f)(i)))
We can show that it works:
scala> wrap((i: Int) => i + 1)
res0: Int => Option[Int] = <function1>
scala> wrap((i: Int, s: String, t: String) => (s * i) + t)
res1: (Int, String, String) => Option[String] = <function3>
scala> res1(3, "foo", "bar")
res2: Option[String] = Some(foofoofoobar)
Note the appropriate static return types. Now for how it works:
The FnToProduct type class provides evidence that some type F is a FunctionN (for some N) that can be converted into a function from some HList to the original output type. The HList function (a Function1, to be precise) is the Out type member of the instance, or the second type parameter of the FnToProduct.Aux helper.
FnFromProduct does the reverse—it's evidence that some F is a Function1 from an HList to some output type that can be converted into a function of some arity to that output type.
In our wrap method, we use FnToProduct.Aux to constrain the Out of the FnToProduct instance for F in such a way that we can refer to the HList parameter list and the O result type in the type of our FnFromProduct instance. The implementation is then pretty straightforward—we just apply the instances in the appropriate places.
This may all seem very complicated, but once you've worked with this kind of generic programming in Scala for a while it becomes more or less intuitive, and we'd of course be happy to answer more specific questions about your use case.