Converting nested case classes to nested Maps using Shapeless - scala

I am trying to solve [this][1] question using Shapeless, in summary it's about converting a nested case class to Map[String,Any], here is the example:
case class Person(name:String, address:Address)
case class Address(street:String, zip:Int)
val p = Person("Tom", Address("Jefferson st", 10000))
The goal is to convert p to following:
Map("name" -> "Tom", "address" -> Map("street" -> "Jefferson st", "zip" -> 10000))
I am trying to do it using Shapeless LabelledGeneric, here is what I have so far:
import shapeless._
import record._, syntax.singleton._
import ops.record._
import shapeless.ops.record._
def writer[T,A<:HList,H<:HList](t:T)
(implicit lGeneric:LabelledGeneric.Aux[T,A],
kys:Keys.Aux[A,H],
vls:Values[A]) = {
val tGen = lGeneric.to(t)
val keys = Keys[lGeneric.Repr].apply
val values = Values[lGeneric.Repr].apply(tGen)
println(keys)
println(values)
}
I am trying to have a recursive writer check each value and try to make Map for each element in value. The above code works fine but when I want to iterate over values with a sample Poly, using the following code I got these errors.
values.map(identity)
//or
tGen.map(identity)
Error:(75, 19) could not find implicit value for parameter mapper: shapeless.ops.hlist.FlatMapper[shapeless.poly.identity.type,vls.Out]
values.flatMap(identity)
^
Error:(75, 19) not enough arguments for method flatMap: (implicit mapper: shapeless.ops.hlist.FlatMapper[shapeless.poly.identity.type,vls.Out])mapper.Out.
Unspecified value parameter mapper.
values.flatMap(identity)
^
I don't know why I'm getting that error. I also would be happy to know if there is an easier way to do the whole thing using Shapeless.
[1]: Scala macros for nested case classes to Map and other way around

Any time you want to perform an operation like flatMap on an HList whose type isn't statically known, you'll need to provide evidence (in the form of an implicit parameter) that the operation is actually available for that type. This is why the compiler is complaining about missing FlatMapper instances—it doesn't know how to flatMap(identity) over an arbitrary HList without them.
A cleaner way to accomplish this kind of thing would be to define a custom type class. Shapeless already provides a ToMap type class for records, and we can take it as a starting point, although it doesn't provide exactly what you're looking for (it doesn't work recursively on nested case classes).
We can write something like the following:
import shapeless._, labelled.FieldType, record._
trait ToMapRec[L <: HList] { def apply(l: L): Map[String, Any] }
Now we need to provide instances for three cases. The first case is the base case—the empty record—and it's handled by hnilToMapRec below.
The second case is the case where we know how to convert the tail of the record, and we know that the head is something that we can also recursively convert (hconsToMapRec0 here).
The final case is similar, but for heads that don't have ToMapRec instances (hconsToMapRec1). Note that we need to use a LowPriority trait to make sure that this instance is prioritized properly with respect to hconsToMapRec0—if we didn't, the two would have the same priority and we'd get errors about ambiguous instances.
trait LowPriorityToMapRec {
implicit def hconsToMapRec1[K <: Symbol, V, T <: HList](implicit
wit: Witness.Aux[K],
tmrT: ToMapRec[T]
): ToMapRec[FieldType[K, V] :: T] = new ToMapRec[FieldType[K, V] :: T] {
def apply(l: FieldType[K, V] :: T): Map[String, Any] =
tmrT(l.tail) + (wit.value.name -> l.head)
}
}
object ToMapRec extends LowPriorityToMapRec {
implicit val hnilToMapRec: ToMapRec[HNil] = new ToMapRec[HNil] {
def apply(l: HNil): Map[String, Any] = Map.empty
}
implicit def hconsToMapRec0[K <: Symbol, V, R <: HList, T <: HList](implicit
wit: Witness.Aux[K],
gen: LabelledGeneric.Aux[V, R],
tmrH: ToMapRec[R],
tmrT: ToMapRec[T]
): ToMapRec[FieldType[K, V] :: T] = new ToMapRec[FieldType[K, V] :: T] {
def apply(l: FieldType[K, V] :: T): Map[String, Any] =
tmrT(l.tail) + (wit.value.name -> tmrH(gen.to(l.head)))
}
}
Lastly we provide some syntax for convenience:
implicit class ToMapRecOps[A](val a: A) extends AnyVal {
def toMapRec[L <: HList](implicit
gen: LabelledGeneric.Aux[A, L],
tmr: ToMapRec[L]
): Map[String, Any] = tmr(gen.to(a))
}
And then we can demonstrate that it works:
scala> p.toMapRec
res0: Map[String,Any] = Map(address -> Map(zip -> 10000, street -> Jefferson st), name -> Tom)
Note that this won't work for types where the nested case classes are in a list, tuple, etc., but you could extend it to those cases pretty straightforwardly.

I have a problem with an approach provided by Travis Brown.
Some of nesting case classes are not converted to Map https://scalafiddle.io/sf/cia2jTa/0.
The answer was found here.
To correct the solution just wrap ToMapRec[T] in implicit parameters to Lazy[ToMapRec[T]]. Corrected fiddle https://scalafiddle.io/sf/cia2jTa/1

Related

Shapeless generic ZipWith with static poly

I have some code:
case class Name(name: String) extends StaticAnnotation
case class Example(#Name("baz") foo: String, bar: Int)
object nameAnnotationZip2 extends Poly2 {
implicit val newName: Case.Aux[Symbol, Some[Name], String] =
at((_, name) => name.value.name)
implicit val existingName: Case.Aux[Symbol, None.type, String] =
at((field, _) => field.name)
}
def renameRecord[
A,
Record <: HList,
Fields <: HList,
Annotations <: HList,
ZippedWith <: HList
](a: A)(implicit
generic: LabelledGeneric.Aux[A, Record],
fields: Keys.Aux[Record, Fields],
annotations: Annotations.Aux[Name, A, Annotations],
zippedWith: ZipWith.Aux[
Fields,
Annotations,
nameAnnotationZip2.type,
ZippedWith
]
) = {
val record = generic.to(a)
println(fields())
println(annotations())
zippedWith(fields(), annotations())
}
renameRecord(Example("qix", 5))
which attempts to create a record from an instance with renamed keys based on an annotation. unfortunately it always gives me a
could not find implicit value for parameter zippedWith: shapeless.ops.hlist.ZipWith.Aux[Fields,Annotations,nameAnnotationZip2.type,ZippedWith]
I haven't added the ZipWithKeys bit yet, but I believe I need to work around this issue first.
I believe I'm simply using Poly totally wrong somehow, but the compiler message isn't helping me find my error.
edit/part two:
I can work around by using Zip instead of ZipWith, converting to Sized, simply mapping the seq without Poly, and converting back to HList. what's the runtime performance difference between doing that and the above?
The type of Fields isn't Symbol, it's Symbol ## "foo". Case's generic parameters aren't variant so nameAnnotationZip2 isn't defined for the elements of Fields.
Adding a generic parameter to the cases fixes the issue
object nameAnnotationZip2 extends Poly2 {
implicit def newName[K <: Symbol]: Case.Aux[K, Some[Name], String] =
at((_, name) => name.value.name)
implicit def existingName[K <: Symbol]: Case.Aux[K, None.type, String] =
at((field, _) => field.name)
}

How to compare types of different, generic HLists?

So I have a user defined input of keys, and the user is also expected to provide serializers for these keys. I am trying to statically ensure that the serializers provided work with the user provided types. Example user usage would be something like:
case class Keys(key1: String, key2: Int)
val keys = Keys("Foo", 2)
val keyDescriptor = (StringSerializer, IntSerializer)
toSerializedKeys(keys, keyDescriptor)
but I would like something like this to fail at compile time:
val keys = Keys("Foo", 2)
val keyDescriptor = (StringSerializer, StringSerializer)
toSerializedKeys(keys, keyDescriptor)
To abstract over arity and attempt to provide compile time type matching on the keys/serializers I'm trying to use shapeless. My current solution works fine given explicit HLists. But as soon as I try to apply this to generic HLists things fall apart.
Current solution:
Assume we have the following user provided serializers:
trait SerializerFn[T] {
def toBytes(t: T): ByteBuffer
}
object StringSerializerFn extends SerializerFn[String] {
override def toBytes(t: String): ByteBuffer = ???
}
object IntSerializerFn extends SerializerFn[Int] {
override def toBytes(t: Int): ByteBuffer = ???
}
I currently create a type class, helper function, and instances to handle serialization of HLists:
trait Serializer[HA <: HList, HB <: HList] {
def toBufs(keys: HA)(fns: HB): List[ByteBuffer]
}
object Serializer extends SerializerInstances {
def toBufs[HA <: HList, HB <: HList](keys: HA)(fns: HB)(
implicit
serializer: Serializer[HA, HB]
): List[ByteBuffer] = {
serializer.toBufs(keys)(fns)
}
}
trait SerializerInstances {
implicit val hnilSerializer: Serializer[HNil, HNil] =
new Serializer[HNil, HNil] {
override def toBufs(keys: HNil)(fns: HNil): List[ByteBuffer] = Nil
}
implicit def hconsSerializer[H, Fn <: SerializerFn[H], TA <: HList, TB <: HList](
implicit
hconsSerializer: Serializer[TA, TB],
): Serializer[H :: TA, Fn :: TB] =
new Serializer[H :: TA, Fn :: TB] {
override def toBufs(keys: H :: TA)(fns: Fn :: TB): List[ByteBuffer] =
List(fns.head.toBytes(keys.head)) ::: hconsSerializer.toBufs(keys.tail)(fns.tail)
}
}
The static type checking I'm going for here is happening in hconsSerializer where we restrict the head of both HLists to be an H and SerializerFn[H], respectively.
And to test it, this compiles fine:
val testKeys = "3" :: "4" :: HNil
val testFns = StringSerializerFn :: StringSerializerFn :: HNil
val testSerializedKeys = Serializer.toBufs(testKeys)(testFns)
and as expected, this doesn't compile:
val testKeys = "3" :: "4" :: HNil
val testFns = StringSerializerFn :: IntSerializerFn :: HNil
val testSerializedKeys = Serializer.toBufs(testKeys)(testFns)
However, when trying to generalize this (so that we can actually use this in a constructor that we provide to users), things fall apart, (I've omitted the Generic conversion from the user provided Product to HList for brevity):
def toSerializedKeys[ReprA <: HList, ReprB <: HList](
keys: ReprA,
fns: ReprB
): List[ByteBuffer] =
Serializer.toBufs(keys)(fns)
For reference, intellij complains that the implicit Serializer can't be found, and explicitly passing hconsSerializer shows:
Required: Serializer[ ReprA, ReprB]
Found: Serializer[Nothing :: HNil, Nothing :: HNil]
and Scalac says:
[error] found : [H, Fn <: com.twitter.beam.io.manhattan.SerializerFn[H], TA <: shapeless.HList, TB <: shapeless.HList]com.twitter.beam.io.manhattan.Serializer[H :: TA,Fn :: TB]
[error] required: com.twitter.beam.io.manhattan.Serializer[ReprA,ReprB]
What kind of implicit evidence do I need to be able to perform this kind of recursive type checking on generic HLists? Is accomplishing something like this even possible? I'm new to Shapeless.
Also, if comparing HList types like this isn't possible. Another solution could be something that allowed the user to define individual implicit key serializers, instead of a full KeyDescriptor that we convert to an HList, e.g something like:
implicit val intSerializer = IntSerializerFn
implicit val strSerizlizer = StringSerializerFn
However, I tried something like this and then tried mapping over the Keys HList with a Poly1 that takes an implicit SerializerFn, but I ran into similar issues when trying to work with a generic HLists.
If you are using Scala3 (aka dotty), here is a simple solution using stdlib only for your purpose:
import scala.deriving.Mirror.ProductOf as PF
import java.nio.ByteBuffer
trait SerializerFn[T]:
def toBytes(t: T): ByteBuffer
object StringSerializerFn extends SerializerFn[String]:
override def toBytes(t: String): ByteBuffer = ???
object IntSerializerFn extends SerializerFn[Int]:
override def toBytes(t: Int): ByteBuffer = ???
case class SI(key1: String, key2: Int)
case class SII(key1: String, key2: Int, key3: Int)
def toSerializedKeys[K <: Product, S <: Tuple](keys: K, sers: S)
(using pf: PF[K], ev: S <:< Tuple.Map[pf.MirroredElemTypes, SerializerFn]): Unit = {}
// success
toSerializedKeys(SI("a", 1), (StringSerializerFn, IntSerializerFn))
toSerializedKeys(SII("a", 1, 2), (StringSerializerFn, IntSerializerFn, IntSerializerFn))
// compile fail
// toSerializedKeys(SI("a", 1), (StringSerializerFn, StringSerializerFn))
// toSerializedKeys(SII("a", 1, 2), (IntSerializerFn, IntSerializerFn, IntSerializerFn))
I guess you just missed evidence serializer: Serializer[ReprA, ReprB] in
def toSerializedKeys[ReprA <: HList, ReprB <: HList](
keys: ReprA,
fns: ReprB
): List[ByteBuffer] =
Serializer.toBufs(keys)(fns)
So just replace this with
def toSerializedKeys[ReprA <: HList, ReprB <: HList](
keys: ReprA,
fns: ReprB
)(implicit serializer: Serializer[ReprA, ReprB]): List[ByteBuffer] =
Serializer.toBufs(keys)(fns)
Then
case class Keys(key1: String, key2: Int)
val keys = Keys("Foo", 2)
val keyDescriptor = (StringSerializer, IntSerializer)
toSerializedKeys(keys, keyDescriptor)
compiles (as expected) and
val keys = Keys("Foo", 2)
val keyDescriptor = (StringSerializer, StringSerializer)
toSerializedKeys(keys, keyDescriptor)
doesn't (as expected).
Just noticed that Alec advised the same in comments.

Shapeless: what the difference between these two approaches of instance derivation?

Can someone explain me what the difference between these two approaches for typeclass instance derivation (specifically for Option[A])?
1.
trait MyTrait[A] {...}
object MyTrait extends LowPriority {
// instances for primitives
}
trait LowPriority extends LowestPriority {
final implicit def generic[A, H <: HList](
implicit gen: Generic.Aux[A, H],
h: Lazy[MyTrait[H]]
): MyTrait[A] = ???
final implicit val hnil: MyTrait[HNil] = ???
final implicit def product[H, T <: HList](
implicit h: Lazy[MyTrait[H]],
t: Lazy[MyTrait[T]]
): MyTrait[H :: T] = ???
}
// special instances for Options
trait LowestPriority {
implicit def genericOption[A, Repr <: HList](
implicit gen: Generic.Aux[A, Repr],
hEncoder: Lazy[MyTrait[Option[Repr]]]
): MyTrait[Option[A]] = ???
implicit val hnilOption: MyTrait[Option[HNil]] = ???
implicit def productOption1[H, T <: HList](
implicit
head: Lazy[MyTrait[Option[H]]],
tail: Lazy[MyTrait[Option[T]]],
notOption: H <:!< Option[Z] forSome { type Z }
): MyTrait[Option[H :: T]] = ???
implicit def product2[H, T <: HList](
implicit
head: Lazy[MyTrait[Option[H]]],
tail: Lazy[MyTrait[Option[T]]
): MyTrait[Option[Option[H] :: T]] = ???
}
trait MyTrait[A] {...}
object MyTrait extends LowPriority {
// instances for primitives
}
trait LowPriority {
// deriving instances for options from existing non-option instances
final implicit def forOption[A](implicit instance: MyTrait[A]): MyTrait[Option[A]] = ??? // <<<----
final implicit def generic[A, H <: HList](
implicit gen: Generic.Aux[A, H],
h: Lazy[MyTrait[H]]
): MyTrait[A] = ???
final implicit val hnil: MyTrait[HNil] = ???
final implicit def product[H, T <: HList](
implicit h: Lazy[MyTrait[H]],
t: Lazy[MyTrait[T]]
): MyTrait[H :: T] = ???
}
I tried both and they worked correctly, but i'm not sure that they will produce the same results for all cases (maybe i've missed something).
Do we really need LowestPriority instances for this?
Am i right if i would say that the first approach gives us just a little bit more flexibility?
I assuming that by "worked correctly" you mean "compiled" or "worked for some simple use case".
Both of your examples deal with generic product types, but not with generic sum types, so there is no risk that e.g. Option[A] could get derived using Some[A] :+: None :+: CNil, which would enforce some ambiguity. So (as far as I can tell) you could write the second version like:
trait MyTrait[A] {...}
object MyTrait extends LowPriority {
// instances for primitives
// deriving instances for options from existing non-option instances
final implicit def forOption[A](implicit instance: MyTrait[A]): MyTrait[Option[A]] = ???
}
trait LowPriority {
// <<<----
final implicit def hcons[A, H <: HList](
implicit gen: Generic.Aux[A, H],
h: Lazy[MyTrait[H]]
): MyTrait[A] = ???
final implicit val hnil: MyTrait[HNil] = ???
final implicit def product[H, T <: HList](
implicit h: Lazy[MyTrait[H]],
t: Lazy[MyTrait[T]]
): MyTrait[H :: T] = ???
}
and it would derive things correctly.
But how 1. and 2. differs?
In second version you can derive MyTrait[Option[A]] if you can derive for A, and you can derive for any A which is primitive/option/product - so Option[Option[String]], Option[String] and Option[SomeCaseClass] should all work. It should also work if this SomeCaseClass contains fields which are Options, or other case classes which are Options, etc.
Version 1. is slightly different:
at first you are looking for primitives
then you try to derive for a product (so e.g. Option would not be handled here)
then you do something weird:
genericOption assumes that you created a Option[Repr], and then I guess map it using Repr
in order to build that Repr you take Option[HNil] and prepend types inside Option using productOption, which would break if someone used Option as a field
so you "fix" that by prepending an Option in a special case product2
I guess, you tested that only against case classes, because the first version would not work for:
Option for primitives (Option[String], Option[Int] or whatever you defined as primitive)
nested options (Option[Option[String]])
options for custom defined types which are not case classes but have manually defined instances:
class MyCustomType
object MyCustomType {
implicit val myTrait: MyTrait[MyCustomType]
}
implicitly[Option[MyCustomType]]
For that reason any solution with implicit def forOption[A](implicit instance: MyTrait[A]): MyTrait[Option[A]] is simpler and more bulletproof.
Depending on what you put directly into companion low-priority implicits might be or might not be needed:
if you defined coproducts then manual support for e.g. Option, List, Either could conflict with shapeless derived ones
if you manually implemented MyTrait implicit for some type in its companion object then it would have the same priority as implicits directly in MyTrait - so if it could be derived using shapeless you could have conflicts
For that reason it makes sense to put shapeless implicits in LowPriorityImplicits but primitives, and manual codecs for List, Option, Either, etc directly in companion. That is, unless you defined some e.g. Option[String] implicits directly in companion which could clash with "Option[A] with implicit for A".
Since I don't know your exact use case I cannot tell for sure, but I would probably go with the seconds approach, or most likely with the one I implemented in the snippet above.
Actually it's hard to say without right hand sides and actual implementations.
From information you provided it doesn't follow that the two type classes behave equivalently.
For example in the 1st approach you consider some special cases, so theoretically it's possible that you redefine some general behavior in special case differently.
By the way, Option[A] is a coproduct of Some[A] and None.type (List[A] is a coproduct of scala.::[A] and Nil.type) and sometimes it's easier to derive a type class for coproducts than for Option[A] (or List[A]).

Shapeless: How to express <: case class type param for Generix.Aux

I've got the following code to basically iterate through the fields of a case class and map them using a Poly to the same type and use ToList[HL, Out]
For simplicity we can assume the Poly does this:
object Schema extends Poly1 {
implicit def caseInt = at[Int](_ => "I'm an int")
implicit def caseString = at[String](_ => "Oh boy a string")
}
def infer[V1 <: Product, Out <: HList, MapperOut <: HList](v1: V1)(
implicit gen: Generic.Aux[V1, Out],
map: Mapper.Aux[Schema.type, Out, MapperOut],
to: ToList[MapperOut, String]
): List[String] = to (gen to v1 map Schema)
This is all very straightforward and works very well for simple scenarios:
case class Test(id: Int, text: String)
val list = infer(Test(2, "text"))
// List("I'm an int", "Oh boy a string")
Now going out to where the buses don't run:
class Automagical[T <: Product with Serializable : TypeTag] {
def instance: T
// The typetag bit is needed for something else
def convert: List[String] = infer(instance)
}
Sadly any call to above fails with:
could not find implicit value for parameter gen: shapeless.Generic.Aux[T,Out]
Bonus
How can I improve the infer method by not requiring an instance of T at all? Obviously type inference is fine, but I do need to somehow materialise a List[String] from an HList[Lub] and map over something.
Given I only ever care about the types, is it possible to derive a concrete instance of List[String] by only knowing the types to be poly mapped encoded as an HList?
Something like:
def infer[V1 <: Product, Out <: HList, MapperOut <: HList]()(
implicit gen: Generic.Aux[V1, Out],
map: Mapper.Aux[Schema.type, Out, MapperOut],
to: ToList[MapperOut, String]
): List[String] = {
// Magically build an HList with samples from its types.
// Or an alternative approach that gives the same outcome
val reifiedInstance = reify[Out]
to (reifiedInstance map Schema)
}
To ensure that convert has access to the implicit parameters for infer, they have to be present when the Automagical instance is created:
abstract class Automagical[T <: Product with Serializable : TypeTag,
Out <: HList, MapperOut <: HList]
(implicit
gen: Generic.Aux[T, Out],
map: Mapper.Aux[Schema.type, Out, MapperOut],
to: ToList[MapperOut, String]) {
def instance: T
// The typetag bit is needed for something else
def convert: List[String] = infer(instance)
}

Implicit Generic.Aux missing on conversion from Shapeless HList to case class

I just recently started learning scala and today I decided I wanted to write a CSV parser that would load nicely into case classes but store the data in rows (lists) of Shapeless's HList object so that I could get some exposure to type-level programming.
Here's what I have so far:
// LoadsCsv.scala
import shapeless._
import scala.collection.mutable
trait LoadsCsv[A, T <: HList] {
val rows: mutable.MutableList[T] = new mutable.MutableList[T]
def convert(t: T)(implicit gen: Generic.Aux[A, T]): A = gen.from(t)
def get(index: Int): A = {
convert(rows(index))
}
def load(file: String): Unit = {
val lines = io.Source.fromFile(file).getLines()
lines.foreach(line => rows += parse(line.split(",")))
}
def parse(line: Array[String]): T
}
And the object that's loading the data set:
// TennisData.scala
import shapeless._
case class TennisData(weather:String, low:Int, high:Int, windy:Boolean, play:Boolean)
object TennisData extends LoadsCsv[TennisData, String :: Int :: Int :: Boolean :: Boolean :: HNil] {
load("tennis.csv")
override def parse(line: Array[String]) = {
line(0) :: line(1).toInt :: line(2).toInt :: line(3).toBoolean :: line(4).toBoolean :: HNil
}
}
Things seem to be working alright until I added the get() with the conversion from the HList to the case class where I now get a compilation error. Why isn't the implicit getting loaded and what can I do to fix it or otherwise convert from the HList to the case class?
Error:(14, 17) could not find implicit value for parameter gen: shapeless.Generic.Aux[A,T]
return convert(rows(index))
^
I've been reading the shapeless documentation and it mentions that this area had been in flux between version 1 and 2, but I believe things should be working on my version of shapeless and scala so I suspect I've just done something incorrectly.
https://github.com/milessabin/shapeless/wiki/Migration-guide:-shapeless-1.2.4-to-2.0.0#iso-is-now-generic
For reference, I'm running scala 2.11.6 and shapeless 2.2.2
You're very close. The problem is that Scala isn't going to propagate implicit requirements up the call chain automatically for you. If you need a Generic[A, T] instance to call convert, then you'll have to make sure that one's in scope every time you call convert convert. If A and T are fixed (and are actually an case class-HList pair), Shapeless will generate one for you. In your get method, however, the compiler still knows nothing about A and T except that T is an HList, so you need to require the instance again in order to call convert:
def get(index: Int)(implicit gen: Generic.Aux[A, T]): A = convert(rows(index))
Everything should work just fine after this change.
Note that you could also require the instance at the trait level by adding an (abstract) method like the following:
implicit def genA: Generic.Aux[A, T]
Then any class implementing LoadsCsv could have an implicit val genA parameter (or could supply the instance in some other way).