Dynamically create extensible record in shapeless 2.0 - scala

I need to produce an extensible record given an HList of keys and a map of values, here's a MWE of what I'm trying to achieve (you can copy/paste this in any REPL with shapeless 2.0 available, in order to reproduce the issue)
import shapeless._; import syntax.singleton._; import record._
case class Foo[T](column: Symbol)
val cols = Foo[String]('column1) :: HNil
val values = Map("column1" -> "value1")
object toRecord extends Poly1 {
implicit def Foo[T] = at[Foo[T]] { foo =>
val k = foo.column.name
val v = values.get(k)
(k ->> v)
}
}
val r = cols.map(toRecord)
// r: shapeless.::[Option[String] with shapeless.record.KeyTag[k.type,Option[String]] forSome { val k: String },shapeless.HNil] = Some(value1) :: HNil
val value = r("column1")
// error: No field String("column1") in record shapeless.::[Option[String] with shapeless.record.KeyTag[k.type,Option[String]] forSome { val k: String },shapeless.HNil]
val value = r("column1")
If I try defining the record manually everything works as expected
val q = ("column1" ->> Some("value1")) :: HNil
// q: shapeless.::[Some[String] with shapeless.record.KeyTag[String("column1"),Some[String]],shapeless.HNil] = Some(value1) :: HNil
q("column1")
// Some[String] = Some(value1)
Clearly the difference is that in one case the KeyTag has type
KeyTag[String("column1"), Some[String]]
and in the (non-working) other
KeyTag[k.type,Option[String]] forSome { val k: String }
I sense the issue is with the string k not being statically known, but I have no clue on how to fix this.
Generally speaking, is there a way of dynamically generating an extensible record from a list of keys?
I fear the answer is to use a macro, but I'd be glad if another solution existed.

This isn't too bad if you can change your Foo definition a bit to allow it to keep track of the singleton type of the column key (note that I've removed the unused T type parameter):
import shapeless._; import syntax.singleton._; import record._
case class Foo[K <: Symbol](column: Witness.Aux[K])
val cols = Foo('column1) :: HNil
val values = Map("column1" -> "value1")
object toRecord extends Poly1 {
implicit def atFoo[K <: Symbol] = at[Foo[K]] { foo =>
field[K](values.get(foo.column.value.name))
}
}
val r = cols.map(toRecord)
And then:
scala> val value = r('column1)
value: Option[String] = Some(value1)
Note that I've changed your string key ("column1") to a symbol, since that's what we've put into the record.

Related

Scala 3 quotes: get default case class field value [duplicate]

Is there a clean way to access the default values of a case class fields when performing type class derivation in Scala 3 using Mirrors? For example:
case class Foo(s: String = "bar", i: Int, d: Double = Math.PI)
Mirror.Product.MirroredElemLabels will be set to ("s", "i", "d"). Is there anything like: (Some["bar"], None, Some[3.141592653589793])?
If not could this be achieved using Macros? Can I use the Mirrors and Macros simultaneously to derive a type class instance?
You'll have to write a macro working with methods named like <init>$default$1, <init>$default$2, ... in companion object
import scala.quoted.*
inline def printDefaults[T]: Unit = ${printDefaultsImpl[T]}
def printDefaultsImpl[T](using Quotes, Type[T]): Expr[Unit] = {
import quotes.reflect.*
(1 to 3).map(i =>
TypeRepr.of[T].typeSymbol
.companionClass
.declaredMethod(s"$$lessinit$$greater$$default$$$i")
.headOption
.flatMap(_.tree.asInstanceOf[DefDef].rhs)
).foreach(println)
'{()}
}
printDefaults[Foo]
//Some(Literal(Constant(bar)))
//None
//Some(Select(Ident(Math),PI))
Mirrors and macros can work together:
import scala.quoted.*
import scala.deriving.*
trait Default[T] {
type Out <: Tuple
def defaults: Out
}
object Default {
transparent inline given mkDefault[T](using
m: Mirror.ProductOf[T],
s: ValueOf[Tuple.Size[m.MirroredElemTypes]]
): Default[T] =
new Default[T] {
type Out = Tuple.Map[m.MirroredElemTypes, Option]
def defaults = getDefaults[T](s.value).asInstanceOf[Out]
}
inline def getDefaults[T](inline s: Int): Tuple = ${getDefaultsImpl[T]('s)}
def getDefaultsImpl[T](s: Expr[Int])(using Quotes, Type[T]): Expr[Tuple] = {
import quotes.reflect.*
val n = s.asTerm.underlying.asInstanceOf[Literal].constant.value.asInstanceOf[Int]
val terms: List[Option[Term]] =
(1 to n).toList.map(i =>
TypeRepr.of[T].typeSymbol
.companionClass
.declaredMethod(s"$$lessinit$$greater$$default$$$i")
.headOption
.flatMap(_.tree.asInstanceOf[DefDef].rhs)
)
def exprOfOption[T](oet: Option[Expr[T]])(using Type[T], Quotes): Expr[Option[T]] = oet match {
case None => Expr(None)
case Some(et) => '{Some($et)}
}
val exprs: List[Option[Expr[Any]]] = terms.map(_.map(_.asExprOf[Any]))
val exprs1: List[Expr[Option[Any]]] = exprs.map(exprOfOption)
Expr.ofTupleFromSeq(exprs1)
}
}
Usage:
val d = summon[Default[Foo]]
summon[d.Out =:= (Option[String], Option[Int], Option[Double])] // compiles
d.defaults // (Some(bar),None,Some(3.141592653589793))
As Dmytro suggests, information is carried in methods <init>default$x of the class companion object.
However, Quotes discourages accessing a symbol's tree in a macro:
https://github.com/lampepfl/dotty/blob/main/library/src/scala/quoted/Quotes.scala#L3628.
Symbol's tree is lost, unless program is compiled with -Yretain-trees)
It is better to let the macro evaluate <init>default$x, rather than copy the right hand side of its definition.
One can do so by expressing terms as :
val terms: List[Option[Term]] =
(1 to n).toList.map(i =>
TypeRepr.of[T].typeSymbol
.companionClass
.declaredMethod(s"$$lessinit$$greater$$default$$$i")
.headOption
.map(Select(Ref(TypeRepr.of[T].typeSymbol.companionModule),_))
)

Type Class Derivation accessing default values

Is there a clean way to access the default values of a case class fields when performing type class derivation in Scala 3 using Mirrors? For example:
case class Foo(s: String = "bar", i: Int, d: Double = Math.PI)
Mirror.Product.MirroredElemLabels will be set to ("s", "i", "d"). Is there anything like: (Some["bar"], None, Some[3.141592653589793])?
If not could this be achieved using Macros? Can I use the Mirrors and Macros simultaneously to derive a type class instance?
You'll have to write a macro working with methods named like <init>$default$1, <init>$default$2, ... in companion object
import scala.quoted.*
inline def printDefaults[T]: Unit = ${printDefaultsImpl[T]}
def printDefaultsImpl[T](using Quotes, Type[T]): Expr[Unit] = {
import quotes.reflect.*
(1 to 3).map(i =>
TypeRepr.of[T].typeSymbol
.companionClass
.declaredMethod(s"$$lessinit$$greater$$default$$$i")
.headOption
.flatMap(_.tree.asInstanceOf[DefDef].rhs)
).foreach(println)
'{()}
}
printDefaults[Foo]
//Some(Literal(Constant(bar)))
//None
//Some(Select(Ident(Math),PI))
Mirrors and macros can work together:
import scala.quoted.*
import scala.deriving.*
trait Default[T] {
type Out <: Tuple
def defaults: Out
}
object Default {
transparent inline given mkDefault[T](using
m: Mirror.ProductOf[T],
s: ValueOf[Tuple.Size[m.MirroredElemTypes]]
): Default[T] =
new Default[T] {
type Out = Tuple.Map[m.MirroredElemTypes, Option]
def defaults = getDefaults[T](s.value).asInstanceOf[Out]
}
inline def getDefaults[T](inline s: Int): Tuple = ${getDefaultsImpl[T]('s)}
def getDefaultsImpl[T](s: Expr[Int])(using Quotes, Type[T]): Expr[Tuple] = {
import quotes.reflect.*
val n = s.asTerm.underlying.asInstanceOf[Literal].constant.value.asInstanceOf[Int]
val terms: List[Option[Term]] =
(1 to n).toList.map(i =>
TypeRepr.of[T].typeSymbol
.companionClass
.declaredMethod(s"$$lessinit$$greater$$default$$$i")
.headOption
.flatMap(_.tree.asInstanceOf[DefDef].rhs)
)
def exprOfOption[T](oet: Option[Expr[T]])(using Type[T], Quotes): Expr[Option[T]] = oet match {
case None => Expr(None)
case Some(et) => '{Some($et)}
}
val exprs: List[Option[Expr[Any]]] = terms.map(_.map(_.asExprOf[Any]))
val exprs1: List[Expr[Option[Any]]] = exprs.map(exprOfOption)
Expr.ofTupleFromSeq(exprs1)
}
}
Usage:
val d = summon[Default[Foo]]
summon[d.Out =:= (Option[String], Option[Int], Option[Double])] // compiles
d.defaults // (Some(bar),None,Some(3.141592653589793))
As Dmytro suggests, information is carried in methods <init>default$x of the class companion object.
However, Quotes discourages accessing a symbol's tree in a macro:
https://github.com/lampepfl/dotty/blob/main/library/src/scala/quoted/Quotes.scala#L3628.
Symbol's tree is lost, unless program is compiled with -Yretain-trees)
It is better to let the macro evaluate <init>default$x, rather than copy the right hand side of its definition.
One can do so by expressing terms as :
val terms: List[Option[Term]] =
(1 to n).toList.map(i =>
TypeRepr.of[T].typeSymbol
.companionClass
.declaredMethod(s"$$lessinit$$greater$$default$$$i")
.headOption
.map(Select(Ref(TypeRepr.of[T].typeSymbol.companionModule),_))
)

Given a HList how do you get a HMap

I have a HList of Symbols. I want to convert this into HMap with the HList values as the keys and the value mapping set to a generated object based on the key. The generated object holds state hence though default mapping can be known subsequent state is not.
Well... the thing is that Shapeless HMap is not as straight forward as HList, but the following can get you started
import shapeless._
import poly._
val hList = 'omg1 :: 'omg2 :: 'omg3 :: HNil
// lets assume that you want to map Symbols to kv-pairs String -> Int
// hence your HMap will have kv-pairs "omg1" -> 1, "omg2" -> 2...
class BiMapIS[K, V]
implicit val stringToInt = new BiMapIS[String, Int]
object myFoldPolyFunc extends Poly {
implicit def caseSymbol = use(
(hmap: HMap[BiMapIS], elem: Symbol) =>
hmap + (elem.name -> elem.name.last.toString.toInt)
)
}
val hMap = hList.foldLeft(HMap.empty[BiMapIS])(myFoldPolyFunc)
val v1 = hMap.get("omg1")
// Some(1)
val v2 = hMap.get("omg2")
// Some(2)
Without further information and based on a very superficial knowledge about HList (my only perception is a List of Tuple2), I guess you could do that via iteration. A possible pseudo code would be:
// Presume tuple._1 is the key
val hmap = hlist.map{ tuple => (tuple._1, doSomething(tuple._1))}.toMap

Different types in Map Scala

I need a Map where I put different types of values (Double, String, Int,...) in it, key can be String.
Is there a way to do this, so that I get the correct type with map.apply(k) like
val map: Map[String, SomeType] = Map()
val d: Double = map.apply("double")
val str: String = map.apply("string")
I already tried it with a generic type
class Container[T](element: T) {
def get: T = element
}
val d: Container[Double] = new Container(4.0)
val str: Container[String] = new Container("string")
val m: Map[String, Container] = Map("double" -> d, "string" -> str)
but it's not possible since Container takes an parameter. Is there any solution to this?
This is not straightforward.
The type of the value depends on the key. So the key has to carry the information about what type its value is. This is a common pattern. It is used for example in SBT (see for example SettingsKey[T]) and Shapeless Records (Example). However, in SBT the keys are a huge, complex class hierarchy of its own, and the HList in shapeless is pretty complex and also does more than you want.
So here is a small example of how you could implement this. The key knows the type, and the only way to create a Record or to get a value out of a Record is the key. We use a Map[Key, Any] internally as storage, but the casts are hidden and guaranteed to succeed. There is an operator to create records from keys, and an operator to merge records. I chose the operators so you can concatenate Records without having to use brackets.
sealed trait Record {
def apply[T](key:Key[T]) : T
def get[T](key:Key[T]) : Option[T]
def ++ (that:Record) : Record
}
private class RecordImpl(private val inner:Map[Key[_], Any]) extends Record {
def apply[T](key:Key[T]) : T = inner.apply(key).asInstanceOf[T]
def get[T](key:Key[T]) : Option[T] = inner.get(key).asInstanceOf[Option[T]]
def ++ (that:Record) = that match {
case that:RecordImpl => new RecordImpl(this.inner ++ that.inner)
}
}
final class Key[T] {
def ~>(value:T) : Record = new RecordImpl(Map(this -> value))
}
object Key {
def apply[T] = new Key[T]
}
Here is how you would use this. First define some keys:
val a = Key[Int]
val b = Key[String]
val c = Key[Float]
Then use them to create a record
val record = a ~> 1 ++ b ~> "abc" ++ c ~> 1.0f
When accessing the record using the keys, you will get a value of the right type back
scala> record(a)
res0: Int = 1
scala> record(b)
res1: String = abc
scala> record(c)
res2: Float = 1.0
I find this sort of data structure very useful. Sometimes you need more flexibility than a case class provides, but you don't want to resort to something completely type-unsafe like a Map[String,Any]. This is a good middle ground.
Edit: another option would be to have a map that uses a (name, type) pair as the real key internally. You have to provide both the name and the type when getting a value. If you choose the wrong type there is no entry. However this has a big potential for errors, like when you put in a byte and try to get out an int. So I think this is not a good idea.
import reflect.runtime.universe.TypeTag
class TypedMap[K](val inner:Map[(K, TypeTag[_]), Any]) extends AnyVal {
def updated[V](key:K, value:V)(implicit tag:TypeTag[V]) = new TypedMap[K](inner + ((key, tag) -> value))
def apply[V](key:K)(implicit tag:TypeTag[V]) = inner.apply((key, tag)).asInstanceOf[V]
def get[V](key:K)(implicit tag:TypeTag[V]) = inner.get((key, tag)).asInstanceOf[Option[V]]
}
object TypedMap {
def empty[K] = new TypedMap[K](Map.empty)
}
Usage:
scala> val x = TypedMap.empty[String].updated("a", 1).updated("b", "a string")
x: TypedMap[String] = TypedMap#30e1a76d
scala> x.apply[Int]("a")
res0: Int = 1
scala> x.apply[String]("b")
res1: String = a string
// this is what happens when you try to get something out with the wrong type.
scala> x.apply[Int]("b")
java.util.NoSuchElementException: key not found: (b,Int)
This is now very straightforward in shapeless,
scala> import shapeless._ ; import syntax.singleton._ ; import record._
import shapeless._
import syntax.singleton._
import record._
scala> val map = ("double" ->> 4.0) :: ("string" ->> "foo") :: HNil
map: ... <complex type elided> ... = 4.0 :: foo :: HNil
scala> map("double")
res0: Double with shapeless.record.KeyTag[String("double")] = 4.0
scala> map("string")
res1: String with shapeless.record.KeyTag[String("string")] = foo
scala> map("double")+1.0
res2: Double = 5.0
scala> val map2 = map.updateWith("double")(_+1.0)
map2: ... <complex type elided> ... = 5.0 :: foo :: HNil
scala> map2("double")
res3: Double = 5.0
This is with shapeless 2.0.0-SNAPSHOT as of the date of this answer.
I finally found my own solution, which worked best in my case:
case class Container[+T](element: T) {
def get[T]: T = {
element.asInstanceOf[T]
}
}
val map: Map[String, Container[Any]] = Map("a" -> Container[Double](4.0), "b" -> Container[String]("test"))
val double: Double = map.apply("a").get[Double]
val string: String = map.apply("b").get[String]
(a) Scala containers don't track type information for what's placed inside them, and
(b) the return "type" for an apply/get method with a simple String parameter/key is going to be static for a given instance of the object the method is to be applied to.
This feels very much like a design decision that needs to be rethought.
I don't think there's a way to get bare map.apply() to do what you'd want. As the other answers suggest, some sort of container class will be necessary. Here's an example that restricts the values to be only certain types (String, Double, Int, in this case):
sealed trait MapVal
case class StringMapVal(value: String) extends MapVal
case class DoubleMapVal(value: Double) extends MapVal
case class IntMapVal(value: Int) extends MapVal
val myMap: Map[String, MapVal] =
Map("key1" -> StringMapVal("value1"),
"key2" -> DoubleMapVal(3.14),
"key3" -> IntMapVal(42))
myMap.keys.foreach { k =>
val message =
myMap(k) match { // map.apply() in your example code
case StringMapVal(x) => "string: %s".format(x)
case DoubleMapVal(x) => "double: %.2f".format(x)
case IntMapVal(x) => "int: %d".format(x)
}
println(message)
}
The main benefit of the sealted trait is compile-time checking for non-exhaustive matches in pattern matching.
I also like this approach because it's relatively simple by Scala standards. You can go off into the weeds for something more robust, but in my opinion you're into diminishing returns pretty quickly.
If you want to do this you'd have to specify the type of Container to be Any, because Any is a supertype of both Double and String.
val d: Container[Any] = new Container(4.0)
val str: Container[Any] = new Container("string")
val m: Map[String, Container[Any]] = Map("double" -> d, "string" -> str)
Or to make things easier, you can change the definition of Container so that it's no longer type invariant:
class Container[+T](element: T) {
def get: T = element
override def toString = s"Container($element)"
}
val d: Container[Double] = new Container(4.0)
val str: Container[String] = new Container("string")
val m: Map[String, Container[Any]] = Map("double" -> d, "string" -> str)
There is a way but it's complicated. See Unboxed union types in Scala. Essentially you'll have to type the Map to some type Int |v| Double to be able to hold both Int and Double. You'll also pay a high price in compile times.

Is there such a thing as bidirectional maps in Scala?

I'd like to link 2 columns of unique identifiers and be able to get a first column value by a second column value as well as a second column value by a first column value. Something like
Map(1 <-> "one", 2 <-> "two", 3 <-> "three")
Is there such a facility in Scala?
Actually I need even more: 3 columns to select any in a triplet by another in a triplet (individual values will never be met more than once in the entire map). But a 2-column bidirectional map can help too.
Guava has a bimap that you can use along with
import scala.collection.JavaConversions._
My BiMap approach:
object BiMap {
private[BiMap] trait MethodDistinctor
implicit object MethodDistinctor extends MethodDistinctor
}
case class BiMap[X, Y](map: Map[X, Y]) {
def this(tuples: (X,Y)*) = this(tuples.toMap)
private val reverseMap = map map (_.swap)
require(map.size == reverseMap.size, "no 1 to 1 relation")
def apply(x: X): Y = map(x)
def apply(y: Y)(implicit d: BiMap.MethodDistinctor): X = reverseMap(y)
val domain = map.keys
val codomain = reverseMap.keys
}
val biMap = new BiMap(1 -> "A", 2 -> "B")
println(biMap(1)) // A
println(biMap("B")) // 2
Of course one can add syntax for <-> instead of ->.
Here's a quick Scala wrapper for Guava's BiMap.
import com.google.common.{collect => guava}
import scala.collection.JavaConversions._
import scala.collection.mutable
import scala.languageFeature.implicitConversions
class MutableBiMap[A, B] private (
private val g: guava.BiMap[A, B] = new guava.HashBiMap[A, B]()) {
def inverse: MutableBiMap[B, A] = new MutableBiMap[B, A](g.inverse)
}
object MutableBiMap {
def empty[A, B]: MutableBiMap[A, B] = new MutableBiMap()
implicit def toMap[A, B] (x: MutableBiMap[A, B]): mutable.Map[A,B] = x.g
}
I have a really simple BiMap in Scala:
case class BiMap[A, B](elems: (A, B)*) {
def groupBy[X, Y](pairs: Seq[(X, Y)]) = pairs groupBy {_._1} mapValues {_ map {_._2} toSet}
val (left, right) = (groupBy(elems), groupBy(elems map {_.swap}))
def apply(key: A) = left(key)
def apply[C: ClassTag](key: B) = right(key)
}
Usage:
val biMap = BiMap(1 -> "x", 2 -> "y", 3 -> "x", 1 -> "y")
assert(biMap(1) == Set("x", "y"))
assert(biMap("x") == Set(1, 3))
I don't think it exists out of the box, because the generic behavior is not easy to extract
How to handle values matching several keys in a clean api?
However for specific cases here is a good exercise that might help. It must be updated because no hash is used and getting a key or value is O(n).
But the idea is to let you write something similar to what you propose, but using Seq instead of Map...
With the help of implicit and trait, plus find, you could emulate what you need with a kind of clean api (fromKey, fromValue).
The specificities is that a value is not supposed to appear in several places... In this implementation at least.
trait BiMapEntry[K, V] {
def key:K
def value:V
}
trait Sem[K] {
def k:K
def <->[V](v:V):BiMapEntry[K, V] = new BiMapEntry[K, V]() { val key = k; val value = v}
}
trait BiMap[K, V] {
def fromKey(k:K):Option[V]
def fromValue(v:V):Option[K]
}
object BiMap {
implicit def fromInt(i:Int):Sem[Int] = new Sem[Int] {
def k = i
}
implicit def fromSeq[K, V](s:Seq[BiMapEntry[K, V]]) = new BiMap[K, V] {
def fromKey(k:K):Option[V] = s.find(_.key == k).map(_.value)
def fromValue(v:V):Option[K] = s.find(_.value == v).map(_.key)
}
}
object test extends App {
import BiMap._
val a = 1 <-> "a"
val s = Seq(1 <-> "a", 2 <-> "b")
println(s.fromKey(2))
println(s.fromValue("a"))
}
Scala is immutable and values are assigned as reference not copy, so memory footprint will for reference/pointer storage only, which it's better to use to two maps, with type A being key for first and type being B being key for second mapped to B and A respectively, than tun time swapping of maps. And the swapping implementation also has it's own memory footprint and the newly swapped hash-map will also be there in memory till the execution of parent call back and the garbage collector call. And if the the swapping of map is required frequently than virtually your are using equally or more memory than the naive two maps implementation at starting.
One more approach you can try with single map is this(will work only for getting key using mapped value):
def getKeyByValue[A,B](map: Map[A,B], value: B):Option[A] = hashMap.find((a:A,b:B) => b == value)
Code for Scala implementation of find by key:
/** Find entry with given key in table, null if not found.
*/
#deprecatedOverriding("No sensible way to override findEntry as private findEntry0 is used in multiple places internally.", "2.11.0")
protected def findEntry(key: A): Entry =
findEntry0(key, index(elemHashCode(key)))
private[this] def findEntry0(key: A, h: Int): Entry = {
var e = table(h).asInstanceOf[Entry]
while (e != null && !elemEquals(e.key, key)) e = e.next
e
}