Hi, I've been trying to unify collection of nested maps.
So I want to implement a method with signature:
def unifyMaps(seq: Seq[Map[String, Map[WordType, Int]]]): Map[String, Map[WordType, Int]]
(WordType is a Java Enum.) The first approach was to do manual map-merging.
def unifyMapsManually(seq: IndexedSeq[Map[String, Map[WordType, Int]]]): Map[String, Map[WordType, Int]] = {
seq reduce { (acc, newMap) =>
acc ++ newMap.map { case (k, v) =>
val nestedMap = acc.getOrElse(k, Map.empty)
k -> (nestedMap ++ v.map { case (k2, v2) => k2 -> (nestedMap.getOrElse(k2, 0) + v2) })
}
}
}
It works, but what I'm doing here is recursively applying the exact same pattern, so I thought I'd make a recursive-generic version.
Second approach:
def unifyTwoMapsRecursively(m1: Map[String, Map[WordType, Int]], m2: Map[String, Map[WordType, Int]]): Map[String, Map[WordType, Int]] = {
def unifyTwoMaps[K, V](nestedMapOps: (V, (V, V) => V))(m1: Map[K, V], m2: Map[K, V]): Map[K, V] = {
nestedMapOps match {
case (zero, add) =>
m1 ++ m2.map { case (k, v) => k -> add(m1.getOrElse(k, zero), v) }
}
}
val intOps = (0, (a: Int, b: Int) => a + b)
val mapOps = (Map.empty[WordType, Int], unifyTwoMaps(intOps) _)
unifyTwoMaps(mapOps)(m1, m2)
}
But it fails with:
Error:(90, 18) type mismatch;
found : (scala.collection.immutable.Map[pjn.wierzba.DictionaryCLP.WordType,Int], (Map[Nothing,Int], Map[Nothing,Int]) => Map[Nothing,Int])
required: (scala.collection.immutable.Map[_ <: pjn.wierzba.DictionaryCLP.WordType, Int], (scala.collection.immutable.Map[_ <: pjn.wierzba.DictionaryCLP.WordType, Int], scala.collection.immutable.Map[_ <: pjn.wierzba.DictionaryCLP.WordType, Int]) => scala.collection.immutable.Map[_ <: pjn.wierzba.DictionaryCLP.WordType, Int])
unifyTwoMaps(mapOps)(m1, m2)
^
So ok, I have no idea about upper bound on map key, but the curried function clearly is not inferred correctly. I had similar error with intOps, so I tried to provide exact types:
val mapOps = (Map.empty[WordType, Int], unifyTwoMaps(intOps)(_: Map[String, Map[WordType, Int]], _: Map[String, Map[WordType, Int]]))
But this time it fails with:
Error:(89, 67) type mismatch;
found : Map[String,Map[pjn.wierzba.DictionaryCLP.WordType,Int]]
required: Map[?,Int]
val mapOps = (Map.empty[WordType, Int], unifyTwoMaps(intOps)(_: Map[String, Map[WordType, Int]], _: Map[String, Map[WordType, Int]]))
^
And this time I have absolutely no idea what to try next to get it working.
EDIT: I've found solution to my problem, but I'm still wondering why do I get type mismatch error in this code snippet:
val mapOps = (Map.empty[WordType, Int], unifyTwoMaps(intOps) _)
According to this answer scala type inference works per parameter's list - this is exactly what I've been doing here for currying purposes. My unifyTwoMaps function takes two parameters' lists and I'm trying to infer just the second one.
Solution to generic-recursive solution
Ok, so after spending morning on it I've finally understood that I've been providing wrong exact types.
val mapOps = (Map.empty[WordType, Int], unifyTwoMaps(intOps)(_: Map[String, Map[WordType, Int]], _: Map[String, Map[WordType, Int]]))
Should've been
val mapOps = (Map.empty[WordType, Int], unifyTwoMaps(intOps)(_: Map[WordType, Int], _: Map[WordType, Int]))
Because I needed to pass the type of Map's V, Map[WordType, Int], and not the type of whole outer map. And now it works!
Solution to underlying problem of nested map merging
Well, abstracting over maps' V zero and add should ring a bell, I've been reinventing Monoid. So I thought I'd try Scalaz |+| Semigroups operator solution from this answer.
import scalaz.Scalaz._
def unifyMapsWithScalaz(seq: Seq[Map[String, Map[WordType, Int]]]): Map[String, Map[WordType, Int]] = {
seq reduce (_ |+| _)
}
And it works!
What's interesting is that I already saw that post before trying my solution, but I thought that I'm not sure it'd work for nested data structure, especially with my map's keys being Java Enum. I thought I'd have to provide some custom implementation extending Semigroups's typeclass.
But as it turned out during my reinventing-the-wheel implementation, the enum is only needed as a passed type and map key and it works pretty well. Well done Scalaz!
Well, that would've made a good blog post actually..
EDIT: but I still don't understand why I had this type inference problem in the first place, I've updated the question.
Related
I am struggling to obtained the types of the arguments of a defined function in Scala. For example Funcion1[T1, T2].
Since Java will eliminate the types checking (compiler warning: is unchecked since it is eliminated by erasure), I would like to find a way to match that function with their types.
The goal is to be able to have same functionality as:
val fnInput = {x: Map[String, Double] => x}
fnInput match {
case f: Function1[Map[String, Double], Map[String, Double]] => ???
case f: Function1[T1, T2] => ???
case f: Function2[T1, T2, T3] => ???
}
But, checking the arguments types.
Updated: so far my solution will go into using the following tools
import scala.reflect.runtime.universe._
def getType[T: TypeTag](obj: T) = typeOf[T]
val t = getType({x: Map[String, Any] => x})
// check first argument
typeOf[Map[String, Int]] <:< t.typeArgs(0)
// check return of Function1
typeOf[Map[String, Int]] <:< t.typeArgs(1)
// t.typeArgs.length will return the number of arguments +1
Do you believe that this is a good approach?
I am sorry for such non-descriptive title, but I really don't know how to express this better.
class Foo[T]
Seq(new Foo[String], new Foo[Int]).groupBy(_ => 1).map { case (k, Seq(v)) => k -> v }.toMap
<console>:12: error: Cannot prove that (Int, Foo[_146]) forSome { type _146 >: Int with String } <:< (T, U).
WTF?
If I use .mapValues instead of .map, it works. Also, making Foo covariant fixes it too, but in that case I end up with Map[Int,Foo[Any]]
What's going on here? Any ideas?
Without variance, you create a somewhat "nonsensical" sequence:
class Foo[T]
val in = Seq(new Foo[String], new Foo[Int]) // Seq[_ >: Int with String]]
There is simply no common LUB between Foo[String] and Foo[Int]. You may assign it an existential type:
val in = Seq[Foo[_]](new Foo[String], new Foo[Int])
Then we can try to continue:
val in = Seq[Foo[_]](new Foo[String], new Foo[Int])
val g = in.groupBy(_ => 1) // Map[Int, Seq[Foo[_]]]
// the next line would produce a match error, thus make it a `def`
def m = g.map { case (k, Seq(v)) => k -> v } // Iterable[(Int, Foo[_])]
def p = m.toMap // cannot prove that (Int, Foo[_]) <:< (T, U)
Again the existential type bites you here in disallowing a useful inference for the value type. You can enforce it again:
def p = m.toMap[Int, Foo[_]] // Map[Int,Foo[_]]
AFAIK, Scalac will not infer existential types for you.
If you are thinking you have Foo[Any] here, you need to add a variance annotation:
class Foo[+T]
val in = Seq(new Foo[String], new Foo[Int]) // Seq[Foo[Any]]
def m = in.groupBy(_=>1).map {case (k,Seq(v)) => k->v}.toMap // Map[Int,Foo[Any]]
Lets say I want to store a Map[String, Function1] where the parameter and return type of the Function1 can vary. How would I go about storing a Function1[String, String] and Function1[Int, Int] in the same Map.
I've tried Function1[AnyRef, AnyRef] but Function1[String, String] isn't a Function1[AnyRef, AnyRef] so it fails to compile.
If you have only two possible value types, you can wrap values in Either:
val m = Map[String, Either[Int => Int, String => String]]()
If you want to store more than two different types, create your own wrapper, or use something like Coproduct from shapeless.
Does this look like what you're after?
scala> val m = Map[String, Function1[_,_]]()
m: scala.collection.immutable.Map[String,Function1[_, _]] = Map()
scala> val f1 = (i:Int) => i*3
f1: Int => Int = <function1>
scala> val f2 = (b: Boolean) => if (b) "YES" else "NO"
f2: Boolean => String = <function1>
scala> m + ("xcx" -> f2) + ("rtr" -> f1)
res59: scala.collection.immutable.Map[String,Function1[_, _]] = Map(xcx -> <function1>, rtr -> <function1>)
I've recently come across a problem. I'm trying to flatten "tail-nested" tuples in a compiler-friendly way, and I've come up with the code below:
implicit def FS[T](x: T): List[T] = List(x)
implicit def flatten[T,V](x: (T,V))(implicit ft: T=>List[T], fv: V=>List[T]) =
ft(x._1) ++ fv(x._2)
This above code works well for flattening tuples I am calling "tail-nested" like the ones below.
flatten((1,2)) -> List(1,2)
flatten((1,(2,3))) -> List(1,2,3)
flatten((1,(2,(3,4)))) -> List(1,2,3,4)
However, I seek to make my solution more robust. Consider a case where I have a list of these higher-kinded "tail-nested" tuples.
val l = List( (1,2), (1,(2,3)), (1,(2,(3,4))) )
The inferred type signature of this would be List[(Int, Any)] and this poses a problem for an operation such as map, which would fail with:
error: No implicit view available from Any => List[Int]
This error makes sense to me because of the nature of my recursive implicit chain in the flatten function. However, I was wondering: is there any way I can make my method of flattening the tuples more robust so that higher order functions such as map mesh well with it?
EDIT:
As Bask.ws pointed out, the Product trait offers potential for a nice solution. The below code illustrates this:
def flatten(p: Product): List[_] = p.productIterator.toList.flatMap {x => x match {
case pr: Product => flatten(pr)
case _ => List(x)
}}
The result type of this new flatten call is always List[Any]. My problem would be solved if there was a way to have the compiler tighten this bound a bit. In parallel to my original question, does anyone know if it is possible to accomplish this?
UPD Compile-time fail solution added
I have one solution that may suit you. Types of your first 3 examples are resolved in compile time: Int, Tuple2[Int, Int], Tuple2[Int, Tuple2[Int, Int]]. For you example with the list you have heterogeneous list with actual type List[(Int, Any)] and you have to resolve the second type in runtime or it maybe can be done by macro. So you may want to actually write implicit def flatten[T](x: (T,Any)) as your error advises you
Here is the fast solution. It gives a couple of warnings, but it works nicely:
implicit def FS[T](x: T): List[T] = List(x)
implicit def FP[T](x: Product): List[T] = {
val res = (0 until x.productArity).map(i => x.productElement(i) match {
case p: Product => FP[T](p)
case e: T => FS(e)
case _ => sys.error("incorrect element")
})
res.toList.flatten
}
implicit def flatten[T](x: (T,Any))(implicit ft: T=>List[T], fp: Product =>List[T]) =
ft(x._1) ++ (x._2 match {
case p: Product => fp(p)
case t: T => ft(t)
})
val l = List( (1,2), (1,(2,3)), (1,(2,(3,4))) )
scala> l.map(_.flatten)
res0: List[List[Int]] = List(List(1, 2), List(1, 2, 3), List(1, 2, 3, 4))
UPD
I have researched problem a little bit more, and I have found simple solution to make homogeneus list, which can fail at compile time. It is fully typed without Any and match and looks like compiler now correctly resolves nested implicits
case class InfiniteTuple[T](head: T, tail: Option[InfiniteTuple[T]] = None) {
def flatten: List[T] = head +: tail.map(_.flatten).getOrElse(Nil)
}
implicit def toInfiniteTuple[T](x: T): InfiniteTuple[T] = InfiniteTuple(x)
implicit def toInfiniteTuple2[T, V](x: (T, V))(implicit ft: V => InfiniteTuple[T]): InfiniteTuple[T] =
InfiniteTuple(x._1, Some(ft(x._2)))
def l: List[InfiniteTuple[Int]] = List( (1,2), (1,(2,3)), (1,(2,(3,4)))) //OK
def c: List[InfiniteTuple[Int]] = List( (1,2), (1,(2,3)), (1,(2,(3,"44"))))
//Compile-time error
//<console>:11: error: No implicit view available from (Int, (Int, java.lang.String)) => InfiniteTuple[Int]
Then you can implement any flatten you want. For example, one above:
scala> l.map(_.flatten)
res0: List[List[Int]] = List(List(1, 2), List(1, 2, 3), List(1, 2, 3, 4))
Suppose I have a map m: Map[Any, Int]. Now I would like to take only entries (String, Int) from m and create a new map m1: Map[String, Int] with those entries.
I am trying to do the following:
val m1: Map[String, Int] = m collect {case e:(String, Int) => e}
It seems working but I get a warning: non variable type-argument String in type pattern (String, Int) is unchecked since it is eliminated by erasure.
How can I get rid of the warning?
you probably want:
val m1: Map[String, Int] = m collect {case (k:String, v:Int) => k->v}
(Just for reference. What you want is virtualeyes’s answer.)
val m1: Map[String, Int] = m flatMap { e =>
e._1 match {
case e1: String => Some(e1 -> e._2)
case _ => None
}
}
Careful testing will show that your solution actually matches everything in the map, not just entries of type (String,Int). The warning from the compiler is telling you that the types of your match will be thrown away at runtime so your code is actually doing something like this:
val m1: Map[String, Int] = m collect {case e:Tuple2[Any,Any] => e.asInstanceOf[Tuple2[String,Int]]}
And the asInstanceOf call will not blow up since it only casts to a Tuple2 and the (String,Int) bit gets lost again because of erasure. You'll get a nasty failure when you try to iterate against the result though ...
Try this in the REPL
val m:Map[String,Int] = Map("2" -> 3, 3 -> 4, "6" -> 10) collect {case e:(String, Int) => e}
// Oops, thought we would get '2'
m size
// Nothing wrong with this code except m contains some
// hidden nasties which causes us to blow up
for ((s:String, i:Int) <- m) yield s.length + i
`