I want to add keys as shown below:
val x = false
val y = Map[String,String](
"name" -> "AB",
if(x) "placement"->"pc" else null
)
println(y)
Note that its an immutable map, this works when x is true but throws runtime error when x is false, I tried to put else Unit but that doesn't work as well.
Can someone point me the most elegant as well as optimized syntax to do so?
One more option is
val y = Map[String, String](
Seq("name" -> "AB") ++
(if (x) Seq("placement" -> "pc") else Seq()): _*
)
or
val y = (Seq("name" -> "AB") ++
(if (x) Seq("placement" -> "pc") else Seq())).toMap
or
val y = (Seq("name" -> "AB") ++
Option.when(x)("placement" -> "pc").toSeq).toMap
This is not so shorter but maybe slightly less confusing than if (...) Map(...) else List(...).
Sorted it like this:
val x = true
val y = Map[String,String](
"name" -> "AB"
) ++ (if(x) Map[String,String]("place"->"pc") else Nil)
still wondering if this is good or something better should be used.
Related
Having this code
def mergeWith[K, X, Y, Z](xs: mutable.LinkedHashMap[K, X], ys: mutable.LinkedHashMap[K, Y])(f: (X, Y) => Z): mutable.LinkedHashMap[K, Z] =
xs.flatMap {
case (k, x) => ys.get(k).map(k -> f(x, _))
}
it gives me this:
val map1 = LinkedHashMap(4 -> (4), 7 -> (4,7))
val map2 = LinkedHashMap(3 -> (3), 6 -> (3,6), 7 -> (3,7))
val merged = mergeWith(map1,map2){ (x, y) => (x, y) }
merged: scala.collection.mutable.LinkedHashMap[Int,(Any, Any)] = Map(7 -> ((4,7),(3,7)))
But what i want is this:
merged: scala.collection.mutable.LinkedHashMap[Int,(Any, Any)] = Map(3 -> (3), 4 -> (4), 6 -> (3,6), 7 -> ((4,7),(3,7)))
How to modify my code to obtain it?
It can't be done with the current mergeWith() signature. In particular, you're trying to create a LinkedHashMap[K,Z] but there is no Z input. The only way to get a Z is to invoke f() which requires both X and Y as passed parameters.
So if xs is type LinkedHashMap[Int,Char] and has element (2 -> 'w'), and ys is type LinkedHashMap[Int,Long] and has element (8 -> 4L), how are you going to invoke f(c:Char, l:Long) so that you have a [K,Z] entry for both keys 2 and 8? Not possible.
If the mergeWith() signature can be simplified you might do something like this.
def mergeWith[K,V](xs: collection.mutable.LinkedHashMap[K, V]
,ys: collection.mutable.LinkedHashMap[K, V]
)(f: (V, V) => V): collection.mutable.LinkedHashMap[K,V] = {
val ns = collection.mutable.LinkedHashMap[K,V]()
(xs.keySet ++ ys.keySet).foreach{ k =>
if (!xs.isDefinedAt(k)) ns.update(k, ys(k))
else if (!ys.isDefinedAt(k)) ns.update(k, xs(k))
else ns.update(k, f(xs(k), ys(k)))
}
ns
}
This produces the desired result for the example you've given, but it has a number of undesirable qualities, not the least of which is the mutable data structures.
BTW, there is no such thing as a Tuple1 so (4) is the same thing as 4. And whenever you see type Any, it's a pretty good sign that your design needs a re-think.
I am new to spark-scala development. I am trying to create map values in spark using scala but getting nothing printed
def createMap() : Map[String, Int] = {
var tMap:Map[String, Int] = Map()
val tDF = spark.sql("select a, b, c from temp")
for (x <- tDF) {
val k = x.getAs[Long](0) + "|" + x.getAs[Long](1)
val v = x.getAs[Int](2)
tMap += ( k -> v )
println( k -> v ) ///----------This print values
}
println("Hellllooooooooo1")
for ((k,v) <- tMap) println("key = " + k+ ", value= " + v) ////------This prints nothing
println("Hellllooooooooo2")
return tMap
}
Please suggest.
user8598832 gives how to do it properly (for some value of properly). The reason your approach doesn't work is that you're adding (k, v) to the map in an executor, but the println occurs in the driver, which generally won't see the map(s) in the executor(s) (to the extent that it might, that's just an artifact of running it in local mode not in a distributed mode).
The "right" (if collecting to driver is ever right) way to do it:
import org.apache.spark.sql.functions._
tDF.select(concat_ws("|", col("a"), col("b")), col("c")).as[(String, Int)].rdd.collectAsMap
What is a simple approach to testing whether a Map[A,B] is bijective, namely for
val m1 = Map( 1 -> "a", 2 -> "b")
val m2 = Map( 1 -> "a", 2 -> "a")
we have that m1 is bijective unlike m2.
You could do
val m = Map(1 -> "a", 2 -> "a")
val isBijective = m.values.toSet.size == m.size
A bijection is one-to-one and onto. The mapping defined by a Map is definitely onto. Every value has a corresponding key.
The only way it's not one-to-one is if two keys map to the same value. But that means that we'll have fewer values than keys.
As explained here -
For a pairing between X and Y (where Y need not be different from X) to be a bijection, four properties must hold:
1) each element of X must be paired with at least one element of Y
This is inherent nature of Map. In some cases it can be mapped to None which is again one of the element of Y.
2) no element of X may be paired with more than one element of Y,
Again inherent neture of Map.
3) each element of Y must be paired with at least one element of X, and
Every element in Y will have some association with X otherwise it won't exist.
4) no element of Y may be paired with more than one element of X.
Map doesnt have this constraint. So we need to check for this. If Y contains duplicates then it is violated.
So, sufficient check should be "No duplicates in Y")
val a = scala.collection.mutable.Map[Int, Option[Int]]()
a(10) = None
a(20) = None
if(a.values.toArray.distinct.size != a.values.size) println("No") else println("Yes") // No
val a = Map("a"->10, "b"->20)
if(a.values.toArray.distinct.size != a.values.size) println("No") else println("Yes") // Yes
val a = scala.collection.mutable.Map[Int, Option[Int]]()
a(10) = Some(100)
a(20) = None
a(30) = Some(300)
if(a.values.toArray.distinct.size != a.values.size) println("No") else println("Yes") // Yes
Here is code I wrote to average coordinate values contained within the values of a Map :
val averaged = Map((2,10) -> List((2.0,11.0), (5.0,8.0)))
//> averaged : scala.collection.immutable.Map[(Int, Int),List[(Double, Double)
//| ]] = Map((2,10) -> List((2.0,11.0), (5.0,8.0)))
averaged.mapValues(m => {
val s1 = m.map(m => m._1).sum
val s2 = m.map(m => m._2).sum
(s1 / m.size , s2 / m.size)
}) //> res0: scala.collection.immutable.Map[(Int, Int),(Double, Double)] = Map((2,
//| 10) -> (3.5,9.5))
This code works as expected but the mapValues function requires number of passes equals to length of the List. Is there a more idiomatic method of achieving same using Scala ?
If I'm understanding your question correctly, you are asking if it is possible to avoid the traversal of m on each access. The mapValues method returns a view of a Map, meaning that there will be repeated work on access. To avoid that, just use map instead:
val averaged = Map((2, 10) -> List((2.0, 11.0), (5.0, 8.0)))
val result = averaged.map {
case (key, m) =>
val (s1, s2) = m.unzip
(s1.sum / m.size, s2.sum / m.size)
}
println(result)
// Map((2,10) -> (3.5,9.5))
Using unzip additionally means that the code won't traverse m more than once.
I have a Map like:
Map("product1" -> List(Product1ObjectTypes), "product2" -> List(Product2ObjectTypes))
where ProductObjectType has a field usage. Based on the other field (counter) I have to update all ProductXObjectTypes.
The issue is that this update depends on previous ProductObjectType, and I can't find a way to get previous item when iterating over mapValues of this map. So basically, to update current usage I need: CurrentProduct1ObjectType.counter - PreviousProduct1ObjectType.counter.
Is there any way to do this?
I started it like:
val reportsWithCalculatedUsage =
reportsRefined.flatten.flatten.toList.groupBy(_._2.product).mapValues(f)
but I don't know in mapValues how to access previous list item.
I'm not sure if I understand completely, but if you want to update the values inside the lists based on their predecessors, this can generally be done with a fold:
case class Thing(product: String, usage: Int, counter: Int)
val m = Map(
"product1" -> List(Thing("Fnord", 10, 3), Thing("Meep", 0, 5))
//... more mappings
)
//> Map(product1 -> List(Thing(Fnord,10,3), Thing(Meep,0,5)))
m mapValues { list => list.foldLeft(List[Thing]()){
case (Nil, head) =>
List(head)
case (tail, head) =>
val previous = tail.head
val current = head copy (usage = head.usage + head.counter - previous.counter)
current :: tail
} reverse }
//> Map(product1 -> List(Thing(Fnord,10,3), Thing(Meep,2,5)))
Note that regular map is an unordered collection, you need to use something like TreeMap to have predictable order of iteration.
Anyways, from what I understand you want to get pairs of all values in a map. Try something like this:
scala> val map = Map(1 -> 2, 2 -> 3, 3 -> 4)
scala> (map, map.tail).zipped.foreach((t1, t2) => println(t1 + " " + t2))
(1,2) (2,3)
(2,3) (3,4)