I have Some() type Map[String, String], such as
Array[Option[Any]] = Array(Some(Map(String, String)
I want to return it as
Array(Map(String, String))
I've tried few different ways of extracting it-
Let's say if
val x = Array(Some(Map(String, String)
val x1 = for (i <- 0 until x.length) yield { x.apply(i) }
but this returns IndexedSeq(Some(Map)), which is not what I want.
I tried pattern matching,
x.foreach { i =>
i match {
case Some(value) => value
case _ => println("nothing") }}
another thing I tried that was somewhat successful was that
x.apply(0).get.asInstanceOf[Map[String, String]]
will do something what I want, but it only gets 0th index of the entire array and I'd want all the maps in the array.
How can I extract Map type out of Some?
If you want an Array[Any] from your Array[Option[Any]], you can use this for expression:
for {
opt <- x
value <- opt
} yield value
This will put the values of all the non-empty Options inside a new array.
It is equivalent to this:
x.flatMap(_.toArray[Any])
Here, all options will be converted to an array of either 0 or 1 element. All these arrays will then be flattened back to one single array containing all the values.
Generally, the pattern is either to use transformations on the Option[T], like map, flatMap, filter, etc.
The problem is, we'll need to add a type cast to retrieve the underlying Map[String, String] from Any. So we'll use flatten to remove any potentially None types and unwrap the Option, and asInstanceOf to retreive the type:
scala> val y = Array(Some(Map("1" -> "1")), Some(Map("2" -> "2")), None)
y: Array[Option[scala.collection.immutable.Map[String,String]]] = Array(Some(Map(1 -> 1)), Some(Map(2 -> 2)), None)
scala> y.flatten.map(_.asInstanceOf[Map[String, String]])
res7: Array[Map[String,String]] = Array(Map(1 -> 1), Map(2 -> 2))
Also when you talk just about single value you can try Some("test").head and for null simply Some(null).flatten
Related
I have a situation here
I have two strins
val keyMap = "anrodiApp,key1;iosApp,key2;xyz,key3"
val tentMap = "androidApp,tenant1; iosApp,tenant1; xyz,tenant2"
So what I want to add is to create a nested immutable nested map like this
tenant1 -> (andoidiApp -> key1, iosApp -> key2),
tenant2 -> (xyz -> key3)
So basically want to group by tenant and create a map of keyMap
Here is what I tried but is done using mutable map which I do want, is there a way to create this using immmutable map
case class TenantSetting() {
val requesterKeyMapping = new mutable.HashMap[String, String]()
}
val requesterKeyMapping = keyMap.split(";")
.map { keyValueList => keyValueList.split(',')
.filter(_.size==2)
.map(keyValuePair => (keyValuePair[0],keyValuePair[1]))
.toMap
}.flatten.toMap
val config = new mutable.HashMap[String, TenantSetting]
tentMap.split(";")
.map { keyValueList => keyValueList.split(',')
.filter(_.size==2)
.map { keyValuePair =>
val requester = keyValuePair[0]
val tenant = keyValuePair[1]
if (!config.contains(tenant)) config.put(tenant, new TenantSetting)
config.get(tenant).get.requesterKeyMapping.put(requester, requesterKeyMapping.get(requester).get)
}
}
The logic to break the strings into a map can be the same for both as it's the same syntax.
What you had for the first string was not quite right as the filter you were applying to each string from the split result and not on the array result itself. Which also showed in that you were using [] on keyValuePair which was of type String and not Array[String] as I think you were expecting. Also you needed a trim in there to cope with the spaces in the second string. You might want to also trim the key and value to avoid other whitespace issues.
Additionally in this case the combination of map and filter can be more succinctly done with collect as shown here:
How to convert an Array to a Tuple?
The use of the pattern with 2 elements ensures you filter out anything with length other than 2 as you wanted.
The iterator is to make the combination of map and collect more efficient by only requiring one iteration of the collection returned from the first split (see comments below).
With both strings turned into a map it just needs the right use of groupByto group the first map by the value of the second based on the same key to get what you wanted. Obviously this only works if the same key is always in the second map.
def toMap(str: String): Map[String, String] =
str
.split(";")
.iterator
.map(_.trim.split(','))
.collect { case Array(key, value) => (key.trim, value.trim) }
.toMap
val keyMap = toMap("androidApp,key1;iosApp,key2;xyz,key3")
val tentMap = toMap("androidApp,tenant1; iosApp,tenant1; xyz,tenant2")
val finalMap = keyMap.groupBy { case (k, _) => tentMap(k) }
Printing out finalMap gives:
Map(tenant2 -> Map(xyz -> key3), tenant1 -> Map(androidApp -> key1, iosApp -> key2))
Which is what you wanted.
in my main program i receive inputs like -
key1=value1 key2=value2
Now what I want is to create a map out of it. I know the imperative way of doing this where I would get Array[String] that can be foreach and then split by "=" and then key and value can be used to form a Map.
is there a good functional and readable way to achieve this?
Also It will be great if I can avoid mutable Map and I want to avoid initial Dummy value initialization.
def initialize(strings: Array[String]): Unit = {
val m = collection.mutable.Map("dummy" -> "dummyval")
strings.foreach(
s => {
val keyVal:Array[String] = s.split("=")
m += keyVal(0) -> keyVal(1)
})
println(m)
}
you can just use toMap().
However, converting from array to tuple is not quite trivial:
How to convert an Array to a Tuple?
scala> val ar = Array("key1=value1","key2=value2")
ar: Array[String] = Array(key1=value1, key2=value2)
scala> ar.collect(_.split("=") match { case Array(x,y) => (x,y)}).toMap
res10: scala.collection.immutable.Map[String,String] = Map(key1 -> value1, key2 -> value2)
Maybe you have to call Function.unlift for intellij
val r = ar.collect(Function.unlift(_.split("=") match { case Array(x, y) => Some(x, y)})).toMap
similar to above but using only 'map'
ar.map(_.split("=")).map(a=>(a(0), a(1))).toMap
You can use Scopt to do the command line argument parsing in a neat way.
This post is essentially about how to build joint and marginal histograms from a (String, String) RDD. I posted the code that I eventually used below as the answer.
I have an RDD that contains a set of tuples of type (String,String) and since they aren't unique I want to get a look at how many times each String, String combination occurs so I use countByValue like so
val PairCount = Pairs.countByValue().toSeq
which gives me a tuple as output like this ((String,String),Long) where long is the number of times that the (String, String) tuple appeared
These Strings can be repeated in different combinations and I essentially want to run word count on this PairCount variable so I tried something like this to start:
PairCount.map(x => (x._1._1, x._2))
But the output the this spits out is String1->1, String2->1, String3->1, etc.
How do I output a key value pair from a map job in this case where the key is going to be one of the String values from the inner tuple, and the value is going to be the Long value from the outter tuple?
Update:
#vitalii gets me almost there. the answer gets me to a Seq[(String,Long)], but what I really need is to turn that into a map so that I can run reduceByKey it afterwards. when I run
PairCount.flatMap{case((x,y),n) => Seq[x->n]}.toMap
for each unique x I get x->1
for example the above line of code generates mom->1 dad->1 even if the tuples out of the flatMap included (mom,30) (dad,59) (mom,2) (dad,14) in which case I would expect toMap to provide mom->30, dad->59 mom->2 dad->14. However, I'm new to scala so I might be misinterpreting the functionality.
how can I get the Tuple2 sequence converted to a map so that I can reduce on the map keys?
If I correctly understand question, you need flatMap:
val pairCountRDD = pairs.countByValue() // RDD[((String, String), Int)]
val res : RDD[(String, Int)] = pairCountRDD.flatMap { case ((s1, s2), n) =>
Seq(s1 -> n, s2 -> n)
}
Update: I didn't quiet understand what your final goal is, but here's a few more examples that may help you, btw code above is incorrect, I have missed the fact that countByValue returns map, and not RDD:
val pairs = sc.parallelize(
List(
"mom"-> "dad", "dad" -> "granny", "foo" -> "bar", "foo" -> "baz", "foo" -> "foo"
)
)
// don't use countByValue, if pairs is large you will run out of memmory
val pairCountRDD = pairs.map(x => (x, 1)).reduceByKey(_ + _)
val wordCount = pairs.flatMap { case (a,b) => Seq(a -> 1, b ->1)}.reduceByKey(_ + _)
wordCount.take(10)
// count in how many pairs each word occur, keys and values:
val wordPairCount = pairs.flatMap { case (a,b) =>
if (a == b) {
Seq(a->1)
} else {
Seq(a -> 1, b ->1)
}
}.reduceByKey(_ + _)
wordPairCount.take(10)
to get the histograms for the (String,String) RDD I used this code.
val Hist_X = histogram.map(x => (x._1-> 1.0)).reduceByKey(_+_).collect().toMap
val Hist_Y = histogram.map(x => (x._2-> 1.0)).reduceByKey(_+_).collect().toMap
val Hist_XY = histogram.map(x => (x-> 1.0)).reduceByKey(_+_)
where histogram was the (String,String) RDD
I have following list:
val list = List(("name1",20),("name2",20),("name1",30),("name2",30),
("name3",40),("name3",30),("name3",20))
I want following output:
List(("name3",40))
I tried following:
val distElements = list.map(_._2).distinct
list.groupBy(_._1).map{ case(k,v) =>
val h = v.map(_._2)
if(distElements.equals(h)) List.empty else distElements.diff(h)
}.flatten
But this is not I am looking for.
Can anybody give answer/hint me to get expected output.
I understand the question as looking for the element of the list whose _2 (number) occurs only once.
val list = List(("name1",20),("name2",20),("name1",30),("name2",30),
("name3",40),("name3",30),("name3",20))
First you group by the _2 element, which gives you a map whose keys are lists of all elements with the same _2:
val g = list.groupBy(_._2) // Map[Int, List[(String, Int)]]
Now you can filter those entries that consists only of one element:
val opt = g.collectFirst { // Option[(String, Int)]
case (_, single :: Nil) => single
}
Or (if you are expecting possibly more than one distinct value)
val col = g.collect { // Map[String, Int]
case (_, single :: Nil) => single
}
Seems to me that you're looking to match against both the value of the left hand and the right hand at the same time while also preserving the type of collection you're looking at, a List. I would use collect:
val out = myList.collect{
case item # ("name3", 40) => item
}
which combines a PartialFunction with filter and map like qualities. In this case, it filters out any value for which the PartialFunction is not defined while mapping the values which match. Here, I've only allowed for a singular match.
Consider the following from the Scala interpreter:
scala> JSON.parseFull("""{"name":"jack","greeting":"hello world"}""")
res6: Option[Any] = Some(Map(name -> jack, greeting -> hello world))
Why is the Map returned in Some() thing? And how do I work with it?
I want to put the values in an xml template:
<test>
<name>name goes here</name>
<greeting>greeting goes here</greeting>
</test>
What is the Scala way of getting my map out of Some(thing) and getting those values in the xml?
You should probably use something like this:
res6 collect { case x: Map[String, String] => renderXml(x) }
Where:
def renderXml(m: Map[String, String]) =
<test><name>{m.get("name") getOrElse ""}</name></test>
The collect method on Option[A] takes a PartialFunction[A, B] and is a combination of filter (by a predicate) and map (by a function). That is:
opt collect pf
opt filter (a => pf isDefinedAt a) map (a => pf(a))
Are both equivalent. When you have an optional value, you should use map, flatMap, filter, collect etc to transform the option in your program, avoiding extracting the option's contents either via a pattern-match or via the get method. You should never, ever use Option.get - it is the canonical sign that you are doing it wrong. Pattern-matching should be avoided because it represents a fork in your program and hence adds to cyclomatic complexity - the only time you might wish to do this might be for performance
Actually you have the issue that the result of the parseJSON method is an Option[Any] (the reason is that it is an Option, presumably, is that the parsing may not succeed and Option is a more graceful way of handling null than, well, null).
But the issue with my code above is that the case x: Map[String, String] cannot be checked at runtime due to type erasure (i.e. scala can check that the option contains a Map but not that the Map's type parameters are both String. The code will get you an unchecked warning.
An Option is returned because parseFull has different possible return values depending on the input, or it may fail to parse the input at all (giving None). So, aside from an optional Map which associates keys with values, an optional List can be returned as well if the JSON string denoted an array.
Example:
scala> import scala.util.parsing.json.JSON._
import scala.util.parsing.json.JSON._
scala> parseFull("""{"name":"jack"}""")
res4: Option[Any] = Some(Map(name -> jack))
scala> parseFull("""[ 100, 200, 300 ]""")
res6: Option[Any] = Some(List(100.0, 200.0, 300.0))
You might need pattern matching in order to achieve what you want, like so:
scala> parseFull("""{"name":"jack","greeting":"hello world"}""") match {
| case Some(m) => Console println ("Got a map: " + m)
| case _ =>
| }
Got a map: Map(name -> jack, greeting -> hello world)
Now, if you want to generate XML output, you can use the above to iterate over the key/value pairs:
import scala.xml.XML
parseFull("""{"name":"jack","greeting":"hello world"}""") match {
case Some(m: Map[_,_]) =>
<test>
{
m map { case (k,v) =>
XML.loadString("<%s>%s</%s>".format(k,v,k))
}
}
</test>
case _ =>
}
parseFull returns an Option because the string may not be valid JSON (in which case it will return None instead of Some).
The usual way to get the value out of a Some is to pattern match against it like this:
result match {
case Some(map) =>
doSomethingWith(map)
case None =>
handleTheError()
}
If you're certain the input will always be valid and so you don't need to handle the case of invalid input, you can use the get method on the Option, which will throw an exception when called on None.
You have two separate problems.
It's typed as Any.
Your data is inside an Option and a Map.
Let's suppose we have the data:
val x: Option[Any] = Some(Map("name" -> "jack", "greeting" -> "hi"))
and suppose that we want to return the appropriate XML if there is something to return, but not otherwise. Then we can use collect to gather those parts that we know how to deal with:
val y = x collect {
case m: Map[_,_] => m collect {
case (key: String, value: String) => key -> value
}
}
(note how we've taken each entry in the map apart to make sure it maps a string to a string--we wouldn't know how to proceed otherwise. We get:
y: Option[scala.collection.immutable.Map[String,String]] =
Some(Map(name -> jack, greeting -> hi))
Okay, that's better! Now if you know which fields you want in your XML, you can ask for them:
val z = for (m <- y; name <- m.get("name"); greet <- m.get("greeting")) yield {
<test><name>{name}</name><greeting>{greet}</greeting></test>
}
which in this (successful) case produces
z: Option[scala.xml.Elem] =
Some(<test><name>jack</name><greeting>hi</greeting></test>)
and in an unsuccessful case would produce None.
If you instead want to wrap whatever you happen to find in your map in the form <key>value</key>, it's a bit more work because Scala doesn't have a good abstraction for tags:
val z = for (m <- y) yield <test>{ m.map { case (tag, text) => xml.Elem(null, tag, xml.Null, xml.TopScope, xml.Text(text)) }}</test>
which again produces
z: Option[scala.xml.Elem] =
Some(<test><name>jack</name><greeting>hi</greeting></test>)
(You can use get to get the contents of an Option, but it will throw an exception if the Option is empty (i.e. None).)