I have a DStream with tuple of Date and String as value, I want to convert the String part of the tuple to HashSet[String], when I try to map it inside another map it is not working for some reason, here is what I tried so far, any pointers would help
val enTextTupleStream: DStream[(Date, String)]
def extractStrings(data: String): HashSet[String]= HashSet(data)
val ddstream = enTextTupleStream.map {
t =>
val e1 = t._1
val e2 = t._2.map(extractStrings)
}
Related
I am trying some basic logic using scala . I tried the below code but it throws error .
scala> val data = ("HI",List("HELLO","ARE"))
data: (String, List[String]) = (HI,List(HELLO, ARE))
scala> data.flatmap( elem => elem)
<console>:22: error: value flatmap is not a member of (String, List[String])
data.flatmap( elem => elem)
Expected Output :
(HI,HELLO,ARE)
Could some one help me to fix this issue?
You are trying to flatMap over a tuple, which won't work. The following will work:
val data = List(List("HI"),List("HELLO","ARE"))
val a = data.flatMap(x => x)
This will be very trivial in scala:
val data = ("HI",List("HELLO","ARE"))
println( data._1 :: data._2 )
what exact data structure are you working with?
If you are clear about you data structure:
type rec = (String, List[String])
val data : rec = ("HI",List("HELLO","ARE"))
val f = ( v: (String, List[String]) ) => v._1 :: v._2
f(data)
A couple of observations:
Currently there is no flatten method for tuples (unless you use shapeless).
flatMap cannot be directly applied to a list of elements which are a mix of elements and collections.
In your case, you can make element "HI" part of a List:
val data = List(List("HI"), List("HELLO","ARE"))
data.flatMap(identity)
Or, you can define a function to handle your mixed element types accordingly:
val data = List("HI", List("HELLO","ARE"))
def flatten(l: List[Any]): List[Any] = l.flatMap{
case x: List[_] => flatten(x)
case x => List(x)
}
flatten(data)
You are trying to flatMap on Tuple2 which is not available in current api
If you don't want to change your input, you can extract the values from Tuple2 and the extract the values for second tuple value as below
val data = ("HI",List("HELLO","ARE"))
val output = (data._1, data._2(0), data._2(1))
println(output)
If that's what you want:
val data = ("HI",List("HELLO,","ARE").mkString(""))
println(data)
>>(HI,HELLO,ARE)
In the below code, encoded is a JSON string. The JSON.parseFull() function is returning an object of the form: Some(Map(...)). I am using .get to extract the Map, but am unable to index it as the compiler sees it as type Any. Is there any to provide the compiler visibility that it is, in fact, a map?
val parsed = JSON.parseFull(encoded)
val mapped = parsed.get
You can utilize the collect with pattern matching to match on the type:
scala> val parsed: Option[Any] = Some(Map("1" -> List("1")))
parsed: Option[Any] = Some(Map(1 -> List(1)))
scala> val mapped = parsed.collect{
case map: Map[String, Any] => map
}
mapped: Option[Map[String,Any]] = Some(Map(1 -> List(1)))
You can do something like the following in the case of a List value to get values from the List:
scala> mapped.get.map{ case(k, List(item1)) => item1}
res0: scala.collection.immutable.Iterable[Any] = List(1)
I was able to use a combination of the get function and pattern matching similar to what was posted in Tanjin's response to get the desired result.
object ReadFHIR {
def fatal(msg: String) = throw new Exception(msg)
def main (args: Array[String]): Unit = {
val fc = new FhirContext()
val client = fc.newRestfulGenericClient("http://test.fhir.org/r2")
val bundle = client.search().forResource("Observation")
.prettyPrint()
.execute()
val jsonParser = fc.newJsonParser()
val encoded = jsonParser.encodeBundleToString(bundle)
val parsed = JSON.parseFull(encoded)
val mapped: Map[String, Any] = parsed.get match{
case map: Map[String, Any] => map
}
println(mapped("resourceType"))
}
}
I am trying to append filename to each record in the file. I thought if the RDD is Array it would have been easy for me to do it.
Some help with converting RDD type or solving this problem would be much appreciated!
In (String, String) type
scala> myRDD.first()(1)
scala><console>:24: error: (String, String) does not take parametersmyRDD.first()(1)
In Array(string)
scala> myRDD.first()(1)
scala> res1: String = abcdefgh
My function:
def appendKeyToValue(x: Array[Array[String]){
for (i<-0 to (x.length - 1)) {
var key = x(i)(0)
val pattern = new Regex("\\.")
val key2 = pattern replaceAllIn(key1,"|")
var tempvalue = x(i)(1)
val finalval = tempvalue.split("\n")
for (ab <-0 to (finalval.length -1)){
val result = (I am trying to append filename to each record in the filekey2+"|"+finalval(ab))
}
}
}
If you have a RDD[(String, String)], you can access the first tuple field of the first tuple by calling
val firstTupleField: String = myRDD.first()._1
If you want to convert a RDD[(String, String)] into a RDD[Array[String]] you can do the following
val arrayRDD: RDD[Array[String]] = myRDD.map(x => Array(x._1, x._2))
You may also employ a partial function to destructure the tuples:
val arrayRDD: RDD[Array[String]] = myRDD.map { case (a,b) => Array(a, b) }
I can't grasp the map method I guess..
Trying to read a file :
val messagesMap = XML.loadFile(messageXMLFile).map(parseMessageXML)
where the method parseMessageXML is defined as :
def parseMessageXML(xml : scala.xml.Node) = {
val nodes = xml \\ "add"
nodes.map({
node =>
val obj = new AdMessage(node)
println("adding an AdMessage " + obj.toString)
(obj.MessageId -> obj)
}).toMap
}
Can anybody please explain why I end up with a Seq[Map[String, AdMessage]] and not a just a Map[String, AdMessage] ?
map transforms each element of your Seq into an another element.
For instance:
scala> Seq("One", "Two", "Three").map(_.length())
res0: Seq[Int] = List(3, 3, 5)
Each String is mapped into an Int thanks to the length function. Therefore the original type is Seq[String] and the final type is Seq[Int]
In your case, parseMessageXML transforms a Node into a Map[String, AdMessage], so the original type is Seq[Node] and the final type is Seq[Map[String, AdMessage]].
In your case, assuming you just want to transform the content of the file into a Map[String, AdMessage]:
val messagesMap = parseMessageXML(XML.loadFile(messageXMLFile))
why this is NOT valid?
var a = List[(String,String)]
a = a :: ("x","y")
basically i want to init a list of tuple and add a tuple to the list.
Here you have two mistakes.
The first one is that you are trying to instantiate List which an abstract class
I believe that what you are trying to do is the following
var a : List[(String,String)] = List()
This will create a list of an empty list of tuples.
The second is that you are trying to add an element which is not actually a tuple so I believe that you should try the following
a = a:+(("x","y"))
Here you are defining a tuples and adding it to your List a
You can instantiate an empty List[(String, String)] in many ways:
val list = List[(String, String)]()
val list = List.empty[(String, String)]
val list : List[(String, String)] = Nil
val list : List[(String, String)] = List()
val list = Nil : List[(String, String)]
val list : List[(String, String)] = List.empty
Note, too, that the default instantiations of Traversable, Iterable, Seq, and LinearSeq all also return a List, so you can use those too, (e.g. Seq.empty[(String, String)]).
And then you can add elements using :+:
val newList = list :+ ("x", "y")
val newList = list :+ ("x" -> "y")
You were using the cons method ::, which is used for prepending. In Scala, methods that end in : are right associative:
val newList = ("x" -> "y") :: list
You can also call them using the regular dot syntax if you want them to be left associative, like normal methods:
val newList = list.::("x" -> "y")
The reason this method is right associative is because it prepends elements, allowing you to do things like:
val newList = ("x" -> "y") :: ("a" -> "b") :: list
and retain the expected order (the same order that it appears in the code).
Simple straight way in declaration:
val l=List("type"->"number","min"->"2","step"->"1","value"->"2")
You could also write the add like this if you want to add to an existing Array :
var a : List[(String,String)] = List()
a:+=(("x","y"))