Extracting Object from Some() and Using it - scala

In the below code, encoded is a JSON string. The JSON.parseFull() function is returning an object of the form: Some(Map(...)). I am using .get to extract the Map, but am unable to index it as the compiler sees it as type Any. Is there any to provide the compiler visibility that it is, in fact, a map?
val parsed = JSON.parseFull(encoded)
val mapped = parsed.get

You can utilize the collect with pattern matching to match on the type:
scala> val parsed: Option[Any] = Some(Map("1" -> List("1")))
parsed: Option[Any] = Some(Map(1 -> List(1)))
scala> val mapped = parsed.collect{
case map: Map[String, Any] => map
}
mapped: Option[Map[String,Any]] = Some(Map(1 -> List(1)))
You can do something like the following in the case of a List value to get values from the List:
scala> mapped.get.map{ case(k, List(item1)) => item1}
res0: scala.collection.immutable.Iterable[Any] = List(1)

I was able to use a combination of the get function and pattern matching similar to what was posted in Tanjin's response to get the desired result.
object ReadFHIR {
def fatal(msg: String) = throw new Exception(msg)
def main (args: Array[String]): Unit = {
val fc = new FhirContext()
val client = fc.newRestfulGenericClient("http://test.fhir.org/r2")
val bundle = client.search().forResource("Observation")
.prettyPrint()
.execute()
val jsonParser = fc.newJsonParser()
val encoded = jsonParser.encodeBundleToString(bundle)
val parsed = JSON.parseFull(encoded)
val mapped: Map[String, Any] = parsed.get match{
case map: Map[String, Any] => map
}
println(mapped("resourceType"))
}
}

Related

No implicits found for parameter ev: Any <:< (T_, U_)

I'm trying to create a map based on conditions. Here's the general workflow in pseudocode:
def createMyMap(input: String): Map[String, String] = {
val stringArray = input.split(",")
stringArray.map(element => {
if (condition) {
newKey -> newVal
}
}).toMap
}
and I see two compile errors:
No implicits found for parameter ev: Any <:< (T_, U_) in the toMap call
For the method createMyMap
Type mismatch.
Required: scala.Predef.Map [String, String]
Found: scala.collection.immutable.Map[Nothing, Nothing]
This makes sense since the compiler doesn't know how to create the map if the condition isn't fulfilled. For example, if I add this in the method:
if (condition) {
newKey -> new Val
} else {
null
}
then it'll compile. I'm just not too sure how to approach the else - how do I avoid this kind of problem? I'm running into this because I only want to create a map entry if a condition is fulfilled.
It's not clear how newKey and newVal are derived, but here is the template code using collect
def createMyMap(input: String): Map[String, String] =
input.split(",").collect {
case s if <condition> =>
newKey -> newVal
}.to(Map)
e.g.
def createMyMap(input: String): Map[String, String] =
input.split(",").collect {
case s if s.contains('.') =>
s -> "data"
}.to(Map)
You have a few good options that I'll summarize from the comments:
filter + map + toMap
// stringArray: Traversable[String]
// condition: String => Boolean
// transform: String => (String, String)
val result: Map[String, String] = stringArray
.filter(condition)
.map(transform)
.toMap
map + flatten + toMap
val result: Map[String, String] = stringArray
.map { k =>
if (condition(k)) Some(transform(k))
else None
}
.flatten
.toMap
flatMap + toMap
val result: Map[String, String] = stringArray
.flatMap { k =>
if (condition(k)) Some(transform(k))
else None
}
.toMap
collect + toMap
val result: Map[String, String] = stringArray
.collect {
case k if condition(s) => transform(k)
}
.toMap
See documentation.
In short, methods 3 and 4 are especially clean (although all are good IMO). However, the one that is semantically the most readable IMO is 4, which uses collect (with option 1 being a close second).

How do I get map item with Option[Long]?

Below I have some scala that gets a key from an object map m, this works when I have a value just fine, but what happens when I have an Option[Long]? How can I get the map item?
object Main extends App {
val mainKey: Long = 12345
val m: Map[Long, String] = Map(mainKey -> "bar")
println(m.get(mainKey)) // "bar"
val key: Option[Long] = None
println(m.get(key)) // syntax error needs Long
println(m.getOrElse(key, None)) // syntax error needs Long
}
val mainKey: Long = 12345
val m: Map[Long, String] = Map(mainKey -> "bar")
println(m.get(mainKey)) // "bar"
val key: Option[Long] = None
println(key.flatMap(m.get))
prints:
Some(bar)
None
https://scastie.scala-lang.org/6CdEul3sTvGa9gMyamKFEQ

No TypeTag available for case class Type

I want to generate a method which will convert an Object into a Map[String, _], and later back from Map[String, _] to Object.
I generate the initial object as follows:
case class Name (firstName : String, lastName : String)
case class Documents (idx: String, name: Name, code: String)
val mName1 = Name("Roger", "Rabbit")
val myDoc = Documents("12", mName1, "ABCD")
Then following method converts a given Map[String, _] into an Object:
def fromMap[T : TypeTag: ClassTag ](m: Map[String,_]) = {
val rm = runtimeMirror(classTag[T].runtimeClass.getClassLoader)
val classTest = typeOf[T].typeSymbol.asClass
val classMirror = rm.reflectClass(classTest)
val constructor = typeOf[T].decl(termNames.CONSTRUCTOR).asMethod
val constructorMirror = classMirror.reflectConstructor(constructor)
val constructorArgs = constructor.paramLists.flatten.map( (param: Symbol) => {
val paramName = param.name.toString
if(param.typeSignature <:< typeOf[Option[Any]])
m.get(paramName)
else
m.get(paramName).getOrElse(throw new IllegalArgumentException("Map is missing required parameter named " + paramName))
})
constructorMirror(constructorArgs:_*).asInstanceOf[T]
}
And inside the following method I convert the initial Object into a Map[String, _], and back to Object (by invoking the method above):
def fromMapToObject(input: Any) : Unit= {
println("input: "+input)
//Converting an Object into a Map
val r = currentMirror.reflect(input)
val docAsMapValues = r.symbol.typeSignature.members.toStream
.collect{case s : TermSymbol if !s.isMethod => r.reflectField(s)}
.map(r => r.symbol.name.toString.trim -> r.get)
.toMap
println("intermediate: "+docAsMapValues)
val obj = fromMap[Documents](docAsMapValues)
println("output: "+obj)
}
So if I call:
fromMapToObject(myDoc)
Input and output will match.
Problem, trying to go a step further, I want now to do the same with the field name, which is of type Name. But I want this step to be generic, in the sense that without knowing what is the type of the field name, I could convert it into a Map[String, _], and from Map[String, _] back to Object.
So what I will do now in fromMapToObject is:
Extract from the input a Map[String, _]
Extract from the input a Map[String, Types]
Convert the value of the field name from Name into a Map[String, _]
Revert the 3rd step to get back an Object of type Name
This is how I am trying to approach this new scenario:
def fromMapToObject[T: TypeTag: ClassTag](input: Any) : Unit = {
println("input: "+input)
//Converting an Object into a Map
val r = currentMirror.reflect(input)
val docAsMapValues = r.symbol.typeSignature.members.toStream
.collect{case s : TermSymbol if !s.isMethod => r.reflectField(s)}
.map(r => r.symbol.name.toString.trim -> r.get)
.toMap
val docAsMapTypes = r.symbol.typeSignature.members.toStream
.collect{case s : TermSymbol if !s.isMethod => r.reflectField(s)}
.map(r => r.symbol.name.toString.trim -> r.symbol.typeSignature)
.toMap
// Here I extract from the map the value and type of the attribute name
val nameType = docAsMapValues("name")
val nameValue = docAsMapValues("name")
// Converting Name into a map
val r2 = currentMirror.reflect(nameValue)
val nameAsMapValues = r2.symbol.typeSignature.members.toStream
.collect{case s : TermSymbol if !s.isMethod => r2.reflectField(s)}
.map(r2 => r2.symbol.name.toString.trim -> r2.get)
.toMap
type nameT = nameType.type
val obj = fromMap[nameT](nameAsMapValues)
}
But I am getting the following error when compiling in Intellij:
Error:(111, 29) No TypeTag available for nameT
val obj = fromMap[nameT](nameAsMapValues)
I would like to know how could I convert that runtime.universe.Type which is returned from r.symbol.typeSignature into a TypeTag : ClassTag
I'm not completely certain that I'm interpreting your question correctly, but from what I understand this can be solved pretty nicely and type safely via shapeless. To start with, you want to convert your Document to a Map. shapeless can do this out of the box for you with one of the Typeclasses in the ops folder. If we bundle that up into a function, with some machinery to pull everything together, we get something like:
import shapeless._
def ObjectToMap[A, Repr <: HList](obj: A)(
implicit
gen: LabelledGeneric.Aux[A,Repr], //Convert to generic representation
toMap: ops.record.ToMap[Repr] //Convert generic representation to Map[Symbol,_]
) = toMap(gen.to(obj))
which outputs
val m = ObjectToMap(myDoc)
println(m) //Map('code -> ABCD, 'name -> Name(Roger,Rabbit), 'idx -> 12)
Going the other direction is a little bit more complicated. There exists a ops.maps.FromMap typeclass. However, we want to be able to specify the type parameter, and then let the compiler still verify that the generic representation is an HList, to match up with FromMap's signature. Since dependent types don't work with other variables defined in the same parameter list, and we only get one implicit parameter list, we need to resort to a little bit of trickery to curry the type parameters:
trait MapToObject[A]{
def from[Repr <: HList](m: Map[_,_])(
implicit
gen: LabelledGeneric.Aux[A,Repr],
fromMap: ops.maps.FromMap[Repr]
): Option[A] = fromMap(m).map(gen.from)
}
object MapToObject{
def apply[A](
implicit gen: LabelledGeneric[A]
): MapToObject[A] = new MapToObject[A]{}
}
When we run the output of the previous chunk through that we get:
val doc = MapToObject[Documents].from(m)
println(doc) //Some(Documents(12,Name(Roger,Rabbit),ABCD))
type nameT = nameType.type is incorrect. nameType.type is (singleton) type of this specific variable nameType and you want type of name field. This accidentally worked because actually you don't use T in fromMap2 (runtimeMirror(classTag[T].runtimeClass.getClassLoader) can be replaced with currentMirror there).
You wanted to call your original fromMap inside fromMapToObject. You know universe.Type of name and it's enough to find TypeTag and ClassTag implicit parameters for fromMap.
But it's not enough to find T. The thing is that since you use runtime reflection you know universe.Type (and TypeTag, ClassTag) at runtime. But in order to call fromMap you need to know T at compile time. So one way is to use compile-time reflection i.e. macros. Other way is to avoid T and use value parameters like you did.
Case class to map in Scala
Scala: convert map to case class
I was able to find a solution. Since I wasn't able to get classTag and typeTag I modified the function toMap as follow:
def fromMap2[T : ClassTag ](m: Map[String,_], mSymbol: Symbol, mType :Type): Any = {
val rm = runtimeMirror(classTag[T].runtimeClass.getClassLoader)
val classTest = mSymbol.asClass
val classMirror = rm.reflectClass(classTest)
val constructor = mType.decl(termNames.CONSTRUCTOR).asMethod
val constructorMirror = classMirror.reflectConstructor(constructor)
val constructorArgs = constructor.paramLists.flatten.map( (param: Symbol) => {
val paramName = param.name.toString
if(param.typeSignature <:< typeOf[Option[Any]])
m.get(paramName)
else
m.get(paramName).getOrElse(throw new IllegalArgumentException("Map is missing required parameter named " + paramName))
})
constructorMirror(constructorArgs:_*).asInstanceOf[T]
}
So now I need to pass the Type and Symbol of T. I can get these two values like follow:
//Converting an Object into a Map
val r = currentMirror.reflect(input)
val mapValues = r.symbol.typeSignature.members.toStream
.collect{case s : TermSymbol if !s.isMethod => r.reflectField(s)}
.map(r => r.symbol.name.toString.trim -> r.get)
.toMap
val mapTypes = r.symbol.typeSignature.members.toStream
.collect{case s : TermSymbol if !s.isMethod => r.reflectField(s)}
.map(r => r.symbol.name.toString.trim -> r.symbol.typeSignature)
.toMap
val mapTypesSymbols = r.symbol.typeSignature.members.toStream
.collect{case s : TermSymbol if !s.isMethod => r.reflectField(s)}
.map(r => r.symbol.name.toString.trim -> r.symbol.typeSignature.typeSymbol)
.toMap
val nameType = mapTypes("name")
val nameTypeSymbols = mapTypesSymbols("name")
val nameValue = mapValues("name")
// Converting Name into a map
val r2 = currentMirror.reflect(nameValue)
val nameAsMapValues = r2.symbol.typeSignature.members.toStream
.collect{case s : TermSymbol if !s.isMethod => r2.reflectField(s)}
.map(r2 => r2.symbol.name.toString.trim -> r2.get)
.toMap
type nameT = nameType.type
val obj = fromMap2[nameT](nameAsMapValues, nameTypeSymbols, nameType)
Even if this works, I believe it is very anti pattern. So I will leave the question open in case someone could point out ways to improve it.

Case class with empty parameter

I am new to scala and I am trying to extract few columns based below is my code
case class Extract(e1:String,e2:String,e3:String){
override def toString = e1+","+e2+","+e3
}
object ScalaSpark {
def main(args: Array[String])
{
val textfile = sc.textFile("/user/cloudera/xxxx/File2")
val word = textfile.filter(x => x.length > 0).map(_.split("\\|"))
val pid = word.filter(_.contains("SSS"))
val pidkeys = pid.map(tuple => Extract(tuple(0),tuple(3),tuple(7)))
val obx = word.filter(_.contains("HHH"))
val obxkeys = obx.map(tuple => Extract(tuple(0),tuple(5)))
val rddall = pidkeys.unionAll(obxkeys)
rddall.saveAsTextFile("/user/xxxx/xxxx/rddsum1")
}
}
What I am trying with this code is to extract 3 values from row containing SSS and 2 values from row contatining HHH but when i am executing this i am getting below error
error: not enough arguments for method apply: (e1: String, e2: String, e3: String)Extract in object Extract.
I then tried using Opt[String] = None but that also didn't worked i don't know how to sort out this problem please help.
EDIT:
I used Option[String] and my code is written below
case class Extract(e1:String,e2:String,e3:Option[String]){
override def toString = e1+","+e2+","+e3
}
object ScalaSpark {
def main(args: Array[String])
{
val textfile = sc.textFile("/user/cloudera/xxxx/File2")
val word = textfile.filter(x => x.length > 0).map(_.split('|'))
val pid = word.filter(_.contains("SSS"))
val pidkeys = pid.map(tuple => Extract(tuple(0),tuple(5),tuple(8)))
val obx = word.filter(_.contains("HHH"))
val obxkeys = obx.map(tuple => Extract(tuple(0),tuple(5), None))
val rddall = pidkeys.union(obxkeys)
rddall.coalesce(1).saveAsTextFile("/user/xxx/xxx/rddsum1")
}
}
but i am getting below error
error: type mismatch;
found : String
required: Option[String]
val pidkeys = pid.map(tuple => Header(tuple(0),tuple(5),tuple(8)))
^
<console>:38: error: type mismatch;
found : org.apache.spark.rdd.RDD[Extract]
required: org.apache.spark.rdd.RDD[Nothing]
Note: Header >: Nothing, but class RDD is invariant in type T.
You may wish to define T as -T instead. (SLS 4.5)
val rddall = pidkeys.union(obxkeys)
As far as I understand your Extract case class have 3 parameters. Last of them is optional.
If so you should declare it this way:
case class Extract(s1: String, s2: String, s3: Option[String])
and use it either Extract("some string", "other string", Some("optional string")) or Extract("some string", "other string", None).

How do I parse a x-www-url-encoded string into a Map[String, String] using Lift?

From Lift, I'm getting a string of the form
TOKEN=EC%2d454178600K772032D&TIMESTAMP=2011%2d06%2d29T13%3a10%3a58Z&CORRELATIONID=cbd56e97cad38&ACK=Success&VERSION=64&BUILD=1936884
from the response of an HTTP request.
Although it's probably ultra-trivial, I can't find the Lift function that parses this into a nice Map[String, String]. Any help?
From Lift's Req.scala:
// calculate the query parameters
lazy val queryStringParam: (List[String], Map[String, List[String]]) = {
val params: List[(String, String)] =
for {
queryString <- request.queryString.toList
nameVal <- queryString.split("&").toList.map(_.trim).filter(_.length > 0)
(name, value) <- nameVal.split("=").toList match {
case Nil => Empty
case n :: v :: _ => Full((urlDecode(n), urlDecode(v)))
case n :: _ => Full((urlDecode(n), ""))
}} yield (name, value)
val names: List[String] = params.map(_._1).distinct
val nvp: Map[String, List[String]] = params.foldLeft(Map[String, List[String]]()) {
case (map, (name, value)) => map + (name -> (map.getOrElse(name, Nil) ::: List(value)))
}
(names, nvp)
}
I haven't seen any Lift's implementation for that. You can achieve this with something like this:
val input = "TOKEN=EC%2d454178600K772032D&TIMESTAMP=2011%2d06%2d29T13%3a10%3a58Z&CORRELATIONID=cbd56e97cad38&ACK=Success&VERSION=64&BUILD=1936884"
val res = input.split('&') map { str =>
val pair = str.split('=')
(pair(0) -> pair(1))
} toMap
note: it assumes, that you have a well-formed string. In your code you should probably check if the string is ok.
I put together a small Scala library to help do this: https://github.com/theon/scala-uri
You can parse a uri and get the parameters into a Map[String,List[String]] like so:
val uri = parseUri("http://example.com?one=1&two=2").query.params
It also has a DSL for building URLs with query strings:
val uri = "http://example.com" ? ("one" -> 1) & ("two" -> 2)
scala> val queryParams = "a=4&b=5"
scala> queryParams.split("&").toList.map(_.trim).filter(_.length > 0).flatMap(x => {
val s = x.split('=')
Map[String, String](s(0) -> s(1))
}).toMap[String, String]
res0: scala.collection.immutable.Map[String,String] = Map(a -> 4, b -> 5)