I have 2 RDD's that I joined them together using left join. As a result, the fields of the right RDD are now defined as Option as they might be None (null). when writing the result to a file it looks something like this: Some(value) for example: Some('value1'), Some('Value2').
How can I remove the 'Some' / remove the Option from the field definition?
If you have an Option[String] and turn this into a String, you still need to handle the case where your value is None.
For example you can turn None's into empty strings:
val myInput: Option[String] = ...
val myOutput: String = myInput.getOrElse("")
Or into null's:
val myInput: Option[String] = ...
val myOutput: String = myInput.orNull
Or not write them at all:
val myInput: Option[String] = ...
// Does nothing if myInput is None
myInput.foreach(writeToFile)
Related
Let's say I have a list of myObjects
case class myObject(id:Int)
val myObjects = List(myObjects(5), myObjects(4))
I need to get the id of the last object in the type Option[String]
I can do this using
val lastId: Option[String] = Some(myObjects.last.id.toString)
That
's all well and good, but the problem is I need this to work in the case the List is empty. In this situation, I need lastId to None
I have tried
val lastId: Option[String] = myObjects.lastOption.map(_.id.toString).getOrElse(None)
val lastId: Option[String] = myObjects.lastOption.map(_.id.toString.getOrElse(None))
and several variations of this but can't get it to work for both lists with objects, and empty lists
What is the best way to do this?
myObjects.lastOption.map(_.id.toString)
testing:
List(myObject(5), myObject(4)).lastOption.map(_.id.toString)
//res0: Option[String] = Some(4)
List.empty[myObject].lastOption.map(_.id.toString)
//res1: Option[String] = None
New to Scala, continue to struggle with Option related code. I have a HashMap built of Case Class instances that themselves contain hash maps with Case Class instance values. It is not clear to me how to access properties of the retrieved Class instances:
import collection.mutable.HashMap
case class InnerClass(name: String, age: Int)
case class OuterClass(name: String, nestedMap: HashMap[String, InnerClass])
// Load some data...hash maps are mutable
val innerMap = new HashMap[String, InnerClass]()
innerMap += ("aaa" -> InnerClass("xyz", 0))
val outerMap = new HashMap[String, OuterClass]()
outerMap += ("AAA" -> OuterClass("XYZ", innerMap))
// Try to retrieve data
val outerMapTest = outerMap.getOrElse("AAA", None)
val nestedMap = outerMapTest.nestedMap
This produces error: value nestedMap is not a member of Option[ScalaFiddle.OuterClass]
// Try to retrieve data a different way
val outerMapTest = outerMap.getOrElse("AAA", None)
val nestedMap = outerMapTest.nestedMap
This produces error: value nestedMap is not a member of Product with Serializable
Please advise on how I would go about getting access to outerMapTest.nestedMap. I'll eventually need to get values and properties out of the nestedMap HashMap as well.
Since you are using .getOrElse("someKey", None) which returns you a type Product (not the actual type as you expect to be OuterClass)
scala> val outerMapTest = outerMap.getOrElse("AAA", None)
outerMapTest: Product with Serializable = OuterClass(XYZ,Map(aaa -> InnerClass(xyz,0)))
so Product either needs to be pattern matched or casted to OuterClass
pattern match example
scala> outerMapTest match { case x : OuterClass => println(x.nestedMap); case _ => println("is not outerclass") }
Map(aaa -> InnerClass(xyz,0))
Casting example which is a terrible idea when outerMapTest is None, (pattern matching is favored over casting)
scala> outerMapTest.asInstanceOf[OuterClass].nestedMap
res30: scala.collection.mutable.HashMap[String,InnerClass] = Map(aaa -> InnerClass(xyz,0))
But better way of solving it would simply use .get which very smart and gives you Option[OuterClass],
scala> outerMap.get("AAA").map(outerClass => outerClass.nestedMap)
res27: Option[scala.collection.mutable.HashMap[String,InnerClass]] = Some(Map(aaa -> InnerClass(xyz,0)))
For key that does not exist, gives you None
scala> outerMap.get("I dont exist").map(outerClass => outerClass.nestedMap)
res28: Option[scala.collection.mutable.HashMap[String,InnerClass]] = None
Here are some steps you can take to get deep inside a nested structure like this.
outerMap.lift("AAA") // Option[OuterClass]
.map(_.nestedMap) // Option[HashMap[String,InnerClass]]
.flatMap(_.lift("aaa")) // Option[InnerClass]
.map(_.name) // Option[String]
.getOrElse("no name") // String
Notice that if either of the inner or outer maps doesn't have the specified key ("aaa" or "AAA" respectively) then the whole thing will safely result in the default string ("no name").
A HashMap will return None if a key is not found so it is unnecessary to do getOrElse to return None if the key is not found.
A simple solution to your problem would be to use get only as below
Change your first get as
val outerMapTest = outerMap.get("AAA").get
you can check the output as
println(outerMapTest.name)
println(outerMapTest.nestedMap)
And change the second get as
val nestedMap = outerMapTest.nestedMap.get("aaa").get
You can test the outputs as
println(nestedMap.name)
println(nestedMap.age)
Hope this is helpful
You want
val maybeInner = outerMap.get("AAA").flatMap(_.nestedMap.get("aaa"))
val maybeName = maybeInner.map(_.name)
Which if your feeling adventurous you can get with
val name: String = maybeName.get
But that will throw an error if its not there. If its a None
you can access the nestMap using below expression.
scala> outerMap.get("AAA").map(_.nestedMap).getOrElse(HashMap())
res5: scala.collection.mutable.HashMap[String,InnerClass] = Map(aaa -> InnerClass(xyz,0))
if "AAA" didnt exist in the outerMap Map object then the below expression would have returned an empty HashMap as indicated in the .getOrElse method argument (HashMap()).
I have a list of String in Scala, each String has a key/value format as follows:
<row Id="25780063" PostTypeId="2" ParentId="25774527" CreationDate="2014-09-11T05:56:29.900" />
Each String may have some extra key/value. I'd like to extract the value for a few keys for each string. Here is the pattern I've defined but it is not working properly
val idPattern = "Id=(.*).r
val typePattern = "PostTypeId=(.*)".r
How can I correctly extract the value for 'Id' and 'PostTypeId'?
Making it unanchored says find instead of match all input.
scala> val id = """Id="([^"]*)"""".r.unanchored
id: scala.util.matching.UnanchoredRegex = Id="([^"]*)"
scala> """stuff Id="something" more""" match { case id(x) => x }
res7: String = something
scala> id.findFirstIn("""stuff Id="something" more""")
res8: Option[String] = Some(Id="something")
First you have to define the regex as valid stable identifiers.
val IdPattern = "Id=(.*).r
val TypePattern = "PostTypeId=(.*)".r
Note the initial uppercase, required for pattern matching (or use backquotes if really want it lowercased).
Then,
aString match {
case IdPattern(group) => println(s"id=$group")
case TypePattern(group) => println(s"type=$group")
}
Does exist any native function in scala that does the equivalent this?
def strConvert[T](v: Option[T]): String = {
if (v.isDefined)
v.get.toString
else
""
}
For generic T, you can avoid the if with map -- v.map(_.toString).getOrElse("")
scala> Some(1).fold("")(_.toString)
res0: String = 1
scala> None.fold("")(_.toString)
res1: String = ""
Option is a monad. In FP, when working with monads, you define what you want to happen if the monad contains a state (i.e. a value other than None). Otherwise, you move on.
val ostring: Option[String] = functionThatGetsOptionString()
ostring.map { s =>
functionThatUsesString(s)
}
Within the map function, s will contain the raw string if the Option is Some(String) otherwise it won't execute the inner body if Option is None.
I'm searching for a possiblity in scala to call a function and get an Option as result which is "None" iff either an Exception is raised when calling the method or the method return null. Otherwise the Option should have the value of the result.
I know that Try can be used for the first part, but I don't know how to handle the second part:
val result = Try(myFunction).toOption()
If the method now returns null (because it is not a scala function but a Java function), result is Some(null) instead of None.
As I know there is only 1 method in scala standard library to convert null to None - Option.apply(x), so you have to use it manually:
val result = Try(myFunction).toOption.flatMap{Option(_)}
// or
val result = Try(Option(myFunction)).toOption.flatten
You could create your own helper method like this:
implicit class NotNullOption[T](val t: Try[T]) extends AnyVal {
def toNotNullOption = t.toOption.flatMap{Option(_)}
}
scala> Try(null: String).toNotNullOption
res0: Option[String] = None
scala> Try("a").toNotNullOption
res1: Option[String] = Some(a)
You can also do this:
val result = Try(myFunction).toOption.filter(_ != null)
which looks and feels better then .flatten or .flatMap(Option(_))
You can also do pattern matching as:
val result = myFunction() match {
case null => None
case _ => Some(_)
}
but the answer of #senia looks more "scala style"