I have this scala code that involves java:
val message = "Our servers encountered an error" +
(if (ex.getMessage == null) "" else (": " + ex.getMessage))
What is the scala best way to write it?
In scala you do not want to deal with null, it is more convenient to use Option
"Our servers encountered an error while processing your request " +
Option(ex.getMessage).map(":"+_).getOrElse("")
Probably something along the lines of:
val exMessage = Option(ex.getMessage).fold("")(m =>s": $m")
val message = s"Our servers encountered an error$exMessage"
or in one line:
s"Our servers encountered an error${ Option(ex.getMessage).fold("")(m =>s": $m") }"
Though, not the most efficient way. Nor will it make your code any clearer.
EDIT:
This will (as well as the original) probably yield an undesired result if ex,getMessage returns an empty String.
Additionally, as has been mentioned, using Option in such a situation is a good way to obfuscate your code. So here is an example that should be more readable:
def errorMessage(ex: Throwable): String= {
val formattedMsg = ex.getMessage match {
case msg: String if msg.nonEmpty => s": $msg"
case _ => ""
}
s"Our servers encountered an error$formattedMsg"
}
Related
I'd like to move:
val kafkaPartitionOffset = kafkaConsumer.endOffsets(consumedPartitions.asJava)
into a try catch block like so:
val kafkaPartitionOffset : SomeClass =
try {
kafkaConsumer.endOffsets(consumedPartitions.asJava)
} catch {
case e: Exception => {
log.error(s"${consumerGroupId} Could not get Kafka offset", e)
None
}
}
But I'm having trouble on what the SomeClass should be. I've tried Map[TopicPartition, Long] but it says Type mismatch. Any help is appreciated, thank you!
Update: I've also tried Any but I'm unable to do a kafkaPartitionOffset.get(topicPartition) below (get is highlighted red with error message cannot resolve symbol get:
for((topicPartition,OffsetAndMetadata) <- mapTopicPartitionOffset){
val bbCurrentOffset = OffsetAndMetadata.get(topicPartition)
// latest offset
val partitionLatestOffset = kafkaPartitionOffset.get(topicPartition)
// Log for a particular partition
val delta = partitionLatestOffset - bbCurrentOffset
topicOffsetList += delta.abs
}
Take a look at this:
val x = try {
throw new RuntimeException("runtime ex")
"some string"
} catch { case _: RuntimeException => 2 }
The compiler needs to know the type of x before runtime, since x can be used somewhere else in your code, right? So the compiler says:
"Hmm, what is a type that this literal "some string" is of that type,
and also, literal 2 is of that type?"
So it looks for the lowest super-type of both String and Int, which is Any in this case! so val x: Any = .... Now I'm not aware of what this expression kafkaConsumer.endOffsets(...) returns, in case it returns Option[T], then SomeClass would also be Option[T] since you're returning None in the catch block, and if not, do not use None there just because nothing else would fit, there are better approaches to exception handling.
But anyways, Scala provides some utility types to avoid this kind of try catch as much as possible, I would recommend you to use Try in this case.
val kafkaPartitionOffset: Try[Whatever-endOffsets-returns] =
Try(kafkaConsumer.endOffsets(consumedPartitions.asJava))
By the way, the title of the question doesn't match the actual question, please consider changing the title :)
I am trying to learn Spark and Scala. I am working on a scenario to identify the lines that start with H or I. Below is my code
def startWithHorI(s:String):String=
{
if(s.startsWith("I")
return s
if(s.startsWith("H")
return s
}
val fileRDD=sc.textFile("wordcountsample.txt")
val checkRDD=fileRDD.map(startWithHorI)
checkRDD.collect
It is throwing an error while creating the function Found:Unit Required:Boolean.
From research I understood that it is not able to recognize the return as Unit means void. Could someone help me.
There are a few things wrong with your def, we will start there:
It is throwing the error because according to the code posted, your syntax is incomplete and the def is defined improperly:
def startWithHorI(s:String): String=
{
if(s.startsWith("I")) // missing extra paren char in original post
s // do not need return statement
if(s.startsWith("H")) // missing extra paren char in original post
s // do not need return statement
}
This will still return an error because we are expecting a String when the compiler sees that it's returning an Any. We cannot do this if we do not have an else case (what will be returned when s does not start with H or I?) - the compiler will see this as an Any return type. The correction for this would be to have an else condition that ultimately returns a String.
def startWithHorI(s: String): String = {
if(s.startsWith("I")) s else "no I"
if(s.startsWith("H")) s else "no H"
}
If you don't want to return anything, then an Option is worth looking at for a return type.
Finally we can achieve what you are doing via filter - no need to map with a def:
val fileRDD = sc.textFile("wordcountsample.txt")
val checkRDD = fileRDD.filter(s => s.startsWith("H") || s.startsWith("I"))
checkRDD.collect
While passing any function to rdd.map(fn) make sure that fn covers all possible scenarios.
If you want to completely avoid strings which does not start with either H or I then use flatMap and return Option[String] from your function.
Example:
def startWithHorI(s:String): Option[String]=
{
if(s.startsWith("I") || s.startsWith("H")) Some(s)
else None
}
Then,
sc.textFile("wordcountsample.txt").flatMap(startWithHorI)
This will remove all rows not starting with H or I.
In general, to minimize run-time errors try to create total functions which handles all possible values of the arguments.
Something like below would work for you?
val fileRDD=sc.textFile("wordcountsample.txt")
fileRDD.collect
Array[String] = Array("Hello ", Hello World, Instragram, Good Morning)
val filterRDD=fileRDD.filter( x=> (x(0) == 'H'||x(0) == 'I'))
filterRDD.collect()
Array[String] = Array("Hello ", Hello World, Instragram)
I've been learning/experimenting with Scala in the REPL.
By the way, and as a side note, I'm quite impressed so far. I'm finding the language beautiful.
The following happened and need an explanation of what is going on.
Thanks in advance for any help given.
At the REPL entered:
def withMarks(mark: String)(body: => Unit){
println(mark + " Init")
body
println(mark + " End")
}
val a = "Testing clojure with paremeter by name as control structure"
withMarks("***"){
println(a)
println("more expressions")
}
Everything worked as expected.
Than happened what I consider weird, out of ignorance I suspect. I entered some more stuff:
class FileAsIterable{
def iterator = scala.io.Source.fromFile("/Users/MacBookProRetina/Google Drive/NewControl.scala").getLines()
}
val newIterator = new FileAsIterable with Iterable[String]
When evaluating the last line the REPL prints:
newIterator: FileAsIterable with Iterable[String] = (def withMarks(mark: String)(body: => Unit){, println(mark + " Init"), body, println(mark + " End"), }, val a = "Hola Mundo", withMarks("***"){, println(a), })
I keep getting the same result even after restarting the terminal in the Mac, and running the scala REPL at different directory locations.
Don't know how the newIterator val got connected to the the withMarks def.
Nevermind. I was just confused by the contents of the file
Folks,
I've got the following map and structural type defined:
type Mergeable = { def mergeFrom(data: Array[Byte]): com.google.protobuf.GeneratedMessageLite }
val dispatchMap = Map(
1 -> (ZombieSighting.defaultInstance.asInstanceOf[Mergeable], "akka://UlyssesAgenda/user/ServerCore/DispatchTarget")
)
Basically what I'm doing is defining a map that says "when I read protobuf message type 1 off the wire, create a ZombieSighting from the bytes, and dispatch that to the actor found at the indicated string".
So, this is the code I have today that creates a message, an actor, and sends the message to the actor:
val dispatchInfo = dispatchMap.get(messageType)
val theMessage = dispatchInfo.map { _._1.mergeFrom(theBytes) }.get
val actorPath = dispatchInfo.map { _._2 }
val targetActor = actorPath.map { SocketServer.system.actorFor(_) }
targetActor.map { _ ! theMessage }
When I look at this, all I can think of is how many lines of non-functional programming this looks like and I can't help but think there's a much more elegant way to do this. I know I can write functions that take tuples as parameters and return modified tuples, but I don't think that buys me anything in terms of clean idiomatic scala here.
My gut instinct says there's a single map statement I can run on "dispatchMap.get(messageType)" that will give me an instance of an Akka actor based on tuple._2 and a merged protobuf object based on tuple._1.
Can someone suggest a way to pimp this code and reduce the number of lines and make it more functional? e.g.
val (targetActor, theMessage) = ????
targetActor ! theMessage
Edit: here's my first attempt at a refactor.. Is this the "Scala way" to do it?
val (msg, actor) = dispatchMap.get(messageType).map {
case (m:Mergeable, path:String) =>
(m.mergeFrom(theBytes), SocketServer.system.actorFor(path) )
}.get
actor ! msg
Edit 2: Here's a version suggested by the commenter below, but I don't know if this is really idiomatic or not:
def dispatchMessage: IO.Iteratee[Unit] =
repeat {
for {
(messageType, byteString) <- readMessage
} yield {
for (
(protoMessage, actorPath) <- dispatchMap.get(messageType);
theMessage = protoMessage.mergeFrom(byteString.toArray);
targetActor = SocketServer.system.actorFor(actorPath) )
yield
targetActor ! theMessage
}
}
?
Your code might be written like this:
val akkaResponseFuture =
for ((bytes, actorPath) <- dispatchMap.get(messageType);
theMessage <- bytes.mergeFrom(theBytes);
targetActor = SocketServer.system.actorFor(actorPath))
yield
targetActor ! theMessage
This may very well not be right. I did it without trying to create working code, since your sample is not self-contained. Also, theBytes is nowhere defined.
But I'm pretty sure you can use a for comprehension to clarify and streamline it.
I want to filter out empty strings to put them into an Option. So I quickly built this now:
def StrictOption(s: String) = s match {
case s if s != null && s.trim.length() > 0 => Some(s)
case _ => None
}
Question: is this maybe already somewhere in the standard library?
I don't think there's one single method in the standard library to do this, but you can do this much more tersely than your implementation.
Option(s).filter(_.trim.nonEmpty)
If you care at all about performance then
if (s.trim.isEmpty) None else Some(s)
is only 4 characters longer than Ben James's solution, and runs 3 times faster, in my benchmark (47 vs 141).
Starting Scala 2.13, for those not expecting nulls (non-Java context), Option::unless and Option::when are now an alternative option:
// val str = "hello"
Option.unless(str.isEmpty)(str)
// Option[String] = Some(hello)
Option.when(str.nonEmpty)(str)
// Option[String] = Some(hello)
// val str: String = ""
Option.unless(str.isEmpty)(str)
// Option[String] = None
Option.when(str.nonEmpty)(str)
// Option[String] = None
There's nothing built in; Ben's filter is the best brief version if performance isn't an issue (e.g. you're validating user input). Typically, performance will not be an issue.
Also, note that it's a little strange to use match when you're not actually matching anything; it's just more boilerplate to get an if-else statement. Just say
if (s ne null && s.trim.length > 0) Some(s) else None
which is about as fast and brief as anything, unless you want to write your own is-it-whitespace method. Note that trim uses a peculiar criterion: anything above \u0020 (i.e. ' ') is not trimmed, and anything equal or below is. So you could also write your own trimmed-string-is-empty detector, if performance of this operation was particularly important:
def ContentOption(s: String): Option[String] = {
if (s ne null) {
var i = s.length-1
while (i >= 0) {
if (s.charAt(i) > ' ') return Some(s)
i -= 1
}
}
None
}
This could also be achieved with a for-comprehension
val res = for (v <- Option(s) if s.nonEmpty) yield v
Option("something") produces Some("something")
Option(null) produces None