Scala alternative to series of if statements that append to a list? - scala

I have a Seq[String] in Scala, and if the Seq contains certain Strings, I append a relevant message to another list.
Is there a more 'scalaesque' way to do this, rather than a series of if statements appending to a list like I have below?
val result = new ListBuffer[Err]()
val malformedParamNames = // A Seq[String]
if (malformedParamNames.contains("$top")) result += IntegerMustBePositive("$top")
if (malformedParamNames.contains("$skip")) result += IntegerMustBePositive("$skip")
if (malformedParamNames.contains("modifiedDate")) result += FormatInvalid("modifiedDate", "yyyy-MM-dd")
...
result.toList

If you want to use some scala iterables sugar I would use
sealed trait Err
case class IntegerMustBePositive(msg: String) extends Err
case class FormatInvalid(msg: String, format: String) extends Err
val malformedParamNames = Seq[String]("$top", "aa", "$skip", "ccc", "ddd", "modifiedDate")
val result = malformedParamNames.map { v =>
v match {
case "$top" => Some(IntegerMustBePositive("$top"))
case "$skip" => Some(IntegerMustBePositive("$skip"))
case "modifiedDate" => Some(FormatInvalid("modifiedDate", "yyyy-MM-dd"))
case _ => None
}
}.flatten
result.toList
Be warn if you ask for scala-esque way of doing things there are many possibilities.
The map function combined with flatten can be simplified by using flatmap
sealed trait Err
case class IntegerMustBePositive(msg: String) extends Err
case class FormatInvalid(msg: String, format: String) extends Err
val malformedParamNames = Seq[String]("$top", "aa", "$skip", "ccc", "ddd", "modifiedDate")
val result = malformedParamNames.flatMap {
case "$top" => Some(IntegerMustBePositive("$top"))
case "$skip" => Some(IntegerMustBePositive("$skip"))
case "modifiedDate" => Some(FormatInvalid("modifiedDate", "yyyy-MM-dd"))
case _ => None
}
result

Most 'scalesque' version I can think of while keeping it readable would be:
val map = scala.collection.immutable.ListMap(
"$top" -> IntegerMustBePositive("$top"),
"$skip" -> IntegerMustBePositive("$skip"),
"modifiedDate" -> FormatInvalid("modifiedDate", "yyyy-MM-dd"))
val result = for {
(k,v) <- map
if malformedParamNames contains k
} yield v
//or
val result2 = map.filterKeys(malformedParamNames.contains).values.toList

Benoit's is probably the most scala-esque way of doing it, but depending on who's going to be reading the code later, you might want a different approach.
// Some type definitions omitted
val malformations = Seq[(String, Err)](
("$top", IntegerMustBePositive("$top")),
("$skip", IntegerMustBePositive("$skip")),
("modifiedDate", FormatInvalid("modifiedDate", "yyyy-MM-dd")
)
If you need a list and the order is siginificant:
val result = (malformations.foldLeft(List.empty[Err]) { (acc, pair) =>
if (malformedParamNames.contains(pair._1)) {
pair._2 ++: acc // prepend to list for faster performance
} else acc
}).reverse // and reverse since we were prepending
If the order isn't significant (although if the order's not significant, you might consider wanting a Set instead of a List):
val result = (malformations.foldLeft(Set.empty[Err]) { (acc, pair) =>
if (malformedParamNames.contains(pair._1)) {
acc ++ pair._2
} else acc
}).toList // omit the .toList if you're OK with just a Set
If the predicates in the repeated ifs are more complex/less uniform, then the type for malformations might need to change, as they would if the responses changed, but the basic pattern is very flexible.

In this solution we define a list of mappings that take your IF condition and THEN statement in pairs and we iterate over the inputted list and apply the changes where they match.
// IF THEN
case class Operation(matcher :String, action :String)
def processInput(input :List[String]) :List[String] = {
val operations = List(
Operation("$top", "integer must be positive"),
Operation("$skip", "skip value"),
Operation("$modify", "modify the date")
)
input.flatMap { in =>
operations.find(_.matcher == in).map { _.action }
}
}
println(processInput(List("$skip","$modify", "$skip")));
A breakdown
operations.find(_.matcher == in) // find an operation in our
// list matching the input we are
// checking. Returns Some or None
.map { _.action } // if some, replace input with action
// if none, do nothing
input.flatMap { in => // inputs are processed, converted
// to some(action) or none and the
// flatten removes the some/none
// returning just the strings.

Related

Exception handling in Spark Scala UDF

def parse_values(value: String) = {
val values = value.split(",").map(_.trim)
values.foldLeft(Array[(Int, Double)]()) {
case (acc, present) =>
val Array(k, v) = present.split(",")(0).split(":")
acc :+ (k.trim.toInt, v.trim.toDouble)
}
I am currently using the above UDF to parse a column of string into an array of keys and values.
"50:63.25,100:58.38" to [[50,63.2], [100,58.38]].
In some cases, the string is "\N" and I am unable to parse the column value.
If the string is "\N" then I should return an empty array. Can anyone help me to handle this exception or help me adding a new case? I am new to spark-scala.
Error: scala.MatchError: [Ljava.lang.String;#497cb6a9 (of class [Ljava.lang.String;)
You need to check that the resulting Array has two elements. You need a pattern matching like this to avoid that parse error:
def parse_values(value: String) = {
val values = value.split(",").map(_.trim)
values.foldLeft(Array[(Int, Double)]()) {
case (acc, present) =>
val Array(k, v) = {
present.split(",")(0).split(":") match {
case Array(_) => Array("0", "0.0")
case arr => arr
}
}
acc :+ (k.trim.toInt, v.trim.toDouble)
}
}

Scala: How to add match vals to a list val

I have a few vals that match for matching values
Here is an example:
val job_ = Try(jobId.toInt) match {
case Success(value) => jobs.findById(value).map(_.id)
.getOrElse( Left(WrongValue("jobId", s"$value is not a valid job id")))
case Failure(_) => jobs.findByName(jobId.toString).map(_.id)
.getOrElse( Left(WrongValue("jobId", s"'$jobId' is not a known job title.")))
}
// Here the value arrives as a string e.i "yes || no || true || or false" then converted to a boolean
val bool_ = bool.toLowerCase() match {
case "yes" => true
case "no" => false
case "true" => true
case "false" => false
case other => Left(Invalid("bool", s"wrong value received"))
}
Note: invalid case is case class Invalid(x: String, xx: String)
above i'm looking for a given job value and checking whether it exist in the db or not,
No I have a few of these and want to add to a list, here is my list val and flatten it:
val errors = List(..all my vals errors...).flatten // <--- my_list_val (how do I include val bool_ and val job_)
if (errors.isEmpty) { do stuff }
My result should contain errors from val bool_ and val job_
THANK!
You need to fix the types first. The type of bool_ is Any. Which does not give you something you can work with.
If you want to use Either, you need to use it everwhere.
Then, the easiest approach would be to use a for comprehension (I am assuming you're dealing with Either[F, T] here, where WrongValue and Invalid are both sub-classes of F and you're not really interested in the errors).
for {
foundJob <- job_
_ <- bool_
} yield {
// do stuff
}
Note, that in Scala >= 2.13 you can use toIntOption when converting the String to Int:
vaj job_: Either[F, T] = jobId.toIntOption match {
case Some(value) => ...
case _ => ...
}
Also, in case expressions, you can use alternatives when you have the same statement for several cases:
val bool_: Either[F, Boolean] = bool.toLowerCase() match {
case "yes" | "true" => Right(true)
case "no" | "false" => Right(false)
case other => Left(Invalid("bool", "wrong value received"))
}
So, according to your question, and your comments, these are the types you're dealing with.
type ID = Long //whatever id is
def WrongValue(x: String, xx: String) :String = "?-?-?"
case class Invalid(x: String, xx: String)
Now let's create a couple of error values.
val job_ :Either[String,ID] = Left(WrongValue("x","xx"))
val bool_ :Either[Invalid,Boolean] = Left(Invalid("x","xx"))
To combine and report them you might do something like this.
val errors :List[String] =
List(job_, bool_).flatMap(_.swap.toOption.map(_.toString))
println(errors.mkString(" & "))
//?-?-? & Invalid(x,xx)
After checking types as #cbley explained. You can just do a filter operation with pattern matching on your list:
val error = List(// your variables ).filter(_ match{
case Left(_) => true
case _ => false
})

Build conditionally list avoiding mutations

Let's say I want to build list of Pizza's ingredients conditionally:
val ingredients = scala.collection.mutable.ArrayBuffer("tomatoes", "cheese")
if (!isVegetarian()) {
ingredients += "Pepperoni"
}
if (shouldBeSpicy()) {
ingredients += "Jalapeno"
}
//etc
Is there functional way to build this array using immutable collections?
I thought about:
val ingredients = List("tomatoes", "cheese") ++ List(
if (!isVegetarian()) Some("Pepperoni") else None,
if (shouldBeSpicy()) Some("Jalapeno") else None
).flatten
but is there better way?
Here is another possible way that is closer to #Antot but IMHO is much simpler.
What is unclear in your original code is where isVegetarian and shouldBeSpicy actually come from. Here I assume that there is a PizzaConf class as following to provide those configuration settings
case class PizzaConf(isVegetarian: Boolean, shouldBeSpicy: Boolean)
Assuming this, I think the simplest way is to have a allIngredients of List[(String, Function1[PizzaConf, Boolean])] type i.e. one that stores ingredients and functions to check their corresponding availability. Given that buildIngredients becomes trivial:
val allIngredients: List[(String, Function1[PizzaConf, Boolean])] = List(
("Pepperoni", conf => conf.isVegetarian),
("Jalapeno", conf => conf.shouldBeSpicy)
)
def buildIngredients(pizzaConf: PizzaConf): List[String] = {
allIngredients
.filter(_._2(pizzaConf))
.map(_._1)
}
or you can merge filter and map using collect as in following:
def buildIngredients(pizzaConf: PizzaConf): List[String] =
allIngredients.collect({ case (ing, cond) if cond(pizzaConf) => ing })
Your original approach is not bad. I would probably just stick with list:
val ingredients =
List("tomatoes", "cheese") ++
List("Pepperoni", "Sausage").filter(_ => !isVegetarian) ++
List("Jalapeno").filter(_ => shouldBeSpicy)
Which makes it easy to add more ingredients connected to a condition (see "Sausage" above)
You could start with the full list of ingredients and then filter out the ingredients not passing the conditions:
Set("tomatoes", "cheese", "Pepperoni", "Jalapeno")
.filter {
case "Pepperoni" => !isVegetarian;
case "Jalapeno" => shouldBeSpicy;
case _ => true // ingredients by default
}
which for:
val isVegetarian = true
val shouldBeSpicy = true
would return:
Set(tomatoes, cheese, Jalapeno)
This can be achieved by creating a sequence of predicates, which defines the conditions applied to filter the ingredients.
// available ingredients
val ingredients = Seq("tomatoes", "cheese", "ham", "mushrooms", "pepper", "salt")
// predicates
def isVegetarian(ingredient: String): Boolean = ingredient != "ham"
def isSpicy(ingredient: String): Boolean = ingredient == "pepper"
def isSalty(ingredient: String): Boolean = ingredient == "salt"
// to negate another predicate
def not(predicate: (String) => Boolean)(ingr: String): Boolean = !predicate(ingr)
// sequences of conditions for different pizzas:
val vegeterianSpicyPizza: Seq[(String) => Boolean] = Seq(isSpicy, isVegetarian)
val carnivoreSaltyNoSpices: Seq[(String) => Boolean] = Seq(not(isSpicy), isSalty)
// main function: builds a list of ingredients for specified conditions!
def buildIngredients(recipe: Seq[(String) => Boolean]): Seq[String] = {
ingredients.filter(ingredient => recipe.exists(_(ingredient)))
}
println("veg spicy: " + buildIngredients(vegeterianSpicyPizza))
// veg spicy: List(tomatoes, cheese, mushrooms, pepper, salt)
println("carn salty: " + buildIngredients(carnivoreSaltyNoSpices))
// carn salty: List(tomatoes, cheese, ham, mushrooms, salt)
Inspired by other answers, I came up with something like this:
case class If[T](conditions: (Boolean, T)*) {
def andAlways(values: T*): List[T] =
conditions.filter(_._1).map(_._2).toList ++ values
}
It could be used like:
val isVegetarian = false
val shouldBeSpicy = true
val ingredients = If(
!isVegetarian -> "Pepperoni",
shouldBeSpicy -> "Jalapeno",
).andAlways(
"Cheese",
"Tomatoes"
)
Still waiting for a better option :)
If any ingredient will only need testing against one condition, you could do something like this:
val commonIngredients = List("Cheese", "Tomatoes")
val nonVegetarianIngredientsWanted = {
if (!isVegetarian)
List("Pepperoni")
else
List.empty
}
val spicyIngredientsWanted = {
if (shouldBeSpicy)
List("Jalapeno")
else
List.empty
}
val pizzaIngredients = commonIngredients ++ nonVegetarianIngredientsWanted ++ spicyIngredientsWanted
This doesn't work if you have ingredients which are tested in two categories: for example if you have spicy sausage then that should only be included if !isVegetarian and spicyIngredientsWanted. One method of doing this would be to test both conditions together:
val (optionalIngredients) = {
(nonVegetarianIngredientsWanted, spicyIngredientsWanted) match {
case (false, false) => List.empty
case (false, true) => List("Jalapeno")
case (true, false) => List("Pepperoni")
case (true, true) => List("Pepperoni, Jalapeno, Spicy Sausage")
}
val pizzaIngredients = commonIngredients ++ optionalIngredients
This can be extended to test any number of conditions, though of course the number of case arms needed extends exponentially with the number of conditions tested.

Scala: dynamically joining data frames

I have data split into multiple files. I want to load and join the files.
I'd like to build a dynamic function that
1. will join n data files into a single data frame
2. given the input of file location and join column (e.g., pk)
I think this can be done with foldLeft, but I am not quite sure how:
Here is my code so far:
#throws
def dataJoin(path:String, fileNames:String*): DataFrame=
{
try
{
val dfList:ArrayBuffer[DataFrame]=new ArrayBuffer
for(fileName <- fileNames)
{
val df:DataFrame=DataFrameUtils.openFile(spark, s"$path${File.separator}$fileName")
dfList += df
}
dfList.foldLeft
{
(df,df1) => joinDataFrames(df,df1, "UID")
}
}
catch
{
case e:Exception => throw new Exception(e)
}
}
def joinDataFrames(df:DataFrame,df1:DataFrame, joinColum:String): Unit =
{
df.join(df1, Seq(joinColum))
}
foldLeft might indeed be suitable here, but it requires a "zero" element to start the folding from (in addition to the folding function). In this case, that "zero" can be the first DataFrame:
dfList.tail.foldLeft(dfList.head) { (df1, df2) => df1.join(df2, "UID") }
To avoid errors, you probably want to make sure the list isn't empty before trying to access that first item - one way of doing that would be using pattern matching.
dfList match {
case head :: tail => tail.foldLeft(head) { (df1, df2) => df1.join(df2, "UID") }
case Nil => spark.emptyDataFrame
}
Lastly, it's simpler, safer and more idiomatic to map over a collection instead of iterating over it and populated another (empty, mutable) collection:
val dfList = fileNames.map(fileName => DataFrameUtils.openFile(spark, s"$path${File.separator}$fileName"))
Altogether:
def dataJoin(path:String, fileNames: String*): DataFrame = {
val dfList = fileNames
.map(fileName => DataFrameUtils.openFile(spark, s"$path${File.separator}$fileName"))
.toList
dfList match {
case head :: tail => tail.foldLeft(head) { (df1, df2) => df1.join(df2, "UID") }
case Nil => spark.emptyDataFrame
}
}

Scala: how to traverse stream/iterator collecting results into several different collections

I'm going through log file that is too big to fit into memory and collecting 2 type of expressions, what is better functional alternative to my iterative snippet below?
def streamData(file: File, errorPat: Regex, loginPat: Regex): List[(String, String)]={
val lines : Iterator[String] = io.Source.fromFile(file).getLines()
val logins: mutable.Map[String, String] = new mutable.HashMap[String, String]()
val errors: mutable.ListBuffer[(String, String)] = mutable.ListBuffer.empty
for (line <- lines){
line match {
case errorPat(date,ip)=> errors.append((ip,date))
case loginPat(date,user,ip,id) =>logins.put(ip, id)
case _ => ""
}
}
errors.toList.map(line => (logins.getOrElse(line._1,"none") + " " + line._1,line._2))
}
Here is a possible solution:
def streamData(file: File, errorPat: Regex, loginPat: Regex): List[(String,String)] = {
val lines = Source.fromFile(file).getLines
val (err, log) = lines.collect {
case errorPat(inf, ip) => (Some((ip, inf)), None)
case loginPat(_, _, ip, id) => (None, Some((ip, id)))
}.toList.unzip
val ip2id = log.flatten.toMap
err.collect{ case Some((ip,inf)) => (ip2id.getOrElse(ip,"none") + "" + ip, inf) }
}
Corrections:
1) removed unnecessary types declarations
2) tuple deconstruction instead of ulgy ._1
3) left fold instead of mutable accumulators
4) used more convenient operator-like methods :+ and +
def streamData(file: File, errorPat: Regex, loginPat: Regex): List[(String, String)] = {
val lines = io.Source.fromFile(file).getLines()
val (logins, errors) =
((Map.empty[String, String], Seq.empty[(String, String)]) /: lines) {
case ((loginsAcc, errorsAcc), next) =>
next match {
case errorPat(date, ip) => (loginsAcc, errorsAcc :+ (ip -> date))
case loginPat(date, user, ip, id) => (loginsAcc + (ip -> id) , errorsAcc)
case _ => (loginsAcc, errorsAcc)
}
}
// more concise equivalent for
// errors.toList.map { case (ip, date) => (logins.getOrElse(ip, "none") + " " + ip) -> date }
for ((ip, date) <- errors.toList)
yield (logins.getOrElse(ip, "none") + " " + ip) -> date
}
I have a few suggestions:
Instead of a pair/tuple, it's often better to use your own class. It gives meaningful names to both the type and its fields, which makes the code much more readable.
Split the code into small parts. In particular, try to decouple pieces of code that don't need to be tied together. This makes your code easier to understand, more robust, less prone to errors and easier to test. In your case it'd be good to separate producing your input (lines of a log file) and consuming it to produce a result. For example, you'd be able to make automatic tests for your function without having to store sample data in a file.
As an example and exercise, I tried to make a solution based on Scalaz iteratees. It's a bit longer (includes some auxiliary code for IteratorEnumerator) and perhaps it's a bit overkill for the task, but perhaps someone will find it helpful.
import java.io._;
import scala.util.matching.Regex
import scalaz._
import scalaz.IterV._
object MyApp extends App {
// A type for the result. Having names keeps things
// clearer and shorter.
type LogResult = List[(String,String)]
// Represents a state of our computation. Not only it
// gives a name to the data, we can also put here
// functions that modify the state. This nicely
// separates what we're computing and how.
sealed case class State(
logins: Map[String,String],
errors: Seq[(String,String)]
) {
def this() = {
this(Map.empty[String,String], Seq.empty[(String,String)])
}
def addError(date: String, ip: String): State =
State(logins, errors :+ (ip -> date));
def addLogin(ip: String, id: String): State =
State(logins + (ip -> id), errors);
// Produce the final result from accumulated data.
def result: LogResult =
for ((ip, date) <- errors.toList)
yield (logins.getOrElse(ip, "none") + " " + ip) -> date
}
// An iteratee that consumes lines of our input. Based
// on the given regular expressions, it produces an
// iteratee that parses the input and uses State to
// compute the result.
def logIteratee(errorPat: Regex, loginPat: Regex):
IterV[String,List[(String,String)]] = {
// Consumes a signle line.
def consume(line: String, state: State): State =
line match {
case errorPat(date, ip) => state.addError(date, ip);
case loginPat(date, user, ip, id) => state.addLogin(ip, id);
case _ => state
}
// The core of the iteratee. Every time we consume a
// line, we update our state. When done, compute the
// final result.
def step(state: State)(s: Input[String]): IterV[String, LogResult] =
s(el = line => Cont(step(consume(line, state))),
empty = Cont(step(state)),
eof = Done(state.result, EOF[String]))
// Return the iterate waiting for its first input.
Cont(step(new State()));
}
// Converts an iterator into an enumerator. This
// should be more likely moved to Scalaz.
// Adapted from scalaz.ExampleIteratee
implicit val IteratorEnumerator = new Enumerator[Iterator] {
#annotation.tailrec def apply[E, A](e: Iterator[E], i: IterV[E, A]): IterV[E, A] = {
val next: Option[(Iterator[E], IterV[E, A])] =
if (e.hasNext) {
val x = e.next();
i.fold(done = (_, _) => None, cont = k => Some((e, k(El(x)))))
} else
None;
next match {
case None => i
case Some((es, is)) => apply(es, is)
}
}
}
// main ---------------------------------------------------
{
// Read a file as an iterator of lines:
// val lines: Iterator[String] =
// io.Source.fromFile("test.log").getLines();
// Create our testing iterator:
val lines: Iterator[String] = Seq(
"Error: 2012/03 1.2.3.4",
"Login: 2012/03 user 1.2.3.4 Joe",
"Error: 2012/03 1.2.3.5",
"Error: 2012/04 1.2.3.4"
).iterator;
// Create an iteratee.
val iter = logIteratee("Error: (\\S+) (\\S+)".r,
"Login: (\\S+) (\\S+) (\\S+) (\\S+)".r);
// Run the the iteratee against the input
// (the enumerator is implicit)
println(iter(lines).run);
}
}