I want to chain a bunch of filters but do not want the overhead associated with creating multiple lists.
type StringFilter = (String) => Boolean
def nameFilter(value: String): StringFilter =
(s: String) => s == value
def lengthFilter(length: Int): StringFilter =
(s: String) => s.length == length
val list = List("Apple", "Orange")
Problem is this builds a list after each filter:
list.filter(nameFilter("Apples")).filter(lengthFilter(5))
// list of string -> list of name filtered string -> list of name and length filtered string
I want:
// list of string -> list of name and length filtered string
I find out which filters are needed at run-time so I must add filters dynamically.
// Not sure how to implement add function.
val filterPipe: StringFilter = ???
// My preferred DSL (or very close to it)
filterPipe.add(nameFilter("Apples")
filterPipe.add(lengthFilter(5))
// Must have DSL
list.filter(filterPipe)
How can I implement filterPipe?
Is there some way to recursively AND the filter conditions together in a filterPipe (which is itself a StringFilter)?
You can use withFilter:
list.withFilter(nameFilter("Apples")).withFilter(lengthFilter(5))...
A blog post suggest another alternative using an implicit class to allow aggregating multiple predicates using custom operators
implicit class Predicate[A](val pred: A => Boolean) {
def apply(x: A) = pred(x)
def &&(that: A => Boolean) = new Predicate[A](x => pred(x) && that(x))
def ||(that: A => Boolean) = new Predicate[A](x => pred(x) || that(x))
def unary_! = new Predicate[A](x => !pred(x))
}
Then you can apply the predicate chain as follows
list.filter { (nameFilter("Apple") && lengthFilter(5)) (_) }
You can also chain the predicates dynamically
val list = List("Apple", "Orange", "Meat")
val isFruit = nameFilter("Apple") || nameFilter("Orange")
val isShort = lengthFilter(5)
list.filter { (isFruit && isShort) (_) }
As you can see the benefit of this approach compared to the withFilter approach is that you can combine the predicates arbitrarily
Consider also a view on the filters, like this,
list.view.filter(nameFilter("Apples")).filter(lengthFilter(5))
This prevents intermediate collections, namely for each entry in list it applies the subsequent filters.
Related
I'm fresh with scala and udf, now I would like to write a udf which accept 3 parameters from a dataframe columns(one of them is array), for..loop current array, parse and return a case class which will be used afterwards. here's a my code roughly:
case class NewFeatures(dd: Boolean, zz: String)
val resultUdf = udf((arrays: Option[Row], jsonData: String, placement: Int) => {
for (item <- arrays) {
val aa = item.getAs[Long]("aa")
val bb = item.getAs[Long]("bb")
breakable {
if (aa <= 0 || bb <= 0) break
}
val cc = item.getAs[Long]("cc")
val dd = cc > 0
val jsonData = item.getAs[String]("json_data")
val jsonDataObject = JSON.parseFull(jsonData).asInstanceOf[Map[String, Any]]
var zz = jsonDataObject.getOrElse("zz", "").toString
NewFeatures(dd, zz)
}
})
when I run it, it will get exception:
java.lang.UnsupportedOperationException: Schema for type Unit is not supported
how should I modify above udf
First of all, try better naming for your variables, for instance in your case, "arrays" is of type Option[Row]. Here, for (item <- arrays) {...} is basically a .map method, using map on Options, you should provide a function, that uses Row and returns a value of some type (~= signature: def map[V](f: Row => V): Option[V], what you want in your case: def map(f: Row => NewFeatures): Option[NewFeature]). While you're breaking out of this map in some circumstances, so there's no assurance for the compiler that the function inside map method would always return an instance of NewFeatures. So it is Unit (it only returns on some cases, and not all).
What you want to do could be enhanced in something similar to this:
val funcName: (Option[Row], String, Int) => Option[NewFeatures] =
(rowOpt, jsonData, placement) => rowOpt.filter(
/* your break condition */
).map { row => // if passes the filter predicate =>
// fetch data from row, create new instance
}
I have 2 Lists: lista and listb. For each element in lista, I want to check if a_type of each element is in b_type of listb. If true, get the b_name for corresponding b_type and construct an object objc. And, then I should return the list of of constructed objc.
Is there a way to do this in Scala and preferably without any mutable collections?
case class obja = (a_id: String, a_type: String)
case class objb = (b_id: String, b_type: String, b_name: String)
case class objc = (c_id: String, c_type: String, c_name: String)
val lista: List[obja] = List(...)
val listb: List[objb] = List(...)
def getNames(alist: List[obja], blist: List[objb]): List[objc] = ???
Lookup in lists requires traversal in O(n) time, this is inefficient. Therefore, the first thing you do is to create a map from b_type to b_name:
val bTypeToBname = listb.map(b => (b.b_type, b_name)).toMap
Then you iterate through lista, look up in the map whether there is a corresponding b_name for a given a.a_type, and construct the objc:
val cs = for {
a <- lista
b_name <- bTypeToBname.get(a.a_type)
} yield objc(a.a_id, a.a_type, b_name)
Notice how Scala for-comprehensions automatically filter those cases for which bTypeToBname(a.a_type) isn't defined: then the corresponding a is simply skipped. This because we use bTypeToBname.get(a.a_type) (which returns an Option), as opposed to calling bTypeToBname(a.a_type) directly (this would lead to a NoSuchElementException). As far as I understand, this filtering is exactly the behavior you wanted.
case class A(aId: String, aType: String)
case class B(bId: String, bType: String, bName: String)
case class C(cId: String, cType: String, cName: String)
def getNames(aList: List[A], bList: List[B]): List[C] = {
val bMap: Map[String, B] = bList.map(b => b.bType -> b)(collection.breakOut)
aList.flatMap(a => bMap.get(a.aType).map(b => C(a.aId, a.aType, b.bName)))
}
Same as Andrey's answer but without comprehension so you can see what's happening inside.
// make listb into a map from type to name for efficiency
val bs = listb.map(b => b.b_type -> b_name).toMap
val listc: Seq[objc] = lista
.flatMap(a => // flatmap to exclude types not in listb
bs.get(a.a_type) // get an option from blist
.map(bName => objc(a.a_id, a.a_type, bName)) // if there is a b name for that type, make an objc
)
Is it possible (or even worthwhile) to try to write the below code block without a var? It works with a var. This is not for an interview, it's my first attempt at scala (came from java).
The problem: Fit people as close to the front of a theatre as possible, while keeping each request (eg. Jones, 4 tickets) in a single theatre section. The theatre sections, starting at the front, are sized 6, 6, 3, 5, 5... and so on. I'm trying to accomplish this by putting together all of the potential groups of ticket requests, and then choosing the best fitting group per section.
Here are the classes. A SeatingCombination is one possible combination of SeatingRequest (just the IDs) and the sum of their ticketCount(s):
class SeatingCombination(val idList: List[Int], val seatCount: Int){}
class SeatingRequest(val id: Int, val partyName: String, val ticketCount: Int){}
class TheatreSection(val sectionSize: Int, rowNumber: Int, sectionNumber: Int) {
def id: String = rowNumber.toString + "_"+ sectionNumber.toString;
}
By the time we get to the below function...
1.) all of the possible combinations of SeatingRequest are in a list of SeatingCombination and ordered by descending size.
2.) all of the TheatreSection are listed in order.
def getSeatingMap(groups: List[SeatingCombination], sections: List[TheatreSection]): HashMap[Int, TheatreSection] = {
var seatedMap = new HashMap[Int, TheatreSection]
for (sect <- sections) {
val bestFitOpt = groups.find(g => { g.seatCount <= sect.sectionSize && !isAnyListIdInMap(seatedMap, g.idList) })
bestFitOpt.filter(_.idList.size > 0).foreach(_.idList.foreach(seatedMap.update(_, sect)))
}
seatedMap
}
def isAnyListIdInMap(map: HashMap[Int, TheatreSection], list: List[Int]): Boolean = {
(for (id <- list) yield !map.get(id).isEmpty).reduce(_ || _)
}
I wrote the rest of the program without a var, but in this iterative section it seems impossible. Maybe with my implementation strategy it's impossible. From what else I've read, a var in a pure function is still functional. But it's been bothering me I can't think of how to remove the var, because my textbook told me to try to avoid them, and I don't know what I don't know.
You can use foldLeft to iterate on sections with a running state (and again, inside, on your state to add iteratively all the ids in a section):
sections.foldLeft(Map.empty[Int, TheatreSection]){
case (seatedMap, sect) =>
val bestFitOpt = groups.find(g => g.seatCount <= sect.sectionSize && !isAnyListIdInMap(seatedMap, g.idList))
bestFitOpt.
filter(_.idList.size > 0).toList. //convert option to list
flatMap(_.idList). // flatten list from option and idList
foldLeft(seatedMap)(_ + (_ -> sect))) // add all ids to the map with sect as value
}
By the way, you can simplify the second method using exists and map.contains:
def isAnyListIdInMap(map: HashMap[Int, TheatreSection], list: List[Int]): Boolean = {
list.exists(id => map.contains(id))
}
list.exists(predicate: Int => Boolean) is a Boolean which is true if the predicate is true for any element in list.
map.contains(key) checks if map is defined at key.
If you want to be even more concise, you don't need to give a name to the argument of the predicate:
list.exists(map.contains)
Simply changing var to val should do it :)
I think, you may be asking about getting rid of the mutable map, not of the var (it doesn't need to be var in your code).
Things like this are usually written recursively in scala or using foldLeft, like other answers suggest. Here is a recursive version:
#tailrec
def getSeatingMap(
groups: List[SeatingCombination],
sections: List[TheatreSection],
result: Map[Int, TheatreSection] = Map.empty): Map[Int, TheatreSection] = sections match {
case Nil => result
case head :: tail =>
val seated = groups
.iterator
.filter(_.idList.nonEmpty)
.filterNot(_.idList.find(result.contains).isDefined)
.find(_.seatCount <= head.sectionSize)
.fold(Nil)(_.idList.map(id => id -> sect))
getSeatingMap(groups, tail, result ++ seated)
}
btw, I don't think you need to test every id in list for presence in the map - should suffice to just look at the first one. You could also make it a bit more efficient, probably, if instead of checking the map every time to see if the group is already seated, you'd just drop it from the input list as soon as the section is assigned.
#tailrec
def selectGroup(
sect: TheatreSection,
groups: List[SeatingCombination],
result: List[SeatingCombination] = Nil
): (List[(Int, TheatreSection)], List[SeatingCombination]) = groups match {
case Nil => (Nil, result)
case head :: tail
if(head.idList.nonEmpty && head.seatCount <= sect.sectionSize) => (head.idList.map(_ -> sect), result.reverse ++ tail)
case head :: tail => selectGroup(sect, tail, head :: result)
}
and then in getSeatingMap:
...
case head :: tail =>
val(seated, remaining) => selectGroup(sect, groups)
getSeatingMap(remaining, tail, result ++ seated)
Here is how I was able to achieve without using the mutable.HashMap, the suggestion by the comment to use foldLeft was used to do it:
class SeatingCombination(val idList: List[Int], val seatCount: Int){}
class SeatingRequest(val id: Int, val partyName: String, val ticketCount: Int){}
class TheatreSection(val sectionSize: Int, rowNumber: Int, sectionNumber: Int) {
def id: String = rowNumber.toString + "_"+ sectionNumber.toString;
}
def getSeatingMap(groups: List[SeatingCombination], sections: List[TheatreSection]): Map[Int, TheatreSection] = {
sections.foldLeft(Map.empty[Int, TheatreSection]) { (m, sect) =>
val bestFitOpt = groups.find(g => {
g.seatCount <= sect.sectionSize && !isAnyListIdInMap(m, g.idList)
}).filter(_.idList.nonEmpty)
val newEntries = bestFitOpt.map(_.idList.map(_ -> sect)).getOrElse(List.empty)
m ++ newEntries
}
}
def isAnyListIdInMap(map: Map[Int, TheatreSection], list: List[Int]): Boolean = {
(for (id <- list) yield map.get(id).isDefined).reduce(_ || _)
}
In my Play application, I'm creating an Enumeratee using the filter function:
val activeTeams = Enumeratee.filter[Team](teamIsActive)
My problem is that my teamIsActive function returns a Team => Future[Boolean] and the Enumeratee.filter method takes a Team => Boolean as parameter:
def teamIsActive: (Team) => Future[Boolean] = {
team: Team =>
val teamSize = Future[Int] = teamRepository.membersOf(team).map {
members => members.size
}
teamSize.map(_ > 0)
}
So, how can I use my Future[Boolean] with my Enumeratee
I think this isn't possible. You cannot resolve the future inside your predicate function. Instead you should fetch the team sizes first and then create a predicate function with the list of team sizes.
val teamSizes: Future[Map[Team, Int]] = teamRepository.sizes()
def teamIsActive(sizes: Map[Team, Int]): (Team) => Boolean = { team: Team =>
sizes.getOrElse(team, 0) > 0
}
teamSizes.map { sizes =>
val activeTeams = Enumeratee.filter[Team](teamIsActive(sizes))
}
This reduces also the number of queries to your repository. On the other side it can increase the number of data fetched from your repository. But I don't know your data structure.
I wish to find a match within a List and return values dependant on the match. The CollectFirst works well for matching on the elements of the collection but in this case I want to match on the member swEl of the element rather than on the element itself.
abstract class CanvNode (var swElI: Either[CSplit, VistaT])
{
private[this] var _swEl: Either[CSplit, VistaT] = swElI
def member = _swEl
def member_= (value: Either[CSplit, VistaT] ){ _swEl = value; attach}
def attach: Unit
attach
def findVista(origV: VistaIn): Option[Tuple2[CanvNode,VistaT]] = member match
{
case Right(v) if (v == origV) => Option(this, v)
case _ => None
}
}
def nodes(): List[CanvNode] = topNode :: splits.map(i => List(i.n1, i.n2)).flatten
//Is there a better way of implementing this?
val temp: Option[Tuple2[CanvNode, VistaT]] =
nodes.map(i => i.findVista(origV)).collectFirst{case Some (r) => r}
Do I need a View on that, or will the collectFirst method ensure the collection is only created as needed?
It strikes me that this must be a fairly general pattern. Another example could be if one had a List member of the main List's elements and wanted to return the fourth element if it had one. Is there a standard method I can call? Failing that I can create the following:
implicit class TraversableOnceRichClass[A](n: TraversableOnce[A])
{
def findSome[T](f: (A) => Option[T]) = n.map(f(_)).collectFirst{case Some (r) => r}
}
And then I can replace the above with:
val temp: Option[Tuple2[CanvNode, VistaT]] =
nodes.findSome(i => i.findVista(origV))
This uses implicit classes from 2.10, for pre 2.10 use:
class TraversableOnceRichClass[A](n: TraversableOnce[A])
{
def findSome[T](f: (A) => Option[T]) = n.map(f(_)).collectFirst{case Some (r) => r}
}
implicit final def TraversableOnceRichClass[A](n: List[A]):
TraversableOnceRichClass[A] = new TraversableOnceRichClass(n)
As an introductory side node: The operation you're describing (return the first Some if one exists, and None otherwise) is the sum of a collection of Options under the "first" monoid instance for Option. So for example, with Scalaz 6:
scala> Stream(None, None, Some("a"), None, Some("b")).map(_.fst).asMA.sum
res0: scalaz.FirstOption[java.lang.String] = Some(a)
Alternatively you could put something like this in scope:
implicit def optionFirstMonoid[A] = new Monoid[Option[A]] {
val zero = None
def append(a: Option[A], b: => Option[A]) = a orElse b
}
And skip the .map(_.fst) part. Unfortunately neither of these approaches is appropriately lazy in Scalaz, so the entire stream will be evaluated (unlike Haskell, where mconcat . map (First . Just) $ [1..] is just fine, for example).
Edit: As a side note to this side note: apparently Scalaz does provide a sumr that's appropriately lazy (for streams—none of these approaches will work on a view). So for example you can write this:
Stream.from(1).map(Some(_).fst).sumr
And not wait forever for your answer, just like in the Haskell version.
But assuming that we're sticking with the standard library, instead of this:
n.map(f(_)).collectFirst{ case Some(r) => r }
I'd write the following, which is more or less equivalent, and arguably more idiomatic:
n.flatMap(f(_)).headOption
For example, suppose we have a list of integers.
val xs = List(1, 2, 3, 4, 5)
We can make this lazy and map a function with a side effect over it to show us when its elements are accessed:
val ys = xs.view.map { i => println(i); i }
Now we can flatMap an Option-returning function over the resulting collection and use headOption to (safely) return the first element, if it exists:
scala> ys.flatMap(i => if (i > 2) Some(i.toString) else None).headOption
1
2
3
res0: Option[java.lang.String] = Some(3)
So clearly this stops when we hit a non-empty value, as desired. And yes, you'll definitely need a view if your original collection is strict, since otherwise headOption (or collectFirst) can't reach back and stop the flatMap (or map) that precedes it.
In your case you can skip findVista and get even more concise with something like this:
val temp = nodes.view.flatMap(
node => node.right.toOption.filter(_ == origV).map(node -> _)
).headOption
Whether you find this clearer or just a mess is a matter of taste, of course.