How to dynamically generate parallel futures with for-yield - scala

I have below code:
val f1 = Future(genA1)
val f2 = Future(genA2)
val f3 = Future(genA3)
val f4 = Future(genA4)
val results: Future[Seq[A]] = for {
a1 <- f1
a2 <- f2
a3 <- f3
a4 <- f4
} yield Seq(a, b, c, d)
Now I have a requirement to optionally exclude a2, how to modified the code? ( with map or flatMap is also acceptable)
Further more, say if I have M possible future needs to be aggregated like above, and N of M could be optionally excluded against some flag (biz logic), how should I handle it?
thanks in advance!
Leon


In question1, I understand that you want to exclude one entry (e.g B) from the sequence given some logic and in question2, you want to supress N entries from a total of M, and have the future computed on those results. We could generalize both cases to something like this:
// Using a map as simple example, but 'generators' could be a function that creates the required computation
val generators = Map('a' -> genA1, 'b' -> genA1, 'c' -> genA3, 'd' -> genA4)
...
// shouldAccept(k) => Business logic to decide which computations should be executed.
val selectedGenerators = generators.filter{case (k,v) => shouldAccept(k)}
// Create Seq[Future] from the selected computations
val futures = selectedGenerators.map{case (k,v) => Future(v)}
// Create Future[Seq[_]] to have the result of computing all entries.
val result = Future.sequence(futures)
In general, what I think you are looking for is Future.sequence, which takes a Seq[Future[_]] and produces a Future[Seq[_]], which is basically what you are doing "by hand" with the for-comprehension.

Related

Scala how to avoid var during some override condition

Functional programming style of coding guideline says we should not use null or var in Scala for better functional programming code.
I want to perform some operation like below
var a = 2
if(condition==true){
a = a * 3 /*someOperation*/
}
if(condition2==true){
a = a * 6 /*someOperation*/
}
if(condition3==true){
a = a * 8 /*someOperation*/
}
val b = a * 2/*someOperation*/
So now how to avoid var in this condition and replace it with val?

The simplest way to avoid var with multiple conditions is to use temporary values
val a1 = 2
val a2 = if (condition) a1*3 else a1
val a3 = if (condition2) a2*6 else a2
val a = if (condition3) a3*8 else a3
val b = a * 2/*someOperation*/
In the real code you would give a1, a2, and a3 meaningful names to describe the result of each stage of processing.
If you are bothered about having these extra values in scope, put it in a block:
val a = {
val a1 = 2
val a2 = if (condition) a1*3 else a1
val a3 = if (condition2) a2*6 else a2
if (condition3) a3*8 else a3
}
Update
If you want a more functional approach, collect the conditions and modifications together and apply them in turn, like this:
val mods: List[(Boolean, Int=>Int)] = List(
(condition, _*3),
(condition2, _*6),
(condition3, _*8)
)
val a = mods.foldLeft(2){ case (a,(cond, mod)) => if (cond) mod(a) else a }
This is really only appropriate when either the conditions or the modifications are more complex, and keeping them together makes the code clearer.

val a = 2 * (if (condition) 3 else 1)
val b = 2 * a
Or, perhaps...
val a = 2
val b = 2 * (if (condition) a*3 else a)
It depends on if/how a is used after these operations.

I think you might have oversimplified your example, because we know the value of a when writing the code so you could just write it out like this:
val a = if (condition) 2 else 6
val b = a * 2
Assuming your real operation is more complex and can't be precalculated like that, then you might find a pattern match like this is a nicer way to do it:
val a = (condition, 2) match {
case (true, z) =>
z * 3
case (false, z) =>
z
}
val b = a * 2

You can try the following approach:
type Modification = Int => Int
type ModificationNo = Int
type Conditions = Map[ModificationNo, Boolean]
val modifications: List[(Modification, ModificationNo)] =
List[Modification](
a => a * 3,
a => a * 6,
a => a * 8
).zipWithIndex
def applyModifications(initial: Int, conditions: Conditions): Int =
modifications.foldLeft[Int](initial) {
case (currentA, (modificationFunc, modificationNo)) =>
if (conditions(modificationNo)) modificationFunc(currentA)
else currentA
}
val a: Int = applyModifications(initial = 2,
conditions = Map(0 -> true, 1 -> false, 2 -> true))
It may look complicated but this approach allows additional flexibility if number of conditions is big enough.
Now when you have to add more conditions, you don't need to write new if-statements and further reassignments to var. Just add a modification function to an existing list of

there is no 1 perfect solution.
sometimes it is ok to use var if it simplifies the code and limited in scope of a single function.
that being said, this is how I would do it in functional way:
val op1: Int => Int =
if (condition1) x => x * 3
else identity
val op2: Int => Int =
if (condition2) x => x * 6
else identity
val op3: Int => Int =
if (condition3) x => x * 8
else identity
val op = op1 andThen op2 andThen op3
// can also be written as
// val op = Seq(op1, op2, op3).reduceLeft(_ andThen _)
val a = 2
val b = op(a) * 2

The easiest way it to wrap your variable into a monad, so that you .map over it. The simplest monad is an Option, so you could write:
val result = Option(a).map {
case a if condition => a*2
case a => a
}.map {
case a if condition2 => a*6
case a => a
}.fold(a) {
case a if condition3 => a*8
case a => a
}
(The last operation is fold instead of map so that you end up with the "raw" value for the result, rather than an option. Equivalently, you could write it as a .map, and then add .getOrElse(a) at the end).
When you have many conditional operations like this, or many use cases where you have to repeat the pattern, it might help to put them into a list, and then traverse that list instead:
def applyConditionals[T](toApply: (() => Boolean, T => T)*) = toApply
.foldLeft(a) {
case (a, (cond, oper)) if cond() => oper(a)
case (a, _) => a
}
val result = applyConditionals[Int] (
(() => condition, _ * 2),
(() => condition2, _ * 6),
(() => condition3, _ * 8)
)

The important point is that FP is a whole new paradigm of programming. Its so fundamentally different that sometimes you can not take an excerpt of imperative code and try to convert it to functional code.
The difference applies not just to code but to the way of thinking towards solving a problem. Functional programming requires you to think in terms of chained mathematical computation which are more or less independent of each other (which means that each of these mathematical computation should not be changing anything outside of its own environment).
Functional programming totally avoids mutation of state. So, if your solution has a requirement to have a variable x which has a value 10 at one point and other value 100 at some other point... then your solution is not functional. And you can not write function code for a solution which is not functional.
Now, if you look at your case (assuming you do not actually need a to be 2 and then change to 6 after sometime) and try to convert it into chain of independent mathematical computation, then the simplest one is following,
val a = if (condition) 2 else 6
val b = a * 2

Scala apply list of functions to a object

I have a lit of functions
val f1 = (x: Int) => x + 1
val f2 = (x: Int) => x + 2
val f3 = (x: Int) => x + 3
I have a single value:
val data = 5
I want to apply all the functions to the value and return single value. So
f3(f2(f1(data)))
And must return 11.
Now, if I have a seq of functions:
val funcs = Seq(f1, f2, f3)
How can I get 11 from applying all the functions to the data variable ? What is the scala-way to do that ?

yet another way to doing it using chain method in the Function helper object
Function.chain(funcs)(data)

What you are looking for is foldLeft. Indeed, for each function, you apply it to the previous result:
funcs.foldLeft(data){ (previousres, f) => f(previousres) }

you can use foldLeft:
val result = funcs.foldLeft(5)((acc,curr) => curr(acc) )

Basically, you are trying to achieve Function Composition here. So, you could use compose and andThen methods here as:
val data = 5
val funcs = Seq(f1, f2, f3)
//using compose
val result1 = funcs.reduce((a,b) => a.compose(b))(data)
//using andThen
val result2 = funcs.reduce((a,b) => a.andThen(b))(data)
Both result1 and result2 will be 11 in your example.
Please note that the way andThen and compose operate are different. You could see Functional Composition for more information.

For comprehension - execute futures in order

If I have the following for comprehension, futures will be executed in order: f1, f2, f3:
val f = for {
r1 <- f1
r2 <- f2(r1)
r3 <- f3(r2)
} yield r3
For this one however, all the futures are started at the same time:
val f = for {
r1 <- f1
r2 <- f2
r3 <- f3
} yield ...
How can I enforce the order?(I want this order of execution f1, f2, f3)

It does matter what f1, f2, f3 are: a future will start executing a soon as it is created. In your first case, f2(r1) must be a function returning a future, so the future begins executing when the function is called, which happens when r1 becomes available.
If the second case is the same (f2 is a function), then behavior will be the same as the first case, your futures will be executed sequentially, one after the other.
But if you create the futures outside the for, and just assign them to variables f1, f2, f3, then by the time you get inside the comprehension, they are already running.

Future are eager constructs, that is, once created you can not dictate when they get processed. If the Future already exists when you attempt to use it in a for-comprehension, you've already lost the ability to sequence it's execution order.
If you want to enforce ordering on a method that accepts Future arguments then you'll need to wrap the evaluation in a thunk:
def foo(ft: => Future[Thing], f2: => Future[Thing]): Future[Other] = for{
r1 <- ft
r2 <- f2
} yield something(r1, r2)
If, on the other hand, you want to define the Future within a method body, then instead of val use a def
def foo() ={
def f1 = Future{ code here... }
def f2 = Future{ code here... }
for{
r1 <- f1
r2 <- f2
} yield something(r1, r2)

Executing futures in for comprehension is default behavior. It is good when few tasks are processed parrallel without any blocking.
But if you want to preserve procecessing order you have to ways:
Send result of first task to second like in your example
use andThen operator
val allposts = mutable.Set[String]()
Future {
session.getRecentPosts
} andThen {
posts => allposts ++= posts
} andThen {
posts =>
clearAll()
for (post <- allposts) render(post)
}

scala: best way to merge two mutable maps of mutable sets

What is the best way in scala to merge two mutable maps of mutable sets? The operation must be commutative. The things I've tried seem ugly...
import scala.collection.mutable
var d1 = mutable.Map[String, mutable.SortedSet[String]]()
var d2 = mutable.Map[String, mutable.SortedSet[String]]()
// adding some elements. Accumulating nimals with the set of sounds they make.
d1.getOrElseUpdate("dog", mutable.SortedSet[String]("woof"))
d2.getOrElseUpdate("cow", mutable.SortedSet[String]("moo"))
d2.getOrElseUpdate("dog", mutable.SortedSet[String]("woof", "bark"))
magic (that is commutative!)
scala.collection.mutable.Map[String,scala.collection.mutable.SortedSet[String]] =
Map(dog -> TreeSet(bark, woof), cow -> TreeSet(moo))
Basically, I want to override the definition for ++ to merge the sets for matching map keys. Notice that
d1 ++ d2 gives the right answer, while d2 ++ d1 does not (so ++ is not commutative here).

For every key that would be in the resulting Map, you have to merge (++) the value Sets from d1 and d2 for that key.
For mutable.Maps and mutable.Sets, when you are updating one of the Maps, the implementation is really straightforward:
for ((key, values) <- d2)
d1.getOrElseUpdate(key, mutable.SortedSet.empty) ++= values
You can actually create an empty mutable.Map, and use that code to update it with d1 and d2 (and other Maps if needed) in any order.
You can wrap this operation in a following function:
val d1 = mutable.Map[String, mutable.SortedSet[String]](
"dog" -> mutable.SortedSet("woof"),
"cow" -> mutable.SortedSet("moo"))
val d2 = mutable.Map[String, mutable.SortedSet[String]](
"dog" -> mutable.SortedSet("woof", "bark"))
def updateMap[A, B : Ordering]( // `Ordering` is a requirement for `SortedSet`
d1: mutable.Map[A, mutable.SortedSet[B]])(
// `Iterable`s are enough here, but allow to pass a `Map[A, Set[B]]`
d2: Iterable[(A, Iterable[B])]
): Unit =
for ((key, values) <- d2)
d1.getOrElseUpdate(key, mutable.SortedSet.empty) ++= values
// You can call
// `updateMap(d1)(d2)` or
// `updateMap(d2)(d1)` to achieve the same result (but in different variables)
For immutable Maps one possible implementation is:
(
for (key <- d1.keySet ++ d2.keySet)
yield key -> (d1.getOrElse(key, Set.empty) ++ d2.getOrElse(key, Set.empty))
).toMap
Other, likely more effective, but probably slightly more complex implementations are also possible.

Scala list of tuples of different size zip issues?

Hi my two lists as follows:
val a = List((1430299869,"A",4200), (1430299869,"A",0))
val b = List((1430302366,"B",4100), (1430302366,"B",4200), (1430302366,"B",5000), (1430302366,"B",27017), (1430302366,"B",80), (1430302366,"B",9300), (1430302366,"B",9200), (1430302366,"A",5000), (1430302366,"A",4200), (1430302366,"A",80), (1430302366,"A",443), (1430302366,"C",4100), (1430302366,"C",4200), (1430302366,"C",27017), (1430302366,"C",5000), (1430302366,"C",80))
when I used zip two lists as below :
val c = a zip b
it returns results as
List(((1430299869,A,4200),(1430302366,B,4100)), ((1430299869,A,0),(1430302366,B,4200)))
Not all lists of tuples, how can I zip all above data?
EDIT
expected results as combine of two lists like :
List((1430299869,"A",4200), (1430299869,"A",0),(1430302366,"B",4100), (1430302366,"B",4200), (1430302366,"B",5000), (1430302366,"B",27017), (1430302366,"B",80), (1430302366,"B",9300), (1430302366,"B",9200), (1430302366,"A",5000), (1430302366,"A",4200), (1430302366,"A",80), (1430302366,"A",443), (1430302366,"C",4100), (1430302366,"C",4200), (1430302366,"C",27017), (1430302366,"C",5000), (1430302366,"C",80))
Second Edit
I tried this :
val d = for(((a,b,c),(d,e,f)) <- (a zip b)if(b.equals(e) && c.equals(f))) yield (d,e,f)
but it gives empty results because of (a zip b) but I replaced a zip b as a ++ b then it shows following error :
constructor cannot be instantiated to expected type;
So how can I get matching tuples?

Just add one list to another:
a ++ b

According to your 2nd edit, what you need is:
for {
(a1,b1,c) <- a //rename extracted to a1 and b1 to avoid confusion
(d,e,f) <- b
if b1.equals(e) && c.equals(f)
} yield (d,e,f)
Or:
for {
(a1, b1, c) <- a
(d, `b1`, `c`) <- b //enclosing it in backticks avoids capture and matches against already defined values
} yield (d, b1, c)
Zipping won't help since you need to compare all tuples in a with all tuples in b , it seems.

a zip b creates a list of pairs of elements from a and b.
What you're most likely looking for is list concatenation, which is a ++ b

On zipping (pairing) all data in the lists, consider first a briefer input for illustrating the case,
val a = (1 to 2).toList
val b = (10 to 12).toList
Then for instance a for comprehension may convey the needs,
for (i <- a; j <- b) yield (i,j)
which delivers
List((1,10), (1,11), (1,12),
(2,10), (2,11), (2,12))
Update
From OP latest update, consider a dedicated filtering function,
type triplet = (Int,String,Int)
def filtering(key: triplet, xs: List[triplet]) =
xs.filter( v => key._2 == v._2 && key._3 == v._3 )
and so apply it with flatMap,
a.flatMap(filtering(_, b))
List((1430302366,A,4200))
One additional step is to encapsulate this in an implicit class,
implicit class OpsFilter(val keys: List[triplet]) extends AnyVal {
def filtering(xs: List[triplet]) = {
keys.flatMap ( key => xs.filter( v => key._2 == v._2 && key._3 == v._3 ))
}
}
and likewise,
a.filtering(b)
List((1430302366,A,4200))