Related
Given a collection in Scala, I'd like to traverse this collection and for each object I'd like to emit (yield) from 0 to multiple elements that should be joined together into a new collection.
For example, I expect something like this:
val input = Range(0, 15)
val output = input.somefancymapfunction((x) => {
if (x % 3 == 0)
yield(s"${x}/3")
if (x % 5 == 0)
yield(s"${x}/5")
})
to build an output collection that will contain
(0/3, 0/5, 3/3, 5/5, 6/3, 9/3, 10/5, 12/3)
Basically, I want a superset of what filter (1 → 0..1) and map (1 → 1) allows to do: mapping (1 → 0..n).
Solutions I've tried
Imperative solutions
Obviously, it's possible to do so in non-functional maneer, like:
var output = mutable.ListBuffer()
input.foreach((x) => {
if (x % 3 == 0)
output += s"${x}/3"
if (x % 5 == 0)
output += s"${x}/5"
})
Flatmap solutions
I know of flatMap, but it again, either:
1) becomes really ugly if we're talking about arbitrary number of output elements:
val output = input.flatMap((x) => {
val v1 = if (x % 3 == 0) {
Some(s"${x}/3")
} else {
None
}
val v2 = if (x % 5 == 0) {
Some(s"${x}/5")
} else {
None
}
List(v1, v2).flatten
})
2) requires usage of mutable collections inside it:
val output = input.flatMap((x) => {
val r = ListBuffer[String]()
if (x % 3 == 0)
r += s"${x}/3"
if (x % 5 == 0)
r += s"${x}/5"
r
})
which is actually even worse that using mutable collection from the very beginning, or
3) requires major logic overhaul:
val output = input.flatMap((x) => {
if (x % 3 == 0) {
if (x % 5 == 0) {
List(s"${x}/3", s"${x}/5")
} else {
List(s"${x}/3")
}
} else if (x % 5 == 0) {
List(s"${x}/5")
} else {
List()
}
})
which is, IMHO, also looks ugly and requires duplicating the generating code.
Roll-your-own-map-function
Last, but not least, I can roll my own function of that kind:
def myMultiOutputMap[T, R](coll: TraversableOnce[T], func: (T, ListBuffer[R]) => Unit): List[R] = {
val out = ListBuffer[R]()
coll.foreach((x) => func.apply(x, out))
out.toList
}
which can be used almost like I want:
val output = myMultiOutputMap[Int, String](input, (x, out) => {
if (x % 3 == 0)
out += s"${x}/3"
if (x % 5 == 0)
out += s"${x}/5"
})
Am I really overlooking something and there's no such functionality in standard Scala collection libraries?
Similar questions
This question bears some similarity to Can I yield or map one element into many in Scala? — but that question discusses 1 element → 3 elements mapping, and I want 1 element → arbitrary number of elements mapping.
Final note
Please note that this is not the question about division / divisors, such conditions are included purely for illustrative purposes.
Rather than having a separate case for each divisor, put them in a container and iterate over them in a for comprehension:
val output = for {
n <- input
d <- Seq(3, 5)
if n % d == 0
} yield s"$n/$d"
Or equivalently in a collect nested in a flatMap:
val output = input.flatMap { n =>
Seq(3, 5).collect {
case d if n % d == 0 => s"$n/$d"
}
}
In the more general case where the different cases may have different logic, you can put each case in a separate partial function and iterate over the partial functions:
val output = for {
n <- input
f <- Seq[PartialFunction[Int, String]](
{case x if x % 3 == 0 => s"$x/3"},
{case x if x % 5 == 0 => s"$x/5"})
if f.isDefinedAt(n)
} yield f(n)
You can also use some functional library (e.g. scalaz) to express this:
import scalaz._, Scalaz._
def divisibleBy(byWhat: Int)(what: Int): List[String] =
(what % byWhat == 0).option(s"$what/$byWhat").toList
(0 to 15) flatMap (divisibleBy(3) _ |+| divisibleBy(5))
This uses the semigroup append operation |+|. For Lists this operation means a simple list concatenation. So for functions Int => List[String], this append operation will produce a function that runs both functions and appends their results.
If you have complex computation, during which you should sometimes add some elements to operation global accumulator, you can use popular approach named Writer Monad
Preparation in scala is somewhat bulky but results are extremely composable thanks to Monad interface
import scalaz.Writer
import scalaz.syntax.writer._
import scalaz.syntax.monad._
import scalaz.std.vector._
import scalaz.syntax.traverse._
type Messages[T] = Writer[Vector[String], T]
def yieldW(a: String): Messages[Unit] = Vector(a).tell
val output = Vector.range(0, 15).traverse { n =>
yieldW(s"$n / 3").whenM(n % 3 == 0) >>
yieldW(s"$n / 5").whenM(n % 5 == 0)
}.run._1
Here is my proposition for a custom function, might be better with pimp my library pattern
def fancyMap[A, B](list: TraversableOnce[A])(fs: (A => Boolean, A => B)*) = {
def valuesForElement(elem: A) = fs collect { case (predicate, mapper) if predicate(elem) => mapper(elem) }
list flatMap valuesForElement
}
fancyMap[Int, String](0 to 15)((_ % 3 == 0, _ + "/3"), (_ % 5 == 0, _ + "/5"))
You can try collect:
val input = Range(0,15)
val output = input.flatMap { x =>
List(3,5) collect { case n if (x%n == 0) => s"${x}/${n}" }
}
System.out.println(output)
I would us a fold:
val input = Range(0, 15)
val output = input.foldLeft(List[String]()) {
case (acc, value) =>
val acc1 = if (value % 3 == 0) s"$value/3" :: acc else acc
val acc2 = if (value % 5 == 0) s"$value/5" :: acc1 else acc1
acc2
}.reverse
output contains
List(0/3, 0/5, 3/3, 5/5, 6/3, 9/3, 10/5, 12/3)
A fold takes an accumumlator (acc), a collection, and a function. The function is called with the initial value of the accumumator, in this case an empty List[String], and each value of the collection. The function should return an updated collection.
On each iteration, we take the growing accumulator and, if the inside if statements are true, prepend the calculation to the new accumulator. The function finally returns the updated accumulator.
When the fold is done, it returns the final accumulator, but unfortunately, it is in reverse order. We simply reverse the accumulator with .reverse.
There is a nice paper on folds: A tutorial on the universality and expressiveness of fold, by Graham Hutton.
Hi my two lists as follows:
val a = List((1430299869,"A",4200), (1430299869,"A",0))
val b = List((1430302366,"B",4100), (1430302366,"B",4200), (1430302366,"B",5000), (1430302366,"B",27017), (1430302366,"B",80), (1430302366,"B",9300), (1430302366,"B",9200), (1430302366,"A",5000), (1430302366,"A",4200), (1430302366,"A",80), (1430302366,"A",443), (1430302366,"C",4100), (1430302366,"C",4200), (1430302366,"C",27017), (1430302366,"C",5000), (1430302366,"C",80))
when I used zip two lists as below :
val c = a zip b
it returns results as
List(((1430299869,A,4200),(1430302366,B,4100)), ((1430299869,A,0),(1430302366,B,4200)))
Not all lists of tuples, how can I zip all above data?
EDIT
expected results as combine of two lists like :
List((1430299869,"A",4200), (1430299869,"A",0),(1430302366,"B",4100), (1430302366,"B",4200), (1430302366,"B",5000), (1430302366,"B",27017), (1430302366,"B",80), (1430302366,"B",9300), (1430302366,"B",9200), (1430302366,"A",5000), (1430302366,"A",4200), (1430302366,"A",80), (1430302366,"A",443), (1430302366,"C",4100), (1430302366,"C",4200), (1430302366,"C",27017), (1430302366,"C",5000), (1430302366,"C",80))
Second Edit
I tried this :
val d = for(((a,b,c),(d,e,f)) <- (a zip b)if(b.equals(e) && c.equals(f))) yield (d,e,f)
but it gives empty results because of (a zip b) but I replaced a zip b as a ++ b then it shows following error :
constructor cannot be instantiated to expected type;
So how can I get matching tuples?
Just add one list to another:
a ++ b
According to your 2nd edit, what you need is:
for {
(a1,b1,c) <- a //rename extracted to a1 and b1 to avoid confusion
(d,e,f) <- b
if b1.equals(e) && c.equals(f)
} yield (d,e,f)
Or:
for {
(a1, b1, c) <- a
(d, `b1`, `c`) <- b //enclosing it in backticks avoids capture and matches against already defined values
} yield (d, b1, c)
Zipping won't help since you need to compare all tuples in a with all tuples in b , it seems.
a zip b creates a list of pairs of elements from a and b.
What you're most likely looking for is list concatenation, which is a ++ b
On zipping (pairing) all data in the lists, consider first a briefer input for illustrating the case,
val a = (1 to 2).toList
val b = (10 to 12).toList
Then for instance a for comprehension may convey the needs,
for (i <- a; j <- b) yield (i,j)
which delivers
List((1,10), (1,11), (1,12),
(2,10), (2,11), (2,12))
Update
From OP latest update, consider a dedicated filtering function,
type triplet = (Int,String,Int)
def filtering(key: triplet, xs: List[triplet]) =
xs.filter( v => key._2 == v._2 && key._3 == v._3 )
and so apply it with flatMap,
a.flatMap(filtering(_, b))
List((1430302366,A,4200))
One additional step is to encapsulate this in an implicit class,
implicit class OpsFilter(val keys: List[triplet]) extends AnyVal {
def filtering(xs: List[triplet]) = {
keys.flatMap ( key => xs.filter( v => key._2 == v._2 && key._3 == v._3 ))
}
}
and likewise,
a.filtering(b)
List((1430302366,A,4200))
In Scala language, I want to write a function that yields odd numbers within a given range. The function prints some log when iterating even numbers. The first version of the function is:
def getOdds(N: Int): Traversable[Int] = {
val list = new mutable.MutableList[Int]
for (n <- 0 until N) {
if (n % 2 == 1) {
list += n
} else {
println("skip even number " + n)
}
}
return list
}
If I omit printing logs, the implementation become very simple:
def getOddsWithoutPrint(N: Int) =
for (n <- 0 until N if (n % 2 == 1)) yield n
However, I don't want to miss the logging part. How do I rewrite the first version more compactly? It would be great if it can be rewritten similar to this:
def IWantToDoSomethingSimilar(N: Int) =
for (n <- 0 until N) if (n % 2 == 1) yield n else println("skip even number " + n)
def IWantToDoSomethingSimilar(N: Int) =
for {
n <- 0 until N
if n % 2 != 0 || { println("skip even number " + n); false }
} yield n
Using filter instead of a for expression would be slightly simpler though.
I you want to keep the sequentiality of your traitement (processing odds and evens in order, not separately), you can use something like that (edited) :
def IWantToDoSomethingSimilar(N: Int) =
(for (n <- (0 until N)) yield {
if (n % 2 == 1) {
Option(n)
} else {
println("skip even number " + n)
None
}
// Flatten transforms the Seq[Option[Int]] into Seq[Int]
}).flatten
EDIT, following the same concept, a shorter solution :
def IWantToDoSomethingSimilar(N: Int) =
(0 until N) map {
case n if n % 2 == 0 => println("skip even number "+ n)
case n => n
} collect {case i:Int => i}
If you will to dig into a functional approach, something like the following is a good point to start.
First some common definitions:
// use scalaz 7
import scalaz._, Scalaz._
// transforms a function returning either E or B into a
// function returning an optional B and optionally writing a log of type E
def logged[A, E, B, F[_]](f: A => E \/ B)(
implicit FM: Monoid[F[E]], FP: Pointed[F]): (A => Writer[F[E], Option[B]]) =
(a: A) => f(a).fold(
e => Writer(FP.point(e), None),
b => Writer(FM.zero, Some(b)))
// helper for fixing the log storage format to List
def listLogged[A, E, B](f: A => E \/ B) = logged[A, E, B, List](f)
// shorthand for a String logger with List storage
type W[+A] = Writer[List[String], A]
Now all you have to do is write your filtering function:
def keepOdd(n: Int): String \/ Int =
if (n % 2 == 1) \/.right(n) else \/.left(n + " was even")
You can try it instantly:
scala> List(5, 6) map(keepOdd)
res0: List[scalaz.\/[String,Int]] = List(\/-(5), -\/(6 was even))
Then you can use the traverse function to apply your function to a list of inputs, and collect both the logs written and the results:
scala> val x = List(5, 6).traverse[W, Option[Int]](listLogged(keepOdd))
x: W[List[Option[Int]]] = scalaz.WriterTFunctions$$anon$26#503d0400
// unwrap the results
scala> x.run
res11: (List[String], List[Option[Int]]) = (List(6 was even),List(Some(5), None))
// we may even drop the None-s from the output
scala> val (logs, results) = x.map(_.flatten).run
logs: List[String] = List(6 was even)
results: List[Int] = List(5)
I don't think this can be done easily with a for comprehension. But you could use partition.
def getOffs(N:Int) = {
val (evens, odds) = 0 until N partition { x => x % 2 == 0 }
evens foreach { x => println("skipping " + x) }
odds
}
EDIT: To avoid printing the log messages after the partitioning is done, you can change the first line of the method like this:
val (evens, odds) = (0 until N).view.partition { x => x % 2 == 0 }
In a for-comprehension, I can't just put a print statement:
def prod (m: Int) = {
for (a <- 2 to m/(2*3);
print (a + " ");
b <- (a+1) to m/a;
c = (a*b)
if (c < m)) yield c
}
but I can circumvent it easily with a dummy assignment:
def prod (m: Int) = {
for (a <- 2 to m/(2*3);
dummy = print (a + " ");
b <- (a+1) to m/a;
c = (a*b)
if (c < m)) yield c
}
Being a side effect, and only used (so far) in code under development, is there a better ad hoc solution?
Is there a serious problem why I shouldn't use it, beside being a side effect?
update showing the real code, where adapting one solution is harder than expected:
From the discussion with Rex Kerr, the necessity has risen to show the original code, which is a bit more complicated, but did not seem to be relevant for the question (2x .filter, calling a method in the end), but when I tried to apply Rex' pattern to it I failed, so I post it here:
def prod (p: Array[Boolean], max: Int) = {
for (a <- (2 to max/(2*3)).
filter (p);
dummy = print (a + " ");
b <- (((a+1) to max/a).
filter (p));
if (a*b <= max))
yield (em (a, b, max)) }
Here is my attempt -- (b * a).filter is wrong, because the result is an int, not a filterable collection of ints:
// wrong:
def prod (p: Array[Boolean], max: Int) = {
(2 to max/(2*3)).filter (p).flatMap { a =>
print (a + " ")
((a+1) to max/a).filter (p). map { b =>
(b * a).filter (_ <= max).map (em (a, b, max))
}
}
}
Part II belongs to the comments, but can't be read, if written there - maybe I delete it in the end. Please excuse.
Ok - here is Rex last answer in code layout:
def prod (p: Array[Boolean], max: Int) = {
(2 to max/(2*3)).filter (p).flatMap { a =>
print (a + " ")
((a+1) to max/a).filter (b => p (b)
&& b * a < max).map { b => (m (a, b, max))
}
}
}
This is how you need to write it:
scala> def prod(m: Int) = {
| for {
| a <- 2 to m / (2 * 3)
| _ = print(a + " ")
| b <- (a + 1) to (m / a)
| c = a * b
| if c < m
| } yield c
| }
prod: (m: Int)scala.collection.immutable.IndexedSeq[Int]
scala> prod(20)
2 3 res159: scala.collection.immutable.IndexedSeq[Int] = Vector(6, 8, 10, 12, 14
, 16, 18, 12, 15, 18)
Starting Scala 2.13, the chaining operation tap, has been included in the standard library, and can be used with minimum intrusiveness wherever we need to print some intermediate state of a pipeline:
import util.chaining._
def prod(m: Int) =
for {
a <- 2 to m / (2 * 3)
b <- (a + 1) to (m / a.tap(println)) // <- a.tap(println)
c = a * b
if c < m
} yield c
prod(20)
// 2
// 3
// res0: IndexedSeq[Int] = Vector(6, 8, 10, 12, 14, 16, 18, 12, 15, 18)
The tap chaining operation applies a side effect (in this case println) on a value (in this case a) while returning the value (a) untouched:
def tap[U](f: (A) => U): A
It's very convenient when debugging as you can use a bunch of taps without having to modify the code:
def prod(m: Int) =
for {
a <- (2 to m.tap(println) / (2 * 3)).tap(println)
b <- (a + 1) to (m / a.tap(println))
c = (a * b).tap(println)
if c < m
} yield c
I generally find that style of coding rather difficult to follow, since loops and intermediate results and such get all mixed in with each other. I would, instead of a for loop, write something like
def prod(m: Int) = {
(2 to m/(2*3)).flatMap { a =>
print(a + " ")
((a+1) to m/a).map(_ * a).filter(_ < m)
}
}
This also makes adding print statements and such easier.
It doesn't seem like good style to put a side-effecting statement within a for-comprehension (or indeed in the middle of any function), execept for debugging in which case it doesn't really matter what you call it ("debug" seems like a good name).
If you really need to, I think you'd be better separating your concerns somewhat by assigning an intermediate val, e.g. (your original laid out more nicely):
def prod (p: Array[Boolean], max: Int) = {
for {
a <- (2 to max / (2 * 3)) filter p
debug = print (a + " ")
b <- ((a + 1) to max / a) filter p
if a * b <= max
} yield em(a, b, max)
}
becomes
def prod2 (p: Array[Boolean], max: Int) = {
val as = (2 to max / (2 * 3)) filter p
for(a <- as) print(a + " ")
as flatMap {a =>
for {
b <- ((a + 1) to max / a) filter p
if a * b <= max
} yield em(a, b, max)
}
}
I want to implement a lazy iterator that yields the next element in each call, in a 3-level nested loop.
Is there something similar in scala to this snippet of c#:
foreach (int i in ...)
{
foreach (int j in ...)
{
foreach (int k in ...)
{
yield return do(i,j,k);
}
}
}
Thanks, Dudu
Scala sequence types all have a .view method which produces a lazy equivalent of the collection. You can play around with the following in the REPL (after issuing :silent to stop it from forcing the collection to print command results):
def log[A](a: A) = { println(a); a }
for (i <- 1 to 10) yield log(i)
for (i <- (1 to 10) view) yield log(i)
The first will print out the numbers 1 to 10, the second will not until you actually try to access those elements of the result.
There is nothing in Scala directly equivalent to C#'s yield statement, which pauses the execution of a loop. You can achieve similar effects with the delimited continuations which were added for scala 2.8.
If you join iterators together with ++, you get a single iterator that runs over both. And the reduceLeft method helpfully joins together an entire collection. Thus,
def doIt(i: Int, j: Int, k: Int) = i+j+k
(1 to 2).map(i => {
(1 to 2).map(j => {
(1 to 2).iterator.map(k => doIt(i,j,k))
}).reduceLeft(_ ++ _)
}).reduceLeft(_ ++ _)
will produce the iterator you want. If you want it to be even more lazy than that, you can add .iterator after the first two (1 to 2) also. (Replace each (1 to 2) with your own more interesting collection or range, of course.)
You can use a Sequence Comprehension over Iterators to get what you want:
for {
i <- (1 to 10).iterator
j <- (1 to 10).iterator
k <- (1 to 10).iterator
} yield doFunc(i, j, k)
If you want to create a lazy Iterable (instead of a lazy Iterator) use Views instead:
for {
i <- (1 to 10).view
j <- (1 to 10).view
k <- (1 to 10).view
} yield doFunc(i, j, k)
Depending on how lazy you want to be, you may not need all of the calls to iterator / view.
If your 3 iterators are generally small (i.e., you can fully iterate them without concern for memory or CPU) and the expensive part is computing the result given i, j, and k, you can use Scala's Stream class.
val tuples = for (i <- 1 to 3; j <- 1 to 3; k <- 1 to 3) yield (i, j, k)
val stream = Stream(tuples: _*) map { case (i, j, k) => i + j + k }
stream take 10 foreach println
If your iterators are too large for this approach, you could extend this idea and create a Stream of tuples that calculates the next value lazily by keeping state for each iterator. For example (although hopefully someone has a nicer way of defining the product method):
def product[A, B, C](a: Iterable[A], b: Iterable[B], c: Iterable[C]): Iterator[(A, B, C)] = {
if (a.isEmpty || b.isEmpty || c.isEmpty) Iterator.empty
else new Iterator[(A, B, C)] {
private val aItr = a.iterator
private var bItr = b.iterator
private var cItr = c.iterator
private var aValue: Option[A] = if (aItr.hasNext) Some(aItr.next) else None
private var bValue: Option[B] = if (bItr.hasNext) Some(bItr.next) else None
override def hasNext = cItr.hasNext || bItr.hasNext || aItr.hasNext
override def next = {
if (cItr.hasNext)
(aValue get, bValue get, cItr.next)
else {
cItr = c.iterator
if (bItr.hasNext) {
bValue = Some(bItr.next)
(aValue get, bValue get, cItr.next)
} else {
aValue = Some(aItr.next)
bItr = b.iterator
(aValue get, bValue get, cItr.next)
}
}
}
}
}
val stream = product(1 to 3, 1 to 3, 1 to 3).toStream map { case (i, j, k) => i + j + k }
stream take 10 foreach println
This approach fully supports infinitely sized inputs.
I think the below code is what you're actually looking for... I think the compiler ends up translating it into the equivalent of the map code Rex gave, but is closer to the syntax of your original example:
scala> def doIt(i:Int, j:Int) = { println(i + ","+j); (i,j); }
doIt: (i: Int, j: Int)(Int, Int)
scala> def x = for( i <- (1 to 5).iterator;
j <- (1 to 5).iterator ) yield doIt(i,j)
x: Iterator[(Int, Int)]
scala> x.foreach(print)
1,1
(1,1)1,2
(1,2)1,3
(1,3)1,4
(1,4)1,5
(1,5)2,1
(2,1)2,2
(2,2)2,3
(2,3)2,4
(2,4)2,5
(2,5)3,1
(3,1)3,2
(3,2)3,3
(3,3)3,4
(3,4)3,5
(3,5)4,1
(4,1)4,2
(4,2)4,3
(4,3)4,4
(4,4)4,5
(4,5)5,1
(5,1)5,2
(5,2)5,3
(5,3)5,4
(5,4)5,5
(5,5)
scala>
You can see from the output that the print in "doIt" isn't called until the next value of x is iterated over, and this style of for generator is a bit simpler to read/write than a bunch of nested maps.
Turn the problem upside down. Pass "do" in as a closure. That's the entire point of using a functional language
Iterator.zip will do it:
iterator1.zip(iterator2).zip(iterator3).map(tuple => doSomething(tuple))
Just read the 20 or so first related links that are show on the side (and, indeed, where shown to you when you first wrote the title of your question).