I have a composition of combinators in Scala, and the last one is .top, which I could use as .top(num)(Ordering[(Int, Int)].reverse) depending on a boolean parameter.
How do I implement this composition of combinators to use or not use .reverse depending on the boolean parameter, in the same line? I mean, without creating another val to indicate whether .reverse is used?
val mostPopularHero = sparkContext
.textFile("resource/marvel/Marvel-graph.txt") // build up superhero co-apperance data
.map(countCoOccurrences) // convert to (hero ID, number of connections) RDD
.reduceByKey((x, y) => x + y) // combine entries that span more than one line
.map(x => (x._2, x._1)) // flip it from (hero ID, number of connections) to (number of connections, hero ID)
.top(num)(Ordering[(Int, Int)].reverse)
Solution 0
As nicodp has already pointed out, if you have a boolean variable b in scope, you can simply replace the expression
Ordering[(Int, Int)]
by an if-expression
if (b) Ordering[(Int, Int)] else Ordering[(Int, Int)].reverse
I have to admit that this is the shortest and clearest solution I could come up with.
However... I didn't quite like that the expression Ordering[(Int, Int)] appears in the code twice. It doesn't really matter in this case, because it's short, but what if the expression were a bit longer? Apparently, even Ruby has something for such cases.
So, I tried to come up with some ways to not repeat the subexpression Ordering[(Int, Int)]. The nicest solution would be if we had a default Id-monad implementation in the standard library, because then we could simply wrap the one value in pure, and then map it using the boolean.
But there is no Id in standard library. So, here are a few other proposals, just for the case that the expression in question becomes longer:
Solution 1
You can use blocks as expressions in scala, so you can replace the above
Ordering[(Int, Int)] by:
{val x = Ordering[(Int, Int)]; if (b) x else x.reverse}
Update: Wait! This is shorter than the version with repetition! ;)
Solution 2
Define the function that conditionally reverses an ordering, declare Ordering[(Int, Int)] as the type of the argument, and then
instead of re-typing Ordering[(Int, Int)] as an expression, use implicitly:
((x: Ordering[(Int, Int)]) => if (b) x else x.reverse)(implicitly)
Solution 3
We don't have Id, but we can abuse constructors and eliminators of other functors. For example, one could wrap the complex expression in a List or Option, then map it, then unpack the result. Here is a variant with Some:
Some(Ordering[(Int, Int)]).map{ x => if(b) x else x.reverse }.get
Ideally, this would have been Id instead of Some. Notice that Solution 1 does something similar with the default ambient monad.
Solution 4
Finally, if the above pattern occurs more than once in your code, it might be worth it to introduce some extra syntax to deal with it:
implicit class ReversableOrderingOps[X](ord: Ordering[X]) {
def reversedIf(b: Boolean): Ordering[X] = if (b) ord.reverse else ord
}
Now you can define orderings like this:
val myConditionHolds = true
val myOrd = Ordering[(Int, Int)] reversedIf myConditionHolds
or use it in your lengthy expression directly:
val mostPopularHero = sparkContext
.textFile("resource/marvel/Marvel-graph.txt")
.map(countCoOccurrences)
.reduceByKey((x, y) => x + y)
.map(x => (x._2, x._1))
.top(num)(Ordering[(Int, Int)] reversedIf myConditionHolds)
I'm not quite sure if you have access to the boolean parameter here or not, but you can work this out as follows:
.top(num)(if (booleanParameter) Ordering[(Int, Int)].reverse else Ordering[(Int, Int)])
Related
Like the title says what is the intuition behind recursive algos with streams like:
val fibs: LazyList[Int] = (0 #:: fibs).scanLeft(1)(_ + _)
and
val fibs: LazyList[Int] = 0 #:: 1 #:: (fibs.zip(fibs.tail).map{ t => t._1 + t._2 })
How do they unfold? What is the base case for such algos (if it's Nil, why it's so?) and how do they progress towards fibs.take(5) e.g.?
EDIT.
I do understand there is no base case for a lazily defined Stream, as several people pointed out below. Rather, my question concerns what's the base case when infinite stream gets evaluated like in fibs.take(5)(the answer is Nil I believe, please correct me if I'm wrong) and what are the calculation steps in evaluating fibs.take(5)
It's say there are 2 things at play here:
recursive syntax making use of LazyList API
corecursive mathematics behind unfolding
So, let's start with a few words about API and syntax:
#:: takes lazy value and prepends it to LazyList definition, here it is fibs which makes its definition recursive on code level
LazyList lazily evaluates its arguments and then caches/memoizes them for future use letting us access already computed values immediately
However, the mechanism underneath is actually corecursive.
Let's see what is recursion when it comes to data using List as an example:
List(1,2,3,4)
This can be also written as
1 :: 2 :: 3 :: 4 :: Nil
Which is the same as
( ( ( Nil.::(4) ).::(3) ).::(2) ).::(1)
You can see that we:
take Nil
create ::(4, Nil) value which we use to
create ::(3, ::(4, Nil)) value
and so on
In other words, we have to start with some base case and build the whole things from-bottom-up. Such values by definition have to be finite and cannot be used to express series of (possibly) infinite computation.
But there exist an alternative which allows you to express such computations - corecursion and codata.
With corecursion you have a tuple:
the last computed value
a function which can take the value and return the next tuple (next value + next function!)
nothing prevent you from using the same function as second element of the tuple but it's good to have a choice
For instance you could define infinite series of LazyList(1, 2, 3, 4, 5, 6, ...) like:
// I use case class since
// type Pair = (Int, Int => Pair)
// would be illegal in Scala
final case class Pair(value: Int, f: Int => Pair)
val f: Int => Pair = n => Pair(n + 1, f)
Pair(1, f)
Then you would take Pair, get value out of it (1 initially) and use it to generate new Pairs (Pair(2, f), Pair(3, f), ...).
Structure which would use corecursion to generate its values would be called codata (so LazyList can be considered codata).
Same story with Fibonacci sequence, you could define it corecursively with
(Int, Int) as value (initialized to (0, 1)
val f: (Int, Int) => Pair = { case (n, m) => Pair((m, n + m), f } as function
finally, you'd have to pick _1 out of every generated (Int, Int) pair
However, LazyList's API gives you some nice tools so that you don't have to do this manually:
it memoizes (caches) computed values so you can access list(0), list(1), etc, they aren't forgotten right after use
it gives you methods like .map, .flatMap .scanLeft and so on, so while internally it might have more complex types used for corecursion, you are only seeing the final result that you need
Obviously, all of that is done lazily, by codata's definition: at each step you can only know values defined so far, and how to generate next of out it.
That leads us to your example:
val fibs: LazyList[Int] = (0 #:: fibs).scanLeft(1)(_ + _)
You can think of it as something that:
starts with a pair (0, f)
where the f takes this 0 argument, and combines it with 1 to create (0, 1) tuple
and then constructs next fs which trace the previous value, and passes it along current value to the function passed into scanLeft
where all the shenanigans with intermediate values and functions and memoization are handled internally by API
So if you asked me, the "base case" of such algos is a pair of value and function returning pair, run over and over again.
How do they unfold?
They don't. The #:: function takes a by-name argument, which means that it's evaluated lazily.
What is the base case for such algos (if it's Nil, why it's so?).
There is no "base case", these recursive definitions yield infinite streams:
scala> val fibs: LazyList[Int] = (0 #:: fibs).scanLeft(1)(_ + _)
val fibs: LazyList[Int] = LazyList(<not computed>)
scala> fibs.size
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
(Note the "<not computed>" token, which hints at the laziness)
In the below two scenarios, I have flatMap function invoked on a list. In both the cases, the map portion of the flatMap function returns an array which has a iterator. In the first case, the code errors out where in the second case, it produces the expected result.
Scenario-1
val x = List("abc","cde")
x flatMap ( e => e.toArray)
<console>:13: error: polymorphic expression cannot be instantiated to expected type;
found : [B >: Char]Array[B]
required: scala.collection.GenTraversableOnce[?]
x flatMap ( e => e.toArray)
Scenario-2
val x = List("abc,def")
x flatMap ( e => e.split(",") )
res1: List[String] = List(abc, def) //Result
Can you please help why in the first case, it is not behaving as expected ?
I think the difference is that in scenario 1 you actually have a Array[B] where B is some yet-to-be-decided supertype of Char. Normally the compiler would look for an implicit conversion to GenTraversableOnce but because B is not yet known you run into a type inference issue / limitation.
You can help type inference by filling in B.
List("abc", "cde").flatMap(_.toArray[Char])
Or even better, you don't need flatMap in this case. Just call flatten.
List("abc", "cde").flatten
It is worth keeping in mind that Array is not a proper Scala collection so compiler has to work harder to first convert it to something that fits with the rest of Scala collections
implicitly[Array[Char] <:< GenTraversableOnce[Char]] // error (Scala 2.12)
implicitly[Array[Char] <:< IterableOnce[Char]] // error (Scala 2.13)
Hence because flatMap takes a function
String => GenTraversableOnce[Char]
but we are passing in
String => Array[Char]
the compiler first has to find an appropriate implicit conversion of Array[Char] to GenTraversableOnce[Char]. Namely this should be wrapCharArray
scala.collection.immutable.List.apply[String]("abc", "cde").flatMap[Char](((e: String) => scala.Predef.wrapCharArray(scala.Predef.augmentString(e).toArray[Char]((ClassTag.Char: scala.reflect.ClassTag[Char])))))
or shorter
List("abc", "cde").flatMap(e => wrapCharArray(augmentString(e).toArray))
however due to type inference reasons explained by Jasper-M, the compiler is unable to select wrapCharArray conversion. As Daniel puts it
Array tries to be a Java Array and a Scala Collection at the same
time. It mostly succeeds, but fail at both for some corner cases.
Perhaps this is one of those corner cases. For these reasons it is best to avoid Array unless performance or compatibility reasons dictate it. Nevertheless another approach that seems to work in Scala 2.13 is to use to(Collection) method
List("abc","cde").flatMap(_.to(Array))
scala> val x = List("abc","cde")
x: List[String] = List(abc, cde)
scala> x.flatMap[Char](_.toArray)
res0: List[Char] = List(a, b, c, c, d, e)
As the error mentions, your type is wrong.
In the first case, if you appy map not flatmap, you obtain List[Array[Char]]. If you apply flatten to that, you get a List[Char]
In the second case, if you appy map not flatmap, you obtain List[Array[String]]. If you apply flatten to that, you get a List[String]
I suppose you need to transform String to Char in your Array, in order to make it work.
I use Scala 2.13 and still have the same error.
I'm finding that I often have to chain together functions that work on an Option and return a different Option that look something like this:
if(foo.isDefined) someFunctionReturningOption(foo.get) else None
Is there a cleaner way to do this? This pattern gets quite verbose with more complicated variables.
I'm seeing it a fair bit in form handling code that has to deal with optional data. It'll insert None if the value is None or some transformation (which could potentially fail) if there is some value.
This is very much like the ?. operator proposed for C#.
You can use flatMap:
foo.flatMap(someFunctionReturningOption(_))
Or in a for-comprehension:
for {
f <- foo
r <- someFunctionReturningOption(f)
} yield r
The for-comprehension is preferred when chaining multiple instances of these functions together, as they de-sugar to flatMaps.
There're a lot of options (pun intended) but for comprehensions, I guess, is the most convinient in case of chains
for {
x <- xOpt
y <- someFunctionReturningOption(x)
z <- anotherFunctionReturningOption(y)
} yield z
You're looking for flatMap:
foo.flatMap(someFunctionReturningOption)
This fits into the general monadic structure, where a monad wrapping a type uses flatMap to return the same type (e.g. flatMap on Seq[T] returns a Seq).
Option supports map() so when x is an Option[Int] this construct:
if (x.isDefined)
"number %d".format(x.get)
else
None
is easier to write as:
x map (i => "number %d".format(i))
map will keep None unmodified, but it will apply the function you pass to it to any value, and wrap the result back into an Option. For example note how 'x' gets converted to a string message below, but 'y' gets passed along as None:
scala> val x: Option[Int] = Some(3)
x: Option[Int] = Some(3)
scala> val y: Option[Int] = None
y: Option[Int] = None
scala> x map (i => "number %d".format(i))
res0: Option[String] = Some(number 3)
scala> y map (i => "number %d".format(i))
res1: Option[String] = None
In scala, how do I define addition over two Option arguments? Just to be specific, let's say they're wrappers for Int types (I'm actually working with maps of doubles but this example is simpler).
I tried the following but it just gives me an error:
def addOpt(a:Option[Int], b:Option[Int]) = {
a match {
case Some(x) => x.get
case None => 0
} + b match {
case Some(y) => y.get
case None => 0
}
}
Edited to add:
In my actual problem, I'm adding two maps which are standins for sparse vectors. So the None case returns Map[Int, Double] and the + is actually a ++ (with the tweak at stackoverflow.com/a/7080321/614684)
Monoids
You might find life becomes a lot easier when you realize that you can stand on the shoulders of giants and take advantage of common abstractions and the libraries built to use them. To this end, this question is basically about dealing with
monoids (see related questions below for more about this) and the library in question is called scalaz.
Using scalaz FP, this is just:
def add(a: Option[Int], b: Option[Int]) = ~(a |+| b)
What is more this works on any monoid M:
def add[M: Monoid](a: Option[M], b: Option[M]) = ~(a |+| b)
Even more usefully, it works on any number of them placed inside a Foldable container:
def add[M: Monoid, F: Foldable](as: F[Option[M]]) = ~as.asMA.sum
Note that some rather useful monoids, aside from the obvious Int, String, Boolean are:
Map[A, B: Monoid]
A => (B: Monoid)
Option[A: Monoid]
In fact, it's barely worth the bother of extracting your own method:
scala> some(some(some(1))) #:: some(some(some(2))) #:: Stream.empty
res0: scala.collection.immutable.Stream[Option[Option[Option[Int]]]] = Stream(Some(Some(Some(1))), ?)
scala> ~res0.asMA.sum
res1: Option[Option[Int]] = Some(Some(3))
Some related questions
Q. What is a monoid?
A monoid is a type M for which there exists an associative binary operation (M, M) => M and an identity I under this operation, such that mplus(m, I) == m == mplus(I, m) for all m of type M
Q. What is |+|?
This is just scalaz shorthand (or ASCII madness, ymmv) for the mplus binary operation
Q. What is ~?
It is a unary operator meaning "or identity" which is retrofitted (using scala's implicit conversions) by the scalaz library onto Option[M] if M is a monoid. Obviously a non-empty option returns its contents; an empty option is replaced by the monoid's identity.
Q. What is asMA.sum?
A Foldable is basically a datastructure which can be folded over (like foldLeft, for example). Recall that foldLeft takes a seed value and an operation to compose successive computations. In the case of summing a monoid, the seed value is the identity I and the operation is mplus. You can hence call asMA.sum on a Foldable[M : Monoid]. You might need to use asMA because of the name clash with the standard library's sum method.
Some References
Slides and Video of a talk I gave which gives practical examples of using monoids in the wild
def addOpts(xs: Option[Int]*) = xs.flatten.sum
This will work for any number of inputs.
If they both default to 0 you don't need pattern matching:
def addOpt(a:Option[Int], b:Option[Int]) = {
a.getOrElse(0) + b.getOrElse(0)
}
(Repeating comment above in an answer as requested)
You don't extract the content of the option the proper way. When you match with case Some(x), x is the value inside the option(type Int) and you don't call get on that. Just do
case Some(x) => x
Anyway, if you want content or default, a.getOrElse(0) is more convenient
def addOpt(ao: Option[Int], bo: Option[Int]) =
for {
a <- ao
b <- bo
} yield a + b
I have learned the basic difference between foldLeft and reduceLeft
foldLeft:
initial value has to be passed
reduceLeft:
takes first element of the collection as initial value
throws exception if collection is empty
Is there any other difference ?
Any specific reason to have two methods with similar functionality?
Few things to mention here, before giving the actual answer:
Your question doesn't have anything to do with left, it's rather about the difference between reducing and folding
The difference is not the implementation at all, just look at the signatures.
The question doesn't have anything to do with Scala in particular, it's rather about the two concepts of functional programming.
Back to your question:
Here is the signature of foldLeft (could also have been foldRight for the point I'm going to make):
def foldLeft [B] (z: B)(f: (B, A) => B): B
And here is the signature of reduceLeft (again the direction doesn't matter here)
def reduceLeft [B >: A] (f: (B, A) => B): B
These two look very similar and thus caused the confusion. reduceLeft is a special case of foldLeft (which by the way means that you sometimes can express the same thing by using either of them).
When you call reduceLeft say on a List[Int] it will literally reduce the whole list of integers into a single value, which is going to be of type Int (or a supertype of Int, hence [B >: A]).
When you call foldLeft say on a List[Int] it will fold the whole list (imagine rolling a piece of paper) into a single value, but this value doesn't have to be even related to Int (hence [B]).
Here is an example:
def listWithSum(numbers: List[Int]) = numbers.foldLeft((List.empty[Int], 0)) {
(resultingTuple, currentInteger) =>
(currentInteger :: resultingTuple._1, currentInteger + resultingTuple._2)
}
This method takes a List[Int] and returns a Tuple2[List[Int], Int] or (List[Int], Int). It calculates the sum and returns a tuple with a list of integers and it's sum. By the way the list is returned backwards, because we used foldLeft instead of foldRight.
Watch One Fold to rule them all for a more in depth explanation.
reduceLeft is just a convenience method. It is equivalent to
list.tail.foldLeft(list.head)(_)
foldLeft is more generic, you can use it to produce something completely different than what you originally put in. Whereas reduceLeft can only produce an end result of the same type or super type of the collection type. For example:
List(1,3,5).foldLeft(0) { _ + _ }
List(1,3,5).foldLeft(List[String]()) { (a, b) => b.toString :: a }
The foldLeft will apply the closure with the last folded result (first time using initial value) and the next value.
reduceLeft on the other hand will first combine two values from the list and apply those to the closure. Next it will combine the rest of the values with the cumulative result. See:
List(1,3,5).reduceLeft { (a, b) => println("a " + a + ", b " + b); a + b }
If the list is empty foldLeft can present the initial value as a legal result. reduceLeft on the other hand does not have a legal value if it can't find at least one value in the list.
For reference, reduceLeft will error if applied to an empty container with the following error.
java.lang.UnsupportedOperationException: empty.reduceLeft
Reworking the code to use
myList foldLeft(List[String]()) {(a,b) => a+b}
is one potential option. Another is to use the reduceLeftOption variant which returns an Option wrapped result.
myList reduceLeftOption {(a,b) => a+b} match {
case None => // handle no result as necessary
case Some(v) => println(v)
}
The basic reason they are both in Scala standard library is probably because they are both in Haskell standard library (called foldl and foldl1). If reduceLeft wasn't, it would quite often be defined as a convenience method in different projects.
From Functional Programming Principles in Scala (Martin Odersky):
The function reduceLeft is defined in terms of a more general function, foldLeft.
foldLeft is like reduceLeft but takes an accumulator z, as an additional parameter, which is returned when foldLeft is called on an empty list:
(List (x1, ..., xn) foldLeft z)(op) = (...(z op x1) op ...) op x
[as opposed to reduceLeft, which throws an exception when called on an empty list.]
The course (see lecture 5.5) provides abstract definitions of these functions, which illustrates their differences, although they are very similar in their use of pattern matching and recursion.
abstract class List[T] { ...
def reduceLeft(op: (T,T)=>T) : T = this match{
case Nil => throw new Error("Nil.reduceLeft")
case x :: xs => (xs foldLeft x)(op)
}
def foldLeft[U](z: U)(op: (U,T)=>U): U = this match{
case Nil => z
case x :: xs => (xs foldLeft op(z, x))(op)
}
}
Note that foldLeft returns a value of type U, which is not necessarily the same type as List[T], but reduceLeft returns a value of the same type as the list).
To really understand what are you doing with fold/reduce,
check this: http://wiki.tcl.tk/17983
very good explanation. once you get the concept of fold,
reduce will come together with the answer above:
list.tail.foldLeft(list.head)(_)
Scala 2.13.3, Demo:
val names = List("Foo", "Bar")
println("ReduceLeft: "+ names.reduceLeft(_+_))
println("ReduceRight: "+ names.reduceRight(_+_))
println("Fold: "+ names.fold("Other")(_+_))
println("FoldLeft: "+ names.foldLeft("Other")(_+_))
println("FoldRight: "+ names.foldRight("Other")(_+_))
outputs:
ReduceLeft: FooBar
ReduceRight: FooBar
Fold: OtherFooBar
FoldLeft: OtherFooBar
FoldRight: FooBarOther