How does flatmap really work in Scala - scala

I took the scala odersky course and thought that the function that Flatmap takes as arguments , takes an element of Monad and returns a monad of different type.
trait M[T] {
def flatMap[U](f: T => M[U]): M[U]
}
On Monad M[T] , the return type of function is also the same Monad , the type parameter U might be different.
However I have seen examples on internet , where the function returns a completely different Monad. I was under impression that return type of function should the same Monad. Can someone simplify the below to explain how flapmap results in the actual value instead of Option in the list.
Is the List not a Monad in Scala.
val l= List(1,2,3,4,5)
def f(x:int) = if (x>2) Some(x) else None
l.map(x=>f(x))
//Result List[Option[Int]] = List( None , None , Some(3) , Some(4) , Some(5))
l.flatMap(x=>f(x))
//Result: List(3,4,5)

Let's start from the fact that M[T]is not a monad by itself. It's a type constructor. It becomes a monad when it's associated with two operators: bind and return (or unit). There are also monad laws these operators must satisfy, but let's omit them for brevity. In Haskell the type of bind is:
class Monad m where
...
(>>=) :: m a -> (a -> m b) -> m b
where m is a type constructor. Since Scala is OO language bind will look like (first argument is self):
trait M[T] {
def bind[U](f: T => M[U]): M[U]
}
Here M === m, T === a, U === b. bind is often called flatMap. In a pure spherical world in a vacuum that would be a signature of flatMap in OO language. Scala is a very practical language, so the real signature of flatMap for List is:
final def flatMap[B, That](f: (A) ⇒ GenTraversableOnce[B])(implicit bf: CanBuildFrom[List[A], B, That]): That
It's not bind, but will work as a monadic bind if you provide f in the form of (A) => List[B] and also make sure that That is List[B]. On the other hand Scala is not going to watch your back if you provide something different, but will try to find some meaningful conversion (e.g. CanBuildFrom or something else) if it exists.
UPDATE
You can play with scalac flags (-Xlog-implicits, -Xlog-implicit-conversions) to see what's happening:
scala> List(1).flatMap { x => Some(x) }
<console>:1: inferred view from Some[Int] to scala.collection.GenTraversableOnce[?] via scala.this.Option.option2Iterable[Int]: (xo: Option[Int])Iterable[Int]
List(1).flatMap { x => Some(x) }
^
res1: List[Int] = List(1)

Hmm, perhaps confusingly, the signature you gave is not actually correct, since it's really (in simplified form):
def flatMap[B](f: (A) ⇒ GenTraversableOnce[B]): Traversable[B]
Since the compiler is open-source, you can actually see what it's doing (with its full signature):
def flatMap[B, That](f: A => GenTraversableOnce[B])(implicit bf: CanBuildFrom[Repr, B, That]): That = {
def builder = bf(repr) // extracted to keep method size under 35 bytes, so that it can be JIT-inlined
val b = builder
for (x <- this) b ++= f(x).seq
b.result
}
So you can see that there is actually no requirement that the return type of f be the same as the return type of flatMap.

The flatmap found in the standard library is a much more general and flexible method than the monadic bind method like the flatMap from Odersky's example.
For example, the full signature of flatmap on List is
def flatMap[B, That](f: (A) ⇒ GenTraversableOnce[B])(implicit bf: CanBuildFrom[List[A], B, That]): That
Instead of requiring the function passed into flatmap to return a List, it is able to return any GenTraversableOnce object, a very generic type.
flatmap then uses the implicit CanBuildFrom mechanism to determine the appropriate type to return.
So when you use flatmap with a function that returns a List, it is a monadic bind operation, but it lets you use other types as well.

Related

Understanding monads in scala

I'm trying to understand what monads are (not just in scala, but by example using scala). Let's consider the most (in my opinion) simple example of a monad:
scala.Some
As some articles state, every monad in its classic sense should preserve some rules for the flatMap and unit functions.
Here is the definition from scala.Some
#inline final def flatMap[B](f: A => Option[B]): Option[B]
So, understand it better I want to understand it from the category theory standpoint. So, we're considering a monad and it's supposed to be a functor (but between what?).
Here we have to category Option[A] and Option[B] and the flatMap along with the f: A => Option[B] passed into it is supposed to define a Functor between them. But in the tranditional category definition it's a functor from a category to itself.
The category is the category of scala types, where the objects are types and the arrows are functions between values of those types. Option is an endofunctor on this category. For each object (i.e. type) in the Scala category, the Option type constructor maps each type A into a type Option[A].
In addition it maps each arrow f: A => B into an arrow fo: Option[A] => Option[B] which is what Option.map does.
A Monad is a Functor M along with two operations, unit: A => M[A] and join: M[M[A]] => M[A]. For Option, unit(x: A) = Some(x) and join can be defined as:
def join[A](o: Option[Option[A]]): Option[A] = o match {
case None => None
case Some(i) => i
}
flatMap can then be defined as, flatMap(f, m) = join(map(f, m)). Alternatively the monad can be defined using unit and flatMap and join defined as join(m) = flatMap(id, m).

Curry in scala with parametric types

The authors of Functional Programming in Scala give this as the definition of curry in scala:
def curry[A,B,C](f: (A, B) => C): A => (B => C) =
a => b => f(a, b)
However, if we apply it to a function taking parametric types, e.g.:
def isSorted[A](as: Array[A], ordered:(A,A)=>Boolean) =
if(as.size < 2)
true else
as.zip(as.drop(1)).map(ordered.tupled).reduce(_ && _)
Then the result wants A (in isSorted) to be nothing:
scala> curry(isSorted)
res29: Array[Nothing] => (((Nothing, Nothing) => Boolean) => Boolean) = <function1>
This is obviously not what is desired. Should curry be defined differently, or called differently, or is it not practical to implement curry in Scala?
You're running into two separate problems here. The first is that isSorted when it is passed to curry is forced to become monomorphic. The second is that Scala's type inference is failing you here.
This is one of those times where the difference between a function and a method matters in Scala. isSorted is eta-expanded into a function which in turn is a Scala value, not a method. Scala values are always monomorphic, only methods can be polymorphic. For any method of types (A, B) C (this is the syntax for a method type and is different from (A, B) => C which is a function and therefore a value), the default eta-expansion is going to result in the superclass of all functions of that arity, namely (Nothing, Nothing) => Any. This is responsible for all the Nothings you see (you don't have any Anys because isSorted is monomorphic in its return value).
You might imagine despite the monomorphic nature of Scala values though, that you could ideally do something like
def first[A, B](x: A, y: B): A = x
curry(first)(5)(6) // This doesn't compile
This is Scala's local type inference biting you. It works on separate parameter lists from left to right first is the first thing to get a type inferred and as mentioned above, it gets inferred to be (Nothing, Nothing) => Any. This clashes with Ints that follow.
As you've realized, one way of getting around this is annotating your polymorphic method that you pass to curry so that it eta-expands into the correct type. This is almost certainly the way to go.
Another thing you could possibly do (although I'm don't think it'll serve anything except pedagogical purposes) is to curry curry itself and define it as follows:
def curryTwo[A, B, C](x: A)(y: B)(f: (A, B) => C): C = f(x, y)
On the one hand, the below works now because of the left-to-right type inference.
curryTwo(5)(6)(first) // 5
On the other hand, to use curryTwo in the scenarios where you'd want to use curry, you're going to need to need to provide types to Scala's type inference engine anyway.
It turns out I can call curry like this:
curry(isSorted[Int])
Which yields:
scala> curry(isSorted[Int])
res41: Array[Int] => (((Int, Int) => Boolean) => Boolean) = <function1>
See https://stackoverflow.com/a/4593509/21640

Comparing Haskell and Scala Bind/Flatmap Examples

The following bind(>>=) code, in Haskell, does not compile:
ghci> [[1]] >>= Just
<interactive>:38:11:
Couldn't match type ‘Maybe’ with ‘[]’
Expected type: [t] -> [[t]]
Actual type: [t] -> Maybe [t]
In the second argument of ‘(>>=)’, namely ‘Just’
In the expression: [[1]] >>= Just
But, in Scala, it does actually compile and run:
scala> List( List(1) ).flatMap(x => Some(x) )
res1: List[List[Int]] = List(List(1))
Haskell's >>= signature is:
>>= :: Monad m => m a -> (a -> m b) -> m b
So, in [[1]] >>= f, f's type should be: a -> [b].
Why does the Scala code compile?
As #chi explained Scala's flatMap is more general than the Haskell's >>=. The full signature from the Scala docs is:
final def flatMap[B, That](f: (A) ⇒ GenTraversableOnce[B])(implicit bf: CanBuildFrom[List[A], B, That]): That
This implicit isn't relevant for this specific problem, so we could as well use the simpler definition:
final def flatMap[B](f: (A) ⇒ GenTraversableOnce[B]): List[B]
There is only one Problem, Option is no subclass of GenTraversableOnce, here an implicit conversion comes in. Scala defines an implicit conversion from Option to Iterable which is a subclass of Traversable which is a subclass of GenTraversableOnce.
implicit def option2Iterable[A](xo: Option[A]): Iterable[A]
The implicit is defined in the companion object of Option.
A simpler way to see the implicit at work is to assign a Option to an Iterable val:
scala> val i:Iterable[Int] = Some(1)
i: Iterable[Int] = List(1)
Scala uses some defaulting rules, to select List as the implementation of Iterable.
The fact that you can combine different subtypes of TraversableOnce with monad operations comes from the implicit class MonadOps:
implicit class MonadOps[+A](trav: TraversableOnce[A]) {
def map[B](f: A => B): TraversableOnce[B] = trav.toIterator map f
def flatMap[B](f: A => GenTraversableOnce[B]): TraversableOnce[B] = trav.toIterator flatMap f
def withFilter(p: A => Boolean) = trav.toIterator filter p
def filter(p: A => Boolean): TraversableOnce[A] = withFilter(p)
}
This enhances every TraversableOnce with the methods above. The subtypes are free to define more efficient versions on there own, these will shadow the implicit definitions. This is the case for List.
Quoting from the Scala reference for List
final def flatMap[B](f: (A) ⇒ GenTraversableOnce[B]): List[B]
So, flatMap is more general than Haskell's (>>=), since it only requires the mapped function f to generate a traversable type, not necessarily a List.

Folding on Type without Monoid Instance

I'm working on this Functional Programming in Scala exercise:
// But what if our list has an element type that doesn't have a Monoid instance?
// Well, we can always map over the list to turn it into a type that does.
As I understand this exercise, it means that, if we have a Monoid of type B, but our input List is of type A, then we need to convert the List[A] to List[B], and then call foldLeft.
def foldMap[A, B](as: List[A], m: Monoid[B])(f: A => B): B = {
val bs = as.map(f)
bs.foldLeft(m.zero)((s, i) => m.op(s, i))
}
Does this understanding and code look right?
First I'd simplify the syntax of the body a bit:
def foldMap[A, B](as: List[A], m: Monoid[B])(f: A => B): B =
as.map(f).foldLeft(m.zero)(m.ops)
Then I'd move the monoid instance into its own implicit parameter list:
def foldMap[A, B](as: List[A])(f: A => B)(implicit m: Monoid[B]): B =
as.map(f).foldLeft(m.zero)(m.ops)
See the original "Type Classes as Objects and Implicits" paper for more detail about how Scala implements type classes using implicit parameter resolution, or this answer by Rex Kerr that I've also linked above.
Next I'd switch the order of the other two parameter lists:
def foldMap[A, B](f: A => B)(as: List[A])(implicit m: Monoid[B]): B =
as.map(f).foldLeft(m.zero)(m.ops)
In general you want to place the parameter lists containing parameters that change the least often first, in order to make partial application more useful. In this case there may only be one possible useful value of A => B for any A and B, but there are lots of values of List[A].
For example, switching the order allows us to write the following (which assumes a monoid instance for Bar):
val fooSum: List[Foo] => Bar = foldMap(fooToBar)
Finally, as a performance optimization (mentioned by stew above), you could avoid creating an intermediate list by moving the application of f into the fold:
def foldMap[A, B](f: A => B)(as: List[A])(implicit m: Monoid[B]): B =
as.foldLeft(m.zero) {
case (acc, a) => m.op(acc, f(a))
}
This is equivalent and more efficient, but to my eye much less clear, so I'd suggest treating it like any optimization—if you need it, use it, but think twice about whether the gains are really worth the loss of clarity.

Function syntax puzzler in scalaz

Following watching Nick Partidge's presentation on deriving scalaz, I got to looking at this example, which is just awesome:
import scalaz._
import Scalaz._
def even(x: Int) : Validation[NonEmptyList[String], Int]
= if (x % 2 ==0) x.success else "not even: %d".format(x).wrapNel.fail
println( even(3) <|*|> even(5) ) //prints: Failure(NonEmptyList(not even: 3, not even: 5))
I was trying to understand what the <|*|> method was doing, here is the source code:
def <|*|>[B](b: M[B])(implicit t: Functor[M], a: Apply[M]): M[(A, B)]
= <**>(b, (_: A, _: B))
OK, that is fairly confusing (!) - but it references the <**> method, which is declared thus:
def <**>[B, C](b: M[B], z: (A, B) => C)(implicit t: Functor[M], a: Apply[M]): M[C]
= a(t.fmap(value, z.curried), b)
So I have a few questions:
How come the method appears to take a higher-kinded type of one type parameter (M[B]) but can get passed a Validation (which has two type paremeters)?
The syntax (_: A, _: B) defines the function (A, B) => Pair[A,B] which the 2nd method expects: what is happening to the Tuple2/Pair in the failure case? There's no tuple in sight!
Type Constructors as Type Parameters
M is a type parameter to one of Scalaz's main pimps, MA, that represents the Type Constructor (aka Higher Kinded Type) of the pimped value. This type constructor is used to look up the appropriate instances of Functor and Apply, which are implicit requirements to the method <**>.
trait MA[M[_], A] {
val value: M[A]
def <**>[B, C](b: M[B], z: (A, B) => C)(implicit t: Functor[M], a: Apply[M]): M[C] = ...
}
What is a Type Constructor?
From the Scala Language Reference:
We distinguish between first-order
types and type constructors, which
take type parameters and yield types.
A subset of first-order types called
value types represents sets of
(first-class) values. Value types are
either concrete or abstract. Every
concrete value type can be represented
as a class type, i.e. a type
designator (§3.2.3) that refers to a
class1 (§5.3), or as a compound type
(§3.2.7) representing an intersection
of types, possibly with a refinement
(§3.2.7) that further constrains the
types of itsmembers. Abstract value
types are introduced by type
parameters (§4.4) and abstract type
bindings (§4.3). Parentheses in types
are used for grouping. We assume that
objects and packages also implicitly
define a class (of the same name as
the object or package, but
inaccessible to user programs).
Non-value types capture properties of
identifiers that are not values
(§3.3). For example, a type
constructor (§3.3.3) does not directly
specify the type of values. However,
when a type constructor is applied to
the correct type arguments, it yields
a first-order type, which may be a
value type. Non-value types are
expressed indirectly in Scala. E.g., a
method type is described by writing
down a method signature, which in
itself is not a real type, although it
gives rise to a corresponding function
type (§3.3.1). Type constructors are
another example, as one can write type
Swap[m[_, _], a,b] = m[b, a], but
there is no syntax to write the
corresponding anonymous type function
directly.
List is a type constructor. You can apply the type Int to get a Value Type, List[Int], which can classify a value. Other type constructors take more than one parameter.
The trait scalaz.MA requires that it's first type parameter must be a type constructor that takes a single type to return a value type, with the syntax trait MA[M[_], A] {}. The type parameter definition describes the shape of the type constructor, which is referred to as its Kind. List is said to have the kind '* -> *.
Partial Application of Types
But how can MA wrap a values of type Validation[X, Y]? The type Validation has a kind (* *) -> *, and could only be passed as a type argument to a type parameter declared like M[_, _].
This implicit conversion in object Scalaz converts a value of type Validation[X, Y] to a MA:
object Scalaz {
implicit def ValidationMA[A, E](v: Validation[E, A]): MA[PartialApply1Of2[Validation, E]#Apply, A] = ma[PartialApply1Of2[Validation, E]#Apply, A](v)
}
Which in turn uses a trick with a type alias in PartialApply1Of2 to partially apply the type constructor Validation, fixing the type of the errors, but leaving the type of the success unapplied.
PartialApply1Of2[Validation, E]#Apply would be better written as [X] => Validation[E, X]. I recently proposed to add such a syntax to Scala, it might happen in 2.9.
Think of this as a type level equivalent of this:
def validation[A, B](a: A, b: B) = ...
def partialApply1Of2[A, B C](f: (A, B) => C, a: A): (B => C) = (b: B) => f(a, b)
This lets you combine Validation[String, Int] with a Validation[String, Boolean], because the both share the type constructor [A] Validation[String, A].
Applicative Functors
<**> demands the the type constructor M must have associated instances of Apply and Functor. This constitutes an Applicative Functor, which, like a Monad, is a way to structure a computation through some effect. In this case the effect is that the sub-computations can fail (and when they do, we accumulate the failures).
The container Validation[NonEmptyList[String], A] can wrap a pure value of type A in this 'effect'. The <**> operator takes two effectful values, and a pure function, and combines them with the Applicative Functor instance for that container.
Here's how it works for the Option applicative functor. The 'effect' here is the possibility of failure.
val os: Option[String] = Some("a")
val oi: Option[Int] = Some(2)
val result1 = (os <**> oi) { (s: String, i: Int) => s * i }
assert(result1 == Some("aa"))
val result2 = (os <**> (None: Option[Int])) { (s: String, i: Int) => s * i }
assert(result2 == None)
In both cases, there is a pure function of type (String, Int) => String, being applied to effectful arguments. Notice that the result is wrapped in the same effect (or container, if you like), as the arguments.
You can use the same pattern across a multitude of containers that have an associated Applicative Functor. All Monads are automatically Applicative Functors, but there are even more, like ZipStream.
Option and [A]Validation[X, A] are both Monads, so you could also used Bind (aka flatMap):
val result3 = oi flatMap { i => os map { s => s * i } }
val result4 = for {i <- oi; s <- os} yield s * i
Tupling with `<|**|>`
<|**|> is really similar to <**>, but it provides the pure function for you to simply build a Tuple2 from the results. (_: A, _ B) is a shorthand for (a: A, b: B) => Tuple2(a, b)
And beyond
Here's our bundled examples for Applicative and Validation. I used a slightly different syntax to use the Applicative Functor, (fa ⊛ fb ⊛ fc ⊛ fd) {(a, b, c, d) => .... }
UPDATE: But what happens in the Failure Case?
what is happening to the Tuple2/Pair in the failure case?
If any of the sub-computations fails, the provided function is never run. It only is run if all sub-computations (in this case, the two arguments passed to <**>) are successful. If so, it combines these into a Success. Where is this logic? This defines the Apply instance for [A] Validation[X, A]. We require that the type X must have a Semigroup avaiable, which is the strategy for combining the individual errors, each of type X, into an aggregated error of the same type. If you choose String as your error type, the Semigroup[String] concatenates the strings; if you choose NonEmptyList[String], the error(s) from each step are concatenated into a longer NonEmptyList of errors. This concatenation happens below when two Failures are combined, using the ⊹ operator (which expands with implicits to, for example, Scalaz.IdentityTo(e1).⊹(e2)(Semigroup.NonEmptyListSemigroup(Semigroup.StringSemigroup)).
implicit def ValidationApply[X: Semigroup]: Apply[PartialApply1Of2[Validation, X]#Apply] = new Apply[PartialApply1Of2[Validation, X]#Apply] {
def apply[A, B](f: Validation[X, A => B], a: Validation[X, A]) = (f, a) match {
case (Success(f), Success(a)) => success(f(a))
case (Success(_), Failure(e)) => failure(e)
case (Failure(e), Success(_)) => failure(e)
case (Failure(e1), Failure(e2)) => failure(e1 ⊹ e2)
}
}
Monad or Applicative, how shall I choose?
Still reading? (Yes. Ed)
I've shown that sub-computations based on Option or [A] Validation[E, A] can be combined with either Apply or with Bind. When would you choose one over the other?
When you use Apply, the structure of the computation is fixed. All sub-computations will be executed; the results of one can't influence the the others. Only the 'pure' function has an overview of what happened. Monadic computations, on the other hand, allow the first sub-computation to influence the later ones.
If we used a Monadic validation structure, the first failure would short-circuit the entire validation, as there would be no Success value to feed into the subsequent validation. However, we are happy for the sub-validations to be independent, so we can combine them through the Applicative, and collect all the failures we encounter. The weakness of Applicative Functors has become a strength!