I watch this video and starts at 6min35s, it mentioned this graph:
saying that the IO Monad makes it easy to handle concurrency. I got confused on this: how does it work? How do the two for comprehension enable the concurrency (the computation of d and f)?
No, it doesn't enable the concurrency
for comprehensions only help you omitting several parentheses and indents.
The code you referred, translate to [flat]map is strictly equivalent to:
async.boundedQueue[Stuff](100).flatMap{ a =>
val d = computeB(a).flatMap{
b => computeD(b).map{ result =>
result
}
}
val f = computeC(a).flatMap{ c =>
computeE(c).flatMap{ e =>
computeF(e).map{ result =>
result
}
}
}
d.merge(f).map(g => g)
}
See, it only helps you omitting several parentheses and indents (joke)
The concurrency is hidden in flatMap and map
Once you understand how for is translated to flatMap and map, you can implement your concurrency inside them.
As map takes a function as argument, it doesn't mean that the function is executed during execution of map function, you can defer the function to another thread or run it latter. This is how concurrency implemented.
Take Promise and Future as example:
val future: Future = ```some promise```
val r: Future = for (v <- future) yield doSomething(v)
// or
val r: Future = future.map(v => doSomething(v))
r.wait
The function doSomething is not executed during the execution of Future.map function, instead it is called when the promise commits.
Conclusion
How to implement concurrency using for syntax suger:
for will convert into flatMap and map by scala compiler
Write you flatMap and map, where you will get a callback function from argument
Call the function you got whenever and wherever you like
Further reading
The flow control feature of many languages share a same property, they are like delimited continuation shift/reset, where they capture the following execution upto a scope into an function.
JavaScript:
async function() {
...
val yielded = await new Promise((resolve) => shift(resolve))
// resolve will captured execution of following statements upto end of the function.
...captured
}
Haskell:
do {
...
yielded_monad <- ```some monad``` -- shift function is >>= of the monad
...captured
}
Scala:
for {
...
yielded_monad <- ```some monad``` // shift function is flatMap/map of the monad
...captured
} yield ...
next time you see a language feature which capture following execution into a function, you know you can implement flow control using the feature.
The difference between delimited continuation and call/cc is that call/cc capture the whole following execution of the program, but delimited continuation has a scope.
Related
I'm new to functional programming and Scala, and I was checking out the Cats Effect framework and trying to understand what the IO monad does. So far what I've understood is that writing code in the IO block is just a description of what needs to be done and nothing happens until you explicitly run using the unsafe methods provided, and also a way to make code that performs side-effects referentially transparent by actually not running it.
I tried executing the snippet below just to try to understand what it means:
object Playground extends App {
var out = 10
var state = "paused"
def changeState(newState: String): IO[Unit] = {
state = newState
IO(println("Updated state."))
}
def x(string: String): IO[Unit] = {
out += 1
IO(println(string))
}
val tuple1 = (x("one"), x("two"))
for {
_ <- x("1")
_ <- changeState("playing")
} yield ()
println(out)
println(state)
}
And the output was:
13
paused
I don't understand why the assignment state = newState does not run, but the increment and assign expression out += 1 run. Am I missing something obvious on how this is supposed to work? I could really use some help. I understand that I can get this to run using the unsafe methods.
In your particular example, I think what is going on is that regular imperative Scala coded is unaffected by the IO monad--it runs when it normally would under the rules of Scala.
When you run:
for {
_ <- x("1")
_ <- changeState("playing")
} yield ()
this immediately calls x. That has nothing to do with the IO monad; it's just how for comprehensions are defined. The first step is to evaluate the first statement so you can call flatMap on it.
As you observe, you never "run" the monadic result, so the argument to flatMap, the monadic continuation, is never invoked, resulting in no call to changeState. This is specific to the IO monad, as, e.g., the List monad's flatMap would have immediately invoked the function (unless it were an empty list).
This might be a really dumb question but I am trying to understand the logic behind using #flatMap and not just #map in this method definition in Finatra's HttpClient definition:
def executeJson[T: Manifest](request: Request, expectedStatus: Status = Status.Ok): Future[T] = {
execute(request) flatMap { httpResponse =>
if (httpResponse.status != expectedStatus) {
Future.exception(new HttpClientException(httpResponse.status, httpResponse.contentString))
} else {
Future(parseMessageBody[T](httpResponse, mapper.reader[T]))
.transformException { e =>
new HttpClientException(httpResponse.status, s"${e.getClass.getName} - ${e.getMessage}")
}
}
}
}
Why create a new Future when I can just use #map and instead have something like:
execute(request) map { httpResponse =>
if (httpResponse.status != expectedStatus) {
throw new HttpClientException(httpResponse.status, httpResponse.contentString)
} else {
try {
FinatraObjectMapper.parseResponseBody[T](httpResponse, mapper.reader[T])
} catch {
case e => throw new HttpClientException(httpResponse.status, s"${e.getClass.getName} - ${e.getMessage}")
}
}
}
Would this be purely a stylistic difference and using Future.exception is just better style in this case, whereas throwing almost looks like a side-effect (in reality it's not, as it doesn't exit the context of a Future) or is there something more behind it, such as order of execution and such?
Tl;dr:
What's the difference between throwing within a Future vs returning a Future.exception?
From a theoretical point of view, if we take away the exceptions part (they cannot be reasoned about using category theory anyway), then those two operations are completely identical as long as your construct of choice (in your case Twitter Future) forms a valid monad.
I don't want to go into length over these concepts, so I'm just going to present the laws directly (using Scala Future):
import scala.concurrent.ExecutionContext.Implicits.global
// Functor identity law
Future(42).map(x => x) == Future(42)
// Monad left-identity law
val f = (x: Int) => Future(x)
Future(42).flatMap(f) == f(42)
// combining those two, since every Monad is also a Functor, we get:
Future(42).map(x => x) == Future(42).flatMap(x => Future(x))
// and if we now generalise identity into any function:
Future(42).map(x => x + 20) == Future(42).flatMap(x => Future(x + 20))
So yes, as you already hinted, those two approaches are identical.
However, there are three comments that I have on this, given that we are including exceptions into the mix:
Be careful - when it comes to throwing exceptions, Scala Future (probably Twitter too) violates the left-identity law on purpose, in order to trade it off for some extra safety.
Example:
import scala.concurrent.ExecutionContext.Implicits.global
def sneakyFuture = {
throw new Exception("boom!")
Future(42)
}
val f1 = Future(42).flatMap(_ => sneakyFuture)
// Future(Failure(java.lang.Exception: boom!))
val f2 = sneakyFuture
// Exception in thread "main" java.lang.Exception: boom!
As #randbw mentioned, throwing exceptions is not idiomatic to FP and it violates principles such as purity of functions and referential transparency of values.
Scala and Twitter Future make it easy for you to just throw an exception - as long as it happens in a Future context, exception will not bubble up, but instead cause that Future to fail. However, that doesn't mean that literally throwing them around in your code should be permitted, because it ruins the structure of your programs (similarly to how GOTO statements do it, or break statements in loops, etc.).
Preferred practice is to always evaluate every code path into a value instead of throwing bombs around, which is why it's better to flatMap into a (failed) Future than to map into some code that throws a bomb.
Keep in mind referential transparency.
If you use map instead of flatMap and someone takes the code from the map and extracts it out into a function, then you're safer if this function returns a Future, otherwise someone might run it outside of Future context.
Example:
import scala.concurrent.ExecutionContext.Implicits.global
Future(42).map(x => {
// this should be done inside a Future
x + 1
})
This is fine. But after completely valid refactoring (which utilizes the rule of referential transparency), your codfe becomes this:
def f(x: Int) = {
// this should be done inside a Future
x + 1
}
Future(42).map(x => f(x))
And you will run into problems if someone calls f directly. It's much safer to wrap the code into a Future and flatMap on it.
Of course, you could argue that even when using flatMap someone could rip out the f from .flatMap(x => Future(f(x)), but it's not that likely. On the other hand, simply extracting the response processing logic into a separate function fits perfectly with the functional programming's idea of composing small functions into bigger ones, and it's likely to happen.
From my understanding of FP, exceptions are not thrown. This would be, as you said, a side-effect. Exceptions are instead values that are handled at some point in the execution of the program.
Cats (and i'm sure other libraries, too) employs this technique too (https://github.com/typelevel/cats/blob/master/core/src/main/scala/cats/ApplicativeError.scala).
Therefore, the flatMap call allows the exception to be contained within a satisfied Future here and handled at a later point in the program's execution where other exception value handling may also occur.
I have an actor in my play app, that every tick (2 sec) sends a message to some method:
onSomething() : Future[Unit] = {
for {
a <- somethingThatReturnsFuture
b <- anotherThingThatReturnsFuture
}
}
This method has two calls that return future so I decided to use for-comprehension, but is it true that for-comprehension is blocking? So akka could not call this method again even with the 16 instances they run until the method complete?
If I would have my method to work with flatMap/map this will allow akka to have better performance? Like this:
onSomething() : Future[Unit] = {
somethingThatReturnsFuture.flatMap(res1 => {
anotherThingThatReturnsFuture.map(res2 => {
//whatever
})
})
}
thanks
As per Luis' comment, for-comprehensions are just syntactic sugar
Scala’s “for comprehensions” are syntactic sugar for composition of
multiple operations with foreach, map, flatMap, filter or withFilter.
Scala actually translates a for-expression into calls to those
methods, so any class providing them, or a subset of them, can be used
with for comprehensions.
which expands into underlying monadic operations, thus there should be no performance hit over using monadic operations directly. If your methods are independent of each other then you might gain same performance by taking advantage of Futures being eager and start them outside the for-comprehension like so
val aF = somethingThatReturnsFuture()
val bF = anotherThingThatReturnsFuture() // I started without waiting on anyone
for {
a <- aF
b <- bF
} yield {
a + b
}
However if calculation of b depends on a then you will not be able to kick them off in parallel
for {
a <- somethingThatReturnsFuture
b <- anotherThingThatReturnsFuture(a)
} yield {
a + b
}
Here anotherThingThatReturnsFuture "blocks" in the sense of having to wait on somethingThatReturnsFuture.
First of all.
Because the methods you are calling return a Future, none of them will block the Thread execution.
But what it is true, is that flatmap will concatenate sequentially the two operations. I mean, it will call the first method, then it returns inmediately, because it is a Future, and then it will call the second.
This will happen in the two options you have posted before (for comprehension and flatmap) because they are basically the same.
If you want to call the two methods at the same time, (in two different threads), so you don't know which of them will start to execute first, you have to use parallel collections.
But in your case, perhaps it is better to not use them because using futures will guarante that the thread will not block
Does Scala have an equivelent to golangs defer?
from:
http://golang.org/doc/effective_go.html#defer
Go's defer statement schedules a function call (the deferred function) to be run immediately before the function executing the defer returns. It's an unusual but effective way to deal with situations such as resources that must be released regardless of which path a function takes to return. The canonical examples are unlocking a mutex or closing a file.
Scala does not offer defer by design, however you can create it yourself by wrapping
your function in another function, passing an object which keeps track of functions to call.
Example:
class DeferTracker() {
class LazyVal[A](val value:() => A)
private var l = List[LazyVal[Any]]()
def apply(f: => Any) = { l = new LazyVal(() => f) :: l }
def makeCalls() = l.foreach { x => x.value() }
}
def Deferrable[A](context: DeferTracker => A) = {
val dt = new DeferTracker()
val res = context(dt)
dt.makeCalls
res
}
In this example, Deferable would be the wrapping function which calls context and
returns it contents, giving it an object which tracks defer calls.
You can use this construct like this:
def dtest(x:Int) = println("dtest: " + x)
def someFunction(x:Int):Int = Deferrable { defer =>
defer(dtest(x))
println("before return")
defer(dtest(2*x))
x * 3
}
println(someFunction(3))
The output would be:
before return
dtest: 6
dtest: 3
3
I'm aware that this can be solved differently but it is really just an example that
Scala supports the concept of defer without too much fuss.
I can't think of a Scala specific way, but wouldn't this be equivalent (though not as pretty):
try {
// Do stuff
} finally {
// "defer"
}
No. Go has this construct precisely because it doesn't support exceptions and has no try...finally syntax.
Personally, I think it invites a maintenance nightmare; calls to defer can be buried anywhere in a function. Even where responsible coders put the defers right beside the thing to be cleaned up, I think it's less clear than a finally block and as for what it lets the messy coders do... at least finally blocks put all the clean-up in one place.
defer is the opposite of idiomatic Scala. Scala offers monadic ways to control program flow and defer magic does not contribute at all. Monads offer a functional improvement over try...finally which let you
Define your own error handling flow
Manipulate defined flows functionally
Make a function's anticipated errors part of its signature
defer has no place in this.
I am currently playing with Scalaz non-blocking futures aka. Promises. I am struggling to make the following function tail-recursive:
#tailrec
private def repeat( res: Promise[I] ):Promise[I] =
res map p flatMap {
(b:Boolean) =>
if( b ) repeat( res flatMap f ) else res
}
where p is a predicate with type I=>Boolean and f is a concurrent function with
type I=>Promise[I].
The method compiles without the annotation.
Any hints ? Thanks
Your method isn't recursive at all. res is a computation potentially running in another thread. res map p flatMap f will immediately return a promise as far as your method is concerned. The recurrence to repeat will occur in a different process.
In slightly more terse terms, Promise is a continuation monad, and flatMap calls are automatically translated to continuation-passing style for you.
Although this looks tail recursive because the call appears only once in code, you have more than one recursive call - one for each element inside your collection. At least that's what the compiler sees. (Supposing this is a flatMap on some collection; I have no idea what p does return)
You pass the recursion to somewhere as an anonymous function. No one knows how often it will be executed.