The Scenario
In an application I am currently writing I am using cats-effect's IO monad in an IOApp.
If started with a command line argument 'debug', I am delegeting my program flow into a debug loop that waits for user input and executes all kinds of debugging-relevant methods. As soon as the developer presses enter without any input, the application will exit the debug loop and exit the main method, thus closing the application.
The main method of this application looks roughly like this:
import scala.concurrent.{ExecutionContext, ExecutionContextExecutor}
import cats.effect.{ExitCode, IO, IOApp}
import cats.implicits._
object Main extends IOApp {
val BlockingFileIO: ExecutionContextExecutor = ExecutionContext.fromExecutor(blockingIOCachedThreadPool)
def run(args: List[String]): IO[ExitCode] = for {
_ <- IO { println ("Running with args: " + args.mkString(","))}
debug = args.contains("debug")
// do all kinds of other stuff like initializing a webserver, file IO etc.
// ...
_ <- if(debug) debugLoop else IO.unit
} yield ExitCode.Success
def debugLoop: IO[Unit] = for {
_ <- IO(println("Debug mode: exit application be pressing ENTER."))
_ <- IO.shift(BlockingFileIO) // readLine might block for a long time so we shift to another thread
input <- IO(StdIn.readLine()) // let it run until user presses return
_ <- IO.shift(ExecutionContext.global) // shift back to main thread
_ <- if(input == "b") {
// do some debug relevant stuff
IO(Unit) >> debugLoop
} else {
shutDown()
}
} yield Unit
// shuts down everything
def shutDown(): IO[Unit] = ???
}
Now, I want to test if e.g. my run method behaves like expected in my ScalaTests:
import org.scalatest.FlatSpec
class MainSpec extends FlatSpec{
"Main" should "enter the debug loop if args contain 'debug'" in {
val program: IO[ExitCode] = Main.run("debug" :: Nil)
// is there some way I can 'search through the IO monad' and determine if my program contains the statements from the debug loop?
}
}
My Question
Can I somehow 'search/iterate through the IO monad' and determine if my program contains the statements from the debug loop? Do I have to call program.unsafeRunSync() on it to check that?
You could implement the logic of run inside your own method, and test that instead, where you aren't restricted in the return type and forward run to your own implementation. Since run forces your hand to IO[ExitCode], there's not much you can express from the return value. In general, there's no way to "search" an IO value as it just a value that describes a computation that has a side effect. If you want to inspect it's underlying value, you do so by running it in the end of the world (your main method), or for your tests, you unsafeRunSync it.
For example:
sealed trait RunResult extends Product with Serializable
case object Run extends RunResult
case object Debug extends RunResult
def run(args: List[String]): IO[ExitCode] = {
run0(args) >> IO.pure(ExitCode.Success)
}
def run0(args: List[String]): IO[RunResult] = {
for {
_ <- IO { println("Running with args: " + args.mkString(",")) }
debug = args.contains("debug")
runResult <- if (debug) debugLoop else IO.pure(Run)
} yield runResult
}
def debugLoop: IO[Debug.type] =
for {
_ <- IO(println("Debug mode: exit application be pressing ENTER."))
_ <- IO.shift(BlockingFileIO) // readLine might block for a long time so we shift to another thread
input <- IO(StdIn.readLine()) // let it run until user presses return
_ <- IO.shift(ExecutionContext.global) // shift back to main thread
_ <- if (input == "b") {
// do some debug relevant stuff
IO(Unit) >> debugLoop
} else {
shutDown()
}
} yield Debug
// shuts down everything
def shutDown(): IO[Unit] = ???
}
And then in your test:
import org.scalatest.FlatSpec
class MainSpec extends FlatSpec {
"Main" should "enter the debug loop if args contain 'debug'" in {
val program: IO[RunResult] = Main.run0("debug" :: Nil)
program.unsafeRunSync() match {
case Debug => // do stuff
case Run => // other stuff
}
}
}
To search through some monad expression, it would have to be values, not statements, aka reified. That is the core idea behind the (in)famous Free monad. If you were to go through the hassle of expressing your code in some "algebra" as they call (think DSL) it and lift it into monad expression nesting via Free, then yes you would be able to search through it. There are plenty of resources that explain Free monads better than I could google is your friend here.
My general suggestion would be that the general principles of good testing apply everywhere. Isolate the side-effecting part and inject it into the main piece of logic, so that you can inject a fake implementation in testing to allow all sorts of assertions.
Related
Why putStrLn in flatMap followed by a result statement didn't get effectively write to stdout?
object Mgr extends App {
def main1(args: Array[String]) = getStrLn.flatMap { s =>
putStrLn(s) // Why this did not write to console?
UIO.succeed(s)
}
override def run(args: List[String]): URIO[zio.ZEnv, Int] = main1(Array()).fold(_ => 1,
{ x =>
println(x) // only this line wrote to console, why?
0
})
}
Your problem is basically, that you put two effects into single flatMap.
By invoking putStrLn(s) you're not actually printing to console, you're merely creating the description of the action that will print when your program is interpreted and run (when method run is called). And because in your flatmap only last value is returned (in your case UIO.succeed(s)), then only it will be taken into consideration while constructing ZIO program.
You can fix your program by chaining both actions.
You can do it with *> operator:
def main1(args: Array[String]) = getStrLn.flatMap { s =>
putStrLn(s) *> UIO.succeed(s)
}
or you could just put effects into separate flatMaps. But since you want to create side-effect (by printing value), but then pass value further unchanged, you need to use special function tap:
def main1(args: Array[String]) = getStrLn.tap { s =>
putStrLn(s)
}.flatMap { s =>
UIO.succeed(s)
}
Your issue is also described (with other pitfalls) in this great article (look at the first point).
I am creating in Scala and Cats a function that does some I/O and that will be called by other parts of the code. I'm also learning Cats and I want my function to:
Be generic in its effect and use a F[_]
Run on a dedicated thread pool
I want to introduce async boundaries
I assume that all my functions are generic in F[_] up to the main method because I'm trying to follow these Cat's guidelines
But I struggle to make these constraint to work by using ContextShift or ExecutionContext. I have written a full example here and this is an exctract from the example:
object ComplexOperation {
// Thread pool for ComplexOperation internal use only
val cs = IO.contextShift(
ExecutionContext.fromExecutor(Executors.newSingleThreadExecutor())
)
// Complex operation that takes resources and time
def run[F[_]: Sync](input: String): F[String] =
for {
r1 <- Sync[F].delay(cs.shift) *> op1(input)
r2 <- Sync[F].delay(cs.shift) *> op2(r1)
r3 <- Sync[F].delay(cs.shift) *> op3(r2)
} yield r3
def op1[F[_]: Sync](input: String): F[Int] = Sync[F].delay(input.length)
def op2[F[_]: Sync](input: Int): F[Boolean] = Sync[F].delay(input % 2 == 0)
def op3[F[_]: Sync](input: Boolean): F[String] = Sync[F].delay(s"Complex result: $input")
}
This clearly doesn't abstract over effects as ComplexOperation.run needs a ContextShift[IO] to be able to introduce async boundaries. What is the right (or best) way of doing this?
Creating ContextShift[IO] inside ComplexOperation.run makes the function depend on IO which I don't want.
Moving the creation of a ContextShift[IO] on the caller will simply shift the problem: the caller is also generic in F[_] so how does it obtain a ContextShift[IO] to pass to ComplexOperation.run without explicitly depending on IO?
Remember that I don't want to use one global ContextShift[IO] defined at the topmost level but I want each component to decide for itself.
Should my ComplexOperation.run create the ContextShift[IO] or is it the responsibility of the caller?
Am I doing this right at least? Or am I going against standard practices?
So I took the liberty to rewrite your code, hope it helps:
import cats.effect._
object Functions {
def sampleFunction[F[_]: Sync : ContextShift](file: String, blocker: Blocker): F[String] = {
val handler: Resource[F, Int] =
Resource.make(
blocker.blockOn(openFile(file))
) { handler =>
blocker.blockOn(closeFile(handler))
}
handler.use(handler => doWork(handler))
}
private def openFile[F[_]: Sync](file: String): F[Int] = Sync[F].delay {
println(s"Opening file $file with handler 2")
2
}
private def closeFile[F[_]: Sync](handler: Int): F[Unit] = Sync[F].delay {
println(s"Closing file handler $handler")
}
private def doWork[F[_]: Sync](handler: Int): F[String] = Sync[F].delay {
println(s"Calculating the value on file handler $handler")
"The final value"
}
}
object Main extends IOApp {
override def run(args: List[String]): IO[ExitCode] = {
val result = Blocker[IO].use { blocker =>
Functions.sampleFunction[IO](file = "filePath", blocker)
}
for {
data <- result
_ <- IO(println(data))
} yield ExitCode.Success
}
}
You can see it running here.
So, what does this code does.
First, it creates a Resource for the file, since close has to be done, even on guarantee or on failure.
It is using Blocker to run the open and close operations on a blocking thread poo (that is done using ContextShift).
Finally, on the main, it creates a default Blocker for instance, for **IO*, and uses it to call your function; and prints the result.
Fell free to ask any question.
I'm converting Future code to IO. I have code similar to this
def doSomething: Future[Foo] = {
Future {
//some code the result of which we don't care about
}
Future {
//Foo
}
}
And then at the end of the program, I doSomething.unsafeRunSync. How do I convert these Futures to IOs while maintaining the fire-and-forget functionality of the first Future? In using IO's async API, I am worried about accidentally blocking the thread when I later call unsafeRunSync on doSomething.
A solution that uses only cats-effect could use IO.start. This, combined with the fact that you will then never join the resulting Fiber, will look something like this:
import cats.effect._
import cats.implicits._
import scala.concurrent.ExecutionContext.Implicits.global
import scala.concurrent.duration._
object ExampleApp extends App{
val fireAndForget =
IO(println("Side effect pre-sleep")) *>
IO.sleep(2.seconds) *>
IO(println("Side effect post-sleep"))
val toBeUsed = IO{
println("Inside second one")
42
}
val result = for {
fiber <- IO.shift *> fireAndForget.start
res <- toBeUsed.handleErrorWith { error =>
// This is just in case you 'toBeUsed' can actually fail,
// and you might want to cancel the original side-effecting IO
fiber.cancel *> IO.raiseError(error) }
} yield res
println(result.unsafeRunSync())
println("Waiting 3 seconds...")
IO.sleep(3.seconds).unsafeRunSync()
println("Done")
}
This will print (most of the times) something similar to:
Side effect pre-sleep
Inside second one
42 // Up until here, will be printed right away
Waiting 3 seconds... // It will then be waiting a while
Side effect post-sleep // ...at which point the side effecting code terminates
Done
Finally, here are more details about Fiber and IO.shift
I believe that you need to wrap the first Future in such a way that it completes immediately. We ignore exeptions, or catch them however, but they are contained within its own thread. The parameter cb is the promise that needs to complete; so we short-circuit the completion by providing a value immediately.
def firstFuture(implicit ec: ExecutionContext): IO[Unit] = {
IO.async[Unit] { cb =>
ec.execute(() => {
try {
//some code the result of which we don't care about
} catch {
}
})
cb(Right(()))
}
}
In the for-comprehension, the firstFuture will complete immediately even though its thread will have a long-running task active on it.
def doSomething(implicit ec: ExecutionContext): IO[Foo] = {
for {
_ <- firstFuture
IO.async[Foo] { fb =>
// Foo
}
}
}
I am a newbie to scala futures and I have a doubt regarding the return value of scala futures.
So, generally syntax for a scala future is
def downloadPage(url: URL) = Future[List[Int]] {
}
I want to know how to access the List[Int] from some other method which calls this method.
In other words,
val result = downloadPage("localhost")
then what should be the approach to get List[Int] out of the future ?
I have tried using map method but not able to do this successfully.`
The case of Success(listInt) => I want to return the listInt and I am not able to figure out how to do that.
The best practice is that you don't return the value. Instead you just pass the future (or a version transformed with map, flatMap, etc.) to everyone who needs this value and they can add their own onComplete.
If you really need to return it (e.g. when implementing a legacy method), then the only thing you can do is to block (e.g. with Await.result) and you need to decide how long to await.
You need to wait for the future to complete to get the result given some timespan, here's something that would work:
import scala.concurrent.duration._
def downloadPage(url: URL) = Future[List[Int]] {
List(1,2,3)
}
val result = downloadPage("localhost")
val myListInt = result.result(10 seconds)
Ideally, if you're using a Future, you don't want to block the executing thread, so you would move your logic that deals with the result of your Future into the onComplete method, something like this:
result.onComplete({
case Success(listInt) => {
//Do something with my list
}
case Failure(exception) => {
//Do something with my error
}
})
I hope you already solved this since it was asked in 2013 but maybe my answer can help someone else:
If you are using Play Framework, it support async Actions (actually all Actions are async inside). An easy way to create an async Action is using Action.async(). You need to provide a Future[Result]to this function.
Now you can just make transformations from your Future[List[Int]] to Future[Result] using Scala's map, flatMap, for-comprehension or async/await. Here an example from Play Framework documentation.
import play.api.libs.concurrent.Execution.Implicits.defaultContext
def index = Action.async {
val futureInt = scala.concurrent.Future { intensiveComputation() }
futureInt.map(i => Ok("Got result: " + i))
}
You can do something like that. If The wait time that is given in Await.result method is less than it takes the awaitable to execute, you will have a TimeoutException, and you need to handle the error (or any other error).
import scala.concurrent._
import ExecutionContext.Implicits.global
import scala.util.{Try, Success, Failure}
import scala.concurrent.duration._
object MyObject {
def main(args: Array[String]) {
val myVal: Future[String] = Future { silly() }
// values less than 5 seconds will go to
// Failure case, because silly() will not be done yet
Try(Await.result(myVal, 10 seconds)) match {
case Success(extractedVal) => { println("Success Happened: " + extractedVal) }
case Failure(_) => { println("Failure Happened") }
case _ => { println("Very Strange") }
}
}
def silly(): String = {
Thread.sleep(5000)
"Hello from silly"
}
}
The best way I’ve found to think of a Future is a box that will, at some point, contain the thing that you want. The key thing with a Future is that you never open the box. Trying to force open the box will lead you to blocking and grief. Instead, you put the Future in another, larger box, typically using the map method.
Here’s an example of a Future that contains a String. When the Future completes, then Console.println is called:
import scala.concurrent.Future
import scala.concurrent.ExecutionContext.Implicits.global
object Main {
def main(args:Array[String]) : Unit = {
val stringFuture: Future[String] = Future.successful("hello world!")
stringFuture.map {
someString =>
// if you use .foreach you avoid creating an extra Future, but we are proving
// the concept here...
Console.println(someString)
}
}
}
Note that in this case, we’re calling the main method and then… finishing. The string’s Future, provided by the global ExecutionContext, does the work of calling Console.println. This is great, because when we give up control over when someString is going to be there and when Console.println is going to be called, we let the system manage itself. In constrast, look what happens when we try to force the box open:
val stringFuture: Future[String] = Future.successful("hello world!")
val someString = Future.await(stringFuture)
In this case, we have to wait — keep a thread twiddling its thumbs — until we get someString back. We’ve opened the box, but we’ve had to commandeer the system’s resources to get at it.
It wasn't yet mentioned, so I want to emphasize the point of using Future with for-comprehension and the difference of sequential and parallel execution.
For example, for sequential execution:
object FuturesSequential extends App {
def job(n: Int) = Future {
Thread.sleep(1000)
println(s"Job $n")
}
val f = for {
f1 <- job(1)
f2 <- job(2)
f3 <- job(3)
f4 <- job(4)
f5 <- job(5)
} yield List(f1, f2, f3, f4, f5)
f.map(res => println(s"Done. ${res.size} jobs run"))
Thread.sleep(6000) // We need to prevent main thread from quitting too early
}
And for parallel execution (note that the Future are before the for-comprehension):
object FuturesParallel extends App {
def job(n: Int) = Future {
Thread.sleep(1000)
println(s"Job $n")
}
val j1 = job(1)
val j2 = job(2)
val j3 = job(3)
val j4 = job(4)
val j5 = job(5)
val f = for {
f1 <- j1
f2 <- j2
f3 <- j3
f4 <- j4
f5 <- j5
} yield List(f1, f2, f3, f4, f5)
f.map(res => println(s"Done. ${res.size} jobs run"))
Thread.sleep(6000) // We need to prevent main thread from quitting too early
}
According to the Scala Language Specification (§6.19), "An enumerator sequence always starts with a generator". Why?
I sometimes find this restriction to be a hindrance when using for-comprehensions with monads, because it means you can't do things like this:
def getFooValue(): Future[Int] = {
for {
manager = Manager.getManager() // could throw an exception
foo <- manager.makeFoo() // method call returns a Future
value = foo.getValue()
} yield value
}
Indeed, scalac rejects this with the error message '<-' expected but '=' found.
If this was valid syntax in Scala, one advantage would be that any exception thrown by Manager.getManager() would be caught by the Future monad used within the for-comprehension, and would cause it to yield a failed Future, which is what I want. The workaround of moving the call to Manager.getManager() outside the for-comprehension doesn't have this advantage:
def getFooValue(): Future[Int] = {
val manager = Manager.getManager()
for {
foo <- manager.makeFoo()
value = foo.getValue()
} yield value
}
In this case, an exception thrown by foo.getValue() will yield a failed Future (which is what I want), but an exception thrown by Manager.getManager() will be thrown back to the caller of getFooValue() (which is not what I want). Other possible ways of handling the exception are more verbose.
I find this restriction especially puzzling because in Haskell's otherwise similar do notation, there is no requirement that a do block should begin with a statement containing <-. Can anyone explain this difference between Scala and Haskell?
Here's a complete working example showing how exceptions are caught by the Future monad in for-comprehensions:
import scala.concurrent._
import scala.concurrent.duration._
import scala.concurrent.ExecutionContext.Implicits.global
import scala.util.{Try, Success, Failure}
class Foo(val value: Int) {
def getValue(crash: Boolean): Int = {
if (crash) {
throw new Exception("failed to get value")
} else {
value
}
}
}
class Manager {
def makeFoo(crash: Boolean): Future[Foo] = {
if (crash) {
throw new Exception("failed to make Foo")
} else {
Future(new Foo(10))
}
}
}
object Manager {
def getManager(crash: Boolean): Manager = {
if (crash) {
throw new Exception("failed to get manager")
} else {
new Manager()
}
}
}
object Main extends App {
def getFooValue(crashGetManager: Boolean,
crashMakeFoo: Boolean,
crashGetValue: Boolean): Future[Int] = {
for {
manager <- Future(Manager.getManager(crashGetManager))
foo <- manager.makeFoo(crashMakeFoo)
value = foo.getValue(crashGetValue)
} yield value
}
def waitForValue(future: Future[Int]): Unit = {
val result = Try(Await.result(future, Duration("10 seconds")))
result match {
case Success(value) => println(s"Got value: $value")
case Failure(e) => println(s"Got error: $e")
}
}
val future1 = getFooValue(false, false, false)
waitForValue(future1)
val future2 = getFooValue(true, false, false)
waitForValue(future2)
val future3 = getFooValue(false, true, false)
waitForValue(future3)
val future4 = getFooValue(false, false, true)
waitForValue(future4)
}
Here's the output:
Got value: 10
Got error: java.lang.Exception: failed to get manager
Got error: java.lang.Exception: failed to make Foo
Got error: java.lang.Exception: failed to get value
This is a trivial example, but I'm working on a project in which we have a lot of non-trivial code that depends on this behaviour. As far as I understand, this is one of the main advantages of using Future (or Try) as a monad. What I find strange is that I have to write
manager <- Future(Manager.getManager(crashGetManager))
instead of
manager = Manager.getManager(crashGetManager)
(Edited to reflect #RexKerr's point that the monad is doing the work of catching the exceptions.)
for comprehensions do not catch exceptions. Try does, and it has the appropriate methods to participate in for-comprehensions, so you can
for {
manager <- Try { Manager.getManager() }
...
}
But then it's expecting Try all the way down unless you manually or implicitly have a way to switch container types (e.g. something that converts Try to a List).
So I'm not sure your premises are right. Any assignment you made in a for-comprehension can just be made early.
(Also, there is no point doing an assignment inside a for comprehension just to yield that exact value. Just do the computation in the yield block.)
(Also, just to illustrate that multiple types can play a role in for comprehensions so there's not a super-obvious correct answer for how to wrap an early assignment in terms of later types:
// List and Option, via implicit conversion
for {i <- List(1,2,3); j <- Option(i).filter(_ <2)} yield j
// Custom compatible types with map/flatMap
// Use :paste in the REPL to define A and B together
class A[X] { def flatMap[Y](f: X => B[Y]): A[Y] = new A[Y] }
class B[X](x: X) { def map[Y](f: X => Y): B[Y] = new B(f(x)) }
for{ i <- (new A[Int]); j <- (new B(i)) } yield j.toString
Even if you take the first type you still have the problem of whether there is a unique "bind" (way to wrap) and whether to doubly-wrap things that are already the correct type. There could be rules for all these things, but for-comprehensions are already hard enough to learn, no?)
Haskell translates the equivalent of for { manager = Manager.getManager(); ... } to the equivalent of lazy val manager = Manager.getManager(); for { ... }. This seems to work:
scala> lazy val x: Int = throw new Exception("")
x: Int = <lazy>
scala> for { y <- Future(x + 1) } yield y
res8: scala.concurrent.Future[Int] = scala.concurrent.impl.Promise$DefaultPromise#fedb05d
scala> Try(Await.result(res1, Duration("10 seconds")))
res9: scala.util.Try[Int] = Failure(java.lang.Exception: )
I think the reason this can't be done is because for-loops are syntactic sugar for flatMap and map methods (except if you are using a condition in the for-loop, in that case it's desugared with the method withFilter). When you are storing in a immutable variable, you can't use these methods. That's the reason you would be ok using Try as pointed out by Rex Kerr. In that case, you should be able to use map and flatMap methods.