Scalaz Task not starting - scala

I'm trying to make an asyncronous call using scalaz Task.
For some strange reason whenever I attempt to call methodA, although I get a scalaz.concurrent.Task returned I never see the 2nd print statement which leads me to believe that the asynchronous call is never returned. Is there something that I'm missing that is required in order for me to start the task?
for {
foo <- println("we are in this for loop!").asAsyncSuccess()
authenticatedUser <- methodA(parameterA)
foo2 <- println("we are out of this this call!").asAsyncSuccess()
} yield {
authenticatedUser
}
def methodA(parameterA: SomeType): Task[User]

First off, Scalaz Task is lazy: it won't start when you create it or get from some method. It's by design and allows Tasks to combine well. It's not intended to start until you fully assemble your entire program into single main Task, so when you're done, you need to manually start an aggregated Task using one of the perform methods, like unsafePerformSync that will wait for the result of aggregate async computation on a current thread:
(for {
foo <- println("we are in this for loop!").asAsyncSuccess()
authenticatedUser <- methodA(parameterA)
foo2 <- println("we are out of this this call!").asAsyncSuccess()
} yield {
authenticatedUser
}).unsafePerformSync
As you can see from your original code, you've never started any Task. However, you've mentioned that first println printed its message to console. This may be related to asAsyncSuccess method: I've not found it in Scalaz, so I assume it's in implicit class somewhere in your code:
implicit class TaskOps[A](val a: A) extends AnyVal {
def asAsyncSuccess(): Task[A] = Task(a)
}
If you write a helper implicit class like this, it won't convert an effectful expression into lazy Task action, despite the fact that Task.apply
is call-by-name. It will only convert its result, which happens to be just Unit, because its own parameter a is not call-by-name. To work as intended, it needs to be like the following:
implicit class TaskOps[A](a: => A) {
def asAsyncSuccess(): Task[A] = Task(a)
}
You may also ask why first println prints something while second isn't. This is related to for-comprehensions desugaring into method calls. Specifically, compiler turns your code into this:
println("we are in this for loop!").asAsyncSuccess().flatMap(foo =>
methodA(parameterA).flatMap(authenticatedUser =>
println("we are out of this this call!").asAsyncSuccess().map(foo2 =>
authenticatedUser)))
If asAsyncSuccess doesn't respect intended Task semantics, first println gets executed immediately. But its result is then wrapped into Task that is, essentially, just a wrapper over start function, and flatMap implementation on it just combines such functions together without running them. So, methodA or second println won't be executed until you call unsafePerformSync or other perform helper method. Note, however, that second println will still be executed earlier than it should according to Task semantics, just not that earlier.

Related

Mocked methods created within `withObjectMocked` not invoked when called from an `Actor`

I have published a minimal project showcasing my problem at https://github.com/Zwackelmann/mockito-actor-test
In my project I refactored a couple of components from classes to objects in all cases where the class did not really have a meaningful state. Since some of these objects establish connections to external services that need to be mocked, I was happy to see that mockito-scala introduced the withObjectMocked context function, which allows mocking objects within the scope of the function.
This feature worked perfectly for me until I introduced Actors in the mix, which would ignore the mocked functions despite being in the withObjectMocked context.
For an extended explanation what I did check out my github example project from above which is ready to be executed via sbt run.
My goal is to mock the doit function below. It should not be called during tests, so for this demonstration it simply throws a RuntimeException.
object FooService {
def doit(): String = {
// I don't want this to be executed in my tests
throw new RuntimeException(f"executed real impl!!!")
}
}
The FooService.doit function is only called from the FooActor.handleDoit function. This function is called by the FooActor after receiving the Doit message or when invoked directly.
object FooActor {
val outcome: Promise[Try[String]] = Promise[Try[String]]()
case object Doit
def apply(): Behavior[Doit.type] = Behaviors.receiveMessage { _ =>
handleDoit()
Behaviors.same
}
// moved out actual doit behavior so I can compare calling it directly with calling it from the actor
def handleDoit(): Unit = {
try {
// invoke `FooService.doit()` if mock works correctly it should return the "mock result"
// otherwise the `RuntimeException` from the real implementation will be thrown
val res = FooService.doit()
outcome.success(Success(res))
} catch {
case ex: RuntimeException =>
outcome.success(Failure(ex))
}
}
}
To mock Foo.doit I used withObjectMocked as follows. All following code is within this block. To ensure that the block is not left due to asynchronous execution, I Await the result of the FooActor.outcome Promise.
withObjectMocked[FooService.type] {
// mock `FooService.doit()`: The real method throws a `RuntimeException` and should never be called during tests
FooService.doit() returns {
"mock result"
}
// [...]
}
I now have two test setups: The first simply calls FooActor.handleDoit directly
def simpleSetup(): Try[String] = {
FooActor.handleDoit()
val result: Try[String] = Await.result(FooActor.outcome.future, 1.seconds)
result
}
The second setup triggers FooActor.handleDoit via the Actor
def actorSetup(): Try[String] = {
val system: ActorSystem[FooActor.Doit.type] = ActorSystem(FooActor(), "FooSystem")
// trigger actor to call `handleDoit`
system ! FooActor.Doit
// wait for `outcome` future. The 'real' `FooService.doit` impl results in a `Failure`
val result: Try[String] = Await.result(FooActor.outcome.future, 1.seconds)
system.terminate()
result
}
Both setups wait for the outcome promise to finish before exiting the block.
By switching between simpleSetup and actorSetup I can test both behaviors. Since both are executed within the withObjectMocked context, I would expect that both trigger the mocked function. However actorSetup ignores the mocked function and calls the real method.
val result: Try[String] = simpleSetup()
// val result: Try[String] = actorSetup()
result match {
case Success(res) => println(f"finished with result: $res")
case Failure(ex) => println(f"failed with exception: ${ex.getMessage}")
}
// simpleSetup prints: finished with result: mock result
// actorSetup prints: failed with exception: executed real impl!!!
Any suggestions?
withObjectMock relies on the code exercising the mock executing in the same thread as withObjectMock (see Mockito's implementation and see ThreadAwareMockHandler's check of the current thread).
Since actors execute on the threads of the ActorSystem's dispatcher (never in the calling thread), they cannot see such a mock.
You may want to investigate testing your actor using the BehaviorTestKit, which itself effectively uses a mock/stub implementation of the ActorContext and ActorSystem. Rather than spawning an actor, an instance of the BehaviorTestKit encapsulates a behavior and passes it messages which are processed synchronously in the testing thread (via the run and runOne methods). Note that the BehaviorTestKit has some limitations: certain categories of behaviors aren't really testable via the BehaviorTestKit.
More broadly, I'd tend to suggest that mocking in Akka is not worth the effort: if you need pervasive mocks, that's a sign of a poor implementation. ActorRef (especially of the typed variety) is IMO the ultimate mock: encapsulate exactly what needs to be mocked into its own actor with its own protocol and inject that ActorRef into the behavior under test. Then you validate that the behavior under test holds up its end of the protocol correctly. If you want to validate the encapsulation (which should be as simple as possible to the extent that it's obviously correct, but if you want/need to spend effort on getting those coverage numbers up...) you can do the BehaviorTestKit trick as above (and since the only thing the behavior is doing is exercising the mocked functionality, it almost certainly won't be in the category of behaviors which aren't testable with the BehaviorTestKit).

How does the cats-effect IO monad really work?

I'm new to functional programming and Scala, and I was checking out the Cats Effect framework and trying to understand what the IO monad does. So far what I've understood is that writing code in the IO block is just a description of what needs to be done and nothing happens until you explicitly run using the unsafe methods provided, and also a way to make code that performs side-effects referentially transparent by actually not running it.
I tried executing the snippet below just to try to understand what it means:
object Playground extends App {
var out = 10
var state = "paused"
def changeState(newState: String): IO[Unit] = {
state = newState
IO(println("Updated state."))
}
def x(string: String): IO[Unit] = {
out += 1
IO(println(string))
}
val tuple1 = (x("one"), x("two"))
for {
_ <- x("1")
_ <- changeState("playing")
} yield ()
println(out)
println(state)
}
And the output was:
13
paused
I don't understand why the assignment state = newState does not run, but the increment and assign expression out += 1 run. Am I missing something obvious on how this is supposed to work? I could really use some help. I understand that I can get this to run using the unsafe methods.
In your particular example, I think what is going on is that regular imperative Scala coded is unaffected by the IO monad--it runs when it normally would under the rules of Scala.
When you run:
for {
_ <- x("1")
_ <- changeState("playing")
} yield ()
this immediately calls x. That has nothing to do with the IO monad; it's just how for comprehensions are defined. The first step is to evaluate the first statement so you can call flatMap on it.
As you observe, you never "run" the monadic result, so the argument to flatMap, the monadic continuation, is never invoked, resulting in no call to changeState. This is specific to the IO monad, as, e.g., the List monad's flatMap would have immediately invoked the function (unless it were an empty list).

Scala Fork-Join-All With Multiple Generic Types and 1 Generic Unit of Work

I'm attempting to write a method which accepts multiple generic types and takes as an argument a unit of work to execute.
The idea is that the unit of work is a common function that itself is generic. For the sake of example, let's say it's something like the following:
def loadModelRdd[T: TypeTag](sc: SparkContext): RDD[T] = {
...
}
loadModelRdd() will construct an RDD of the given type after some internal processing like loading the Model information, etc.
A prototype method I've been hacking on looks something like the following (non-working):
def forkAll[A : Manifest, B : Manifest](work: => RDD[_]): (RDD[A], RDD[B]) = {
def aFuture = Future { work } // How can I notify that this work call returns type A?
def bFuture = Future { work } // How can I notify that this work call returns type B?
val res = for {
a <- aFuture
b <- bFuture
} yield (a.asInstanceOf[A], b.asInstanceOf[B])
Await.result(res, 10.seconds)
}
This is a shortened version of the code I'm working on as I'm actually looking at accepting as many as 10 different types.
As you can see, the overall goal of the forkAll method is to wrap the unit of work in a Future, fork-join the execution of the unit of work for each type, then return the results as a Tuple'd result. An example consumer statement would be:
val (a, b) = forkAll[ClassA, ClassB](loadModelRdd)
i.e I want to fork-join at this point and wait for the results, but I want the executions to be executed in parallel and then collected back to the Driver (Spark Driver to be specific).
The problem is I'm not sure how to coerce the type returned by the unit of work within forkAll when constructing the Future {} blocks. Without the forkAll, the implementation looked like the following:
val resA = loadModelRdd[ClassA](sc)
val resB = loadModelRdd[ClassB](sc)
...
I am looking at doing this for two reasons:
To abstract the details of fork-join for any unit of work which matches this model.
A version of this code, which explicitly states what the unit of work is, is working in Production and was responsible for cutting execution of a long-running block by close to half. I have a couple of execution steps where this pattern could be applied
Is this something that is possible in Scala's type system? Or should I look at this problem from a different perspective? I've tried a couple of implementations (including one described here) but I haven't quite found one that fits my current view of the problem
Please let me know if there is any additional information needed.
Thanks!
Short answer: Scala does not allow functions with type parameters, so what you want is not exactly possible.
You are attempting to pass a method with a type parameter. Although methods are allowed to have type parameters, functions are not. When you try to pass a method, it acts like an anonymous function, so you must specify a type.
However, since methods do allow type parameters, you can take advantage of this by creating an abstract class that will do your fork/join
abstract class ForkJoin {
protected def work[T]: RDD[T]
def apply[A, B]: (RDD[A], RDD[B]) = {
// Write implementation of fork/join here
(work[A], work[B])
}
}
then overriding the type generic work method so that it does what you want, such as calling some other pre-defined method.
val forkJoin = new ForkJoin {
override protected def work[T]: RDD[T] =
loadModelRdd[T](sc)
}
val (intRdd, stringRdd) = forkJoin[Int, String]
Check out this for a prototype implementation that compiles and runs without issues.

Scala: How to return a Some or Option

I have the following piece of code that I am trying to enhance:
I am using the java.nio.file package to represent a directory or a file as a Path.
So here goes:
import java.nio.file.{Paths,DirectoryStream,Files,
Path,DirectoryIteratorException}
val path: Path = Paths.get(directoryPath)
var directoryStream: Option[DirectoryStream[Path]] = None
// so far so good
try {
directoryStream = Some(Files.newDirectoryStream(pathO))
// this is where i get into trouble
def getMeDirStream: DirectoryStream[Path] =
if (!directoryStream.isEmpty && directoryStream.isDefined)
getMeDirStream.get
else
None
// invoke the iterator() method of dstream here
}
The above piece of code will not compile because I do not know what to return in the else, and right now, for the life of me, I can only come up with None, which the compiler simply does not like and I would like to learn what should be its replacement.
I want this example to be a learning lesson of Option and Some for me.
Okay, this is where I choke. I would like to check if the directoryStream is not empty and is defined, and then if this is the case, I would like to invoke getMeDirStream.get to invoke the iterator() method on it.
The API for Option tells me that invoking the get() method could result in a java.util.NoSuchElementException if the option is empty.
If the directoryStream is empty I want to return something and not None, because IntelliJ is telling me that "Expression of type None.type doesn't conform to expected type DirectoryStream[Path]".
Now, I am being all naive about this.
I would like to know the following:
What should I return in the else other than None?
Should I wrap the getMeDirStream.get in a try-catch with a java.util.NoSuchElementException, even though I am checking if the directoryStream is empty or not.?
What is the purpose of a try-catch in the getMeDirStream.get, if there is indeed such a need?
How can I clean up the above piece of code to incorporate correct checks for being isDefined and for catching appropriate exceptions?
Once I know what to return in the else (and after putting in the appropriate try-catch block if necessary), I would like to invoke the iterator() method on getMeDirStream to do some downstream operations.
Some and None are subtypes of Option, but to be more correct, they are actually two different cases of Option or data constructors. In other words, even though Scala allows you to directly invoke a Some or a None you should still regard their type to be Option. The more important thing to take from this is that you should never under any circumstance invoke Option#get as it is unsafe.
The intention of Option is to indicate the possibility that a value does not exist. If you care about the errors, then you should probably look at using Either instead (or Scalaz's Either called \/).
You can keep the computation within the Option context and then only extract the value later, or provide a default.
def fromTryCatch[A](a: => A): Either[Throwable, A] = try { Right(a) } catch { case e: Throwable => Left(e) }
val getMeDirStream: Option[java.util.Iterator[Path]] =
for {
path <- fromTryCatch(Paths.get(directoryPath)).toOption
directoryStream <- fromTryCatch(Files.newDirectoryStream(pathO)).toOption
} yield directoryStream.iterator()
Later, or right after, you can get the iterator, or provide a default value:
val iterator = getMeDirStream.getOrElse(java.util.Collections.emptyIterator[Path])
Your specific questions are difficult to address because it's unclear exactly what you're trying to achieve. In particular, when you ask what the purpose of the try block is... Well, you wrote it, so only you can answer that.
In general, you never call get on an Option. You either use pattern matching:
option match {
case Some(value) => /* ... */
case None => /* ... */
}
or you use methods like map, flatMap, and foreach (or the equivalent comprehension syntax that gpampara's code uses).
My revision of gpampara's answer:
import scala.collection.convert.wrapAll._
import scala.util.Try
import java.nio.file.{Paths, Files, Path}
val getMeDirStream: Option[Iterator[Path]] =
for {
path <- Try(Paths.get("")).toOption
directoryStream <- Try(Files.newDirectoryStream(path)).toOption
} yield directoryStream.iterator
Changes:
Using Try(...).toOption instead of Either
Using implicits in scala.collection.convert to return the result as a Scala Iterator.
Try is similar to Option. Instead of Some and None, it has Success and Failure subtypes, and the failure case includes a Throwable, whereas None is just a singleton with no additional information.

why does this scala by-name parameter behave weirdly

OK the question might not say much, but here's the deal:
I'm learning scala and decided to make an utility class "FuncThread" with a method which receives a by-name parameter function (I guess its called that because it's a function but without a parameter list) and then starts a thread with a runable which in turn executes the passed function, I wrote such a class as follows:
class FuncThread
{
def runInThread( func: => Unit)
{
val thread = new Thread(new Runnable()
{
def run()
{
func
}
}
thread.start()
}
}
Then I wrote a junit test as follows:
#Test
def weirdBehaivorTest()
{
var executed = false
val util = new FuncThread()
util.runInThread
{
executed = true
}
//the next line makes the test pass....
//val nonSense : () => Unit = () => { Console println "???" }
assertTrue(executed)
}
If I uncomment the second commented line, the test passes but if it remains commented the test fails, is this the correct behaviour? how and when do by-name parameter functions get executed?
I know Scala has the actors library but I wanted to try this since I've always wanted to do this in Java
Is this just a race condition? runInThread starts the thread but your assertion tests 'executed' before the other thread sets it to true. Adding your extra line means more code (and so time) is executed before the test, making it more likely that 'executed' has been set to true
It's also worth noting that (as of Scala 2.8), the construct you were trying to write is available in the standard library
import scala.actors.Futures._
future{
executed = true
}
This construct is actually more powerful than what you're describing, the thread calculation can return a value, and which can be waited for.
import scala.actors.Futures._
//forks off expensive calculation
val expensiveToCalculateNumber:Future[Int] = future{
bigExpensiveCalculation()
}
// do a lot of other stuff
//print out the result of the expensive calculation if it's ready, otherwise wait until it is
println( expensiveToCalculateNumber());