I have a function that returns a fs2.Stream of Measurements.
import cats.effect._
import fs2._
def apply(sds: SerialPort, interval: Int)(implicit cs: ContextShift[IO]): Stream[IO, SdsMeasurement] =
for {
blocker <- Stream.resource(Blocker[IO])
stream <- io.readInputStream(IO(sds.getInputStream), 1, blocker)
.through(SdsStateMachine.collectMeasurements())
} yield stream
Normally it is an infinite Stream, unless I pass it a test flag, in which case it should output one value and halt.
val infiniteSource: Stream[IO, SdsMeasurement] = ...
val source = if (isTest) infiniteSource.take(1) else infiniteSource
source.compile.drain
The infinite Stream works fine. It gives me all Measurements infinitely. The test Stream indeed gives me only the first measurement, nothing more. The problem I have is that the Stream does not return after this last measurements. It blocks forever. What am I doing wrong?
Note: I think I abstracted the essential code, but for more context, please take a look at my project: https://github.com/jkransen/fijnstof/blob/ZIO/src/main/scala/nl/kransen/fijnstof/Main.scala
The code you've presented here looks fine, I don't think the issue lies within that code. If it blocks, then presumably one of the underlying APIs blocks, for instance it might be the close method of the InputStream. What I usually do in such situations is to add log statements before and after every function call that might block.
Related
I'm having trouble mocking the MVar2 method make in my Unit test (using mockito).
I tried this:
private val sq = mock[MVar2[IO, String]]
and when I tried to mock the method like:
when(sq.take).thenReturn(
"testString".pure[IO],
IO.never
)
I get the infinite loop in the test.
I was positive I could mock it like this.
The actual code calling the make is:
def run: F[Unit] =
fs2.Stream
.repeatEval[F, String](signalQ.take)
.map(encode)
.unNone
.evalTap(logMessage)
.evalMap(send)
.compile
.drain
I see two issues.
The first thing, there is no reason to mock IO. You can pretty much do this:
// define this wherever you create sq
val sq = MVar2[IO, String]
// then DI sq and
sq.put("testString") >> run
Then the second thing, how does .repeatEval(io) works? It calls io repeatedly, every time the stream needs another element. So at first it would evaluate io with your mock to:
"testString".pure[IO]
which would compute the first element to "testString" and later to
IO.never
which means that the second element would never appear as this io would never return - neither value nor exception. And your stream halts as you can see. (Which would be according to the spec but apparently not what you want).
If you wanted to get Some(string) if value is present and None if it's absent (to terminate stream on None) you should try:
def run: F[Unit] =
fs2.Stream
.repeatEval[F, Option[String]](signalQ.tryTake)// <-- try to get None on empty
.unNoneTerminate // <-- terminate on None
.map(encode)
.evalTap(logMessage)
.evalMap(send)
.compile
.drain
Basically I mean:
for(v <- Future(long time operation)) yield v*someOtherValue
This expression returns another Future, but the question is, is the v*someOhterValue operation lazy or not? Will this expression block on getting the value of Future(long time operation)?
Or it is like a chain of callbacks?
A short experiment can test this question.
import concurrent._;
import concurrent.ExecutionContext.Implicits.global
import scala.concurrent.duration._
object TheFuture {
def main(args: Array[String]): Unit = {
val fut = for (v <- Future { Thread.sleep(2000) ; 10 }) yield v * 10;
println("For loop is finished...")
println(Await.ready(fut, Duration.Inf).value.get);
}
}
If we run this, we see For loop is finished... almost immediately, and then two seconds later, we see the result. So the act of performing map or similar operations on a future is not blocking.
A map (or, equivalently, your for comprehension) on a Future is not lazy: it will be executed as soon as possible on another thread. However, since it runs on another thread, it isn't blocking, either.
If you want to do the definition and execution of the Future separately, then you have to use something like a Monix Task.
https://monix.io/api/3.0/monix/eval/Task.html
I have:
val observable: Observable[Int] = Observable.from(List(5))
and I can test that the input list is indeed passed on to the observable by testing:
materializeValues(observable) should contain (5)
where materializeValues is:
def materializeValues[T](observable: Observable[T]): List[T] = {
observable.toBlocking.toIterable.toList
}
Now, if I create an observable from a future, I can't seem to use materializeValues for the test as the test times out. So if I have:
val futVal = Future.successful(5)
val observable: Observable[Int] = Observable.from(futVal)
materializeValues(observable) should contain(5)
it times out and does not pass the test. What is different in the process of materializing these two observables, which leads to me not being able to block on it?
Also, what is the idomatic way of testing an observable? Is there any way of doing it without calling toBlocking?
I think the problem is that you use AsyncWordSpecLike (by the way why AsyncWordSpecLike instead of AsyncWordSpec?). AsyncWordSpecLike/AsyncWordSpec are designed to simplify testing Future. Unfortunately Observable is a more powerful abstraction that can't be easily mapped onto a Future.
Particularly AsyncWordSpecLike/AsyncWordSpec allow your tests to return Future[Assertion]. To make it possible it provides custom implicit ExecutionContext that it can force to execute everything and know when all scheduled jobs have finished. However the same custom ExecutionContext is the reason why your second code doesn't work: processing of the scheduled jobs starts only after execution of your test code has finished but your code blocks on the futVal because unlucklily for you callback registered in Future.onComplete is scheduled to be run on the ExecutionContext. It means that you have a kind of dead-lock with your own thread.
I'm not sure what is the official way to test Observable on Scala. In Java I think TestSubscriber is the suggested tool. As I said Observable is fundamentally more powerful thing than Future so I think to test Observable you should avoid using AsyncWordSpecLike/AsyncWordSpec. If you switch to use FlatSpec or WordSpec, you can do something like this:
class MyObservableTestSpec extends WordSpec with Matchers {
import scala.concurrent.ExecutionContext.Implicits.global
val testValue = 5
"observables" should {
"be testable if created from futures" in {
val futVal = Future.successful(testValue)
val observable = Observable.from(futVal)
val subscriber = TestSubscriber[Int]()
observable(subscriber)
subscriber.awaitTerminalEvent
// now after awaitTerminalEvent you can use various subscriber.assertXyz methods
subscriber.assertNoErrors
subscriber.assertValues(testValue)
// or you can use Matchers as
subscriber.getOnNextEvents should contain(testValue)
}
}
}
I already have a Source[T], but I need to pass it to a function that requires a Stream[T].
I could .run the source and materialize everything to a list and then do a .toStream on the result but that removes the lazy/stream aspect that I want to keep.
Is this the only way to accomplish this or am I missing something?
EDIT:
After reading Vladimir's comment, I believe I'm approaching my issue in the wrong way. Here's a simple example of what I have and what I want to create:
// Given a start index, returns a list from startIndex to startIndex+10. Stops at 50.
def getMoreData(startIndex: Int)(implicit ec: ExecutionContext): Future[List[Int]] = {
println(s"f running with $startIndex")
val result: List[Int] = (startIndex until Math.min(startIndex + 10, 50)).toList
Future.successful(result)
}
So getMoreData just emulates a service which returns data by the page to me.
My first goal it to create the following function:
def getStream(startIndex: Int)(implicit ec: ExecutionContext): Stream[Future[List[Int]]]
where the next Future[List[Int]] element in the stream depends on the previous one, taking the last index read from the previous Future's value in the stream. Eg with a startIndex of 0:
getStream(0)(0) would return Future[List[0 until 10]]
getStream(0)(1) would return Future[List[10 until 20]]
... etc
Once I have that function, I then want to create a 2nd function to further map it down to a simple Stream[Int]. Eg:
def getFlattenedStream(stream: Stream[Future[List[Int]]]): Stream[Int]
Streams are beginning to feel like the wrong tool for the job and I should just write a simple loop instead. I liked the idea of streams because the consumer can map/modify/consume the stream as they see fit without the producer needing to know about it.
Scala Streams are a fine way of accomplishing your task within getStream; here is a basic way to construct what you're looking for:
def getStream(startIndex : Int)
(implicit ec: ExecutionContext): Stream[Future[List[Int]]] =
Stream
.from(startIndex, 10)
.map(getMoreData)
Where things get tricky is with your getFlattenedStream function. It is possible to eliminate the Future wrapper around your List[Int] values, but it will require an Await.result function call which is usually a mistake.
More often than not it is best to operate on the Futures and allow asynchronous operations to happen on their own. If you analyze your ultimate requirement/goal it is usually not necessary to wait on a Future.
But, iff you absolutely must drop the Future then here is the code that can accomplish it:
val awaitDuration = 10 Seconds
def getFlattenedStream(stream: Stream[Future[List[Int]]]): Stream[Int] =
stream
.map(f => Await.result(f, awaitDuration))
.flatten
I am learning scalaz stream at the moment, I am confused why repeatEval only evaluate Task.async once.
val result = Process
.repeatEval(Task.async[Unit](t => {
val result = scala.io.Source.fromURL("http://someUrl").mkString
println(".......")
println(result)
}))
result.runLog.run //only print once
However, if I change Task.async to Task.delay. It evaluates the function infinitely. I dont know why is that
val result = Process
.repeatEval(Task.delay({
val result = scala.io.Source.fromURL("http://someUrl").mkString
println(".......")
println(result)
}))
result.runLog.run //print infinitely
Many thanks in advance
As I mention in my answer to your recent question about Task, Task.async takes a function that registers callbacks—not some code that should be executed asynchronously. In the case of the other question, you actually want Task.async, since you're interoperating with a callback-based API.
Here it seems like you probably want Task.apply, not Task.delay. The two look similar, but delay simply suspends the computation—it doesn't use an ExecutorService to run it in a separate thread. You can see this in the following example:
import scalaz._, Scalaz._, concurrent._
val delayTask = Task.delay(Thread.sleep(5000))
val applyTask = Task(Thread.sleep(5000))
Nondeterminism[Task].both(delayTask, delayTask).run
Nondeterminism[Task].both(applyTask, applyTask).run
The delayTask version will take longer.