I'm trying to share a Ref[F, A] between 2 concurrent streams. Below is a simplified example of the actual scenario.
class Container[F[_]](implicit F: Sync[F]) {
private val counter = Ref[F].of(0)
def incrementBy2 = counter.flatMap(c => c.update(i => i + 2))
def printCounter = counter.flatMap(c => c.get.flatMap(i => F.delay(println(i))))
}
In the main function,
object MyApp extends IOApp {
def run(args: List[String]): IO[ExitCode] = {
val s = for {
container <- Ref[IO].of(new Container[IO]())
} yield {
val incrementBy2 = Stream.repeatEval(
container.get
.flatTap(c => c.incrementBy2)
.flatMap(c => container.update(_ => c))
)
.metered(2.second)
.interruptScope
val printStream = Stream.repeatEval(
container.get
.flatMap(_.printCounter)
)
.metered(1.seconds)
incrementBy2.concurrently(printStream)
}
Stream.eval(s)
.flatten
.compile
.drain
.as(ExitCode.Success)
}
}
The updates made by the incrementBy2 are not visible in printStream.
How can I fix this?
I would appreciate any help to understand the mistake in this code.
Thanks
Your code is broken since the class definition, you are not even updating the same Ref
Remember that the point of IO is to be a description of a computation, so Ref[F].of(0) returns a program that when evaluated will create a new Ref, calling multiple flatMaps on it will result in multiple Refs being created.
Also, your is not doing tagless final in the right way (and some may argue that even the right way is not worth it: https://alexn.org/blog/2022/04/18/scala-oop-design-sample/)
This is how I would write your code:
trait Counter {
def incrementBy2: IO[Unit]
def printCounter: IO[Unit]
}
object Counter {
val inMemory: IO[Counter] =
IO.ref(0).map { ref =>
new Counter {
override final val incrementBy2: IO[Unit] =
ref.update(c => c + 2)
override final val printCounter: IO[Unit] =
ref.get.flatMap(IO.println)
}
}
}
object Program {
def run(counter: Counter): Stream[IO, Unit] =
Stream
.repeatEval(counter.printCounter)
.metered(1.second)
.concurrently(
Stream.repeatEval(counter.incrementBy2).metered(2.seconds)
).interruptAfter(10.seconds)
}
object Main extends IOApp.Simple {
override final val run: IO[Unit] =
Stream
.eval(Counter.inMemory)
.flatMap(Program.run)
.compile
.drain
}
PS: I would actually not have printCounter but getCounter
and do the printing in the Program
You can see the code running here.
Related
Trying to wrap my head around how async tasks are chained together, for ex futures and flatMap
val a: Future[Int] = Future { 123 }
val b: Future[Int] = a.flatMap(a => Future { a + 321 })
How would i implement something similar where i build a new Future by waiting for a first and applying a f on the result. I tried looking a scala's source code but got stuck here: https://github.com/scala/scala/blob/2.13.x/src/library/scala/concurrent/Future.scala#L216
i would imagine some code like:
def myFlatMap(a: Future[Int])(f: a => Future[Int]): Future[Int] = {
Future {
// somehow wait for a to complete and the apply 'f'
a.onComplete {
case Success(x) => f(a)
}
}
}
but i guess above will return a Future[Unit] instead. I would just like to understand the "pattern" of how async tasks are chained together.
(Disclaimer: I'm the main maintainer of Scala Futures)
You wouldn't want to implement flatMap/recoverWith/transformWith yourself—it is a very complex feature to implement safely (stack-safe, memory-safe, and concurrency-safe).
You can see what I am talking about here:
https://github.com/scala/scala/blob/2.13.x/src/library/scala/concurrent/impl/Promise.scala#L442
https://github.com/scala/scala/blob/2.13.x/src/library/scala/concurrent/impl/Promise.scala#L307
https://github.com/scala/scala/blob/2.13.x/src/library/scala/concurrent/impl/Promise.scala#L274
There is more reading on the topic here: https://viktorklang.com/blog/Futures-in-Scala-2.12-part-9.html
I found that looking at the source for scala 2.12 was easier for me to understand how Future / Promise was implemented, i learned that Future is in fact a Promise and by looking at the implementation i now understand how "chaining" async tasks can be implemented.
Im posting the code that i wrote to help me better understand what was going on under the hood.
import java.util.concurrent.{ScheduledThreadPoolExecutor, TimeUnit}
import scala.collection.mutable.ListBuffer
trait MyAsyncTask[T] {
type Callback = T => Unit
var result: Option[T]
// register a new callback to be called when task is completed.
def onComplete(callback: Callback): Unit
// complete the given task, and invoke all registered callbacks.
def complete(value: T): Unit
// create a new task, that will wait for the current task to complete and apply
// the provided map function, and complete the new task when completed.
def flatMap[B](f: T => MyAsyncTask[B]): MyAsyncTask[B] = {
val wrapper = new DefaultAsyncTask[B]()
onComplete { x =>
f(x).onComplete { y =>
wrapper.complete(y)
}
}
wrapper
}
def map[B](f: T => B): MyAsyncTask[B] = flatMap(x => CompletedAsyncTask(f(x)))
}
/**
* task with fixed pre-calculated result.
*/
case class CompletedAsyncTask[T](value: T) extends MyAsyncTask[T] {
override var result: Option[T] = Some(value)
override def onComplete(callback: Callback): Unit = {
// we already have the result just call the callback.
callback(value)
}
override def complete(value: T): Unit = () // noop nothing to complete.
}
class DefaultAsyncTask[T] extends MyAsyncTask[T] {
override var result: Option[T] = None
var isCompleted = false
var listeners = new ListBuffer[Callback]()
/**
* register callback, to be called when task is completed.
*/
override def onComplete(callback: Callback): Unit = {
if (isCompleted) {
// already completed just invoke callback
callback(result.get)
} else {
// add the listener
listeners.addOne(callback)
}
}
/**
* trigger all registered callbacks to `onComplete`
*/
override def complete(value: T): Unit = {
result = Some(value)
listeners.foreach { listener =>
listener(value)
}
}
}
object MyAsyncTask {
def apply[T](body: (T => Unit) => Unit): MyAsyncTask[T] = {
val task = new DefaultAsyncTask[T]()
// pass in `complete` as callback.
body(task.complete)
task
}
}
object MyAsyncFlatMap {
/**
* helper to simulate async task.
*/
def delayedCall(body: => Unit, delay: Int): Unit = {
val scheduler = new ScheduledThreadPoolExecutor(1)
val run = new Runnable {
override def run(): Unit = {
body
}
}
scheduler.schedule(run, delay, TimeUnit.SECONDS)
}
def main(args: Array[String]): Unit = {
val getAge = MyAsyncTask[Int] { cb =>
delayedCall({ cb(66) }, 1)
}
val getName = MyAsyncTask[String] { cb =>
delayedCall({ cb("John") }, 2)
}
// same as: getAge.flatMap(age => getName.map(name => (age, name)))
val result: MyAsyncTask[(Int, String)] = for {
age <- getAge
name <- getName
} yield (age, name)
result.onComplete {
case (age, name) => println(s"hello $name $age")
}
}
}
I would like to have a Source.queue (or something analogous to it to push items on a materialized graph) that is instrumented to tell me the current level of saturation of the queue.
I'd like to do that without (re-)implementing the functionality provided by the QueueSource graph stage.
One possible solution I came up with is the following:
object InstrumentedSource {
final class InstrumentedSourceQueueWithComplete[T](
delegate: SourceQueueWithComplete[T],
bufferSize: Int,
)(implicit executionContext: ExecutionContext)
extends SourceQueueWithComplete[T] {
override def complete(): Unit = delegate.complete()
override def fail(ex: Throwable): Unit = delegate.fail(ex)
override def watchCompletion(): Future[Done] = delegate.watchCompletion()
private val buffered = new AtomicLong(0)
private[InstrumentedSource] def onDequeue(): Unit = {
val _ = buffered.decrementAndGet()
}
object BufferSaturationRatioGauge extends RatioGauge {
override def getRatio: RatioGauge.Ratio = RatioGauge.Ratio.of(buffered.get(), bufferSize)
}
lazy val bufferSaturationGauge: RatioGauge = BufferSaturationRatioGauge
override def offer(elem: T): Future[QueueOfferResult] = {
val result = delegate.offer(elem)
result.foreach {
case QueueOfferResult.Enqueued =>
val _ = buffered.incrementAndGet()
case _ => // do nothing
}
result
}
}
def queue[T](bufferSize: Int, overflowStrategy: OverflowStrategy)(
implicit executionContext: ExecutionContext,
materializer: Materializer,
): Source[T, InstrumentedSourceQueueWithComplete[T]] = {
val (queue, source) = Source.queue[T](bufferSize, overflowStrategy).preMaterialize()
val instrumentedQueue = new InstrumentedSourceQueueWithComplete[T](queue, bufferSize)
source.mapMaterializedValue(_ => instrumentedQueue).map { item =>
instrumentedQueue.onDequeue()
item
}
}
}
This mostly appears to work from some manual testing (apart from the fact that buffered is at most eventually consistent with the actual number of items in the queue, which should be fine in my case), but I was wondering if there is are solutions that perhaps make better use of built-in functionalities that I might have missed.
I'm trying to implement an application that controls a camera. Camera commands are represented as a stream of CameraAction objects:
sealed trait CameraMessage
case object Record(recordId: String) extends CameraMessage
case object Stop extends CameraMessage
...
val s = Stream[F, CameraMessage]
Let's say I have a test stream that emits "Record" and emits "Stop" 20 seconds later, after another 20 seconds another "Record" message is emitted and so on, the input stream is infinite.
Then the app consumes "Record" it should create an instance of GStreamer pipeline (i.e. it is an effect) and "run" it, on "Stop" it 'stops' the pipeline and closes it. Then on subsequent "Record" the pattern is repeated with new GStreamer pipeline.
The problem is that I need to pass an instance of impure, mutable object between handles of stream events.
FS2 documentation suggest to use chunks to make a stream stateful, so I tried
def record(gStreamerPipeline: String, fileName: String)
(implicit sync: Sync[F]): F[Pipeline] =
{
... create and open pipeline ...
}
def stopRecording(pipe: Pipeline)(implicit sync: Sync[F]): F[Unit] = {
... stop pipeline, release resources ...
}
def effectPipe(pipelineDef: String)(implicit L: Logger[F]):
Pipe[F, CameraMessage, F[Unit]] = {
type CameraSessionHandle = Pipeline
type CameraStream = Stream[F, CameraSessionHandle]
s: Stream[F, CameraMessage] =>
s.scanChunks(Stream[F, CameraSessionHandle]()) {
case (s: CameraStream, c: Chunk[CameraMessage]) =>
c.last match {
case Some(Record(fileName)) =>
(Stream.bracket(record(pipelineDef, fileName))(p => stopRecording(p)), Chunk.empty)
case Some(StopRecording) =>
(Stream.empty, Chunk(s.compile.drain))
case _ =>
(s, Chunk.empty)
}
}
}
The problem with this code that actual recording does not happen on 'Record' event but rather then the effect of the whole chunk is evaluated, i.e. when 'StopRecording' message arrives the camera is turned on and then immediately turned off again.
How can I pass a "state" without chunking? Or is there any other way to achieve the result I need?
This may be similar to
FS2 Stream with StateT[IO, _, _], periodically dumping state
but the difference is that the state in my case is not a pure data structure but a resource.
I eventually was able so solve it using Mutable Reference pattern as described in https://typelevel.org/blog/2018/06/07/shared-state-in-fp.html
Here is the code:
import cats.effect._
import cats.syntax.all._
import fs2.Stream
import scala.concurrent.{ExecutionContext, ExecutionContextExecutor}
import scala.language.higherKinds
class FRef[F[_], T](implicit sync: Sync[F]) {
private var state: T = _
def set(n: T): F[Unit] = sync.delay(this.state = n)
def get: F[T] = sync.pure(state)
}
object FRef {
def apply[F[_], T](implicit sync: Sync[F]): F[FRef[F, T]] = sync.delay { new FRef() }
}
class CameraImpl(id: String) extends Camera {
override def record(): Unit = {
println(s"Recording $id")
}
override def stop(): Unit = {
println(s"Stopping $id")
}
override def free(): Unit = {
Thread.sleep(500)
println(s"Freeing $id")
}
}
object Camera {
def apply(id: String) = new CameraImpl(id)
}
trait Camera {
def record(): Unit
def stop(): Unit
def free(): Unit
}
sealed trait CameraMessage
case class Record(recordId: String) extends CameraMessage
case object StopRecording extends CameraMessage
class Streamer[F[_]](implicit sync: Sync[F]) {
def record(id: String): F[Camera] = {
sync.delay {
val r = Camera(id)
r.record()
r
}
}
def stopRecording(pipe: Camera): F[Unit] = {
sync.delay {
pipe.stop()
pipe.free()
}
}
def effectPipe(state: FRef[F, Option[Camera]])(
implicit sync: Sync[F]): Stream[F, CameraMessage] => Stream[F, Unit] = {
type CameraStream = Stream[F, Camera]
s: Stream[F, CameraMessage] =>
s.evalMap {
case Record(fileName) =>
for {
r <- record(fileName)
_ <- state.set(Some(r))
} yield ()
case StopRecording =>
for {
s <- state.get
_ <- stopRecording(s.get)
_ <- state.set(None)
} yield ()
}
}
}
object FS2Problem extends IOApp {
import scala.concurrent.duration._
override def run(args: List[String]): IO[ExitCode] = {
implicit val ec: ExecutionContextExecutor = ExecutionContext.global
val streamer = new Streamer[IO]
val s = Stream.awakeEvery[IO](5.seconds).take(10).zipWithIndex.map {
case (_, idx) =>
idx % 2 match {
case 0 =>
Record(s"Record $idx")
case _ =>
StopRecording
}
}
val ss = for {
streamerState <- Stream.eval(FRef[IO, Option[Camera]])
s <- s.through(streamer.effectPipe(streamerState))
} yield ()
ss.compile.drain.map(_ => ExitCode.Success)
}
}
I have an iteration module which can apply an arbitrary function (Build generic reusable iteration module from higher order function) and would love to wrap it into a progressbar.
val things = Range(1,10)
def iterationModule[A](
iterationItems: Seq[A],
functionToApply: A => Any
): Unit = {
iterationItems.foreach(functionToApply)
}
def foo(s:Int) = println(s)
iterationModule[Int](things, foo)
A basic progressbar could look like:
import me.tongfei.progressbar.ProgressBar
val pb = new ProgressBar("Test", things.size)
things.foreach(t=> {
println(t)
pb.step
})
But how can the function which is passed to the iterator module be intercepted and surrounded with a progressbar, i.e. call the pb.step?
An annoying possibility would be to pass the mutable pb object into each function (have it implement an interface).
But is it also possible to intercept and surround the function being passed by this stepping logic?
However, when looping with Seq().par.foreach, this might be problematic.
I need the code to work in Scala 2.11.
edit
A more complex example:
val things = Range(1,100).map(_.toString)
def iterationModule[A](
iterationItems: Seq[A],
functionToApply: A => Any,
parallel: Boolean = false
): Unit = {
val pb = new ProgressBar(functionToApply.toString(), iterationItems.size)
if (parallel) {
iterationItems.par.foreach(functionToApply)
} else {
iterationItems.foreach(functionToApply)
}
}
def doStuff(inputDay: String, inputConfigSomething: String): Unit = println(inputDay + "__"+ inputConfigSomething)
iterationModule[String](things, doStuff(_, "foo"))
The function should be able to take the iteration item and additional parameters.
edit 2
import me.tongfei.progressbar.ProgressBar
val things = Range(1,100).map(_.toString)
def doStuff(inputDay: String, inputConfigSomething: String): Unit = println(inputDay + "__"+ inputConfigSomething)
def iterationModulePb[A](items: Seq[A], f: A => Any, parallel: Boolean = false): Unit = {
val pb = new ProgressBar(f.toString, items.size)
val it = if (parallel) {
items.par.iterator
} else {
items.iterator
}
it.foreach { x =>
f(x)
pb.step()
}
}
iterationModulePb[String](things, doStuff(_, "foo"))
After a little discussion I figured out how to use a Seq with standard iterators.
For Scala 2.13 this would be the most general form.
import me.tongfei.progressbar.ProgressBar
def iterationModule[A](items: IterableOnce[A], f: A => Any): Unit = {
val (it, pb) =
if (items.knowSize != -1)
items.iterator -> new ProgressBar("Test", items.knowSize)
else {
val (iter1, iter2) = items.iterator.split
iter1 -> new ProgressBar("Test", iter2.size)
}
it.foreach { x =>
f(x)
pb.step()
}
}
Note: most of the changes are just to make the code more generic, but the general idea is just to create a function that wraps both the original function and the call to the ProgressBar.
Edit
A simplified solution for 2.11
def iterationModule[A](items: Seq[A], parallel: Boolean = false)
(f: A => Any): Unit = {
val pb = new ProgressBar("test", items.size)
val it = if (parallel) {
items.iterator.par
} else {
items.iterator
}
it.foreach { a =>
f(a)
pb.step()
}
}
I'm trying to figure out how to write this piece of code in an elegant pure-functional style using scalaz7 IO and monad transformers but just can't get my head around it.
Just imagine I have this simple API:
def findUuid(request: Request): Option[String] = ???
def findProfile(uuid: String): Future[Option[Profile]] = redisClient.get[Profile](uuid)
Using this API I can easily write impure function with OptionT transformer like this:
val profileT = for {
uuid <- OptionT(Future.successful(findUuid(request)))
profile <- OptionT(findProfile(uuid))
} yield profile
val profile: Future[Option[Profile]] = profileT.run
As you have noticed - this function contains findProfile() with a side-effect. I want to isolate this effect inside of the IO monad and interpret outside of the pure function but don't know how to combine it all together.
def findProfileIO(uuid: String): IO[Future[Option[Profile]]] = IO(findProfile(uuid))
val profileT = for {
uuid <- OptionT(Future.successful(findUuid(request)))
profile <- OptionT(findProfileIO(uuid)) //??? how to put Option inside of the IO[Future[Option]]
} yield profile
val profile = profileT.run //how to run transformer and interpret IO with the unsafePerformIO()???
Any peaces of advice on how it might be done?
IO is meant more for synchronous effects. Task is more what you want!
See this question and answer: What's the difference between Task and IO in Scalaz?
You can convert your Future to Task and then have an API like this:
def findUuid(request: Request): Option[String] = ???
def findProfile(uuid: String): Task[Option[Profile]] = ???
This works because Task can represent both synchronous and asynchronous operations, so findUuid can also be wrapped in Task instead of IO.
Then you can wrap these in OptionT:
val profileT = for {
uuid <- OptionT(Task.now(findUuid(request)))
profile <- OptionT(findProfileIO(uuid))
} yield profile
Then at the end somewhere you can run it:
profileT.run.attemptRun
Check out this link for converting Futures to Tasks and vice versa: Scalaz Task <-> Future
End up with this piece of code, thought it might be useful for someone (Play 2.6).
Controller's method is a pure function since Task evaluation takes place outside of the controller inside of PureAction ActionBuilder. Thanks to Luka's answer!
Still struggling with new paradigm of Action composition in Play 2.6 though, but this is another story.
FrontendController.scala:
def index = PureAction.pure { request =>
val profileOpt = (for {
uuid <- OptionT(Task.now(request.cookies.get("uuid").map(t => uuidKey(t.value))))
profile <- OptionT(redis.get[Profile](uuid).asTask)
} yield profile).run
profileOpt.map { profileOpt =>
Logger.info(profileOpt.map(p => s"User logged in - $p").getOrElse("New user, suggesting login"))
Ok(views.html.index(profileOpt))
}
}
Actions.scala
Convenient action with Task resolution at the end
class PureAction #Inject()(parser: BodyParsers.Default)(implicit ec: ExecutionContext) extends ActionBuilderImpl(parser) {
self =>
def pure(block: Request[AnyContent] => Task[Result]): Action[AnyContent] = composeAction(new Action[AnyContent] {
override def parser: BodyParser[AnyContent] = self.parser
override def executionContext: ExecutionContext = self.ec
override def apply(request: Request[AnyContent]): Future[Result] = {
val taskResult = block(request)
taskResult.asFuture //End of the world lives here
}
})
}
Converters.scala
Task->Future and Future->Task implicit converters
implicit class FuturePimped[+T](root: => Future[T]) {
import scalaz.Scalaz._
def asTask(implicit ec: ExecutionContext): Task[T] = {
Task.async { register =>
root.onComplete {
case Success(v) => register(v.right)
case Failure(ex) => register(ex.left)
}
}
}
}
implicit class TaskPimped[T](root: => Task[T]) {
import scalaz._
val p: Promise[T] = Promise()
def asFuture: Future[T] = {
root.unsafePerformAsync {
case -\/(ex) => p.failure(ex); ()
case \/-(r) => p.success(r); ()
}
p.future
}
}