Instrument a Source.queue - scala

I would like to have a Source.queue (or something analogous to it to push items on a materialized graph) that is instrumented to tell me the current level of saturation of the queue.
I'd like to do that without (re-)implementing the functionality provided by the QueueSource graph stage.
One possible solution I came up with is the following:
object InstrumentedSource {
final class InstrumentedSourceQueueWithComplete[T](
delegate: SourceQueueWithComplete[T],
bufferSize: Int,
)(implicit executionContext: ExecutionContext)
extends SourceQueueWithComplete[T] {
override def complete(): Unit = delegate.complete()
override def fail(ex: Throwable): Unit = delegate.fail(ex)
override def watchCompletion(): Future[Done] = delegate.watchCompletion()
private val buffered = new AtomicLong(0)
private[InstrumentedSource] def onDequeue(): Unit = {
val _ = buffered.decrementAndGet()
}
object BufferSaturationRatioGauge extends RatioGauge {
override def getRatio: RatioGauge.Ratio = RatioGauge.Ratio.of(buffered.get(), bufferSize)
}
lazy val bufferSaturationGauge: RatioGauge = BufferSaturationRatioGauge
override def offer(elem: T): Future[QueueOfferResult] = {
val result = delegate.offer(elem)
result.foreach {
case QueueOfferResult.Enqueued =>
val _ = buffered.incrementAndGet()
case _ => // do nothing
}
result
}
}
def queue[T](bufferSize: Int, overflowStrategy: OverflowStrategy)(
implicit executionContext: ExecutionContext,
materializer: Materializer,
): Source[T, InstrumentedSourceQueueWithComplete[T]] = {
val (queue, source) = Source.queue[T](bufferSize, overflowStrategy).preMaterialize()
val instrumentedQueue = new InstrumentedSourceQueueWithComplete[T](queue, bufferSize)
source.mapMaterializedValue(_ => instrumentedQueue).map { item =>
instrumentedQueue.onDequeue()
item
}
}
}
This mostly appears to work from some manual testing (apart from the fact that buffered is at most eventually consistent with the actual number of items in the queue, which should be fine in my case), but I was wondering if there is are solutions that perhaps make better use of built-in functionalities that I might have missed.

Related

fs2 - Sharing a Ref with 2 streams

I'm trying to share a Ref[F, A] between 2 concurrent streams. Below is a simplified example of the actual scenario.
class Container[F[_]](implicit F: Sync[F]) {
private val counter = Ref[F].of(0)
def incrementBy2 = counter.flatMap(c => c.update(i => i + 2))
def printCounter = counter.flatMap(c => c.get.flatMap(i => F.delay(println(i))))
}
In the main function,
object MyApp extends IOApp {
def run(args: List[String]): IO[ExitCode] = {
val s = for {
container <- Ref[IO].of(new Container[IO]())
} yield {
val incrementBy2 = Stream.repeatEval(
container.get
.flatTap(c => c.incrementBy2)
.flatMap(c => container.update(_ => c))
)
.metered(2.second)
.interruptScope
val printStream = Stream.repeatEval(
container.get
.flatMap(_.printCounter)
)
.metered(1.seconds)
incrementBy2.concurrently(printStream)
}
Stream.eval(s)
.flatten
.compile
.drain
.as(ExitCode.Success)
}
}
The updates made by the incrementBy2 are not visible in printStream.
How can I fix this?
I would appreciate any help to understand the mistake in this code.
Thanks
Your code is broken since the class definition, you are not even updating the same Ref
Remember that the point of IO is to be a description of a computation, so Ref[F].of(0) returns a program that when evaluated will create a new Ref, calling multiple flatMaps on it will result in multiple Refs being created.
Also, your is not doing tagless final in the right way (and some may argue that even the right way is not worth it: https://alexn.org/blog/2022/04/18/scala-oop-design-sample/)
This is how I would write your code:
trait Counter {
def incrementBy2: IO[Unit]
def printCounter: IO[Unit]
}
object Counter {
val inMemory: IO[Counter] =
IO.ref(0).map { ref =>
new Counter {
override final val incrementBy2: IO[Unit] =
ref.update(c => c + 2)
override final val printCounter: IO[Unit] =
ref.get.flatMap(IO.println)
}
}
}
object Program {
def run(counter: Counter): Stream[IO, Unit] =
Stream
.repeatEval(counter.printCounter)
.metered(1.second)
.concurrently(
Stream.repeatEval(counter.incrementBy2).metered(2.seconds)
).interruptAfter(10.seconds)
}
object Main extends IOApp.Simple {
override final val run: IO[Unit] =
Stream
.eval(Counter.inMemory)
.flatMap(Program.run)
.compile
.drain
}
PS: I would actually not have printCounter but getCounter
and do the printing in the Program
You can see the code running here.

How to propagate context via Kleisli?

Just trying to propagate my tracing context inside Kleisli as it was done originally in the next tutorial.
object TraceLogger {
def log(msg: String): Kleisli[IO, UUID, Unit] = Kleisli { traceId => IO(println(s"[$traceId] $msg")) }
}
trait ServiceStub {
def request(arg: String): Kleisli[IO, UUID, _]
}
trait ClientStub {
def get(arg: String): Kleisli[IO, UUID, _]
}
case class FirstServiceExample(clientStub: ClientStub) extends ServiceStub {
override def request(arg: String): Kleisli[IO, UUID, _] = Kleisli { (context: UUID) =>
val requestComputation = clientStub.get("calling second service!")
TraceLogger.log(arg)
requestComputation(context)
}
}
case class FirstClientExample(service: FirstServiceExample) {
def request(): IO[_] = {
val traceId = UUID.randomUUID()
service.request("root!").run(traceId)
}
}
And now I need to run execution:
val exampleClientStub = new ClientStub() {
override def get(arg: String): Kleisli[IO, UUID, _] = Kleisli.ask
}
val exampleClientService = FirstServiceExample(exampleClientStub)
FirstClientExample(exampleClientService).request().unsafeRunSync()
But, unfortunately, I don't see any logs here. Would you kindly help me to find an issue?
TraceLogger.log(arg) This returns an IO which is just a description of computation; it is doing nothing.
And since you just leave that value alone it is equivalent to just having a 1 in the middle of your code, it is simply discarded.
You need to chain your IOs together to create new IOs that represent "do this and then do that", that is basically what the flatMap method does.
Kleisli { (context: UUID) =>
val requestComputation = clientStub.get("calling second service!")
TraceLogger.log(arg)(context) >> // >> is equivalent to flatMap(_ => )
requestComputation(context)
}
(There is probably a better way to write this, I am not used to Kliesli)
Fabio's series on "Programas as Values" may be very useful: https://systemfw.org/archive.html

FS2 passing resource (or effect) as a state

I'm trying to implement an application that controls a camera. Camera commands are represented as a stream of CameraAction objects:
sealed trait CameraMessage
case object Record(recordId: String) extends CameraMessage
case object Stop extends CameraMessage
...
val s = Stream[F, CameraMessage]
Let's say I have a test stream that emits "Record" and emits "Stop" 20 seconds later, after another 20 seconds another "Record" message is emitted and so on, the input stream is infinite.
Then the app consumes "Record" it should create an instance of GStreamer pipeline (i.e. it is an effect) and "run" it, on "Stop" it 'stops' the pipeline and closes it. Then on subsequent "Record" the pattern is repeated with new GStreamer pipeline.
The problem is that I need to pass an instance of impure, mutable object between handles of stream events.
FS2 documentation suggest to use chunks to make a stream stateful, so I tried
def record(gStreamerPipeline: String, fileName: String)
(implicit sync: Sync[F]): F[Pipeline] =
{
... create and open pipeline ...
}
def stopRecording(pipe: Pipeline)(implicit sync: Sync[F]): F[Unit] = {
... stop pipeline, release resources ...
}
def effectPipe(pipelineDef: String)(implicit L: Logger[F]):
Pipe[F, CameraMessage, F[Unit]] = {
type CameraSessionHandle = Pipeline
type CameraStream = Stream[F, CameraSessionHandle]
s: Stream[F, CameraMessage] =>
s.scanChunks(Stream[F, CameraSessionHandle]()) {
case (s: CameraStream, c: Chunk[CameraMessage]) =>
c.last match {
case Some(Record(fileName)) =>
(Stream.bracket(record(pipelineDef, fileName))(p => stopRecording(p)), Chunk.empty)
case Some(StopRecording) =>
(Stream.empty, Chunk(s.compile.drain))
case _ =>
(s, Chunk.empty)
}
}
}
The problem with this code that actual recording does not happen on 'Record' event but rather then the effect of the whole chunk is evaluated, i.e. when 'StopRecording' message arrives the camera is turned on and then immediately turned off again.
How can I pass a "state" without chunking? Or is there any other way to achieve the result I need?
This may be similar to
FS2 Stream with StateT[IO, _, _], periodically dumping state
but the difference is that the state in my case is not a pure data structure but a resource.
I eventually was able so solve it using Mutable Reference pattern as described in https://typelevel.org/blog/2018/06/07/shared-state-in-fp.html
Here is the code:
import cats.effect._
import cats.syntax.all._
import fs2.Stream
import scala.concurrent.{ExecutionContext, ExecutionContextExecutor}
import scala.language.higherKinds
class FRef[F[_], T](implicit sync: Sync[F]) {
private var state: T = _
def set(n: T): F[Unit] = sync.delay(this.state = n)
def get: F[T] = sync.pure(state)
}
object FRef {
def apply[F[_], T](implicit sync: Sync[F]): F[FRef[F, T]] = sync.delay { new FRef() }
}
class CameraImpl(id: String) extends Camera {
override def record(): Unit = {
println(s"Recording $id")
}
override def stop(): Unit = {
println(s"Stopping $id")
}
override def free(): Unit = {
Thread.sleep(500)
println(s"Freeing $id")
}
}
object Camera {
def apply(id: String) = new CameraImpl(id)
}
trait Camera {
def record(): Unit
def stop(): Unit
def free(): Unit
}
sealed trait CameraMessage
case class Record(recordId: String) extends CameraMessage
case object StopRecording extends CameraMessage
class Streamer[F[_]](implicit sync: Sync[F]) {
def record(id: String): F[Camera] = {
sync.delay {
val r = Camera(id)
r.record()
r
}
}
def stopRecording(pipe: Camera): F[Unit] = {
sync.delay {
pipe.stop()
pipe.free()
}
}
def effectPipe(state: FRef[F, Option[Camera]])(
implicit sync: Sync[F]): Stream[F, CameraMessage] => Stream[F, Unit] = {
type CameraStream = Stream[F, Camera]
s: Stream[F, CameraMessage] =>
s.evalMap {
case Record(fileName) =>
for {
r <- record(fileName)
_ <- state.set(Some(r))
} yield ()
case StopRecording =>
for {
s <- state.get
_ <- stopRecording(s.get)
_ <- state.set(None)
} yield ()
}
}
}
object FS2Problem extends IOApp {
import scala.concurrent.duration._
override def run(args: List[String]): IO[ExitCode] = {
implicit val ec: ExecutionContextExecutor = ExecutionContext.global
val streamer = new Streamer[IO]
val s = Stream.awakeEvery[IO](5.seconds).take(10).zipWithIndex.map {
case (_, idx) =>
idx % 2 match {
case 0 =>
Record(s"Record $idx")
case _ =>
StopRecording
}
}
val ss = for {
streamerState <- Stream.eval(FRef[IO, Option[Camera]])
s <- s.through(streamer.effectPipe(streamerState))
} yield ()
ss.compile.drain.map(_ => ExitCode.Success)
}
}

ReactiveMongo with Akka Streams Custom Source

I am using reactivemongo-akka-stream and trying to transform the AkkaStreamCursor.documentSource. I am and am seeing two problems:
the documentation states that this operation returns a Source[T, NotUsed] but collection.find(query).cursor[BSONDocument].doccumentSource() returns a Source[BSONDocument, Future[State]]. Is there a way to avoid the State object?
Assuming I use Future[State] I am looking to get source of class as below
case class Inner(foo: String, baz: Int)
case class Outer(bar: Inner)
//
implicit object InnerReader extends BSONDocumentReader[Inner]//defined
val getCollection: Future[BSONCollection] = connection.database("db").map(_.collection("things")
def stream()(implicit m: Materializer): Source[Outer, Future[State]] = {
getCollection.map(_.find().cursor[Inner]().documentSource()).map(_.via(Flow[Inner].map(in => Outer(in))))
But instead of getting back a Future[Source[Outer, Future[State]] that i could deal with, this returns a Future[Source[Inner, Future[State]]#Repr[Outer]]
How can bson readers be used with this library?
As per the suggestion by cchantep, I needed to use fromFuture with flatMapConcat:
def stream()(implicit m: Materializer): Source[Outer, NotUsed] = {
val foo = getCollection.map(x => col2Source(x))
fix(foo).via(flowmap(map = Outer(_)))
}
def col2Source(col: BSONCollection): Source[Inner, Future[State]] = {
val cursor: AkkaStreamCursor[Inner] =
col.find(BSONDocument.empty).cursor[Inner]()
cursor.documentSource()
}
def flowmap[In, Out](
map: (In) => Out
): Flow[In, Out, NotUsed] = Flow[In].map(e => map(e))
def fix[Out, Mat](futureSource: Future[Source[Out, Mat]]): Source[Out, NotUsed] = {
Source.fromFuture(futureSource).flatMapConcat(identity)
}

How to create Source and push elements to it manually?

I want to create custom StatefulStage which should works like groupBy method and emit Source[A, Unit] elements but I don't understand how to create instance of Source[A, Unit] and push incoming element to it. Here is stub:
class GroupBy[A, Mat]() extends StatefulStage[A, Source[A, Unit]] {
override def initial: StageState[A, Source[A, Unit]] = new StageState[A, Source[A, Unit]] {
override def onPush(elem: A, ctx: Context[Source[A, Unit]]): SyncDirective = {
val source: Source[A, Unit] = ... // Need to create source here
// and push `elem` to `source` here
emit(List(source).iterator, ctx)
}
}
}
You can use the following snippet for test GroupBy flow (it should print events from created stream):
case class Tick()
case class Event(timestamp: Long, sessionUid: String, traffic: Int)
implicit val system = ActorSystem()
import system.dispatcher
implicit val materializer = ActorMaterializer()
var rnd = Random
rnd.setSeed(1)
val eventsSource = Source
.tick(FiniteDuration(0, SECONDS), FiniteDuration(1, SECONDS), () => Tick)
.map {
case _ => Event(System.currentTimeMillis / 1000, s"session-${rnd.nextInt(5)}", rnd.nextInt(10) * 10)
}
val flow = Flow[Event]
.transform(() => new GroupByUntil)
.map {
(source) => source.runForeach(println)
}
eventsSource
.via(flow)
.runWith(Sink.ignore)
.onComplete(_ => system.shutdown())
Can anybody explain me how to do it?
UPDATE:
I wrote the following onPush method base on this answer but it didn't print events. As I understand I can push element to source only when it running as part of flow but I want to run flow outside of GroupBy in test snippet. If I run flow in GroupBy as in this example then it will process events and send them to Sink.ignore. I think this is a reason why my test snippet didn't print events.
override def onPush(elem: A, ctx: Context[Source[A, Unit]]): SyncDirective = {
val source: Source[A, ActorRef] = Source.actorRef[A](1000, OverflowStrategy.fail)
val flow = Flow[A].to(Sink.ignore).runWith(source)
flow ! elem
emit(List(source.asInstanceOf[Source[A, Unit]]).iterator, ctx)
}
So, how to fix it?