I am using reactivemongo-akka-stream and trying to transform the AkkaStreamCursor.documentSource. I am and am seeing two problems:
the documentation states that this operation returns a Source[T, NotUsed] but collection.find(query).cursor[BSONDocument].doccumentSource() returns a Source[BSONDocument, Future[State]]. Is there a way to avoid the State object?
Assuming I use Future[State] I am looking to get source of class as below
case class Inner(foo: String, baz: Int)
case class Outer(bar: Inner)
//
implicit object InnerReader extends BSONDocumentReader[Inner]//defined
val getCollection: Future[BSONCollection] = connection.database("db").map(_.collection("things")
def stream()(implicit m: Materializer): Source[Outer, Future[State]] = {
getCollection.map(_.find().cursor[Inner]().documentSource()).map(_.via(Flow[Inner].map(in => Outer(in))))
But instead of getting back a Future[Source[Outer, Future[State]] that i could deal with, this returns a Future[Source[Inner, Future[State]]#Repr[Outer]]
How can bson readers be used with this library?
As per the suggestion by cchantep, I needed to use fromFuture with flatMapConcat:
def stream()(implicit m: Materializer): Source[Outer, NotUsed] = {
val foo = getCollection.map(x => col2Source(x))
fix(foo).via(flowmap(map = Outer(_)))
}
def col2Source(col: BSONCollection): Source[Inner, Future[State]] = {
val cursor: AkkaStreamCursor[Inner] =
col.find(BSONDocument.empty).cursor[Inner]()
cursor.documentSource()
}
def flowmap[In, Out](
map: (In) => Out
): Flow[In, Out, NotUsed] = Flow[In].map(e => map(e))
def fix[Out, Mat](futureSource: Future[Source[Out, Mat]]): Source[Out, NotUsed] = {
Source.fromFuture(futureSource).flatMapConcat(identity)
}
Related
Just trying to propagate my tracing context inside Kleisli as it was done originally in the next tutorial.
object TraceLogger {
def log(msg: String): Kleisli[IO, UUID, Unit] = Kleisli { traceId => IO(println(s"[$traceId] $msg")) }
}
trait ServiceStub {
def request(arg: String): Kleisli[IO, UUID, _]
}
trait ClientStub {
def get(arg: String): Kleisli[IO, UUID, _]
}
case class FirstServiceExample(clientStub: ClientStub) extends ServiceStub {
override def request(arg: String): Kleisli[IO, UUID, _] = Kleisli { (context: UUID) =>
val requestComputation = clientStub.get("calling second service!")
TraceLogger.log(arg)
requestComputation(context)
}
}
case class FirstClientExample(service: FirstServiceExample) {
def request(): IO[_] = {
val traceId = UUID.randomUUID()
service.request("root!").run(traceId)
}
}
And now I need to run execution:
val exampleClientStub = new ClientStub() {
override def get(arg: String): Kleisli[IO, UUID, _] = Kleisli.ask
}
val exampleClientService = FirstServiceExample(exampleClientStub)
FirstClientExample(exampleClientService).request().unsafeRunSync()
But, unfortunately, I don't see any logs here. Would you kindly help me to find an issue?
TraceLogger.log(arg) This returns an IO which is just a description of computation; it is doing nothing.
And since you just leave that value alone it is equivalent to just having a 1 in the middle of your code, it is simply discarded.
You need to chain your IOs together to create new IOs that represent "do this and then do that", that is basically what the flatMap method does.
Kleisli { (context: UUID) =>
val requestComputation = clientStub.get("calling second service!")
TraceLogger.log(arg)(context) >> // >> is equivalent to flatMap(_ => )
requestComputation(context)
}
(There is probably a better way to write this, I am not used to Kliesli)
Fabio's series on "Programas as Values" may be very useful: https://systemfw.org/archive.html
I would like to have a Source.queue (or something analogous to it to push items on a materialized graph) that is instrumented to tell me the current level of saturation of the queue.
I'd like to do that without (re-)implementing the functionality provided by the QueueSource graph stage.
One possible solution I came up with is the following:
object InstrumentedSource {
final class InstrumentedSourceQueueWithComplete[T](
delegate: SourceQueueWithComplete[T],
bufferSize: Int,
)(implicit executionContext: ExecutionContext)
extends SourceQueueWithComplete[T] {
override def complete(): Unit = delegate.complete()
override def fail(ex: Throwable): Unit = delegate.fail(ex)
override def watchCompletion(): Future[Done] = delegate.watchCompletion()
private val buffered = new AtomicLong(0)
private[InstrumentedSource] def onDequeue(): Unit = {
val _ = buffered.decrementAndGet()
}
object BufferSaturationRatioGauge extends RatioGauge {
override def getRatio: RatioGauge.Ratio = RatioGauge.Ratio.of(buffered.get(), bufferSize)
}
lazy val bufferSaturationGauge: RatioGauge = BufferSaturationRatioGauge
override def offer(elem: T): Future[QueueOfferResult] = {
val result = delegate.offer(elem)
result.foreach {
case QueueOfferResult.Enqueued =>
val _ = buffered.incrementAndGet()
case _ => // do nothing
}
result
}
}
def queue[T](bufferSize: Int, overflowStrategy: OverflowStrategy)(
implicit executionContext: ExecutionContext,
materializer: Materializer,
): Source[T, InstrumentedSourceQueueWithComplete[T]] = {
val (queue, source) = Source.queue[T](bufferSize, overflowStrategy).preMaterialize()
val instrumentedQueue = new InstrumentedSourceQueueWithComplete[T](queue, bufferSize)
source.mapMaterializedValue(_ => instrumentedQueue).map { item =>
instrumentedQueue.onDequeue()
item
}
}
}
This mostly appears to work from some manual testing (apart from the fact that buffered is at most eventually consistent with the actual number of items in the queue, which should be fine in my case), but I was wondering if there is are solutions that perhaps make better use of built-in functionalities that I might have missed.
I want to create custom StatefulStage which should works like groupBy method and emit Source[A, Unit] elements but I don't understand how to create instance of Source[A, Unit] and push incoming element to it. Here is stub:
class GroupBy[A, Mat]() extends StatefulStage[A, Source[A, Unit]] {
override def initial: StageState[A, Source[A, Unit]] = new StageState[A, Source[A, Unit]] {
override def onPush(elem: A, ctx: Context[Source[A, Unit]]): SyncDirective = {
val source: Source[A, Unit] = ... // Need to create source here
// and push `elem` to `source` here
emit(List(source).iterator, ctx)
}
}
}
You can use the following snippet for test GroupBy flow (it should print events from created stream):
case class Tick()
case class Event(timestamp: Long, sessionUid: String, traffic: Int)
implicit val system = ActorSystem()
import system.dispatcher
implicit val materializer = ActorMaterializer()
var rnd = Random
rnd.setSeed(1)
val eventsSource = Source
.tick(FiniteDuration(0, SECONDS), FiniteDuration(1, SECONDS), () => Tick)
.map {
case _ => Event(System.currentTimeMillis / 1000, s"session-${rnd.nextInt(5)}", rnd.nextInt(10) * 10)
}
val flow = Flow[Event]
.transform(() => new GroupByUntil)
.map {
(source) => source.runForeach(println)
}
eventsSource
.via(flow)
.runWith(Sink.ignore)
.onComplete(_ => system.shutdown())
Can anybody explain me how to do it?
UPDATE:
I wrote the following onPush method base on this answer but it didn't print events. As I understand I can push element to source only when it running as part of flow but I want to run flow outside of GroupBy in test snippet. If I run flow in GroupBy as in this example then it will process events and send them to Sink.ignore. I think this is a reason why my test snippet didn't print events.
override def onPush(elem: A, ctx: Context[Source[A, Unit]]): SyncDirective = {
val source: Source[A, ActorRef] = Source.actorRef[A](1000, OverflowStrategy.fail)
val flow = Flow[A].to(Sink.ignore).runWith(source)
flow ! elem
emit(List(source.asInstanceOf[Source[A, Unit]]).iterator, ctx)
}
So, how to fix it?
I want to write some function that can return different Class objects according to the arguments.
For example, I have some classes that extends akka Actor, and I want to get their classes by passing different Int values. The code below is not correct, but I think you can understand what I mean:
def createActor(num: Int): Unit {
val c: Class = o.getActorClass(num)
system.actorOf(Props[c]) ! "hello"
}
object o {
val m: Map[Int, Class] = Map(1->classOf[Actor1], 2->classOf[Actor2])
def getActorClass(num: Int): Class {
m(num)
}
}
Hope my question is understandable. Thank you!
If you simply return the ActorRef, you should be fine.
def createActor(num: Int): ActorRef = {
val c = o.getActorClass(num)
val actor = system.actorOf(Props(c))
actor ! "hello"
actor
}
I would create a generator method for creating a new mapping under a given actor system like so:
def newMapping(sys: ActorSystem, mappings: (Int, Props)*) = {
assert(sys != null)
val m = mappings.toMap
(i: Int) => sys.actorOf(m(i))
}
Then you can create different mappings with different ActorSystems and ultimately get the effect you desire:
val sys = ActorSystem("sys1")
val myMapping = newMapping(sys, 1 -> Props[Class1], 2 -> Props[Class2])
val actor1 = myMapping(1)
val actor2 = myMapping(2)
I am a newbie in using Play framework. I am struggling to design a model with one-to-many and retrieve it using Json. I am using Play 2.6, Scala 2.11 with Slick 3.0.0. I have two tables: Rack : case class RackRow(id: String, produced: Float, currentHour: Long) and Gpu: case class Gpu(id: String, rackId: String, produced: Float, installedAt: String).
One of my previous error was to put the business logic in the Controller layer. So I am moving the business logic to the model. Below is my RackRepository.scala where I am using Slick:
case class Rack(id: String, produced: Float, currentHour: String, gpuList: Seq[Gpu])
case class RackRow(id: String, produced: Float, currentHour: Long)
case class RackException(message: String) extends Exception(message)
class RackRepository #Inject()(protected val dbConfigProvider: DatabaseConfigProvider, gpuRepository: GpuRepository)
(implicit ec: ExecutionContext) extends HasDatabaseConfigProvider[JdbcProfile] {
import profile.api._
lazy val RackTable = new TableQuery(tag => new RackTable(tag))
def getProfile: JdbcProfile = profile
def database: JdbcBackend#DatabaseDef = db
def create(row: List[RackRow]): Future[Option[Int]] =
db.run(RackTable ++= row)
def insert(row: RackRow): Future[Unit] =
db.run(RackTable += row).map(_ => ())
def updateProduced(rackId: String, produced: Float): Future[Unit] =
db.run(RackTable.filter(_.id === rackId).map(r => r.produced).update(produced)).map(_ => ())
def updateRack(rackId: String, produced: Float, currentHour: Long): Future[Unit] =
db.run(RackTable.filter(_.id === rackId).map(r => (r.produced, r.currentHour)).update(produced, currentHour)).map(_ => ())
def updateRackProduced(id: String): Future[Unit] = {
gpuRepository.getByRack(id).map { seqGpuRow: Seq[GpuRow] =>
val total: Float = seqGpuRow.map(_.produced).sum
update(id, Some(total), Some(System.currentTimeMillis))
}
}
def update(rackId: String, produced: Option[Float], currentHour: Option[Long]): Future[Unit] = {
(produced, currentHour) match {
case (Some(produced), Some(currentHour)) =>
db.run(RackTable.filter(_.id === rackId).map(r => (r.produced, r.currentHour)).update(produced, currentHour)).map(_ => ())
case (Some(produced), None) =>
db.run(RackTable.filter(_.id === rackId).map(r => r.produced).update(produced)).map(_ => ())
case (None, Some(currentHour)) =>
db.run(RackTable.filter(_.id === rackId).map(r => r.currentHour).update(currentHour)).map(_ => ())
case (None, None) => Future("update not executed.")
}
}
def listAllRacksWithSetup(): Future[Setup] = {
list().flatMap { seqRackRow: Seq[RackRow] =>
val futureSeqRackRow: Seq[Future[Rack]] = seqRackRow.map { rackRow: RackRow =>
gpuRepository.getByRack(rackRow.id).map { seqGpuRow: Seq[GpuRow] =>
val seqGpu = seqGpuRow.map(gpuRepository.gpuRowToGpu) // return Seq[Gpu]
Rack(rackRow.id, rackRow.produced, Util.toDate(rackRow.currentHour), seqGpu)
} // return Future[Rack]
}
val futureSeqRack: Future[Seq[Rack]] = Future.sequence(futureSeqRackRow)
futureSeqRack.map(racks => Setup(getProfitPerGpu, racks))
}
}
}
If you see the constructor of the class RackRepository, I am passing the parameter gpuRepository: GpuRepository, that is the other model. I guess this approach is not right. But I did this to use the method gpuRepository.getByRack() in the def listAllRacksWithSetup().
The problem comes when I go to the other model GpuRepository where I cannot pass rackRepository: RackRepository in the constructor because it throws an error about recursive dependency. So, this is why my approach is not right.
How do I design the RackRepository and GpuRepository to one have access to the other? If this is right based on best practices.
I guess during making the question I realize what is the answer. but thanks to everyone that read my question...
I create a lazy val rackRepository = new RackRepository(dbConfigProvider) inside the GpuRepository. I wonder if this is still a good practice with Play framework. If someone could still answer about best practices of MVC in Play framework.
Thanks
class GpuRepository #Inject()(protected val dbConfigProvider: DatabaseConfigProvider)
(implicit ec: ExecutionContext) extends HasDatabaseConfigProvider[JdbcProfile] {
import profile.api._
lazy val GpuTable = new TableQuery(tag => new GpuTable(tag))
lazy val rackRepository = new RackRepository(dbConfigProvider)