How do I get instances of connected Inlet and Outlet in FlowShape? - scala

How do I get instances of connected Inlet and Outlet in FlowShape ? Consider following example
def throttleFlow[T](rate: FiniteDuration) = Flow.fromGraph(GraphDSL.create() { implicit builder =>
import GraphDSL.Implicits._
val ticker = Source.tick(rate, rate, Unit)
val zip = builder.add(Zip[T, Unit.type])
val map = Flow[(T, Unit.type)].map { case (value, _) => value }
val messageExtractor = builder.add(map)
val in = Inlet[T]("Req.in")
val out = Outlet[T]("Req.out")
out ~> zip.in0
ticker ~> zip.in1
zip.out ~> messageExtractor.in
FlowShape.of(in, messageExtractor.out)
})
when I use it in Source.via() I get following exception
Caused by: java.lang.IllegalArgumentException: requirement failed: The output port [Req.out] is not part of the underlying graph.
at scala.Predef$.require(Predef.scala:219)
at akka.stream.impl.StreamLayout$Module$class.wire(StreamLayout.scala:204)
What am I missing ?

The in Inlet and out Outlet are not connected to anything. That is why there is an exception (unfortunately this kind of problem can only be detected at runtime)
You want a flow where the only open inlet is one of the zip inlets (zip.in0, since zip.in1 is connected to the ticker), and the only open outlet is the output of the messageExtractor, so how about this:
def throttleFlow[T](rate: FiniteDuration) = Flow.fromGraph(GraphDSL.create() { implicit builder =>
import GraphDSL.Implicits._
val ticker = Source.tick(rate, rate, ())
val zip = builder.add(Zip[T, Unit])
val map = Flow[(T, Unit)].map { case (value, _) => value }
val messageExtractor = builder.add(map)
ticker ~> zip.in1
zip.out ~> messageExtractor.in
FlowShape.of(zip.in0, messageExtractor.out)
})

Related

Consume a source with two sinks and get the result of one sink

I'd like to consume a Source with two different sinks.
Simplified example:
val source = Source(1 to 20)
val addSink = Sink.fold[Int, Int](0)(_ + _)
val subtractSink = Sink.fold[Int, Int](0)(_ - _)
val graph = GraphDSL.create() { implicit builder =>
import GraphDSL.Implicits._
val bcast = builder.add(Broadcast[Int](2))
source ~> bcast.in
bcast.out(0) ~> addSink
bcast.out(1) ~> subtrackSink
ClosedShape
}
RunnableGraph.fromGraph(graph).run()
val result: Future[Int] = ???
I need to be able to retrieve the result of addSink. RunnableGraph.fromGraph(graph).run() gives me NotUsed,
but I'd like to get an Int (the result of the first fold Sink). Is it possible?
Pass in both sinks to the graph builder's create method, which gives you access to their respective materialized values:
val graph = GraphDSL.create(addSink, subtractSink)((_, _)) { implicit builder =>
(aSink, sSink) =>
import GraphDSL.Implicits._
val bcast = builder.add(Broadcast[Int](2))
source ~> bcast.in
bcast.out(0) ~> aSink
bcast.out(1) ~> sSink
ClosedShape
}
val (addResult, subtractResult): (Future[Int], Future[Int]) =
RunnableGraph.fromGraph(graph).run()
Alternatively, you can forgo the graph DSL and use alsoToMat:
val result: Future[Int] =
Source(1 to 20)
.alsoToMat(addSink)(Keep.right)
.toMat(subtractSink)(Keep.left)
.run()
The above gives you the materialized value of addSink. If you want to get the materialized value of both addSink and subtractSink, use Keep.both:
val (addResult, subtractResult): (Future[Int], Future[Int]) =
Source(1 to 20)
.alsoToMat(addSink)(Keep.right)
.toMat(subtractSink)(Keep.both) // <--
.run()

Websocket with Graph DSL

I am trying to implement a Websocket Login flow with Akka Flow. I get a myriad of nasty runtime exceptions around Inlets, Outlets and Connection issues.
My latest is:
java.lang.IllegalStateException: Illegal GraphDSL usage. Inlets [Map.in] were not returned in the resulting shape and not connected.
Snippet:
object Login {
def graph(system: ActorSystem, future: Future[LoginCommand.UserData], socketUrl: String) =
Source.fromGraph(GraphDSL.create() { implicit builder: GraphDSL.Builder[NotUsed] =>
import GraphDSL.Implicits._
val in = Source.fromFuture(future)
in.named("LoginData")
val fanIn = Zip[LoginResponse, LoginCommand.UserData]
val exasolLogin = builder.add(Http(system).webSocketClientFlow(WebSocketRequest(socketUrl)))
val encryptLoginData = FlowShape(exasolLogin.in, fanIn.out)
val exasolAnnounce = Http(system).webSocketClientFlow(WebSocketRequest(socketUrl))
val announceLogin = Source.single(LoginCommand)
in -> fanIn
announceLogin -> exasolAnnounce -> fanIn
fanIn -> encryptLoginData -> exasolLogin
SourceShape(exasolLogin.out)
})
}
I might be using the DSL totally wrong as I have not yet found a single writeup which explains Graphs, Shapes, Flows, Materialized values in depth.
Could someone point out what I am doing wrong or perhaps how this is supposed to be written?
EDIT 1:
Have now replaced -> with ~> and get nasty compile errors:
object Login {
def graph(system: ActorSystem, future: Future[LoginCommand.UserData], socketUrl: String) =
Source.fromGraph(GraphDSL.create() { implicit builder: GraphDSL.Builder[NotUsed] =>
import GraphDSL.Implicits._
val in = Source.fromFuture(future)
in.named("LoginData")
val fanIn = builder.add(Zip[LoginResponse, LoginCommand.UserData])
val exasolLogin = Http(system).webSocketClientFlow(WebSocketRequest(socketUrl))
val encryptLoginData = Flow[(LoginResponse, LoginCommand.UserData)].map(data => data._1)
val loginDataMessage = Flow[LoginCommand.UserData].map(data => TextMessage("bar"))
val exasolAnnounce = Http(system).webSocketClientFlow(WebSocketRequest(socketUrl))
val announceResponse = Flow[Message].map(data => LoginResponse("key", "mod", "exp"))
val loginMessage = Flow[LoginCommand].map(data => TextMessage("foo"))
val session = builder.add(Flow[Message].map(data => LoginCommand.SessionData(0, 1, "2", "db", "w", 59, 546, 45, "q", "TZ", "TZB")))
in ~> fanIn.in1
Source.single(LoginCommand) ~> loginMessage ~> exasolAnnounce ~> announceResponse ~> fanIn.in0
fanIn.out ~> encryptLoginData ~> loginDataMessage ~> exasolLogin ~> session
SourceShape(session.out)
})
}
which leads to
exasol-client/LoginGraph.scala:42: error: overloaded method value ~> with alternatives:
(to: akka.stream.SinkShape[exasol.LoginCommand.type])(implicit b: akka.stream.scaladsl.GraphDSL.Builder[_])Unit <and>
(to: akka.stream.Graph[akka.stream.SinkShape[exasol.LoginCommand.type], _])(implicit b: akka.stream.scaladsl.GraphDSL.Builder[_])Unit <and>
[Out](flow: akka.stream.FlowShape[exasol.LoginCommand.type,Out])(implicit b: akka.stream.scaladsl.GraphDSL.Builder[_])akka.stream.scaladsl.GraphDSL.Implicits.PortOps[Out] <and>
[Out](junction: akka.stream.UniformFanOutShape[exasol.LoginCommand.type,Out])(implicit b: akka.stream.scaladsl.GraphDSL.Builder[_])akka.stream.scaladsl.GraphDSL.Implicits.PortOps[Out] <and>
[Out](junction: akka.stream.UniformFanInShape[exasol.LoginCommand.type,Out])(implicit b: akka.stream.scaladsl.GraphDSL.Builder[_])akka.stream.scaladsl.GraphDSL.Implicits.PortOps[Out] <and>
[Out](via: akka.stream.Graph[akka.stream.FlowShape[exasol.LoginCommand.type,Out],Any])(implicit b: akka.stream.scaladsl.GraphDSL.Builder[_])akka.stream.scaladsl.GraphDSL.Implicits.PortOps[Out] <and>
[U >: exasol.LoginCommand.type](to: akka.stream.Inlet[U])(implicit b: akka.stream.scaladsl.GraphDSL.Builder[_])Unit
cannot be applied to (akka.stream.FlowShape[exasol.LoginCommand,akka.http.scaladsl.model.ws.TextMessage.Strict])
Source.single(LoginCommand) ~> loginMessage ~> exasolAnnounce ~> announceResponse ~> fanIn.in0
You need something like this:
object Login {
def graph(system: ActorSystem, future: Future[LoginCommand.UserData], socketUrl: String): Source[Message, NotUsed] =
Source.fromGraph(GraphDSL.create() { implicit builder: GraphDSL.Builder[NotUsed] =>
import GraphDSL.Implicits._
val in = Source.fromFuture(future)
in.named("LoginData")
val fanIn = builder.add(Zip[LoginResponse, LoginCommand.UserData])
val exasolLogin = builder.add(Http(system).webSocketClientFlow(WebSocketRequest(socketUrl)))
val encryptLoginData = Flow[(LoginResponse, LoginCommand.UserData)].map(data => TextMessage(data.toString)) //stub
val encryptAnnounceData = Flow[LoginCommand].map(data => TextMessage(data.toString)) //stub
val decryptAnnounceData = Flow[Message].map(message => LoginResponse(message)) //stub
val exasolAnnounce = Http(system).webSocketClientFlow(WebSocketRequest(socketUrl))
val announceLogin = Source.single(LoginCommand)
in ~> fanIn.in1
announceLogin ~> encryptAnnounceData ~> exasolAnnounce ~> decryptAnnounceData ~> fanIn.in0
fanIn.out ~> encryptLoginData ~> exasolLogin
SourceShape(exasolLogin.out)
})
}
Keep in mind, that -> and ~> are different operators (you should use ~>). And you need to add a shape to the builder only if you are going to manually connect the shape inlets and outlets.

Custom Akka Streams

I have a set of stream stages (Sources, Flows and Sinks) which I would like to add some MetaData information to.
Therefore rather than Sources producing A -> (A, StreamMetaData). I've managed to do this using custom stream stages whereby on grab(in) the element, I push(out, (elem, StreamMetaData)). In reality it is not 'converting' the existing Source but passing it to a flow to recreate a new source.
Now I'm trying to implement the below MetaStream stage:
Therefore given that the source is producing tuples of (A, StreamMetaData), I want to pass the A to an existing Flow for some computation to take place and then merge the output produced 'B' with the StreamMetaData. These will then be passed to a Sink which accepts (B, StreamMetaData).
How would you suggest I go about it. I've been informed partial graphs are the best bet and would help in completing such a task. UniformFanOut and UniformFanIn using Unzip((A streamMetaData), A, StreamMetaData) and Zip(A,B)
val fanOut = GraphDSL.create() { implicit b =>
val unzip = b.add(Unzip[T, StreamMetaData])
UniformFanOutShape(unzip.in, unzip.out0, unzip.out1)
}
val fanIn = GraphDSL.create() { implicit b =>
val zip = b.add(Zip[T ,StreamMetaData]())
UniformFanInShape(zip)
}
How can I connect the fanIn and fanOut so as to achieve the same behavior as in the picture?
I had something like this in mind;
def metaFlow[T, B, Mat](flow: Flow[T, B, Mat]): Unit = {
val wrappedFlow =
Flow.fromGraph(GraphDSL.create(){ implicit b =>
import GraphDSL.Implicits._
val unzip: FanOutShape2[(T, StreamMetaData), T, StreamMetaData] = b.add(Unzip[T, StreamMetaData])
val existingFlow = b.add(flow)
val zip: FanInShape2[B,StreamMetaData,(B,StreamMetaData)] = b.add(Zip[B, StreamMetaData])
unzip.out0 ~> existingFlow ~> zip.in0
unzip.out1 ~> zip.in1
FlowShape(unzip.in, zip.out)
})
}
Thanks in advance.
This aprox creating a new SourceShape stacking flow graph can work, a little bit different of your flowShape implementation.
def sourceGraph[A, B](f: A => B, source: Source[(A, StreamMetaData), NotUsed]) = Source.fromGraph(GraphDSL.create() { implicit builder: GraphDSL.Builder[NotUsed] =>
import GraphDSL.Implicits._
val unzip = builder.add(Unzip[A, StreamMetaData]())
val zip = builder.add(Zip[B, StreamMetaData]())
val flow0 = builder.add(Flow[A].map { f(_) })
val flow1 = source ~> unzip.in
unzip.out0 ~> flow0 ~> zip.in0
unzip.out1 ~> zip.in1
SourceShape(zip.out)
})
def flowGraph[A, B](f: A => B) = Flow.fromGraph(GraphDSL.create() { implicit builder =>
import GraphDSL.Implicits._
val unzip = builder.add(Unzip[A, StreamMetaData]())
val zip = builder.add(Zip[B, StreamMetaData]())
val flow0 = builder.add(Flow[A].map { f(_) })
unzip.out0 ~> flow0 ~> zip.in0
unzip.out1 ~> zip.in1
FlowShape(unzip.in, zip.out)
})

Akka Streams override KillSwitch

I have the following Graph:
case class FlowFactory() {
val reactiveConnection = ???
val serviceRabbitConnection = ???
val switch = KillSwitches.single[Routed]
val stream: RunnableGraph[UniqueKillSwitch] = RunnableGraph.fromGraph(GraphDSL.create(switch) { implicit builder: GraphDSL.Builder[UniqueKillSwitch] => sw =>
import GraphDSL.Implicits._
val in = builder.add(Source.fromPublisher(reactiveConnection.consume(???)))
val context = builder.add(contextFlow(serviceRabbitConnection))
val inflate = builder.add(inflateFlow())
val compute = builder.add(computeFlow())
val out = builder.add(Sink.fromSubscriber(reactiveConnection.publish()))
in ~> context ~> inflate ~> compute ~> sw ~> out
ClosedShape
})
val killSwitch = stream.run()
killSwitch.shutdown()
}
When I shutdown the stream, I also need to kill the following connections : reactiveConnection and serviceRabbitConnection.
How do I achieve that, is there a easy way to override KillSwitch's shutdown() method?
Is there a method that is called when the stream is closed?, like onComplete() or onClose()?
You can perform your callback inside the stream, by attaching an additional sink (Sink.onComplete).
val sink1 = Sink.fromSubscriber(reactiveConnection.publish())
val sink2 = Sink.onComplete{
case Success(_) ⇒ println("success!")
case Failure(e) ⇒ println(s"failure - $e")
}
val out = builder.add(Sink.combine(sink1, sink2)(Broadcast(_)))

How do you throttle Flow in the latest Akka (2.4.6)?

How do you throttle Flow in the latest Akka (2.4.6) ? I'd like to throttle Http client flow to limit number of requests to 3 requests per second. I found following example online but it's for old Akka and akka-streams API changed so much that I can't figure out how to rewrite it.
def throttled[T](rate: FiniteDuration): Flow[T, T] = {
val tickSource: Source[Unit] = TickSource(rate, rate, () => ())
val zip = Zip[T, Unit]
val in = UndefinedSource[T]
val out = UndefinedSink[T]
PartialFlowGraph { implicit builder =>
import FlowGraphImplicits._
in ~> zip.left ~> Flow[(T, Unit)].map { case (t, _) => t } ~> out
tickSource ~> zip.right
}.toFlow(in, out)
}
Here is my best attempt so far
def throttleFlow[T](rate: FiniteDuration) = Flow.fromGraph(GraphDSL.create() { implicit builder =>
import GraphDSL.Implicits._
val ticker = Source.tick(rate, rate, Unit)
val zip = builder.add(Zip[T, Unit.type])
val map = Flow[(T, Unit.type)].map { case (value, _) => value }
val messageExtractor = builder.add(map)
val in = Inlet[T]("Req.in")
val out = Outlet[T]("Req.out")
out ~> zip.in0
ticker ~> zip.in1
zip.out ~> messageExtractor.in
FlowShape.of(in, messageExtractor.out)
})
it throws exception in my main flow though :)
private val queueHttp = Source.queue[(HttpRequest, (Any, Promise[(Try[HttpResponse], Any)]))](1000, OverflowStrategy.backpressure)
.via(throttleFlow(rate))
.via(poolClientFlow)
.mapAsync(4) {
case (util.Success(resp), any) =>
val strictFut = resp.entity.toStrict(5 seconds)
strictFut.map(ent => (util.Success(resp.copy(entity = ent)), any))
case other =>
Future.successful(other)
}
.toMat(Sink.foreach({
case (triedResp, (value: Any, p: Promise[(Try[HttpResponse], Any)])) =>
p.success(triedResp -> value)
case _ =>
throw new RuntimeException()
}))(Keep.left)
.run
where poolClientFlow is Http()(system).cachedHostConnectionPool[Any](baseDomain)
Exception is:
Caused by: java.lang.IllegalArgumentException: requirement failed: The output port [Req.out] is not part of the underlying graph.
at scala.Predef$.require(Predef.scala:219)
at akka.stream.impl.StreamLayout$Module$class.wire(StreamLayout.scala:204)
Here is an attempt that uses the throttle method as mentioned by #Qingwei. The key is to not use bindAndHandle(), but to use bind() and throttle the flow of incoming connections before handling them. The code is taken from the implementation of bindAndHandle(), but leaves out some error handling for simplicity. Please don't do that in production.
implicit val system = ActorSystem("test")
implicit val mat = ActorMaterializer()
import system.dispatcher
val maxConcurrentConnections = 4
val handler: Flow[HttpRequest, HttpResponse, NotUsed] = complete(LocalDateTime.now().toString)
def handleOneConnection(incomingConnection: IncomingConnection): Future[Done] =
incomingConnection.flow
.watchTermination()(Keep.right)
.joinMat(handler)(Keep.left)
.run()
Http().bind("127.0.0.1", 8080)
.throttle(3, 1.second, 1, ThrottleMode.Shaping)
.mapAsyncUnordered(maxConcurrentConnections)(handleOneConnection)
.to(Sink.ignore)
.run()