I am trying to implement a Websocket Login flow with Akka Flow. I get a myriad of nasty runtime exceptions around Inlets, Outlets and Connection issues.
My latest is:
java.lang.IllegalStateException: Illegal GraphDSL usage. Inlets [Map.in] were not returned in the resulting shape and not connected.
Snippet:
object Login {
def graph(system: ActorSystem, future: Future[LoginCommand.UserData], socketUrl: String) =
Source.fromGraph(GraphDSL.create() { implicit builder: GraphDSL.Builder[NotUsed] =>
import GraphDSL.Implicits._
val in = Source.fromFuture(future)
in.named("LoginData")
val fanIn = Zip[LoginResponse, LoginCommand.UserData]
val exasolLogin = builder.add(Http(system).webSocketClientFlow(WebSocketRequest(socketUrl)))
val encryptLoginData = FlowShape(exasolLogin.in, fanIn.out)
val exasolAnnounce = Http(system).webSocketClientFlow(WebSocketRequest(socketUrl))
val announceLogin = Source.single(LoginCommand)
in -> fanIn
announceLogin -> exasolAnnounce -> fanIn
fanIn -> encryptLoginData -> exasolLogin
SourceShape(exasolLogin.out)
})
}
I might be using the DSL totally wrong as I have not yet found a single writeup which explains Graphs, Shapes, Flows, Materialized values in depth.
Could someone point out what I am doing wrong or perhaps how this is supposed to be written?
EDIT 1:
Have now replaced -> with ~> and get nasty compile errors:
object Login {
def graph(system: ActorSystem, future: Future[LoginCommand.UserData], socketUrl: String) =
Source.fromGraph(GraphDSL.create() { implicit builder: GraphDSL.Builder[NotUsed] =>
import GraphDSL.Implicits._
val in = Source.fromFuture(future)
in.named("LoginData")
val fanIn = builder.add(Zip[LoginResponse, LoginCommand.UserData])
val exasolLogin = Http(system).webSocketClientFlow(WebSocketRequest(socketUrl))
val encryptLoginData = Flow[(LoginResponse, LoginCommand.UserData)].map(data => data._1)
val loginDataMessage = Flow[LoginCommand.UserData].map(data => TextMessage("bar"))
val exasolAnnounce = Http(system).webSocketClientFlow(WebSocketRequest(socketUrl))
val announceResponse = Flow[Message].map(data => LoginResponse("key", "mod", "exp"))
val loginMessage = Flow[LoginCommand].map(data => TextMessage("foo"))
val session = builder.add(Flow[Message].map(data => LoginCommand.SessionData(0, 1, "2", "db", "w", 59, 546, 45, "q", "TZ", "TZB")))
in ~> fanIn.in1
Source.single(LoginCommand) ~> loginMessage ~> exasolAnnounce ~> announceResponse ~> fanIn.in0
fanIn.out ~> encryptLoginData ~> loginDataMessage ~> exasolLogin ~> session
SourceShape(session.out)
})
}
which leads to
exasol-client/LoginGraph.scala:42: error: overloaded method value ~> with alternatives:
(to: akka.stream.SinkShape[exasol.LoginCommand.type])(implicit b: akka.stream.scaladsl.GraphDSL.Builder[_])Unit <and>
(to: akka.stream.Graph[akka.stream.SinkShape[exasol.LoginCommand.type], _])(implicit b: akka.stream.scaladsl.GraphDSL.Builder[_])Unit <and>
[Out](flow: akka.stream.FlowShape[exasol.LoginCommand.type,Out])(implicit b: akka.stream.scaladsl.GraphDSL.Builder[_])akka.stream.scaladsl.GraphDSL.Implicits.PortOps[Out] <and>
[Out](junction: akka.stream.UniformFanOutShape[exasol.LoginCommand.type,Out])(implicit b: akka.stream.scaladsl.GraphDSL.Builder[_])akka.stream.scaladsl.GraphDSL.Implicits.PortOps[Out] <and>
[Out](junction: akka.stream.UniformFanInShape[exasol.LoginCommand.type,Out])(implicit b: akka.stream.scaladsl.GraphDSL.Builder[_])akka.stream.scaladsl.GraphDSL.Implicits.PortOps[Out] <and>
[Out](via: akka.stream.Graph[akka.stream.FlowShape[exasol.LoginCommand.type,Out],Any])(implicit b: akka.stream.scaladsl.GraphDSL.Builder[_])akka.stream.scaladsl.GraphDSL.Implicits.PortOps[Out] <and>
[U >: exasol.LoginCommand.type](to: akka.stream.Inlet[U])(implicit b: akka.stream.scaladsl.GraphDSL.Builder[_])Unit
cannot be applied to (akka.stream.FlowShape[exasol.LoginCommand,akka.http.scaladsl.model.ws.TextMessage.Strict])
Source.single(LoginCommand) ~> loginMessage ~> exasolAnnounce ~> announceResponse ~> fanIn.in0
You need something like this:
object Login {
def graph(system: ActorSystem, future: Future[LoginCommand.UserData], socketUrl: String): Source[Message, NotUsed] =
Source.fromGraph(GraphDSL.create() { implicit builder: GraphDSL.Builder[NotUsed] =>
import GraphDSL.Implicits._
val in = Source.fromFuture(future)
in.named("LoginData")
val fanIn = builder.add(Zip[LoginResponse, LoginCommand.UserData])
val exasolLogin = builder.add(Http(system).webSocketClientFlow(WebSocketRequest(socketUrl)))
val encryptLoginData = Flow[(LoginResponse, LoginCommand.UserData)].map(data => TextMessage(data.toString)) //stub
val encryptAnnounceData = Flow[LoginCommand].map(data => TextMessage(data.toString)) //stub
val decryptAnnounceData = Flow[Message].map(message => LoginResponse(message)) //stub
val exasolAnnounce = Http(system).webSocketClientFlow(WebSocketRequest(socketUrl))
val announceLogin = Source.single(LoginCommand)
in ~> fanIn.in1
announceLogin ~> encryptAnnounceData ~> exasolAnnounce ~> decryptAnnounceData ~> fanIn.in0
fanIn.out ~> encryptLoginData ~> exasolLogin
SourceShape(exasolLogin.out)
})
}
Keep in mind, that -> and ~> are different operators (you should use ~>). And you need to add a shape to the builder only if you are going to manually connect the shape inlets and outlets.
Related
I'm attempting to apply a separate a filter on 2 output streams as a result of a fanout. I emit 3 objects of Type Test defined as :
case class Test(test: String, tester: Double)
Running the below src does not produce any results :
import akka.NotUsed
import akka.actor.ActorSystem
import akka.stream.ClosedShape
import akka.stream.scaladsl.{Broadcast, Flow, GraphDSL, RunnableGraph, Sink, Source, Zip}
object TestFanOut extends App {
implicit val actorSystem = ActorSystem()
val filter1 = Flow[Test].filter(f => f.test.equalsIgnoreCase("2"))
val filter2 = Flow[Test].filter(f => f.test.equalsIgnoreCase("1"))
val output = Sink.foreach[(Test , Test)](println)
val input = Source.repeat(Test("1" , 23)).take(3)
case class Test(test: String, tester: Double)
val graph = RunnableGraph.fromGraph(
GraphDSL.create() { implicit builder: GraphDSL.Builder[NotUsed] =>
import GraphDSL.Implicits._
val broadcast = builder.add(Broadcast[Test](2))
val zip = builder.add(Zip[Test , Test])
input ~> broadcast
broadcast.out(0) ~> filter1 ~> zip.in0
broadcast.out(1) ~> filter2 ~> zip.in1
zip.out ~> output
ClosedShape
}
)
graph.run()
}
However, changing the filters to :
val filter1 = Flow[Test].filter(f => f.test.equalsIgnoreCase("1"))
val filter2 = Flow[Test].filter(f => f.test.equalsIgnoreCase("1"))
produces :
(Test(1,23.0),Test(1,23.0))
(Test(1,23.0),Test(1,23.0))
(Test(1,23.0),Test(1,23.0))
It seems my filtering logic is not correctly applied ?
When I define the filters as :
val filter1 = Flow[Test].filter(f => f.test.equalsIgnoreCase("2"))
val filter2 = Flow[Test].filter(f => f.test.equalsIgnoreCase("1"))
I expect output of :
(Test(1,23.0),Test(1,23.0))
(Test(1,23.0))
As this this matches one of the fan out streams.
Zip requires output from both Flows. In this case, it won't be possible because of the filter.
Instead, you could use Merge,
val output = Sink.foreach[Test](println)
val graph = RunnableGraph.fromGraph(
GraphDSL.create() { implicit builder: GraphDSL.Builder[NotUsed] =>
import GraphDSL.Implicits._
val broadcast = builder.add(Broadcast[Test](2))
val merge = builder.add(Merge[Test](2))
input ~> broadcast
broadcast.out(0) ~> filter1 ~> merge
broadcast.out(1) ~> filter2 ~> merge
merge ~> output
ClosedShape
}
)
This would print:
Test(1,23.0)
Test(1,23.0)
Test(1,23.0)
On the other hand, if you want zipped content, you could refactor using Option,
val filter1 = Flow[Option[Test]].map(f => f.filter(_.test.equalsIgnoreCase("2")))
val filter2 = Flow[Option[Test]].map(f => f.filter(_.test.equalsIgnoreCase("1")))
val output = Sink.foreach[(Option[Test] , Option[Test])](println)
val input = Source.repeat(Option(Test("1" , 23))).take(3)
val graph = RunnableGraph.fromGraph(
GraphDSL.create() { implicit builder: GraphDSL.Builder[NotUsed] =>
import GraphDSL.Implicits._
val broadcast = builder.add(Broadcast[Option[Test]](2))
val zip = builder.add(Zip[Option[Test] , Option[Test]])
input ~> broadcast
broadcast.out(0) ~> filter1 ~> zip.in0
broadcast.out(1) ~> filter2 ~> zip.in1
zip.out ~> output
ClosedShape
}
)
This would print:
(None,Some(Test(1,23.0)))
(None,Some(Test(1,23.0)))
(None,Some(Test(1,23.0)))
I have a set of stream stages (Sources, Flows and Sinks) which I would like to add some MetaData information to.
Therefore rather than Sources producing A -> (A, StreamMetaData). I've managed to do this using custom stream stages whereby on grab(in) the element, I push(out, (elem, StreamMetaData)). In reality it is not 'converting' the existing Source but passing it to a flow to recreate a new source.
Now I'm trying to implement the below MetaStream stage:
Therefore given that the source is producing tuples of (A, StreamMetaData), I want to pass the A to an existing Flow for some computation to take place and then merge the output produced 'B' with the StreamMetaData. These will then be passed to a Sink which accepts (B, StreamMetaData).
How would you suggest I go about it. I've been informed partial graphs are the best bet and would help in completing such a task. UniformFanOut and UniformFanIn using Unzip((A streamMetaData), A, StreamMetaData) and Zip(A,B)
val fanOut = GraphDSL.create() { implicit b =>
val unzip = b.add(Unzip[T, StreamMetaData])
UniformFanOutShape(unzip.in, unzip.out0, unzip.out1)
}
val fanIn = GraphDSL.create() { implicit b =>
val zip = b.add(Zip[T ,StreamMetaData]())
UniformFanInShape(zip)
}
How can I connect the fanIn and fanOut so as to achieve the same behavior as in the picture?
I had something like this in mind;
def metaFlow[T, B, Mat](flow: Flow[T, B, Mat]): Unit = {
val wrappedFlow =
Flow.fromGraph(GraphDSL.create(){ implicit b =>
import GraphDSL.Implicits._
val unzip: FanOutShape2[(T, StreamMetaData), T, StreamMetaData] = b.add(Unzip[T, StreamMetaData])
val existingFlow = b.add(flow)
val zip: FanInShape2[B,StreamMetaData,(B,StreamMetaData)] = b.add(Zip[B, StreamMetaData])
unzip.out0 ~> existingFlow ~> zip.in0
unzip.out1 ~> zip.in1
FlowShape(unzip.in, zip.out)
})
}
Thanks in advance.
This aprox creating a new SourceShape stacking flow graph can work, a little bit different of your flowShape implementation.
def sourceGraph[A, B](f: A => B, source: Source[(A, StreamMetaData), NotUsed]) = Source.fromGraph(GraphDSL.create() { implicit builder: GraphDSL.Builder[NotUsed] =>
import GraphDSL.Implicits._
val unzip = builder.add(Unzip[A, StreamMetaData]())
val zip = builder.add(Zip[B, StreamMetaData]())
val flow0 = builder.add(Flow[A].map { f(_) })
val flow1 = source ~> unzip.in
unzip.out0 ~> flow0 ~> zip.in0
unzip.out1 ~> zip.in1
SourceShape(zip.out)
})
def flowGraph[A, B](f: A => B) = Flow.fromGraph(GraphDSL.create() { implicit builder =>
import GraphDSL.Implicits._
val unzip = builder.add(Unzip[A, StreamMetaData]())
val zip = builder.add(Zip[B, StreamMetaData]())
val flow0 = builder.add(Flow[A].map { f(_) })
unzip.out0 ~> flow0 ~> zip.in0
unzip.out1 ~> zip.in1
FlowShape(unzip.in, zip.out)
})
I have the following types and declarations:
import scalaz._, Scalaz._
trait Container[T]
type FreeContainer[A] = Free[Container, A]
type FreeFreeContainer[A] = Free[FreeContainer, A]
val fc: FreeContainer[Int]
val ffc: FreeFreeContainer[Int]
val t: Container ~> Id
val tranformed: Int = fc.foldMap(t) //ok
val tranformed2: Int = ffc.foldMap(t) //error
Is it possible to lift Container ~> Id to FreeContainer ~> Id?
Yes, via foldMap:
val t: Container ~> Id
val tt: FreeContainer ~> Id = new (FreeContainer ~> Id) {
def apply[A](fc: FreeContainer[A]): A = fc.foldMap(t)
}
Using the polymorphic lambda syntax of kind-projector, this can be simplified to
val tt: FreeContainer ~> Id = λ[FreeContainer ~> Id](_.foldMap(t))
So you can do
val tranformed2: Int = ffc.foldMap(λ[FreeContainer ~> Id](_.foldMap(t)))
Alternatively, you can just do two consecutive foldMaps, the first one with the identity natural transformation:
val tranformed2: Int = ffc.foldMap(NaturalTransformation.refl[FreeContainer]).foldMap(t)
I have the following Graph:
case class FlowFactory() {
val reactiveConnection = ???
val serviceRabbitConnection = ???
val switch = KillSwitches.single[Routed]
val stream: RunnableGraph[UniqueKillSwitch] = RunnableGraph.fromGraph(GraphDSL.create(switch) { implicit builder: GraphDSL.Builder[UniqueKillSwitch] => sw =>
import GraphDSL.Implicits._
val in = builder.add(Source.fromPublisher(reactiveConnection.consume(???)))
val context = builder.add(contextFlow(serviceRabbitConnection))
val inflate = builder.add(inflateFlow())
val compute = builder.add(computeFlow())
val out = builder.add(Sink.fromSubscriber(reactiveConnection.publish()))
in ~> context ~> inflate ~> compute ~> sw ~> out
ClosedShape
})
val killSwitch = stream.run()
killSwitch.shutdown()
}
When I shutdown the stream, I also need to kill the following connections : reactiveConnection and serviceRabbitConnection.
How do I achieve that, is there a easy way to override KillSwitch's shutdown() method?
Is there a method that is called when the stream is closed?, like onComplete() or onClose()?
You can perform your callback inside the stream, by attaching an additional sink (Sink.onComplete).
val sink1 = Sink.fromSubscriber(reactiveConnection.publish())
val sink2 = Sink.onComplete{
case Success(_) ⇒ println("success!")
case Failure(e) ⇒ println(s"failure - $e")
}
val out = builder.add(Sink.combine(sink1, sink2)(Broadcast(_)))
How do you throttle Flow in the latest Akka (2.4.6) ? I'd like to throttle Http client flow to limit number of requests to 3 requests per second. I found following example online but it's for old Akka and akka-streams API changed so much that I can't figure out how to rewrite it.
def throttled[T](rate: FiniteDuration): Flow[T, T] = {
val tickSource: Source[Unit] = TickSource(rate, rate, () => ())
val zip = Zip[T, Unit]
val in = UndefinedSource[T]
val out = UndefinedSink[T]
PartialFlowGraph { implicit builder =>
import FlowGraphImplicits._
in ~> zip.left ~> Flow[(T, Unit)].map { case (t, _) => t } ~> out
tickSource ~> zip.right
}.toFlow(in, out)
}
Here is my best attempt so far
def throttleFlow[T](rate: FiniteDuration) = Flow.fromGraph(GraphDSL.create() { implicit builder =>
import GraphDSL.Implicits._
val ticker = Source.tick(rate, rate, Unit)
val zip = builder.add(Zip[T, Unit.type])
val map = Flow[(T, Unit.type)].map { case (value, _) => value }
val messageExtractor = builder.add(map)
val in = Inlet[T]("Req.in")
val out = Outlet[T]("Req.out")
out ~> zip.in0
ticker ~> zip.in1
zip.out ~> messageExtractor.in
FlowShape.of(in, messageExtractor.out)
})
it throws exception in my main flow though :)
private val queueHttp = Source.queue[(HttpRequest, (Any, Promise[(Try[HttpResponse], Any)]))](1000, OverflowStrategy.backpressure)
.via(throttleFlow(rate))
.via(poolClientFlow)
.mapAsync(4) {
case (util.Success(resp), any) =>
val strictFut = resp.entity.toStrict(5 seconds)
strictFut.map(ent => (util.Success(resp.copy(entity = ent)), any))
case other =>
Future.successful(other)
}
.toMat(Sink.foreach({
case (triedResp, (value: Any, p: Promise[(Try[HttpResponse], Any)])) =>
p.success(triedResp -> value)
case _ =>
throw new RuntimeException()
}))(Keep.left)
.run
where poolClientFlow is Http()(system).cachedHostConnectionPool[Any](baseDomain)
Exception is:
Caused by: java.lang.IllegalArgumentException: requirement failed: The output port [Req.out] is not part of the underlying graph.
at scala.Predef$.require(Predef.scala:219)
at akka.stream.impl.StreamLayout$Module$class.wire(StreamLayout.scala:204)
Here is an attempt that uses the throttle method as mentioned by #Qingwei. The key is to not use bindAndHandle(), but to use bind() and throttle the flow of incoming connections before handling them. The code is taken from the implementation of bindAndHandle(), but leaves out some error handling for simplicity. Please don't do that in production.
implicit val system = ActorSystem("test")
implicit val mat = ActorMaterializer()
import system.dispatcher
val maxConcurrentConnections = 4
val handler: Flow[HttpRequest, HttpResponse, NotUsed] = complete(LocalDateTime.now().toString)
def handleOneConnection(incomingConnection: IncomingConnection): Future[Done] =
incomingConnection.flow
.watchTermination()(Keep.right)
.joinMat(handler)(Keep.left)
.run()
Http().bind("127.0.0.1", 8080)
.throttle(3, 1.second, 1, ThrottleMode.Shaping)
.mapAsyncUnordered(maxConcurrentConnections)(handleOneConnection)
.to(Sink.ignore)
.run()