I'm not really sure how to ask the question. What I basically want to know is if it's possible to get this graph to return a failure, even though its output is upstream of the failure:
def flow(n: String) = Flow[Int].map{v => println(s"$n: $v"); v + 1}
val g = Source.fromGraph(GraphDSL.create() { implicit b =>
import GraphDSL.Implicits._
val broadcast = b.add(Broadcast[Int](2))
val source = b.add(Source.single(1))
source ~> flow("A") ~> broadcast
broadcast.out(1) ~> flow("B") ~> flow("C").mapAsync(1){ _ => Future.failed(new Exception())} ~> Sink.foreach[Int](f => println("OK"))
SourceShape(broadcast.out(0))
})
Related
I'm attempting to apply a separate a filter on 2 output streams as a result of a fanout. I emit 3 objects of Type Test defined as :
case class Test(test: String, tester: Double)
Running the below src does not produce any results :
import akka.NotUsed
import akka.actor.ActorSystem
import akka.stream.ClosedShape
import akka.stream.scaladsl.{Broadcast, Flow, GraphDSL, RunnableGraph, Sink, Source, Zip}
object TestFanOut extends App {
implicit val actorSystem = ActorSystem()
val filter1 = Flow[Test].filter(f => f.test.equalsIgnoreCase("2"))
val filter2 = Flow[Test].filter(f => f.test.equalsIgnoreCase("1"))
val output = Sink.foreach[(Test , Test)](println)
val input = Source.repeat(Test("1" , 23)).take(3)
case class Test(test: String, tester: Double)
val graph = RunnableGraph.fromGraph(
GraphDSL.create() { implicit builder: GraphDSL.Builder[NotUsed] =>
import GraphDSL.Implicits._
val broadcast = builder.add(Broadcast[Test](2))
val zip = builder.add(Zip[Test , Test])
input ~> broadcast
broadcast.out(0) ~> filter1 ~> zip.in0
broadcast.out(1) ~> filter2 ~> zip.in1
zip.out ~> output
ClosedShape
}
)
graph.run()
}
However, changing the filters to :
val filter1 = Flow[Test].filter(f => f.test.equalsIgnoreCase("1"))
val filter2 = Flow[Test].filter(f => f.test.equalsIgnoreCase("1"))
produces :
(Test(1,23.0),Test(1,23.0))
(Test(1,23.0),Test(1,23.0))
(Test(1,23.0),Test(1,23.0))
It seems my filtering logic is not correctly applied ?
When I define the filters as :
val filter1 = Flow[Test].filter(f => f.test.equalsIgnoreCase("2"))
val filter2 = Flow[Test].filter(f => f.test.equalsIgnoreCase("1"))
I expect output of :
(Test(1,23.0),Test(1,23.0))
(Test(1,23.0))
As this this matches one of the fan out streams.
Zip requires output from both Flows. In this case, it won't be possible because of the filter.
Instead, you could use Merge,
val output = Sink.foreach[Test](println)
val graph = RunnableGraph.fromGraph(
GraphDSL.create() { implicit builder: GraphDSL.Builder[NotUsed] =>
import GraphDSL.Implicits._
val broadcast = builder.add(Broadcast[Test](2))
val merge = builder.add(Merge[Test](2))
input ~> broadcast
broadcast.out(0) ~> filter1 ~> merge
broadcast.out(1) ~> filter2 ~> merge
merge ~> output
ClosedShape
}
)
This would print:
Test(1,23.0)
Test(1,23.0)
Test(1,23.0)
On the other hand, if you want zipped content, you could refactor using Option,
val filter1 = Flow[Option[Test]].map(f => f.filter(_.test.equalsIgnoreCase("2")))
val filter2 = Flow[Option[Test]].map(f => f.filter(_.test.equalsIgnoreCase("1")))
val output = Sink.foreach[(Option[Test] , Option[Test])](println)
val input = Source.repeat(Option(Test("1" , 23))).take(3)
val graph = RunnableGraph.fromGraph(
GraphDSL.create() { implicit builder: GraphDSL.Builder[NotUsed] =>
import GraphDSL.Implicits._
val broadcast = builder.add(Broadcast[Option[Test]](2))
val zip = builder.add(Zip[Option[Test] , Option[Test]])
input ~> broadcast
broadcast.out(0) ~> filter1 ~> zip.in0
broadcast.out(1) ~> filter2 ~> zip.in1
zip.out ~> output
ClosedShape
}
)
This would print:
(None,Some(Test(1,23.0)))
(None,Some(Test(1,23.0)))
(None,Some(Test(1,23.0)))
I'd like to consume a Source with two different sinks.
Simplified example:
val source = Source(1 to 20)
val addSink = Sink.fold[Int, Int](0)(_ + _)
val subtractSink = Sink.fold[Int, Int](0)(_ - _)
val graph = GraphDSL.create() { implicit builder =>
import GraphDSL.Implicits._
val bcast = builder.add(Broadcast[Int](2))
source ~> bcast.in
bcast.out(0) ~> addSink
bcast.out(1) ~> subtrackSink
ClosedShape
}
RunnableGraph.fromGraph(graph).run()
val result: Future[Int] = ???
I need to be able to retrieve the result of addSink. RunnableGraph.fromGraph(graph).run() gives me NotUsed,
but I'd like to get an Int (the result of the first fold Sink). Is it possible?
Pass in both sinks to the graph builder's create method, which gives you access to their respective materialized values:
val graph = GraphDSL.create(addSink, subtractSink)((_, _)) { implicit builder =>
(aSink, sSink) =>
import GraphDSL.Implicits._
val bcast = builder.add(Broadcast[Int](2))
source ~> bcast.in
bcast.out(0) ~> aSink
bcast.out(1) ~> sSink
ClosedShape
}
val (addResult, subtractResult): (Future[Int], Future[Int]) =
RunnableGraph.fromGraph(graph).run()
Alternatively, you can forgo the graph DSL and use alsoToMat:
val result: Future[Int] =
Source(1 to 20)
.alsoToMat(addSink)(Keep.right)
.toMat(subtractSink)(Keep.left)
.run()
The above gives you the materialized value of addSink. If you want to get the materialized value of both addSink and subtractSink, use Keep.both:
val (addResult, subtractResult): (Future[Int], Future[Int]) =
Source(1 to 20)
.alsoToMat(addSink)(Keep.right)
.toMat(subtractSink)(Keep.both) // <--
.run()
I have a set of stream stages (Sources, Flows and Sinks) which I would like to add some MetaData information to.
Therefore rather than Sources producing A -> (A, StreamMetaData). I've managed to do this using custom stream stages whereby on grab(in) the element, I push(out, (elem, StreamMetaData)). In reality it is not 'converting' the existing Source but passing it to a flow to recreate a new source.
Now I'm trying to implement the below MetaStream stage:
Therefore given that the source is producing tuples of (A, StreamMetaData), I want to pass the A to an existing Flow for some computation to take place and then merge the output produced 'B' with the StreamMetaData. These will then be passed to a Sink which accepts (B, StreamMetaData).
How would you suggest I go about it. I've been informed partial graphs are the best bet and would help in completing such a task. UniformFanOut and UniformFanIn using Unzip((A streamMetaData), A, StreamMetaData) and Zip(A,B)
val fanOut = GraphDSL.create() { implicit b =>
val unzip = b.add(Unzip[T, StreamMetaData])
UniformFanOutShape(unzip.in, unzip.out0, unzip.out1)
}
val fanIn = GraphDSL.create() { implicit b =>
val zip = b.add(Zip[T ,StreamMetaData]())
UniformFanInShape(zip)
}
How can I connect the fanIn and fanOut so as to achieve the same behavior as in the picture?
I had something like this in mind;
def metaFlow[T, B, Mat](flow: Flow[T, B, Mat]): Unit = {
val wrappedFlow =
Flow.fromGraph(GraphDSL.create(){ implicit b =>
import GraphDSL.Implicits._
val unzip: FanOutShape2[(T, StreamMetaData), T, StreamMetaData] = b.add(Unzip[T, StreamMetaData])
val existingFlow = b.add(flow)
val zip: FanInShape2[B,StreamMetaData,(B,StreamMetaData)] = b.add(Zip[B, StreamMetaData])
unzip.out0 ~> existingFlow ~> zip.in0
unzip.out1 ~> zip.in1
FlowShape(unzip.in, zip.out)
})
}
Thanks in advance.
This aprox creating a new SourceShape stacking flow graph can work, a little bit different of your flowShape implementation.
def sourceGraph[A, B](f: A => B, source: Source[(A, StreamMetaData), NotUsed]) = Source.fromGraph(GraphDSL.create() { implicit builder: GraphDSL.Builder[NotUsed] =>
import GraphDSL.Implicits._
val unzip = builder.add(Unzip[A, StreamMetaData]())
val zip = builder.add(Zip[B, StreamMetaData]())
val flow0 = builder.add(Flow[A].map { f(_) })
val flow1 = source ~> unzip.in
unzip.out0 ~> flow0 ~> zip.in0
unzip.out1 ~> zip.in1
SourceShape(zip.out)
})
def flowGraph[A, B](f: A => B) = Flow.fromGraph(GraphDSL.create() { implicit builder =>
import GraphDSL.Implicits._
val unzip = builder.add(Unzip[A, StreamMetaData]())
val zip = builder.add(Zip[B, StreamMetaData]())
val flow0 = builder.add(Flow[A].map { f(_) })
unzip.out0 ~> flow0 ~> zip.in0
unzip.out1 ~> zip.in1
FlowShape(unzip.in, zip.out)
})
I am getting the dreadful java.lang.IllegalArgumentException: requirement failed: The inlets [] and outlets [] must correspond to the inlets [Map.in] and outlets [] error message and previous SO answers are not helping me.
This is my graph
val graph = RunnableGraph.fromGraph(GraphDSL.create(sink){ implicit b =>
s =>
import GraphDSL.Implicits._
val broadcast = b.add(Broadcast[Foo](2))
val merge = b.add(Merge[Foo](2))
source ~> func1() ~> broadcast.in
broadcast.out(0).filter(_.country == "US") ~> usSpecificFilter ~> merge.in(0)
broadcast.out(1).filter(_.country != "US") ~> internationalFilter ~> merge.in(1)
merge.out ~> sink
ClosedShape
})
def func1() : Flow[String, Foo, NotUsed] = {...}
From previous threads I could gather that if you have "unused flows" (meaning something which was added to the builder but was not used in the graph) then this error will come.
But as you can see above that I don't have unused flow.
Edit::
val graph = RunnableGraph.fromGraph(GraphDSL.create(sink){ implicit b =>
s =>
import GraphDSL.Implicits._
val broadcast = b.add(Broadcast[Foo](2))
val merge = b.add(Merge[Foo](2))
val s = b.add(sink)
source ~> func1() ~> broadcast.in
broadcast.out(0).filter(_.country == "US") ~> usSpecificFilter ~> merge.in(0)
broadcast.out(1).filter(_.country != "US") ~> internationalFilter ~> merge.in(1)
merge.out ~> s.in
ClosedShape
})
Still does not work!
In your graph, change merge.out ~> sink to merge.out ~> s.in:
val graph = RunnableGraph.fromGraph(GraphDSL.create(sink) { implicit b => s =>
import GraphDSL.Implicits._
val broadcast = b.add(Broadcast[Foo](2))
val merge = b.add(Merge[Foo](2))
source ~> func1() ~> broadcast.in
broadcast.out(0).filter(_.country == "US") ~> usSpecificFilter ~> merge.in(0)
broadcast.out(1).filter(_.country != "US") ~> internationalFilter ~> merge.in(1)
merge.out ~> s.in // <--- changed sink to s.in
ClosedShape
})
I'm trying to build an Akka Stream with a simple cycle in it. After reading the documentation here and having no luck with it I tried to just copy the example code as a starting base, but that also doesn't work. The code compiles (after including a source which is missing from the example) but nothing is printed out. It looks as though something is backpressuring for ever but I don't understand why.
Here's my code, any help would be much appreciated:
import akka.actor.ActorSystem
import akka.stream.{ActorMaterializer, ActorMaterializerSettings}
import akka.stream.scaladsl._
import akka.stream.ClosedShape
object Simulate {
implicit val system = ActorSystem()
implicit val materializer = ActorMaterializer()
def main(args: Array[String]): Unit = {
// Define simulation flowgraph
val g = RunnableGraph.fromGraph(GraphDSL.create() { implicit b =>
import b._
import GraphDSL.Implicits._
val source = add(Source.repeat[Int](1))
val zip = add(ZipWith[Int, Int, Int]((left, right) => left))
val bcast = add(Broadcast[Int](2))
val concat = add(Concat[Int]())
val start = add(Source.single[Int](0))
val sink = add(Sink.ignore)
source ~> zip.in0
zip.out.map { s => println(s); s } ~> bcast ~> sink
concat <~ bcast
zip.in1 <~ concat <~ start
ClosedShape
})
g.run()
}
}
EDIT: it actually seems that the problem is not adding a buffer, but the order inlets / outlets were declared.
This works:
val g = RunnableGraph.fromGraph(GraphDSL.create() { implicit b =>
import GraphDSL.Implicits._
val source = Source.repeat(1)
val start = Source.single(0)
val zip = b.add(ZipWith((left: Int, right: Int) => left))
val bcast = b.add(Broadcast[Int](2))
val concat = b.add(Concat[Int]())
source ~> zip.in0
zip.out.map { s => println(s); s } ~> bcast ~> Sink.ignore
zip.in1 <~ concat <~ start
concat <~ bcast
ClosedShape
})
g.run()
The order of zip.in1 <~ concat <~ start and concat <~ bcast is consistent with what's on the docs.