I am not able to figure out how to stop akka stream Runnable Graph immediately ? How to use killswitch to achieve this? It has been just a few days that I started akka streams. In my case I am reading lines from a file and doing some operations in flow and writing to the sink. What I want to do is, stop reading file immediately whenever I want, and I hope this should possibly stop the whole running graph. Any ideas on this would be greatly appreciated.
Thanks in advance.
Since Akka Streams 2.4.3, there is an elegant way to stop the stream from the outside via KillSwitch.
Consider the following example, which stops stream after 10 seconds.
object ExampleStopStream extends App {
implicit val system = ActorSystem("streams")
implicit val materializer = ActorMaterializer()
import system.dispatcher
val source = Source.
fromIterator(() => Iterator.continually(Random.nextInt(100))).
delay(500.millis, DelayOverflowStrategy.dropHead)
val square = Flow[Int].map(x => x * x)
val sink = Sink.foreach(println)
val (killSwitch, done) =
source.via(square).
viaMat(KillSwitches.single)(Keep.right).
toMat(sink)(Keep.both).run()
system.scheduler.scheduleOnce(10.seconds) {
println("Shutting down...")
killSwitch.shutdown()
}
done.foreach { _ =>
println("I'm done")
Await.result(system.terminate(), 1.seconds)
}
}
The one way have a service or shutdownhookup which can call graph cancellable
val graph=
Source.tick(FiniteDuration(0,TimeUnit.SECONDS), FiniteDuration(1,TimeUnit.SECONDS), Random.nextInt).to(Sink.foreach(println))
val cancellable=graph.run()
cancellable.cancel
The cancellable.cancel can be part of ActorSystem.registerOnTermination
Related
I want to use a function similar to take(n: Int) but in a time dimension:
consume(period: Duration. So I want a stream to terminate if a timeout occurs. I know that I could compile a stream to something like IO[List[T]] and cancel it, but then I'll lose the result. In reality I want to convert an endless stream into a limited one and preserve the results.
More on the wider scope of the problem. I have an endless stream of events from a messaging broker, but I also have rotating credentials to connect to the broker. So what I want is to consume the stream of events for some time, then stop, acquire new credentials, connect again to the broker creating a new stream and concatenate two streams into one.
There is a method, that does exactly this:
/**
* Interrupts this stream after the specified duration has passed.
*/
def interruptAfter[F2[x] >: F[x]: Concurrent: Timer](duration: FiniteDuration): Stream[F2, O]
You need something like that
import scala.util.Random
import scala.concurrent.ExecutionContext
import fs2._
import fs2.concurrent.SignallingRef
implicit val ex = ExecutionContext.global
implicit val t: Timer[IO] = IO.timer(ex)
implicit val cs: ContextShift[IO] = IO.contextShift(ex)
val effect: IO[Long] = IO.sleep(1.second).flatMap(_ => IO{
val next = Random.nextLong()
println("NEXT: " + next)
next
})
val signal = SignallingRef[IO, Boolean](false).unsafeRunSync()
val timer = Stream.sleep(10.seconds).flatMap(_ =>
Stream.eval(signal.set(true)).flatMap(_ =>
Stream.emit(println("Finish")).covary[IO]))
val stream = timer concurrently
Stream.repeatEval(effect).interruptWhen(signal)
stream.compile.drain.unsafeRunSync()
Also if you want to save your result of publishing data you need to have some unbounded Queue from fs2 for converting published data to your result via queue.stream
I have Akka Stream flow which is reading from file using alpakka, processing data and write into a file. I want to stop flow after processed n elements, count the time of duration and call system terminate. How can I achieve it?
My flow looks like that:
val graph = RunnableGraph.fromGraph(GraphDSL.create() { implicit builder: GraphDSL.Builder[NotUsed] =>
import GraphDSL.Implicits._
sourceFile ~> mainFlow ~> sinkFile
ClosedShape
})
graph.run()
Do you have an idea? Thanks
Agreeing with what #Viktor said, first of all you don't need to use the graphDSL to achieve this, and you can use take(n) to complete the graph.
Secondly, you can use mapMaterializedValue to attach a callback to your Sink materialized value (which in turn should materializes to a Future[Something]).
val graph: RunnableGraph[Future[FiniteDuration]] =
sourceFile
.via(mainFlow)
.take(N)
.toMat(sinkFile)(Keep.right)
.mapMaterializedValue { f ⇒
val start = System.nanoTime()
f.map(_ ⇒ FiniteDuration(System.nanoTime() - start, TimeUnit.NANOSECONDS))
}
graph.run().onComplete { duration ⇒
println(s"Elapsed time: $duration")
}
Note that you are going to need an ExecutionContext in scope.
EDIT
Even if you have to use the graphDSL, the same concepts apply. You only need to expose the materialized Future of your sink and map on that.
val graph: RunnableGraph[Future[??Something??]] =
RunnableGraph.fromGraph(GraphDSL.create(sinkFile) { implicit builder: GraphDSL.Builder[Future[Something]] => snk =>
import GraphDSL.Implicits._
sourceFile ~> mainFlow ~> snk
ClosedShape
})
val timedGraph: RunnableGraph[Future[FiniteDuration]] =
graph.mapMaterializedValue { f ⇒
val start = System.nanoTime()
f.map(_ ⇒ FiniteDuration(System.nanoTime() - start, TimeUnit.NANOSECONDS))
}
timedGraph.run().onComplete { duration ⇒
println(s"Elapsed time: $duration")
}
No need for GraphDSL here.
val doneFuture = (sourceFile via mainFlow.take(N) runWith sinkFile) transformWith { _ => system.terminate() }
To obtain time, you can use akka-streams-contrib: https://github.com/akka/akka-stream-contrib/blob/master/contrib/src/main/scala/akka/stream/contrib/Timed.scala
What can be alternative to dynamically changing running graph ? Here is my situation. I have graph that ingests articles into DB. Articles come from 3 plugins in different format. Thus I have several flows
val converterFlow1: Flow[ImpArticle, Article, NotUsed]
val converterFlow2: Flow[NewsArticle, Article, NotUsed]
val sinkDB: Sink[Article, Future[Done]]
// These are being created every time I poll plugins
val sourceContentProvider : Source[ImpArticle, NotUsed]
val sourceNews : Source[NewsArticle, NotUsed]
val sourceCit : Source[Article, NotUsed]
val merged = Source.combine(
sourceContentProvider.via(converterFlow1),
sourceNews.via(converterFlow2),
sourceCit)(Merge(_))
val res = merged
.buffer(10, OverflowStrategy.backpressure)
.toMat(sinkDB)(Keep.both)
.run()
Problem is that I get data from content provider once per 24 hrs, from news once per 2 hrs and last source may come at any time because it's coming from humans.
I realize that graphs are immutable but how I can periodically attach new instances of Source to my graph so that I have single point of throttling of the process of ingesting ?
UPDATE: You can say my data is stream of Source-s, three sources in my case. But I cannot change that because I get instances of Source from external classes (so called plugins). These plugins work independently from my ingestion class. I can't combine them into one gigantic class to have single Source.
Okay, in general the correct way would be to join a stream of sources into a single source, i.e. go from Source[Source[T, _], Whatever] to Source[T, Whatever]. This can be done with flatMapConcat or with flatMapMerge. Therefore, if you can get a Source[Source[Article, NotUsed], NotUsed], you can use one of flatMap* variants and obtain a final Source[Article, NotUsed]. Do it for each of your sources (no pun intended), and then your original approach should work.
I've implemented code based up on answer given by Vladimir Matveev and want to share it with others since it looks like common use-case to me.
I knew about Source.queue which Viktor Klang mentioned but I wasn't aware of flatMapConcat. It's pure awesomeness.
implicit val system = ActorSystem("root")
implicit val executor = system.dispatcher
implicit val materializer = ActorMaterializer()
case class ImpArticle(text: String)
case class NewsArticle(text: String)
case class Article(text: String)
val converterFlow1: Flow[ImpArticle, Article, NotUsed] = Flow[ImpArticle].map(a => Article("a:" + a.text))
val converterFlow2: Flow[NewsArticle, Article, NotUsed] = Flow[NewsArticle].map(a => Article("a:" + a.text))
val sinkDB: Sink[Article, Future[Done]] = Sink.foreach { a =>
Thread.sleep(1000)
println(a)
}
// These are being created every time I poll plugins
val sourceContentProvider: Source[ImpArticle, NotUsed] = Source(List(ImpArticle("cp1"), ImpArticle("cp2")))
val sourceNews: Source[NewsArticle, NotUsed] = Source(List(NewsArticle("news1"), NewsArticle("news2")))
val sourceCit: Source[Article, NotUsed] = Source(List(Article("a1"), Article("a2")))
val (queue, completionFut) = Source
.queue[Source[Article, NotUsed]](10, backpressure)
.flatMapConcat(identity)
.buffer(2, OverflowStrategy.backpressure)
.toMat(sinkDB)(Keep.both)
.run()
queue.offer(sourceContentProvider.via(converterFlow1))
queue.offer(sourceNews.via(converterFlow2))
queue.offer(sourceCit)
queue.complete()
completionFut.onComplete {
case Success(res) =>
println(res)
system.terminate()
case Failure(ex) =>
ex.printStackTrace()
system.terminate()
}
Await.result(system.whenTerminated, Duration.Inf)
I'd still check success of Future returned by queue.offer but in my case these calls will be pretty infrequent.
If you cannot model it as a Source[Source[_,_],_] then I'd consider using a Source.queue[Source[T,_]](queueSize, overflowStrategy): here
What you'll have to be careful about though is what happens if submission fails.
I created a small example graph in which the input elements are passed through to the output and also sent back to a feedback loop that drops everything (using filter).
I expect to get the same behaviour as an identity Flow[T], since the feedback branch drops everything.
What happens instead is that the input elements are emitted as expected but the materialization never completes.
Am I doing something wrong? Is this supposed to happen? Shouldn't the feedback output of the broadcast complete when the input stream completes?
I guess the problem is something similar to the chicken-and-egg scenario described here?
I am using akka-stream-experimental 2.0.3
Cheers
object Test extends App {
implicit val system = ActorSystem()
implicit val mat = ActorMaterializer()
val flow = Flow.fromGraph(GraphDSL.create() { implicit b =>
import GraphDSL.Implicits._
val dropEverything = b.add(Flow[Int].filter(_ => false))
val input = b.add(Merge[Int](2))
val bcast = b.add(Broadcast[Int](2))
input ~> bcast
input.in(1) <~ dropEverything <~ bcast.out(1)
FlowShape(input.in(0), bcast.out(0))
})
val result = Source.single(42).via(flow).runWith(Sink.foreach(println))
try {
// prints 42 but the stream doesn't terminate and the await timeouts
println(Await.result(result, 5.seconds))
} finally {
system.terminate()
}
}
This has been answered here. The cycle never completes because Merge and Broadcast are waiting for each other to complete.
You can change this to val input = b.add(Merge[Int](2, eagerComplete = true)) to prevent this.
Alternatively, you can try val dropEverything = b.add(Flow[Int].take(1).filter(_ => false)) where n is the number of elements from input to process, which is 1 in this case.
I'm trying to build event sourced service with REST interface using scala. I somewhat new to scala, although I'm familiar with functional programming (haskell at beginner level).
So I've build persistent actor and view without major problems. The idea of actors is quite simple I think.
object Main extends App {
val system = ActorSystem("HelloSystem")
val systemActor = system.actorOf(Props[SystemActor], name = "systemactor")
val trajectoryView = system.actorOf(Props[TrajectoryView], name = "trajectoryView")
var datas = List()
val processData = ProcessData(0, List(1,2,3), Coordinates(50, 50))
implicit val timeout = Timeout(5 seconds)
def intialDatas(): List[ProcessData] =
(for (i <- 1 to 3) yield ProcessData(i, List(1,2,3), Coordinates(50 + i, 50 + i)))(collection.breakOut)
val command = RegisterProcessCommand(3, this.intialDatas())
val id = Await.result(systemActor ? command, timeout.duration).asInstanceOf[String]
println(id)
systemActor ! MoveProcessCommand(4, ProcessData(4, List(3,4,5), Coordinates(54, 54)), id)
val processes = Await.result(systemActor ? "get", timeout.duration).asInstanceOf[Set[Process]]
println(processes)
implicit val json4sFormats = DefaultFormats
println(write(processes))
println("*****************")
systemActor ! "print"
val getTrajectoryCommand = GetTrajectoryCommand(id)
Thread.sleep(10000)
trajectoryView ! "print"
// val trajectory = Await.result(trajectoryView ? getTrajectoryCommand, timeout.duration).asInstanceOf[ListBuffer[Coordinates]]
println("******* TRAJECTORY *********")
trajectoryView ! "print"
// println(trajectory)
system.shutdown()
}
I've been able to create a script for playing with actor that I've created.
I've read the tutorials for spray routing, but I've been unable to grasp what exactly should I do to provide REST interface for actors that I've created.
object Boot extends App{
implicit val system = ActorSystem("example")
val systemActor = system.actorOf(Props[SystemActor], name = "systemactor")
val trajectoryView = system.actorOf(Props[TrajectoryView], name = "trajectoryView")
val service = system.actorOf(Props[ProcessesService], "processes-rest-service")
implicit val timeout = Timeout(5 seconds)
IO(Http) ? Http.Bind(service, interface = "localhost", port = 8080)
}
And a service
class ProcessesService(systemActor: ActorRef) extends Actor with HttpService {
def actorRefFactory = context
def receive = runRoute(route)
val json4sFormats = DefaultFormats
implicit val timeout = Timeout(5 seconds)
val route = path("processes") {
get {
respondWithMediaType(`application/json`) {
complete {
write(Await.result(systemActor ? "get", timeout.duration).asInstanceOf[Set[Process]])
}
}
}
}
}
I think I need to somehow pass actorRef for SystemActor to this ProcessesService, but I'm not sure how. Also I'm not sure how should I return a response to the request. I understand that I need to somehow pass the "get" message to SystemActor through ActorRef and then serialize the answer to json, but I don't know how to do that.
I would appreciate help!
In spray you can complete routes with a Future.
You should be able to do something like
complete { systemActor ? "get" }
Json serialization is a separate issue.
Oh, your question is vague. Yes you need to be able to reference an actor within your routes. You could just import the val from boot where you define it. They're just Scala variables so where you put them is up to you.