I'm accessing data from a REST endpoint :
"https://api-public.sandbox.pro.coinbase.com/products/BTC-EUR/ticker"
To access the data once per second I use an infinite loop while(true) { to invoke a message send to the actor once per second which begins the process of invoking the REST request:
The actor to access the data is:
object ProductTickerRestActor {
case class StringData(data: String)
}
class ProductTickerRestActor extends Actor {
override def receive: PartialFunction[Any, Unit] = {
case ProductTickerRestActor.StringData(data) =>
try {
println("in ProductTickerRestActor")
val rData = scala.io.Source.fromURL("https://api-public.sandbox.pro.coinbase.com/products/BTC-EUR/ticker").mkString
println("rData : "+rData)
}
catch {
case e: Exception =>
println("Exception thrown in ProductTickerRestActor: " + e.getMessage)
}
case msg => println(s"I cannot understand ${msg.toString}")
}
}
I start the application using:
object ExchangeModelDataApplication {
def main(args: Array[String]): Unit = {
val actorSystem = ActorSystemConfig.getActorSystem
val priceDataActor = actorSystem.actorOf(Props[ProductTickerRestActor], "ProductTickerRestActor")
val throttler = Throttlers.getThrottler(priceDataActor)
while(true) {
throttler ! ProductTickerRestActor.StringData("test")
Thread.sleep(1000)
}
}
Throttler:
object Throttlers {
implicit val materializer = ActorMaterializer.create(ActorSystemConfig.getActorSystem)
def getThrottler(priceDataActor: ActorRef) = Source.actorRef(bufferSize = 1000, OverflowStrategy.dropNew)
.throttle(1, 1.second)
.to(Sink.actorRef(priceDataActor, NotUsed))
.run()
}
How to run the following code asynchronously instead of blocking using an infinite loop? :
throttler ! ProductTickerRestActor.StringData("test")
Thread.sleep(1000)
Also, the throttler, in this case, maybe redundant as I'm throttling requests within the loop regardless.
I would just use Akka Streams for this along with Akka HTTP. Using Akka 2.6.x, something along these lines would be sufficient for 1 request/second
import akka.actor.ActorSystem
import akka.http.scaladsl.Http
import akka.http.scaladsl.model._
import akka.stream.scaladsl._
import scala.concurrent.duration._
object HTTPRepeatedly {
implicit val system = ActorSystem()
import system.dispatcher
val sourceFromHttp: Source[String, NotUsed] =
Source.repeated("test") // Not sure what "test" is actually used for here...
.throttle(1, 1.second)
.map { str =>
HttpRequest(uri = "https://api-public.sandbox.pro.coinbase.com/products/BTC-EUR/ticker")
}.mapAsync(1) { req =>
Http().singleRequest(req)
}.mapAsync(1)(_.entity.toStrict(1.minute))
.map(_.data.decodeString(java.nio.charset.StandardCharsets.UTF_8))
}
Then you could, for instance (for simplicity, put this in a main within HTTPRepeatedly so the implicits are in scope etc.)
val done: Future[Done] =
sourceFromHttp
.take(10) // stop after 10 requests
.runWith(Sink.foreach { rData => println(s"rData: $rData") })
scala.concurrent.Await.result(done, 11.minute)
system.terminate()
Sending a request every second is not a good idea. If, for some reason, the request is delayed, you are going to get a lot of requests piled up. Instead, send the next request one second after the previous requests completes.
Because this code uses a synchronous GET request, you can just send the next request one second after mkString returns.
But using synchronous requests is not a good way to consume a RESTful API in Akka. It blocks the actor receive method until the request is complete, which can eventually block the whole ActorSystem.
Instead, use Akka Http and singleRequest to perform an asynchronous request.
Http().singleRequest(HttpRequest(uri = "https://api-public.sandbox.pro.coinbase.com/products/BTC-EUR/ticker"))
This returns a Future. Make the new request one second after the request completes (e.g. using onComplete on the Future).
Not only is this safer and more asynchronous, it also provides much more control over the REST API call than fromUrl
Related
I have an external (that is, I cannot change it) Java API which looks like this:
public interface Sender {
void send(Event e);
}
I need to implement a Sender which accepts each event, transforms it to a JSON object, collects some number of them into a single bundle and sends over HTTP to some endpoint. This all should be done asynchronously, without send() blocking the calling thread, with some fixed-size buffer and dropping new events if the buffer is full.
With akka-streams this is quite simple: I create a graph of stages (which uses akka-http to send HTTP requests), materialize it and use the materialized ActorRef to push new events to the stream:
lazy val eventPipeline = Source.actorRef[Event](Int.MaxValue, OverflowStrategy.fail)
.via(CustomBuffer(bufferSize)) // buffer all events
.groupedWithin(batchSize, flushDuration) // group events into chunks
.map(toBundle) // convert each chunk into a JSON message
.mapAsyncUnordered(1)(sendHttpRequest) // send an HTTP request
.toMat(Sink.foreach { response =>
// print HTTP response for debugging
})(Keep.both)
lazy val (eventsActor, completeFuture) = eventPipeline.run()
override def send(e: Event): Unit = {
eventsActor ! e
}
Here CustomBuffer is a custom GraphStage which is very similar to the library-provided Buffer but tailored to our specific needs; it probably does not matter for this particular question.
As you can see, interacting with the stream from non-stream code is very simple - the ! method on the ActorRef trait is asynchronous and does not need any additional machinery to be called. Each event which is sent to the actor is then processed through the entire reactive pipeline. Moreover, because of how akka-http is implemented, I even get connection pooling for free, so no more than one connection is opened to the server.
However, I cannot find a way to do the same thing with FS2 properly. Even discarding the question of buffering (I will probably need to write a custom Pipe implementation which does additional things that we need) and HTTP connection pooling, I'm still stuck with a more basic thing - that is, how to push the data to the reactive stream "from outside".
All tutorials and documentation that I can find assume that the entire program happens inside some effect context, usually IO. This is not my case - the send() method is invoked by the Java library at unspecified times. Therefore, I just cannot keep everything inside one IO action, I necessarily have to finalize the "push" action inside the send() method, and have the reactive stream as a separate entity, because I want to aggregate events and hopefully pool HTTP connections (which I believe is naturally tied to the reactive stream).
I assume that I need some additional data structure, like Queue. fs2 does indeed have some kind of fs2.concurrent.Queue, but again, all documentation shows how to use it inside a single IO context, so I assume that doing something like
val queue: Queue[IO, Event] = Queue.unbounded[IO, Event].unsafeRunSync()
and then using queue inside the stream definition and then separately inside the send() method with further unsafeRun calls:
val eventPipeline = queue.dequeue
.through(customBuffer(bufferSize))
.groupWithin(batchSize, flushDuration)
.map(toBundle)
.mapAsyncUnordered(1)(sendRequest)
.evalTap(response => ...)
.compile
.drain
eventPipeline.unsafeRunAsync(...) // or something
override def send(e: Event) {
queue.enqueue(e).unsafeRunSync()
}
is not the correct way and most likely would not even work.
So, my question is, how do I properly use fs2 to solve my problem?
Consider the following example:
import cats.implicits._
import cats.effect._
import cats.effect.implicits._
import fs2._
import fs2.concurrent.Queue
import scala.concurrent.ExecutionContext
import scala.concurrent.duration._
object Answer {
type Event = String
trait Sender {
def send(event: Event): Unit
}
def main(args: Array[String]): Unit = {
val sender: Sender = {
val ec = ExecutionContext.global
implicit val cs: ContextShift[IO] = IO.contextShift(ec)
implicit val timer: Timer[IO] = IO.timer(ec)
fs2Sender[IO](2)
}
val events = List("a", "b", "c", "d")
events.foreach { evt => new Thread(() => sender.send(evt)).start() }
Thread sleep 3000
}
def fs2Sender[F[_]: Timer : ContextShift](maxBufferedSize: Int)(implicit F: ConcurrentEffect[F]): Sender = {
// dummy impl
// this is where the actual logic for batching
// and shipping over the network would live
val consume: Pipe[F, Event, Unit] = _.evalMap { event =>
for {
_ <- F.delay { println(s"consuming [$event]...") }
_ <- Timer[F].sleep(1.seconds)
_ <- F.delay { println(s"...[$event] consumed") }
} yield ()
}
val suspended = for {
q <- Queue.bounded[F, Event](maxBufferedSize)
_ <- q.dequeue.through(consume).compile.drain.start
sender <- F.delay[Sender] { evt =>
val enqueue = for {
wasEnqueued <- q.offer1(evt)
_ <- F.delay { println(s"[$evt] enqueued? $wasEnqueued") }
} yield ()
enqueue.toIO.unsafeRunAsyncAndForget()
}
} yield sender
suspended.toIO.unsafeRunSync()
}
}
The main idea is to use a concurrent Queue from fs2. Note, that the above code demonstrates that neither the Sender interface nor the logic in main can be changed. Only an implementation of the Sender interface can be swapped out.
I don't have much experience with exactly that library but it should look somehow like that:
import cats.effect.{ExitCode, IO, IOApp}
import fs2.concurrent.Queue
case class Event(id: Int)
class JavaProducer{
new Thread(new Runnable {
override def run(): Unit = {
var id = 0
while(true){
Thread.sleep(1000)
id += 1
send(Event(id))
}
}
}).start()
def send(event: Event): Unit ={
println(s"Original producer prints $event")
}
}
class HackedProducer(queue: Queue[IO, Event]) extends JavaProducer {
override def send(event: Event): Unit = {
println(s"Hacked producer pushes $event")
queue.enqueue1(event).unsafeRunSync()
println(s"Hacked producer pushes $event - Pushed")
}
}
object Test extends IOApp{
override def run(args: List[String]): IO[ExitCode] = {
val x: IO[Unit] = for {
queue <- Queue.unbounded[IO, Event]
_ = new HackedProducer(queue)
done <- queue.dequeue.map(ev => {
println(s"Got $ev")
}).compile.drain
} yield done
x.map(_ => ExitCode.Success)
}
}
We can create a bounded queue that will consume elements from sender and make them available to fs2 stream processing.
import cats.effect.IO
import cats.effect.std.Queue
import fs2.Stream
trait Sender[T]:
def send(e: T): Unit
object Sender:
def apply[T](bufferSize: Int): IO[(Sender[T], Stream[IO, T])] =
for
q <- Queue.bounded[IO, T](bufferSize)
yield
val sender: Sender[T] = (e: T) => q.offer(e).unsafeRunSync()
def stm: Stream[IO, T] = Stream.eval(q.take) ++ stm
(sender, stm)
Then we'll have two ends - one for Java worlds, to send new elements to Sender. Another one - for stream processing in fs2.
class TestSenderQueue:
#Test def testSenderQueue: Unit =
val (sender, stream) = Sender[Int](1)
.unsafeRunSync()// we have to run it preliminary to make `sender` available to external system
val processing =
stream
.map(i => i * i)
.evalMap{ ii => IO{ println(ii)}}
sender.send(1)
processing.compile.toList.start//NB! we start processing in a separate fiber
.unsafeRunSync() // immediately right now.
sender.send(2)
Thread.sleep(100)
(0 until 100).foreach(sender.send)
println("finished")
Note that we push data in the current thread and have to run fs2 in a separate thread (.start).
I've built an Akka actor that queries an API at regular intervals, like this:
val cancellable =
system.scheduler.schedule(0 milliseconds,
5 seconds,
actor,
QueryController(1))
The Actor, in essence is:
object UpdateStatistics {
/**
* Query the controller for the given switch Id
*
* #param dpId Switch's Id
*/
case class QueryController(dpId: Int)
case object Stop
def props: Props = Props[UpdateStatistics]
}
class UpdateStatistics extends Actor with akka.actor.ActorLogging {
import UpdateStatistics._
def receive = {
case QueryController(id) =>
import context.dispatcher
log.info(s"Receiving request to query controller")
Future { FlowCollector.getSwitchFlows(1) } onComplete {
f => self ! f.get
}
case Stop =>
log.info(s"Shuting down")
context stop self
case json: JValue =>
log.info("Getting json response, computing features...")
val features = FeatureExtractor.getFeatures(json)
log.debug(s"Features: $features")
sender ! features
case x =>
log.warning("Received unknown message: {}", x)
}
}
What I am trying to do is get the json:Jvalue message out of UpdateStatistics actor. Reading the Akka docs I thought this may be useful:
implicit val i = inbox()
i.select() {
case x => println(s"Valor Devuelto $x")
}
println(i receive(2.second))
But I do not know how to modify UpdateStatistics actor in order to send the result to the inbox above.
Another option I've read in the docs are event streams.
But I do not think this is the correct way.
Is there a way of achieving what I want to do? Or do I need to use a second Actor to which I send the JSON response?
You probably are looking for the ask pattern in AKKA. This will allow you to return a value to the sender.
import akka.pattern.ask
import akka.util.duration._
implicit val timeout = Timeout(5 seconds)
val future = actor ? QueryController(1)
val result = Await.result(future, timeout.duration).asInstanceOf[JValue]
println(result)
To make this work, you need to send the response to the original sender, rather than self. Also, you should beware the dangers of closing over sender in a future when handling messages.
I am trying to figure out how I can setup a Master Actor that calls the appropriate children, in support of some spray routes where I am trying to emulate db calls. I am new to akka / spray, so just trying to gain a better understanding of how you would properly setup spray -> actors -> db calls (etc.). I can get the response back from the top level actor, but when I try to get it back from one actor level below the parent I can't seem to get anything to work.
When looking at the paths of the actors, it appears that from the way I am making the call from my spray route that I am passing from a temp actor. Below is what I have so far for stubbing this out. This has to be just user error / ignorance, just not sure how to proceed. Any suggestions would be appreciated.
The Demo Spray Service and Redis Actor code snippets below show where I am calling the actor from my route and the multiple actors where I am having the issue (want my route to get response from SummaryActor). Thanks!
Boot:
object Boot extends App {
// we need an ActorSystem to host our application in
implicit val system = ActorSystem("on-spray-can")
// create and start our service actor
val service = system.actorOf(Props[DemoServiceActor], "demo-service")
implicit val timeout = Timeout(5.seconds)
// start a new HTTP server on port 8080 with our service actor as the handler
IO(Http) ? Http.Bind(service, interface = "localhost", port = 8080)
}
Demo Service Actor (For Spray)
class DemoServiceActor extends Actor with Api {
// the HttpService trait defines only one abstract member, which
// connects the services environment to the enclosing actor or test
def actorRefFactory = context
// this actor only runs our route, but you could add
// other things here, like request stream processing
// or timeout handling
def receive = handleTimeouts orElse runRoute(route)
//Used to watch for request timeouts
//http://spray.io/documentation/1.1.2/spray-routing/key-concepts/timeout-handling/
def handleTimeouts: Receive = {
case Timedout(x: HttpRequest) =>
sender ! HttpResponse(StatusCodes.InternalServerError, "Too late")
}
}
//Master trait for handling large APIs
//http://stackoverflow.com/questions/14653526/can-spray-io-routes-be-split-into-multiple-controllers
trait Api extends DemoService {
val route = {
messageApiRouting
}
}
Demo Spray Service (Route):
trait DemoService extends HttpService with Actor {
implicit val timeout = Timeout(5 seconds) // needed for `?` below
val redisActor = context.actorOf(Props[RedisActor], "redisactor")
val messageApiRouting =
path("summary" / Segment / Segment) { (dataset, timeslice) =>
onComplete(getSummary(redisActor, dataset, timeslice)) {
case Success(value) => complete(s"The result was $value")
case Failure(ex) => complete(s"An error occurred: ${ex.getMessage}")
}
}
def getSummary(redisActor: ActorRef, dataset: String, timeslice: String): Future[String] = Future {
val dbMessage = DbMessage("summary", dataset + timeslice)
val future = redisActor ? dbMessage
val result = Await.result(future, timeout.duration).asInstanceOf[String]
result
}
}
Redis Actor (Mock no actual redis client yet)
class RedisActor extends Actor with ActorLogging {
// val pool = REDIS
implicit val timeout = Timeout(5 seconds) // needed for `?` below
val summaryActor = context.actorOf(Props[SummaryActor], "summaryactor")
def receive = {
case msg: DbMessage => {
msg.query match {
case "summary" => {
log.debug("Summary Query Request")
log.debug(sender.path.toString)
summaryActor ! msg
}
}
}
//If not match log an error
case _ => log.error("Received unknown message: {} ")
}
}
class SummaryActor extends Actor with ActorLogging{
def receive = {
case msg: DbMessage =>{
log.debug("Summary Actor Received Message")
//Send back to Spray Route
}
}
}
The first problem with your code is that you need to forward from the master actor to the child so that the sender is properly propagated and available for the child to respond to. So change this (in RedisActor):
summaryActor ! msg
To:
summaryActor forward msg
That's the primary issue. Fix that and your code should start working. There is something else that needs attention though. Your getSummary method is currently defined as:
def getSummary(redisActor: ActorRef, dataset: String, timeslice: String): Future[String] =
Future {
val dbMessage = DbMessage("summary", dataset + timeslice)
val future = redisActor ? dbMessage
val result = Await.result(future, timeout.duration).asInstanceOf[String]
result
}
The issue here is that the ask operation (?) already returns a Future, so there and you are blocking on it to get the result, wrapping that in another Future so that you can return a Future for onComplete to work with. You should be able to simplify things by using the Future returned from ask directly like so:
def getSummary(redisActor: ActorRef, dataset: String, timeslice: String): Future[String] = {
val dbMessage = DbMessage("summary", dataset + timeslice)
(redisActor ? dbMessage).mapTo[String]
}
Just an important comment on the above approaches.
Since the getSummary(...) function returns a Future[String] object and you call it in onComplete(...) function you need to import:
import ExecutionContext.Implicits.global
That way you will have ExecutionContext in scope by letting Future
declare an implicit ExecutionContext parameter.
** If you don't, you will end up getting a mismatching error
since onComplete(...) expects an onComplete Future
magnet Object but you gave a Future[String] Object.
I want to make a request to a server asynchronously involving actors. Say I have 2 actors:
class SessionRetriever extends Actor {
import SessionRetriever._
def receiver = {
Get =>
val s = getSessionIdFromServer() // 1
sender ! Result(s) // 2
}
def getSessionIdFromServer(): String = { ... } // 3
}
object SessionRetriever {
object Get
object Result(s: String)
}
And
class RequestSender extends Actor {
val sActor = context actorOf Props[SessionRetriever]
def receiver = {
// get session id
val sesId = sActor ! SessionRetriever.Get
val res = sendRequestToServer(sesId)
logToFile(res)
context shutdown sActor
}
def sendRequestToServer(sessionId: String): String = { .... }
}
My questions:
val s = getSessionIdFromServer() // 1
sender ! Result(s) // 2
1) getSessionIdFromServer() does a synchronous request to the server. I think it would be much better make a request asynchronous, correct? So it will return Future[String] instead of a plain String.
2) How do I make asynchronous: by using an AsyncHttpClient (if I recall correctly its name) or by wrapping its synchronous body into Future { } ?
3) Should I use blocking { } block ? If yes, then where exactly: inside its body or here val s = blocking { getSessionIdFromServer() } ?
P.S. I'd like not to use async { } and await { } at this point because they are quite high level functions and after all they are build on top on Futures.
you might try this non-blocking way
def receive = {
Get =>
//assume getSessionIdFromServer() run aysnchronize
val f: Future[String] = getSessionIdFromServer()
val client = sender //keep it local to use when future back
f onComplete {
case Success(rep) => client ! Result(rep)
case Failure(ex) => client ! Failed(ex)
}
}
1) If getSessionIdFromServer() is blocking then you should execute it asynchronously from your receive function, otherwise your actor will block each time it receives a new request and will always wait until it receives a new session before processing the next request.
2) Using a Future will "move" the blocking operation to a different thread. So, your actor will not be blocked and will be able to keep processing incoming requests - that's good -, however you are still blocking a thread - not so great. Using the AsyncHttpClient is a good idea. You can explore other non-blocking httpClient, like PlayWebService.
3) I am not quite familiar with blocking so not sure I should advise anything here. From what I understand, it will tell the thread pool that the operation is blocking and that it should spawn a temporary new thread to handle it - this avoids having all your workers being blocked. Again, if you do that your actor will not blocked, but you are still blocking a thread while getting the session from the server.
To summarize: just use an async http client in getSessionIdFromServer if it is possible. Otherwise, use either Future{} or blocking.
To do an asynchronous call with AsyncHttpClient you could deal with the java Future via a scala Promise.
import scala.concurrent.Future
import com.ning.http.client.AsyncHttpClient
import scala.concurrent.Promise
import java.util.concurrent.Executor
object WebClient {
private val client = new AsyncHttpClient
case class BadStatus(status: Int) extends RuntimeException
def get(url: String)(implicit exec: Executor): Future[String] = {
val f = client.prepareGet(url).execute();
val p = Promise[String]()
f.addListener(new Runnable {
def run = {
val response = f.get
if (response.getStatusCode / 100 < 4)
p.success(response.getResponseBodyExcerpt(131072))
else p.failure(BadStatus(response.getStatusCode))
}
}, exec)
p.future
}
def shutdown(): Unit = client.close()
}
object WebClientTest extends App {
import scala.concurrent.ExecutionContext.Implicits.global
WebClient get "http://www.google.com/" map println foreach (_ => WebClient.shutdown())
}
And then deal with the future completion via a callback.
Credit for the code to the awesome reactive programming course at Coursera.
I'm trying to grok Futures and ask pattern in akka.
So, I make two actors, and one asking another to send him back a message. Well, according to akka's Futures documentation, actor should ask(?) for message and it shall give him a Future instanse. Then actor should block (using Await) to get Future results.
Well, I never get my future done. Why is that?
Code is:
package head_thrash
import akka.actor._
import akka.util.Timeout
import scala.concurrent.Await
import scala.concurrent.duration._
object Main extends App {
val system = ActorSystem("actors")
val actor1 = system.actorOf(Props[MyActor], "node_1")
val actor2 = system.actorOf(Props[MyActor], "node_2")
actor2 ! "ping_other"
system.awaitTermination()
Console.println("Bye!")
}
class MyActor extends Actor with ActorLogging {
import akka.pattern.ask
implicit val timeout = Timeout(100.days)
def receive = {
case "ping_other" => {
val selection = context.actorSelection("../node_1")
log.info("Sending ping to node_1")
val result = Await.result(selection ? "ping", Duration.Inf) // <-- Blocks here forever!
log.info("Got result " + result)
}
case "ping" => {
log.info("Sending back pong!")
sender ! "pong"
}
}
}
If I change Duration.Inf to 5.seconds, then actor waits 5 seconds, tells that my future is Timeouted (by throwing TimeoutException), and then other actor finally replies with needed message. So, no async happens. Why? :-(
How should I properly implement that pattern? Thanks.
The official Akka documentation says that Await.result will cause the current thread to block and wait for the Actor to 'complete' the Future with it's reply.
It is strange that your code blocks there forever, do you have only one thread for all your application?
Anyway I guess a more "idiomatic" way to code it would be to use a callback on the future success.
def receive = {
case "ping_other" => {
val selection = context.actorSelection("../node_1")
log.info("Sending ping to node_1")
val future: Future[String] = ask(selection, "ping").mapTo[String]
future.onSuccess {
case result : String ⇒ log.info("Got result " + result)
}
}
...
Two reasons why that doesn't work.
First, "node_1" asks itself and the "ping" will not be processed because it is blocking in waiting for the ask.
Also, there is a shortcoming of actorSelection for relative paths ("../node_1"). It is processed with message passing, and since your actor is blocking it cannot process any other message. This has been improved in upcoming 2.3 version of Akka, but you should avoid blocking anyway.