I'm trying to wrap Kafka's callback-y producer send API as a Future. i.e.:
def sendFuture[K, V](p: KafkaProducer[K, V], rec: ProducerRecord[K, V]): Future[RecordMetadata] = {
val promise = Promise[RecordMetadata]()
p.send(rec, new Callback {
override def onCompletion(metadata: RecordMetadata, exception: Exception): Unit = {
if (metadata != null) {
promise.complete(Success(metadata))
} else {
promise.complete(Failure(exception))
}
}
})
promise.future
}
I deliberately disable the Kafka broker, so that a producer would for a configured time(5000 ms) keep connecting.
In my test, I only Await for 2 seconds and expect to see a TimeoutException thrown.
"a producer can't connect to the broker, we " should {
"see timeout when sending messages" in withProducer { p: KafkaProducer[String, String] =>
val rec = new ProducerRecord[String, String](TEST_TOPIC, "testKey", "testValue")
val f = sendFuture(p, rec)
log.info(">> sent and we wait for at most 1 second...")
intercept[TimeoutException] {
Await.ready(f, 2 seconds)
}
log.info(">> TimeoutException intercepted")
}
}
I would expect Await waits "on parallel" and then fires TimeoutException.
Unfortunately, my test would fail.
It seems that Await, despite the 2-second bound, always waits until the Future is completed with a failure by the send's callback.
I don't quite see the reason.
I think this has more to do with implementation details of Kafka because
I only observe this behaviour when the Future is completed by Kafka send's callback.
That is, replacing sendFuture by dummyFuture passes the test:
def dummyFuture(): Future[Boolean] = {
val promise = Promise[Boolean]()
val thread = new Thread(new Runnable {
override def run(): Unit = {
Thread.sleep(5000)
promise.complete(Success(true))
}
})
thread.start()
promise.future
}
Thanks for any help.
Gist
Kafka Producer API
I found I was wrong to assume that send returns immediately while it's still trying to connect. Await isn't timeout because it starts after send has reached configured timeout and completed the future with a failure.
Related
How can we consume SSE in scala play framework? Most of the resources that I could find were to make an SSE source. I want to reliably listen to SSE events from other services (with autoconnect). The most relevant article was https://doc.akka.io/docs/alpakka/current/sse.html . I implemented this but this does not seem to work (code below). Also the event that I am su
#Singleton
class SseConsumer #Inject()((implicit ec: ExecutionContext) {
implicit val system = ActorSystem()
val send: HttpRequest => Future[HttpResponse] = foo
def foo(x:HttpRequest) = {
try {
println("foo")
val authHeader = Authorization(BasicHttpCredentials("user", "pass"))
val newHeaders = x.withHeaders(authHeader)
Http().singleRequest(newHeaders)
}catch {
case e:Exception => {
println("Exception", e.printStackTrace())
throw e
}
}
}
val eventSource: Source[ServerSentEvent, NotUsed] =
EventSource(
uri = Uri("https://abc/v1/events"),
send,
initialLastEventId = Some("2"),
retryDelay = 1.second
)
def orderStatusEventStable() = {
val events: Future[immutable.Seq[ServerSentEvent]] =
eventSource
.throttle(elements = 1, per = 500.milliseconds, maximumBurst = 1, ThrottleMode.Shaping)
.take(10)
.runWith(Sink.seq)
events.map(_.foreach( x => {
println("456")
println(x.data)
}))
}
Future {
blocking{
while(true){
try{
Thread.sleep(2000)
orderStatusEventStable()
} catch {
case e:Exception => {
println("Exception", e.printStackTrace())
}
}
}
}
}
}
This does not give any exceptions and println("456") is never printed.
EDIT:
Future {
blocking {
while(true){
try{
Await.result(orderStatusEventStable() recover {
case e: Exception => {
println("exception", e)
throw e
}
}, Duration.Inf)
} catch {
case e:Exception => {
println("Exception", e.printStackTrace())
}
}
}
}
}
Added an await and it started working. Able to read 10 messages at a time. But now I am faced with another problem.
I have a producer which can at times produce faster than I can consume and with this code I have 2 issues:
I have to wait until 10 messages are available. How can we take a max. of 10 and a min. of 0 messages?
When the production rate > consumption rate, I am missing few events. I am guessing this is due to throttling. How do we handle it using backpressure?
The issue in your code is that the events: Future would only complete when the stream (eventSource) completes.
I'm not familiar with SSE but the stream likely never completes in your case as it's always listening for new events.
You can learn more in Akka Stream documentation.
Depending on what you want to do with the events, you could just map on the stream like:
eventSource
...
.map(/* do something */)
.runWith(...)
Basically, you need to work with the Akka Stream Source as data is going through it but don't wait for its completion.
EDIT: I didn't notice the take(10), my answer applies only if the take was not here. Your code should work after 10 events sent.
I am playing a bit with the NATS streaming and I have a problem with the subscriber rate limiting. When I set the max in flight to 1 and the timeout to 1 second and I have a consumer which is basically a Thread.sleep(1000) then I get multiple times the same event. I thought by limiting the in flight and using a manual ack this should not happen. How can I get exatly once delivery on very slow consumers?
case class EventBus[I, O](inputTopic: String, outputTopic: String, connection: Connection, eventProcessor: StatefulEventProcessor[I, O]) {
// the event bus could be some abstract class while the `Connection` coulbd be injected using DI
val substritionOptions: SubscriptionOptions = new SubscriptionOptions.Builder()
.setManualAcks(true)
.setDurableName("foo")
.setMaxInFlight(1)
.setAckWait(1, TimeUnit.SECONDS)
.build()
if (!inputTopic.isEmpty) {
connection.subscribe(inputTopic, new MessageHandler() {
override def onMessage(m: Message) {
m.ack()
try {
val event = eventProcessor.deserialize(m.getData)
eventProcessor.onEvent(event)
} catch {
case any =>
try {
val command = new String(m.getData)
eventProcessor.onCommand(command)
} catch {
case any => println(s"de-serialization error: $any")
}
} finally {
println("got event")
}
}
}, substritionOptions)
}
if (!outputTopic.isEmpty) {
eventProcessor.setBus(e => {
try {
connection.publish(outputTopic, eventProcessor.serialize(e))
} catch {
case ex => println(s"serialization error $ex")
}
})
}
}
abstract class StatefulEventProcessor[I, O] {
private var bus: Option[O => Unit] = None
def onEvent(event: I): Unit
def onCommand(command: String): Unit
def serialize(o: O): Array[Byte] =
SerializationUtils.serialize(o.asInstanceOf[java.io.Serializable])
def deserialize(in: Array[Byte]): I =
SerializationUtils.deserialize[I](in)
def setBus(push: O => Unit): Unit = {
if (bus.isDefined) {
throw new IllegalStateException("bus already set")
} else {
bus = Some(push)
}
}
def push(event: O) =
bus.get.apply(event)
}
EventBus("out-1", "out-2", sc, new StatefulEventProcessor[String, String] {
override def onEvent(event: String): Unit = {
Thread.sleep(1000)
push("!!!" + event)
}
override def onCommand(command: String): Unit = {}
})
(0 until 100).foreach(i => sc.publish("out-1", SerializationUtils.serialize(s"test-$i")))
First, there is no exactly once (re)delivery guarantee with NATS Streaming. What MaxInflight gives you, is the assurance that the server will not send new messages to the subscriber until the number of unacknowledged messages is below that number. So in case of MaxInflight(1), you are asking the server to send the next new message only after receiving the ack from the previously delivered message. However, this does not block redelivery of unacknowledged messages.
The server has no guarantee or no knowledge that a message is actually received by a subscriber. This is what the ACK is for, to let the server know that the message was properly processed by the subscriber. If the server would not honor redelivery (even when MaxInflight is reached), then a "lost" message would stall your subscription for ever. Keep in mind that NATS Streaming server and clients are not directly connected to each other with a TCP connection (they are both connected to a NATS server, aka gnatsd).
I want to make a request to a server asynchronously involving actors. Say I have 2 actors:
class SessionRetriever extends Actor {
import SessionRetriever._
def receiver = {
Get =>
val s = getSessionIdFromServer() // 1
sender ! Result(s) // 2
}
def getSessionIdFromServer(): String = { ... } // 3
}
object SessionRetriever {
object Get
object Result(s: String)
}
And
class RequestSender extends Actor {
val sActor = context actorOf Props[SessionRetriever]
def receiver = {
// get session id
val sesId = sActor ! SessionRetriever.Get
val res = sendRequestToServer(sesId)
logToFile(res)
context shutdown sActor
}
def sendRequestToServer(sessionId: String): String = { .... }
}
My questions:
val s = getSessionIdFromServer() // 1
sender ! Result(s) // 2
1) getSessionIdFromServer() does a synchronous request to the server. I think it would be much better make a request asynchronous, correct? So it will return Future[String] instead of a plain String.
2) How do I make asynchronous: by using an AsyncHttpClient (if I recall correctly its name) or by wrapping its synchronous body into Future { } ?
3) Should I use blocking { } block ? If yes, then where exactly: inside its body or here val s = blocking { getSessionIdFromServer() } ?
P.S. I'd like not to use async { } and await { } at this point because they are quite high level functions and after all they are build on top on Futures.
you might try this non-blocking way
def receive = {
Get =>
//assume getSessionIdFromServer() run aysnchronize
val f: Future[String] = getSessionIdFromServer()
val client = sender //keep it local to use when future back
f onComplete {
case Success(rep) => client ! Result(rep)
case Failure(ex) => client ! Failed(ex)
}
}
1) If getSessionIdFromServer() is blocking then you should execute it asynchronously from your receive function, otherwise your actor will block each time it receives a new request and will always wait until it receives a new session before processing the next request.
2) Using a Future will "move" the blocking operation to a different thread. So, your actor will not be blocked and will be able to keep processing incoming requests - that's good -, however you are still blocking a thread - not so great. Using the AsyncHttpClient is a good idea. You can explore other non-blocking httpClient, like PlayWebService.
3) I am not quite familiar with blocking so not sure I should advise anything here. From what I understand, it will tell the thread pool that the operation is blocking and that it should spawn a temporary new thread to handle it - this avoids having all your workers being blocked. Again, if you do that your actor will not blocked, but you are still blocking a thread while getting the session from the server.
To summarize: just use an async http client in getSessionIdFromServer if it is possible. Otherwise, use either Future{} or blocking.
To do an asynchronous call with AsyncHttpClient you could deal with the java Future via a scala Promise.
import scala.concurrent.Future
import com.ning.http.client.AsyncHttpClient
import scala.concurrent.Promise
import java.util.concurrent.Executor
object WebClient {
private val client = new AsyncHttpClient
case class BadStatus(status: Int) extends RuntimeException
def get(url: String)(implicit exec: Executor): Future[String] = {
val f = client.prepareGet(url).execute();
val p = Promise[String]()
f.addListener(new Runnable {
def run = {
val response = f.get
if (response.getStatusCode / 100 < 4)
p.success(response.getResponseBodyExcerpt(131072))
else p.failure(BadStatus(response.getStatusCode))
}
}, exec)
p.future
}
def shutdown(): Unit = client.close()
}
object WebClientTest extends App {
import scala.concurrent.ExecutionContext.Implicits.global
WebClient get "http://www.google.com/" map println foreach (_ => WebClient.shutdown())
}
And then deal with the future completion via a callback.
Credit for the code to the awesome reactive programming course at Coursera.
Update Wed Oct 16th) There was a PullRequest today that gives target info on timeouts.
https://github.com/akka/akka/pull/1780
Akka's timeout exceptions are horrendously unhelpful.
Is there some way to get a useful message about where/what is happening in the timeout?
exceptions like this are not helpful
java.util.concurrent.TimeoutException: Futures timed out after [5000] milliseconds
at akka.dispatch.DefaultPromise.ready(Future.scala:834)
at akka.dispatch.DefaultPromise.ready(Future.scala:811)
at akka.dispatch.Await$.ready(Future.scala:64)
at nl.cwi.crisp.examples.p2p.scala.Network.<init>(Node.scala:136)
at nl.cwi.crisp.examples.p2p.scala.Main$$anonfun$11.apply(Node.scala:164)
at nl.cwi.crisp.examples.p2p.scala.Main$$anonfun$11.apply(Node.scala:164)
at akka.actor.ActorCell.newActor(ActorCell.scala:488)
at akka.actor.ActorCell.create$1(ActorCell.scala:506)
at akka.actor.ActorCell.systemInvoke(ActorCell.scala:591)
at akka.dispatch.Mailbox.processAllSystemMessages(Mailbox.scala:191)
at akka.dispatch.Mailbox.run(Mailbox.scala:160)
at akka.dispatch.ForkJoinExecutorConfigurator$MailboxExecutionTask.exec(AbstractDispatcher.scala:505)
at akka.jsr166y.ForkJoinTask.doExec(ForkJoinTask.java:259)
at akka.jsr166y.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:997)
at akka.jsr166y.ForkJoinPool.runWorker(ForkJoinPool.java:1495)
at akka.jsr166y.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:104)
With the akka code the way it currently is, it's not going to happen. Let's start by taking a look at why. If you look at the PromiseActorRef object you can see:
def apply(provider: ActorRefProvider, timeout: Timeout): PromiseActorRef = {
val result = Promise[Any]()
val scheduler = provider.guardian.underlying.system.scheduler
val a = new PromiseActorRef(provider, result)
implicit val ec = a.internalCallingThreadExecutionContext
val f = scheduler.scheduleOnce(timeout.duration) { result tryComplete Failure(new AskTimeoutException("Timed out")) }
result.future onComplete { _ ‚áí try a.stop() finally f.cancel() }
a
}
This is where the parallel (parallel to the actual actor call) timeout is scheduled. This class has no context of what message it's sending or to what actor ref it was sending it to. That's probably why it just says "Timed Out" which is not very helpful. I'm kinda hoping that the typesafe guys tweak this a little to provide more info, but in case they don't or if you want something in the interim, you could try something like this:
object NewAskPattern{
implicit def ask(ref:ActorRef) = new BetterTimeoutMessageSupportAskableRef(ref)
}
class BetterTimeoutMessageSupportAskableRef(ref: ActorRef) {
import akka.pattern.AskableActorRef
val askRef = new AskableActorRef(ref)
def ask(message: Any)(implicit timeout: Timeout, ec:ExecutionContext): Future[Any] =
(askRef ? message) recover{
case to:TimeoutException =>
val recip = askRef.actorRef.path
val dur = timeout.duration
throw new TimeoutException(s"Timed out sending message $message to recipient $recip using timeout of $dur")
}
def ?(message: Any)(implicit timeout: Timeout, ec:ExecutionContext): Future[Any] =
ask(message)(timeout, ec)
}
class MySlowActor extends Actor{
def receive = {
case any =>
Thread.sleep(5000)
sender ! "bar"
}
}
object NewMessageTest{
import NewAskPattern.ask
def main(args: Array[String]) {
implicit val timeout = Timeout(2 seconds)
val sys = ActorSystem()
import sys.dispatcher
val slow = sys.actorOf(Props[MySlowActor])
val fut = slow ? "foo"
fut onComplete (println(_))
}
}
The general idea here is to wrap the AskableActorRef from the Akka lib and enhance it a little. I'm taking the Future returned by ask and adding a recover combinator to it allowing me to tweak the message when we get a timeout. As this class has the context of what message was being sent and who it was being sent to, it can formulate a more helpful message. Then the NewAskPattern object contains the new implicit to give you the BetterTimeoutMessageSupportAskableRef that allows you to gain this enhanced behavior. Is this a perfect solution? Probably not, but it could be a good starting point for you if you really want this behavior.
What happens when an actor of a future throws an exception?
According to the Akka documentation at http://doc.akka.io/docs/akka/snapshot/scala/futures.html:
It doesn't matter if an Actor or the dispatcher is completing the
Future, if an Exception is caught the Future will contain it instead
of a valid result. If a Future does contain an Exception, calling
Await.result will cause it to be thrown again so it can be handled
properly.
I am not sure this is what I am seeing when running this piece of code:
class Worker extends Actor {
def receive = {
case i: Int => throw new RuntimeException
}
}
implicit val system = ActorSystem("MySystem")
val worker = system.actorOf(Props(new Worker), name="worker")
implicit val timeout = Timeout(5 minutes)
val future = worker ? 0
val res = Await.result(future, 10 seconds)
According to the documentation, Await.result should throw the exception again, but what I am getting is a TimeoutException! Can someone clarify on this?
For actors you need to catch the exception and return it as a failure status. Right now you're not returning anything to the sender so you're getting a timeout exception:
class Worker extends Actor {
def receive = {
case i: Int => {
try {
throw new RuntimeException
sender ! "Some good result"
} catch {
case e: Exception =>
sender ! akka.actor.Status.Failure(e) // Alert the sender of the failure
throw e // Alert any supervisor actor of the failure
}
}
}
}
Futures can handle this a little more gracefully since they always send a result, while actors do not (this would give you the same result as above):
val future = Future {
throw new RuntimeException
}