I have a problem with futures executing SQL queries.
While database connection is good futures asynchronously executing sql queries within sessions from connection pool. TypeSafe Slick helps me to get sessions from the pool.
When database connection broke down the new coming futures can't execute their queries and wait. I don't see any callbacks onComplete.
When the database connection is good again all previous futures are still waiting. Only new futures coming after reconnect can execute their work.
Please advise how to tell already called waiting futures about db reconnection and continue their work, or do callback onComplete Failure.
My configuration for c3p0 ComboPooledDataSource:
val ds = new ComboPooledDataSource
ds.setDriverClass("oracle.jdbc.OracleDriver")
ds.setJdbcUrl(jdbcUrl)
ds.setInitialPoolSize(20)
ds.setMinPoolSize(1)
ds.setMaxPoolSize(40)
ds.setAcquireIncrement(5)
ds.setMaxIdleTime(3600)
//Connection testing
ds.setTestConnectionOnCheckout(false)
ds.setTestConnectionOnCheckin(false)
ds.setIdleConnectionTestPeriod(10)
//Connection recovery
ds.setBreakAfterAcquireFailure(false)
ds.setAcquireRetryAttempts(30)
ds.setAcquireRetryDelay(10000)
val databasePool = Database.forDataSource(ds)
// Typesafe Slick session handling
def withClient[T](body: Session => T) = {
databasePool.withSession(body)
}
Futures creates here:
class RewardActivatorHelper {
private implicit val ec = new ExecutionContext {
val threadPool = Executors.newFixedThreadPool(1000)
def execute(runnable: Runnable) {threadPool.submit(runnable)}
def reportFailure(t: Throwable) {throw t}
}
case class FutureResult(spStart:Long, spFinish:Long)
def activateReward(msg:Msg, time:Long):Unit = {
msg.users.foreach {
user =>
val future:Future[FutureResult] = Future {
val (spStart, spFinish) = OracleClient.rewardActivate(user)
FutureResult(spStart, spFinish)
}
future.onComplete {
case Success(futureResult:FutureResult) =>
//do something
case Failure(e:Throwable) => log.error(e.getMessage())
}
}
}
Problem solved by setTestConnectionOnCheckout(true)
Related
I want to implement a high-throughput server that accepts multiple clients. Every request should query a database, so I need some kind of async behavior.
I followed the ROUTER-to-REQ pattern from documentation + Futures, so I ended with this "architecture":
trait ZmqProtocol extends Protocol {
private val pool = Executors.newCachedThreadPool()
private implicit val ec: ExecutionContextExecutor = ExecutionContext.fromExecutor(pool)
val context: ZMQ.Context = ZMQ.context(1)
val socket: ZMQ.Socket = context.socket(ZMQ.ROUTER)
socket.bind("tcp://*:5555")
override def receiveMessages(): String = {
while (true) {
val address = socket.recv(0)
val empty = socket.recv(0)
val request = socket.recv(0)
Future {
val message = new String(request)
getResponseFromDb(message)
} onComplete {
case Success(response) =>
// Send reply back to client
socket.send(address, ZMQ.SNDMORE)
socket.send("".getBytes, ZMQ.SNDMORE)
socket.send(response.getBytes(), 0)
case Failure(ex) => println(ex)
}
}
"DONE"
}
}
I understand this won't work because I'm sharing socket in Future so I need a better model. I know the ZeroMQ sockets are fast and creating several worker threads would be enough on input side, but if the bottleneck is on the database side and if I need to do some other work while waiting for DB, I presume all my threads would soon be exhausted.
Would it be too much of an overhead if I create new socket and bind on ROUTER in every Future or is there some better solution?
Also, for Scala developers: is there a way to force onComplete being executed on main thread (I suppose it would solve the issue)? Thanks!
I'm again seeking you to share your wisdom with me, the scala padawan!
I'm playing with reactive mongo in scala and while I was writting a test using scalatest, I faced the following issue.
First the code:
"delete" when {
"passing an existent id" should {
"succeed" in {
val testRecord = TestRecord(someString)
Await.result(persistenceService.persist(testRecord), Duration.Inf)
Await.result(persistenceService.delete(testRecord.id), Duration.Inf)
Thread.sleep(1000) // Why do I need that to make the test succeeds?
val thrownException = intercept[RecordNotFoundException] {
Await.result(persistenceService.read(testRecord.id), Duration.Inf)
}
thrownException.getMessage should include(testRecord._id.toString)
}
}
}
And the read and delete methods with the code initializing connection to db (part of the constructor):
class MongoPersistenceService[R](url: String, port: String, databaseName: String, collectionName: String) {
val driver = MongoDriver()
val parsedUri: Try[MongoConnection.ParsedURI] = MongoConnection.parseURI("%s:%s".format(url, port))
val connection: Try[MongoConnection] = parsedUri.map(driver.connection)
val mongoConnection = Future.fromTry(connection)
def db: Future[DefaultDB] = mongoConnection.flatMap(_.database(databaseName))
def collection: Future[BSONCollection] = db.map(_.collection(collectionName))
def read(id: BSONObjectID): Future[R] = {
val query = BSONDocument("_id" -> id)
val readResult: Future[R] = for {
coll <- collection
record <- coll.find(query).requireOne[R]
} yield record
readResult.recover {
case NoSuchResultException => throw RecordNotFoundException(id)
}
}
def delete(id: BSONObjectID): Future[Unit] = {
val query = BSONDocument("_id" -> id)
// first read then call remove. Read will throw if not present
read(id).flatMap { (_) => collection.map(coll => coll.remove(query)) }
}
}
So to make my test pass, I had to had a Thread.sleep right after waiting for the delete to complete. Knowing this is evil usually punished by many whiplash, I want learn and find the proper fix here.
While trying other stuff, I found instead of waiting, entirely closing the connection to the db was also doing the trick...
What am I misunderstanding here? Should a connection to the db be opened and close for each call to it? And not do many actions like adding, removing, updating records with one connection?
Note that everything works fine when I remove the read call in my delete function. Also by closing the connection, I mean call close on the MongoDriver from my test and also stop and start again embed Mongo which I'm using in background.
Thanks for helping guys.
Warning: this is a blind guess, I've no experience with MongoDB on Scala.
You may have forgotten to flatMap
Take a look at this bit:
collection.map(coll => coll.remove(query))
Since collection is Future[BSONCollection] per your code and remove returns Future[WriteResult] per doc, so actual type of this expression is Future[Future[WriteResult]].
Now, you have annotated your function as returning Future[Unit]. Scala often makes Unit as a return value by throwing away possibly meaningful values, which it does in your case:
read(id).flatMap { (_) =>
collection.map(coll => {
coll.remove(query) // we didn't wait for removal
() // before returning unit
})
}
So your code should probably be
read(id).flatMap(_ => collection.flatMap(_.remove(query).map(_ => ())))
Or a for-comprehension:
for {
_ <- read(id)
coll <- collection
_ <- coll.remove(query)
} yield ()
You can make Scala warn you about discarded values by adding a compiler flag (assuming SBT):
scalacOptions += "-Ywarn-value-discard"
There are many questions on SO that combine Futures with Timeout. To be honest, I haven't completely understood how to use them. But it seems I have stumbled upon a problem where I will have to (or maybe not).
I want to throw a TimeoutException if a statement takes more than say 1 minute.To be more clear, currently, this statement tries to get a response from a server but does not throw if the server is not setup. It currently looks like this:
//proper import of exceptions
case class ServerException(exception: Throwable) extends Exception(exception)
//Code that instantiates client and post
val response = try {
client.execute(post)
} catch {
case e#(_: IOException | _: ClientProtocolException) => throw new ServerException(e)
}
To mitigate this problem, I want to introduce a timeout. How do I introduce timeout to this statement such that it throws if no response is got within one minute, else it instantiates response and the program continues as it is?
It's not available in scala Futures. You can switch to scalaz Task - it's a bit different abstraction for async/delayed computations. You can read awesome documentation for it here: http://timperrett.com/2014/07/20/scalaz-task-the-missing-documentation/
import java.util.concurrent.Executors
import scalaz.concurrent.Task
import scala.concurrent.duration._
implicit val scheduledThreadPool =
Executors.newScheduledThreadPool(5)
def executeRequest(req: Request): Task[Response] = ???
val withTimeOut: Task[Response] =
executeRequest(req).timed(1.minute)
Update
Btw you can easily transform your Future to Task, for example it Future is coming from 3rd party lib
object Future2Task {
implicit class Transformer[+T](fut: => Future[T]) {
def toTask(implicit ec: scala.concurrent.ExecutionContext): Task[T] = {
import scala.util.{Failure, Success}
import scalaz.syntax.either._
Task.async {
register =>
fut.onComplete {
case Success(v) => register(v.right)
case Failure(ex) => register(ex.left)
}
}
}
}
}
Timeouts are usually implemented by having an asynchronous timer act as the timeout signal and completing the future in question whenever it or the timer completes.
I believe Akka has a such a timer, but it's pretty simple to roll your own:
object ConcurrencyUtil {
// creates a Future that will complete after a specified duration
object Delay {
def apply(d: Duration): Future[Unit] = {
val p = Promise[Unit]()
val t = new Timer
t.schedule(new TimerTask {
override def run(): Unit = p.success()
}, d.toMillis)
p.future
}
}
implicit class FutureExtensions[T](future: Future[T]) {
def timeout(timeout: Duration) = Future.firstCompletedOf(Seq(
Delay(timeout).map(_ => throw new TimeoutException()),
future
))
}
}
Now you can compose timeout with your future like this:
import ConcurrencyUtil._
val f = someTaskReturningAFuture.timeout(1.minute)
What is now if the task has not completed within 1 minute, the delay will fire, get mapped to throwing a TimeoutException and complete the future f as failed.
Note: This does not address cancellation, i.e. the other future, while no longer being listened for will continue to exist and if it's executing something, continue to execute.
I have a project use both slick and anorm.
I define a method for slick
object DBCache {
def apply(app: play.api.Application) = Cache.getOrElse[Database](app.hashCode.toString){
Database.forDataSource(PlayDB.getDataSource("default")(app))
}
}
private[persist] def inSession[T](block: Session => T) = DBCache(current).withSession(block(_))
And when I can a batch insert method use anorm
def batchInsert(customerAccounts: Seq[Customer]) = DB.withConnection { implicit conn =>
val sql = SQL(insertSql)
val batch = customerAccounts.foldLeft(sql.asBatch) {
(sql, c) => sql.addbatch(xxx)
}
}
It reports
play.api.Application$$anon$1: Execution exception[[MySQLNonTransientConnectionException: No operations allowed after connection closed.]
How to avoid this error
I found the master branch of playframework disable the tracking ,
datasource.setDisableConnectionTracking(
conf.getBoolean("disableConnectionTracking").getOrElse(true))
So I disable it in Global.scala.
There'are still some problem. When sql error happends, connection may be closed. Then I turned to use alibaba's data source https://github.com/alibaba/druid. It works fine!
Update Wed Oct 16th) There was a PullRequest today that gives target info on timeouts.
https://github.com/akka/akka/pull/1780
Akka's timeout exceptions are horrendously unhelpful.
Is there some way to get a useful message about where/what is happening in the timeout?
exceptions like this are not helpful
java.util.concurrent.TimeoutException: Futures timed out after [5000] milliseconds
at akka.dispatch.DefaultPromise.ready(Future.scala:834)
at akka.dispatch.DefaultPromise.ready(Future.scala:811)
at akka.dispatch.Await$.ready(Future.scala:64)
at nl.cwi.crisp.examples.p2p.scala.Network.<init>(Node.scala:136)
at nl.cwi.crisp.examples.p2p.scala.Main$$anonfun$11.apply(Node.scala:164)
at nl.cwi.crisp.examples.p2p.scala.Main$$anonfun$11.apply(Node.scala:164)
at akka.actor.ActorCell.newActor(ActorCell.scala:488)
at akka.actor.ActorCell.create$1(ActorCell.scala:506)
at akka.actor.ActorCell.systemInvoke(ActorCell.scala:591)
at akka.dispatch.Mailbox.processAllSystemMessages(Mailbox.scala:191)
at akka.dispatch.Mailbox.run(Mailbox.scala:160)
at akka.dispatch.ForkJoinExecutorConfigurator$MailboxExecutionTask.exec(AbstractDispatcher.scala:505)
at akka.jsr166y.ForkJoinTask.doExec(ForkJoinTask.java:259)
at akka.jsr166y.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:997)
at akka.jsr166y.ForkJoinPool.runWorker(ForkJoinPool.java:1495)
at akka.jsr166y.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:104)
With the akka code the way it currently is, it's not going to happen. Let's start by taking a look at why. If you look at the PromiseActorRef object you can see:
def apply(provider: ActorRefProvider, timeout: Timeout): PromiseActorRef = {
val result = Promise[Any]()
val scheduler = provider.guardian.underlying.system.scheduler
val a = new PromiseActorRef(provider, result)
implicit val ec = a.internalCallingThreadExecutionContext
val f = scheduler.scheduleOnce(timeout.duration) { result tryComplete Failure(new AskTimeoutException("Timed out")) }
result.future onComplete { _ ‚áí try a.stop() finally f.cancel() }
a
}
This is where the parallel (parallel to the actual actor call) timeout is scheduled. This class has no context of what message it's sending or to what actor ref it was sending it to. That's probably why it just says "Timed Out" which is not very helpful. I'm kinda hoping that the typesafe guys tweak this a little to provide more info, but in case they don't or if you want something in the interim, you could try something like this:
object NewAskPattern{
implicit def ask(ref:ActorRef) = new BetterTimeoutMessageSupportAskableRef(ref)
}
class BetterTimeoutMessageSupportAskableRef(ref: ActorRef) {
import akka.pattern.AskableActorRef
val askRef = new AskableActorRef(ref)
def ask(message: Any)(implicit timeout: Timeout, ec:ExecutionContext): Future[Any] =
(askRef ? message) recover{
case to:TimeoutException =>
val recip = askRef.actorRef.path
val dur = timeout.duration
throw new TimeoutException(s"Timed out sending message $message to recipient $recip using timeout of $dur")
}
def ?(message: Any)(implicit timeout: Timeout, ec:ExecutionContext): Future[Any] =
ask(message)(timeout, ec)
}
class MySlowActor extends Actor{
def receive = {
case any =>
Thread.sleep(5000)
sender ! "bar"
}
}
object NewMessageTest{
import NewAskPattern.ask
def main(args: Array[String]) {
implicit val timeout = Timeout(2 seconds)
val sys = ActorSystem()
import sys.dispatcher
val slow = sys.actorOf(Props[MySlowActor])
val fut = slow ? "foo"
fut onComplete (println(_))
}
}
The general idea here is to wrap the AskableActorRef from the Akka lib and enhance it a little. I'm taking the Future returned by ask and adding a recover combinator to it allowing me to tweak the message when we get a timeout. As this class has the context of what message was being sent and who it was being sent to, it can formulate a more helpful message. Then the NewAskPattern object contains the new implicit to give you the BetterTimeoutMessageSupportAskableRef that allows you to gain this enhanced behavior. Is this a perfect solution? Probably not, but it could be a good starting point for you if you really want this behavior.