Async before and after for creating and dropping scala slick tables in scalatest - scala

I'm trying to figure out a way to have async before and after statements where the next test cases aren't run until the completion of the action inside of the test case. In my case, it is the creating and dropping a table inside of a database
val table = TableQuery[BlockHeaderTable]
val dbConfig: DatabaseConfig[PostgresDriver] = DatabaseConfig.forConfig("databaseUrl")
val database: Database = dbConfig.db
before {
//Awaits need to be used to make sure this is fully executed before the next test case starts
//TODO: Figure out a way to make this asynchronous
Await.result(database.run(table.schema.create), 10.seconds)
}
"BlockHeaderDAO" must "store a blockheader in the database, then read it from the database" in {
//...
}
it must "delete a block header in the database" in {
//...
}
after {
//Awaits need to be used to make sure this is fully executed before the next test case starts
//TODO: Figure out a way to make this asynchronous
Await.result(database.run(table.schema.drop),10.seconds)
}
Is there a simple way I can remove these Await calls inside of my before and after functions?

Unfortunately, #Jeffrey Chung's solution hanged for me (since futureValue actually awaits internally). This is what I ended up doing:
import org.scalatest.{AsyncFreeSpec, FutureOutcome}
import scala.concurrent.Future
class TestTest extends AsyncFreeSpec /* Could be any AsyncSpec. */ {
// Do whatever setup you need here.
def setup(): Future[_] = ???
// Cleanup whatever you need here.
def tearDown(): Future[_] = ???
override def withFixture(test: NoArgAsyncTest) = new FutureOutcome(for {
_ <- setup()
result <- super.withFixture(test).toFuture
_ <- tearDown()
} yield result)
}

The following is the testing approach that Dennis Vriend takes in his slick-3.2.0-test project.
First, define a dropCreateSchema method. This method attempts to create a table; if that attempt fails (because, for example, the table already exists), it drops, then creates, the table:
def dropCreateSchema: Future[Unit] = {
val schema = BlockHeaderTable.schema
db.run(schema.create)
.recoverWith {
case t: Throwable =>
db.run(DBIO.seq(schema.drop, schema.create))
}
}
Second, define a createEntries method that populates the table with some sample data for use in each test case:
def createEntries: Future[Unit] = {
val setup = DBIO.seq(
// insert some rows
BlockHeaderTable ++= Seq(
BlockHeaderTableRow(/* ... */),
// ...
)
).transactionally
db.run(setup)
}
Third, define an initialize method that calls the above two methods sequentially:
def initialize: Future[Unit] = for {
_ <- dropCreateSchema
_ <- createEntries
} yield ()
In the test class, mix in the ScalaFutures trait. For example:
class TestSpec extends FlatSpec
with Matchers
with ScalaFutures
with BeforeAndAfterAll
with BeforeAndAfterEach {
// ...
}
Also in the test class, define an implicit conversion from a Future to a Try, and override the beforeEach method to call initialize:
implicit val timeout: Timeout = 10.seconds
implicit class PimpedFuture[T](self: Future[T]) {
def toTry: Try[T] = Try(self.futureValue)
}
override protected def beforeEach(): Unit = {
blockHeaderRepo.initialize // in this example, initialize is defined in a repo class
.toTry recover {
case t: Throwable =>
log.error("Could not initialize the database", t)
} should be a 'success
}
override protected def afterAll(): Unit = {
db.close()
}
With the above pieces in place, there is no need for Await.

You can simplify #Jeffrey Chung
A simplified dropCreateSchema method:
def dropCreateSchema: Future[Unit] = {
val schema = users.schema
db.run(DBIO.seq(schema.dropIfExists, schema.create))
}
Also in the test class, I simplified beforeEach method that calls initialize. I removed an implicit conversion from a Future to a Try, and use onComplete callback:
override protected def beforeEach(): Unit = {
initialize.onComplete(f =>
f recover {
case t: Throwable =>
log.error("Could not initialize the database", t)
} should be a 'success)
}
override protected def afterAll(): Unit = {
db.close()
}

Related

MVar tryPut returns true and isEmpty also returns true

I wrote simple callback(handler) function which i pass to async api and i want to wait for result:
object Handlers {
val logger: Logger = Logger("Handlers")
implicit val cs: ContextShift[IO] =
IO.contextShift(ExecutionContext.Implicits.global)
class DefaultHandler[A] {
val response: IO[MVar[IO, A]] = MVar.empty[IO, A]
def onResult(obj: Any): Unit = {
obj match {
case obj: A =>
println(response.flatMap(_.tryPut(obj)).unsafeRunSync())
println(response.flatMap(_.isEmpty).unsafeRunSync())
case _ => logger.error("Wrong expected type")
}
}
def getResponse: A = {
response.flatMap(_.take).unsafeRunSync()
}
}
But for some reason both tryPut and isEmpty(when i'd manually call onResult method) returns true, therefore when i calling getResponse it sleeps forever.
This is the my test:
class HandlersTest extends FunSuite {
test("DefaultHandler.test") {
val handler = new DefaultHandler[Int]
handler.onResult(3)
val response = handler.getResponse
assert(response != 0)
}
}
Can somebody explain why tryPut returns true, but nothing puts. And what is the right way to use Mvar/channels in scala?
IO[X] means that you have the recipe to create some X. So on your example, yuo are putting in one MVar and then asking in another.
Here is how I would do it.
object Handlers {
trait DefaultHandler[A] {
def onResult(obj: Any): IO[Unit]
def getResponse: IO[A]
}
object DefaultHandler {
def apply[A : ClassTag]: IO[DefaultHandler[A]] =
MVar.empty[IO, A].map { response =>
new DefaultHandler[A] {
override def onResult(obj: Any): IO[Unit] = obj match {
case obj: A =>
for {
r1 <- response.tryPut(obj)
_ <- IO(println(r1))
r2 <- response.isEmpty
_ <- IO(println(r2))
} yield ()
case _ =>
IO(logger.error("Wrong expected type"))
}
override def getResponse: IO[A] =
response.take
}
}
}
}
The "unsafe" is sort of a hint, but every time you call unsafeRunSync, you should basically think of it as an entire new universe. Before you make the call, you can only describe instructions for what will happen, you can't actually change anything. During the call is when all the changes occur. Once the call completes, that universe is destroyed, and you can read the result but no longer change anything. What happens in one unsafeRunSync universe doesn't affect another.
You need to call it exactly once in your test code. That means your test code needs to look something like:
val test = for {
handler <- TestHandler.DefaultHandler[Int]
_ <- handler.onResult(3)
response <- handler.getResponse
} yield response
assert test.unsafeRunSync() == 3
Note this doesn't really buy you much over just using the MVar directly. I think you're trying to mix side effects inside IO and outside it, but that doesn't work. All the side effects need to be inside.

How to wrap a library async method with a threadsafe async wrapper in scala?

I'm working with a library that has a method that looks something like this.
trait LibraryDependency {
/**
* This method if called while previous future is not resolved will return a failed future
* #return
*/
def foo(): Future[Boolean]
}
Here the method foo() will return a failed future if there is a previous future that was returned by foo() that has not yet completed. Since I cannot modify the library's implementation of the trait, I am trying to wrap it with my own wrapper that provides the behaviour I need.
The behaviour I need is that if there are concurrent calls to foo(), then the other futures will also block until the first future is resolved. I tried to do something like this.
class ThreadSafeLibraryWrapper(delegate: LibraryDependency) extends LibraryDependency {
private val lock: Object = new Object
private implicit val ec: ExecutionContext = ExecutionContext.Implicits.global
/**
* This one will block the other concurrent calls to foo()
* #return
*/
override def foo(): Future[Boolean] = {
val promise = Promise[Boolean]()
lock.synchronized {
val result = delegate.foo()
promise.completeWith(result)
result.onComplete { _ =>
lock.notify()
}
lock.wait()
}
promise.future
}
}
I'm running into the following issue, I'm not sure how to block the thread that is calling this method, and complete the original future, and I get IllegalMonitorStateException.
EDIT: I've solved this by using Await
class ThreadSafeLibraryWrapper(delegate: LibraryDependency) extends LibraryDependency {
private val lock: Object = new Object
private implicit val ec: ExecutionContext = ExecutionContext.Implicits.global
/**
* This one will block the other concurrent calls to foo()
* #return
*/
override def foo(): Future[Boolean] = Future {
lock.synchronized {
Await.result(delegate.foo(), Duration.Inf)
}
}
}
I'm still not sure how to do this by avoiding Await.
If I correctly understood your question, your dependency can run on Future at time, so you would like to have wrapper which will limit access to foo method so to avoid returning failed future. If so, this looks like you need to en-queue next invocations until previous one will complete.
Well, I did some prototyping and I hope it helps:
import java.time.LocalTime.now
import scala.collection.immutable.Queue
import scala.concurrent.{ExecutionContext, Future, Promise}
import scala.util.{Failure, Success, Try}
object Concurrency {
trait LibraryDependency {
/**
* This method if called while previous future is not resolved will return a failed future
* #return
*/
def foo(): Future[Boolean]
}
class DummyLibraryDependency(implicit ec: ExecutionContext) extends LibraryDependency {
override def foo(): Future[Boolean] = {
Future {
println(s"${now()} - started dependency execution")
Thread.sleep(1000)
println(s"${now()} - finished dependency execution")
true
}
}
}
class SafeLibraryDependency(delegate: LibraryDependency)(implicit ec: ExecutionContext) {
private type OnComplete = Try[Boolean] => Unit
private var currentlyRunning: Option[Future[Boolean]] = None
private var queue: Queue[Promise[Boolean]] = Queue[Promise[Boolean]]()
def foo: Future[Boolean] = {
this.synchronized {
currentlyRunning.fold(startDelegateTask(onRunningComplete))(_ => enqueueNextTask())
}
}
private def enqueueNextTask(): Future[Boolean] = {
val promise = Promise[Boolean]()
queue = queue enqueue promise
promise.future
}
private def onRunningComplete(result: Try[Boolean]): Unit = {
this.synchronized {
currentlyRunning = None
if(queue.nonEmpty) {
val (promise, newQueue) = queue.dequeue
queue = newQueue
startDelegateTask { result =>
promise.complete(result)
onRunningComplete(result)
}
}
}
}
private def startDelegateTask(f: OnComplete): Future[Boolean] = {
val task = delegate.foo()
task.onComplete(f)
currentlyRunning = Some(task)
task
}
}
def main(args: Array[String]): Unit = {
import scala.concurrent.ExecutionContext.Implicits.global
val dummyLibraryDependency = new DummyLibraryDependency
val safeLibraryDependency = new SafeLibraryDependency(dummyLibraryDependency)
safeLibraryDependency.foo.onComplete(result => println(s"${now()} - #1 complete with result: $result"))
safeLibraryDependency.foo.onComplete(result => println(s"${now()} - #2 complete with result: $result"))
safeLibraryDependency.foo.onComplete(result => println(s"${now()} - #3 complete with result: $result"))
Thread.sleep(5000)
println("Done")
}
}
So SafeLibraryDependency - this is the wrapper, which limit's invocations to single running Future at time.
On my machine output was next:
19:30:43.666 - started dependency execution
19:30:44.679 - finished dependency execution
19:30:44.681 - started dependency execution
19:30:44.679 - #1 complete with result: Success(true)
19:30:45.681 - finished dependency execution
19:30:45.681 - started dependency execution
19:30:45.681 - #2 complete with result: Success(true)
19:30:46.682 - finished dependency execution
19:30:46.682 - #3 complete with result: Success(true)
Done
Hope this helps you!

Wrapper catching exceptions accured on db.run call

In our project we always injecting dbConfigProvider: DatabaseConfigProvider in our bean objects and then doing database operations with db.run(some query), it returns Future. How can I write a logging wrapper for db.run which will print all sql exceptions.
Example:
class SomeBeanImpl #Inject()(dbConfigProvider: DatabaseConfigProvider) {
private val logger = Logger(getClass)
def someDBQuery() = {
db.run(some wrong sql query) // exception raised in future, I need to print it with logger
}
}
Note:
If I add .onFailure on each db.run call it will mess up my code very badly. That's why I need write this wrapper for all of db.run calls.
If I wrap db.run in some function with different signature, I must change it in many places, which is not the best option. How can I do it implicitly?
You don't need to explicitly create a new wrapper class, you can use the Pimp My Library pattern to create an implicit method which wraps the invocation of db.run and attaches a onFailure to the Future:
object MyExtensions {
class DbExtensions(db: Db) {
def runAndLog(query: String): Future[String] = {
val result = db.run(query)
result.onFailure {
case e => Logger.getLogger("x").error(s"Exception: $e")
}
result
}
}
implicit def dbExtention(db: Db): DbExtensions = new DbExtensions(db)
}
class Db {
def run(query: String): Future[String] = Future.successful("Hello")
}
object App extends Application {
import MyExtensions._
val db = new Db
db.runAndLog("hello")
}
For Scala 2.10 and above, this can be shortened significantly using Implicit Classes:
implicit class DbExtensions(val db: Db) {
def runAndLog(query: String): Future[String] = {
val result = db.run(query)
result.onFailure {
case e => Logger.getLogger("x").error(s"Exception: $e")
}
result
}
}
class Db {
def run(query: String): Future[String] = Future.successful("Hello")
}
object App extends Application {
val db = new Db
db.runAndLog("hello")
}
You can further make DbExtensions extend AnyVal for a performance optimization:
implicit class DbExtensions(val db: Db) extends AnyVal
Make a new class:
case class DBWrapper(db: DatabaseComponent) {
def run(query: String) = db.run(query).onFailure { case e => logger.error(e) }
}
And replace your db wherever you initialize it with DBWrapper(db).
You can make the conversions back and forth implicit too, although I would not recommend it in this case.

How to initialize fixture once for all tests in scalatest?

According to documentation, fixtures can be passed to tests like this:
def withFixture(test: OneArgTest) = {
val f = veryLengthyInitialization(test.configMap)
withFixture(test.toNoArgTest(f))
}
describe("Some test") {
it("should succeed") { f =>
f.doSomething()
// ...
}
// many other tests
// ...
}
Problem is that the initialization is run for each test, and it takes a very long time, so I would like it to run just once. Documentation suggests an alternative:
override def beforeAll(configMap: ConfigMap) {
val fOnce = veryLengthyInitialization(configMap)
// but how do I pass f to the tests?
// ugly workaround:
f = fOnce
}
var f: InitType = null
describe("Some test") {
it("should succeed") { // hack: no parameter, use var instead
f.doSomething()
// ...
}
// many other tests
// ...
}
It works, but my understanding is that null should be avoided, and it's a bit silly to rely on mutation for something as basic as this. What is the correct way to write such tests?
I don't think that there is a solution that is good in all aspects for this issue. One thing I do sometime to avoid the var is structuring the test as follows:
class SomeTest extends Spec {
def initFixtures(): SomeFixtureType = ???
lazy val fixtures = initFixtures()
describe("Some test") {
it("should succeed") in {
fixtures.doSomething()
// ...
}
}
}
In this way I don't need to use the beforeAndAfterAll thing, but you can if you want to guarantee that the fixtures are initialised before the test as follows:
override def beforeAll() {
super.beforeAll()
fixtures
}
It can also be handy when dealing with asynchronous initialisation since you could have something like:
class SomeTest extends Spec {
def initFixtures(): Future[SomeFixtureType] = ???
lazy val fixtures = initFixtures()
describe("Some test") {
it("should succeed") in {
whenReady(fixtures.flatMap {
f.doSomething()
// ...
}) { res => doSomeChecks(res) }
}
}
}
Having both initFixtures and fixtures is most useful in this case as you can have some tests re-initialise the system and act in isolation if they need to do effectful stuffs (e.g. use a database).
I'm not sure if the below is something better:
trait DoSomething {
var f:InitType = _
def doSomething = f.doSomething
}
class MyTestSpec extends DoSomething {
override def beforeAll(configMap: ConfigMap) {
val fOnce = veryLengthyInitialization(configMap)
f = fOnce
}
it("should succeed") {
doSomething()
}
}
For any tests you need a f variable to do something, you could extend the trait and initialize it in your beforeAll methods, in addition, you could even override the doSomething in your test.

Kill or timeout a Future in Scala 2.10

Hi,
I'm using Scala 2.10 with the new futures library and I'm trying to write some code to test an infinite loop. I use a scala.concurrent.Future to run the code with the loop in a separate thread. I would then like to wait a little while to do some testing and then kill off the separate thread/future. I have looked at Await.result but that doesn't actually kill the future. Is there any way to timeout or kill the new Scala 2.10 futures?
I would prefer not having to add external dependencies such as Akka just for this simple part.
Do not try it at home.
import scala.concurrent._
import scala.concurrent.duration._
class MyCustomExecutionContext extends AnyRef with ExecutionContext {
import ExecutionContext.Implicits.global
#volatile var lastThread: Option[Thread] = None
override def execute(runnable: Runnable): Unit = {
ExecutionContext.Implicits.global.execute(new Runnable() {
override def run() {
lastThread = Some(Thread.currentThread)
runnable.run()
}
})
}
override def reportFailure(t: Throwable): Unit = ???
}
implicit val exec = new MyCustomExecutionContext()
val f = future[Int]{ do{}while(true); 1 }
try {
Await.result(f, 10 seconds) // 100% cpu here
} catch {
case e: TimeoutException =>
println("Stopping...")
exec.lastThread.getOrElse(throw new RuntimeException("Not started"))
.stop() // 0% cpu here
}
No - you will have to add a flag that your loop checks. If the flag is set, stop the loop. Make sure the flag is at least volatile.
See Java Concurrency in Practice, p 135-137.
I had a similar problem and wrote the following nonblocking future op:
class TerminationToken(var isTerminated: Boolean)
object TerminationToken { def apply() = new TerminationToken(false) }
implicit class FutureOps[T](future: Future[Option[T]]) {
def terminate(timeout: FiniteDuration, token: TerminationToken): Future[Option[T]] = {
val timeoutFuture = after[Option[T]](timeout, using = context.system.scheduler) {
Future[Option[T]] { token.isTerminated = true; None } }
Future.firstCompletedOf[Option[T]](Seq (future recover { case _ => None }, timeoutFuture))
}
}
Then just create a future that returns an option, and use .terminate(timeout, token) on it