Catching scanamo errors within scala - scala

I'm using scanamo to query a dynamodb table.
I want to create a healthcheck method that will return true as long as dynamo can be scanned (i don't care if there are records in the table, i just want to know that the table is there and can be scanned). And i want false to be returned if the table is not present (i fudge the table name as a test).
This is the scala code i have right now:
trait DynamoTestTrait extends AbstractDynamoConfig {
def test(): Future[List[Either[DynamoReadError, T]]] =
ScanamoAsync.exec(client)(table.consistently.limit(1).scan())
}
and it works when the table is present:
val r = reg.test()
.map(
_.headOption.forall(_.isRight)
)
val e = Await.result(r, scala.concurrent.duration.Duration(5, "seconds"))
but when the table name is incorrect i get an unexpected error:
Unexpected server error: 'Cannot do operations on a non-existent table (Service: AmazonDynamoDBv2; Status Code: 400; Error Code: ResourceNotFoundException;
I would have thought that i was trapping that error in my Left.
How can i trap this error? does scala have a try/catch construct i could use?
I also tried this but the error still gets thrown and not my catch it seems:
trait DynamoTestTrait extends AbstractDynamoConfig {
def test(): Future[Boolean] =
try {
ScanamoAsync
.exec(client)(table.consistently.limit(1).scan())
.map(
_.headOption.forall(_.isRight)
)
} catch {
case e: ResourceNotFoundException => Future.successful(false)
}
}
this seems more scalathonic, but always returns true:
trait DynamoTestTrait extends AbstractDynamoConfig {
def test: Future[Boolean] = {
val result = Try(ScanamoAsync
.exec(client)(table.consistently.limit(1).scan())
.map(
_.headOption.forall(_.isRight)
))
result match{
case Success(v) => Future.successful(true)
case Failure(e) => Future.successful(false)
}
}
}

Related

scalatest - test a method of Future[S] with fallbackTo

Premise: When my API responds to a request for the User object, I want to try enriching it with the properties of case class PartnerView(id: String, vipStatus: Option[Boolean], latestSession: Option[Timestamp]. Since the database can be unreliable at times, I use fallbackTo to provide the values as optional, thus not displaying them in the User JSON response.
The following implementation seems to work so far (running the request through Postman returns the User JSON without the optional values) yet my unit test would complain as if I had an uncaught Exception.
The Service class:
class Service(repo: Repository) {
def get(id: String): Future[Partner] = {
val account = repo.getAccount(id)
val getLatestSession = repo.getLatestSession(id)
val partnerView = (for {
account <- getAccount
latestStartTime <- getLatestSession.map {
case Some(x) => x.scheduledStartTime
case _ => None
}
} yield PartnerView(partnerId, account.vipStatus, latestStartTime))
.fallbackTo(Future.successful(PartnerView(id, None, None)))
partnerView
}
}
The Repository class:
class Repository(database: DatabaseDef, logger: LoggingAdapter) {
def getAccount(id: String): Future[Account] = database.run((...).result.head)
.recover {
case e: Exception =>
logger.error(e, "DB Server went down")
throw e
}
def getLatestSession(id: String): Future[Option[Session]] = database.run((...).result.headOption)
.recover {
case e: Exception =>
logger.error(e, "DB Server went down")
throw e
}
}
The Unit Test:
class ServiceSpec extends AsyncFlatSpec with AsyncMockFactory with OneInstancePerTest {
val mockRepo = mock[Repository]
val service = new Service(mockRepo)
behaviour of "Service"
it should "get an empty PartnerView when the repository get an Exception" in {
(mockRepository.getAccount _)
.expects("partner")
.throwing(new Exception)
service.get("partner")
.map(partnerView => assert(partnerView.id == "partner" && partnerView.vipStatus.isEmpty))
}
}
The test would fail with the message
Testing started at 5:15 p.m. ...
java.lang.Exception was thrown.
{stacktrace here}
I'm expecting the Exception to
By changing the mock setup to below, the test ran successfully:
it should "get an empty PartnerView when the repository get an Exception" in {
(mockRepository.getAccount _)
.expects("partner")
.returning(Future.failed(new Exception))
...
}
since the recover method wraps the Exception inside a Future
Sources:
recover vs recoverWith
official scala article

Unable to print values of a Scala Future by using onComplete & andThen

I am trying to read incremental data from my data source using Scala-Spark. Before hitting the source tables, I am trying to calculate the min & max of partition column that I use in my code in a Future which is present in a class: GetSourceMeta as given below.
def getBounds(keyIdMap:scala.collection.mutable.Map[String, String]): Future[scala.collection.mutable.Map[String, String]] = Future {
var boundsMap = scala.collection.mutable.Map[String, String]()
keyIdMap.keys.foreach(table => if(!keyIdMap(table).contains("Invalid")) {
val minMax = s"select max(insert_tms) maxTms, min(insert_tms) minTms from schema.${table} where source='DB2' and key_id in (${keyIdMap(table)})"
println("MinMax: " + minMax)
val boundsDF = spark.read.format("jdbc").option("url", con.getConUrl()).option("dbtable", s"(${minMax}) as ctids").option("user", con.getUserName()).option("password", con.getPwd()).load()
try {
val maxTms = boundsDF.select("minTms").head.getTimestamp(0).toString + "," + boundsDF.select("maxTms").head.getTimestamp(0).toString
println("Bounds: " + maxTms)
boundsMap += (table -> maxTms)
} catch {
case np: java.lang.NullPointerException => { println("No data found") }
case e: Exception => { println(s"Unknown exception: $e") }
}
}
)
boundsMap.foreach(println)
boundsMap
}
I am calling the above method in my main method as:
object LoadToCopyDB {
val conf = new SparkConf().setAppName("TEST_YEAR").set("some parameters")
def main(args: Array[String]): Unit = {
val spark = SparkSession.builder().config(conf).master("yarn").enableHiveSupport().config("hive.exec.dynamic.partition", "true").config("hive.exec.dynamic.partition.mode", "nonstrict").getOrCreate()
val gsm = new GetSourceMeta()
val minMaxKeyMap = gsm.getBounds(keyIdMap).onComplete {
case Success(values) => values.foreach(println)
case Failure(f) => f.printStackTrace
}
.
.
.
}
Well, the onComplete didn't print any values so I used andThen as below and that didn't help as well.
val bounds: Future[scala.collection.mutable.Map[String, String]] = gpMetaData.getBounds(incrementalIds) andThen {
case Success(outval) => outval.foreach(println)
case Failure(e) => println(e)
}
Earlier the main thread exits without letting the Future: getBounds execute. Hence I couldn't find any println statements from the Future displayed on the terminal. I found out that I need to keep the main thread Await inorder to complete the Future. But when I use Await in main along with onComplete:
Await.result(bounds, Duration.Inf)
The compiler gives an error:
Type mismatch, expected: Awaitable[NotInferedT], actual:Unit
If I declare the val minMaxKeyMap as Future[scala.collection.mutable.Map[String, String] the compiler says: Expression of type Unit doesn't conform to expected type Future[mutable.map[String,String]]
I tried to print the values of bounds after the Await statement but that just prints an empty Map.
I couldn't understand how can to fix this. Could anyone let me know what do I do to make the Future run properly ?
In this kind of cases, is always better to follow the types. The method onComplete only returns Unit, it won´t return a future hence it can´t be passed using Await.
In case you want to return a Future of any type you will have to map or flatmap the value and return an option, for example. In this case, does not matter what you return, you only want Await method to wait for this result and print a trace. You can treat the possible exception in the recover. It would be like that in your code:
val minMaxKeyMap:Future[Option[Any] = gsm.getBounds(keyIdMap).map { values =>
values.foreach(println)
None
}.recover{
case e: Throwable =>
e. printStackTrace
None
}
Note that the recover part has to return an instance of the type.
After that, you can apply the Await to the Future, and you will get the results printed. Is not the prettiest solution but it will work in your case.

Try Failure is not catching exceptions

In the code below I have two Play for Scala functions, the first one catches an exception (this works fine) and in the second one I'm trying to rewrite it using Try.
I have two problems with Try: (1) when the number is negative the method doesn't fail, (2) I need to wrap all the responses with Future.successful.
How to fix this code?
class Test extends Controller {
def test1 = Action.async { request =>
val future = isPositive(-1)
future.map { result =>
Ok("OK, it's positive")
}
.recover {
case e => Ok(e.getMessage)
}
}
def isPositive(i: Int) = Future {
if (i<0)
throw new Exception ( "Number is negative" )
else
i
}
def test2 = Action.async { request =>
isPositiveWithTry(-1) match {
case Success(s) => Future.successful(Ok("OK, it's positive (Try succeded)"))
case Failure(f) => Future.successful(Ok(f.getMessage + " (Try failed)"))
}
}
def isPositiveWithTry(i: Int) : Try[Future[Int]] = Try {
isPositive(i)
}
}
In isPositive method exceptions are already caught by Future
def isPositive(i: Int) = Future {
if (i<0)
throw new Exception ( "Number is negative" )
else
i
}
In the below code
def isPositiveWithTry(i: Int) : Try[Future[Int]] = Try {
isPositive(i)
}
isPositive already catches all expections and Try is always a success.
So, when i is negative. Exception raised are handled by future and try gets a success value, resultant Try is a success. So you get successful Try with a failed Future inside.
Understanding using Grenade example
Assume throwing the exception as blowing up a Grenade.
Assume Future and Try as two layers. When grenade is blasted inside the double layer of Try[Future] i.e Try is around Future and grenade is gone off in the Future.
Now Future withstands the blast and becomes a failed value. As Future already took the damage caused by the damage of exception(grenade). Try will be a success but the value inside the Try is a failed future value. That failed future value is nothing but the exception raised.
Try is redundant when you are using Future
You can refactor your code to below one
Get rid of isPositiveWithTry. This method is not needed.
def isPositive(i: Int) = Future {
if (i<0)
throw new Exception ( "Number is negative" )
else
i
}
def test2 = Action.async { request =>
isPositive(-1).flatMap { _ =>
Future.successful(Ok("OK, it's positive (Try succeded)"))
}.recoverWith {
case f: Throwable => Future.successful(Ok(f.getMessage + " (Try failed)"))
}
}
Again test2 can also be written as
def test2 = Action.async { request =>
isPositive(-1).map { _ =>
Ok("OK, it's positive (Try succeded)")
}.recover {
case f: Throwable => Ok(f.getMessage + " (Try failed)")
}
}
In case isPositive returns Try
def isPositive(i: Int) = Try {
if (i<0)
throw new Exception ( "Number is negative" )
else
i
}
Now test2 will look like
def test2 = Action.async { request =>
isPositive(-1) match {
case Success(s) => Future.successful(Ok("OK, it's positive (Try succeded)"))
case Failure(f) => Future.successful(Ok(f.getMessage + " (Try failed)"))
}
}
Couple points:
1) You need to rewrite your isPositive such that it does not surround itself via a Future. The Future is catching the exception.
def isPositive(i: Int) ={
if (i<0)
throw new Exception ( "Number is negative" )
else
i
}
2) If you have a Try and you want a Future, then you can use the method on the companion object of Future, Future.fromTry. That will take a Try and turn it into the correct state of a Future.

Verify schema using Slick 3

I am looking to use the Slick 3 framework for a Scala application to manage database interactions. I have been able to automatically generate the necessary table objects using Slick, but I also would like an integration test that verifies that the schemas in the database match the schemas in the objects. This is because sometimes tables get altered without my team being alerted, and so we would prefer to catch the change in an integration test instead of a production application.
One way to do this is to simply run a select query on every single table in a test runner. However, I feel like there should be a more direct way. Furthermore, it is not clear to me how to systematically run through all the tables defined in the file, except to manually append the table object to some sequence the test runner moves through. I notice that there is a schema field, but it only has the ability to generate create and drop statements.
Any help would be greatly appreciated. Thank you!
EDIT:
Here is my solution, but I was hoping for a better one:
class TablesIT extends FunSuite with BeforeAndAfter with ScalaFutures {
var db: Database = _
before{ db = Database.forURL( /* personal details */ )}
object ResultMap extends GetResult[Map[String,Any]] { //Object borrowed from http://stackoverflow.com/questions/20262036/slick-query-multiple-tables-databases-with-getting-column-names
def apply(pr: PositionedResult) = {
val rs = pr.rs // <- jdbc result set
val md = rs.getMetaData
val res = (1 to pr.numColumns).map{ i=> md.getColumnName(i) -> rs.getObject(i) }.toMap
pr.nextRow // <- use Slick's advance method to avoid endless loop
res
}
}
def testTableHasCols[A <: Table[_]](table: slick.lifted.TableQuery[A]): Unit = {
whenReady(db.run(table.take(1).result.headOption.asTry)) { case Success(t) => t match {
case Some(r) => logTrace(r.toString)
case None => logTrace("Empty table")
}
case Failure(ex) => fail("Query exception: " + ex.toString)
}
}
def plainSqlSelect[A](query: String)(implicit gr: GetResult[A]): Future[Seq[A]] = {
val stmt = sql"""#$query""".as[A]
db.run(stmt)
}
def compareNumOfCols[A <: Table[_]](table: slick.lifted.TableQuery[A]) = {
val tableName = table.baseTableRow.tableName
val selectStar = whenReady(db.run(sql"""select * from #$tableName limit 1""".as(ResultMap).headOption)) {
case Some(m) => m.size
case None => 0
}
val model = whenReady(db.run(sql"""#${table.take(1).result.statements.head}""".as(ResultMap).headOption)) {
case Some(m) => m.size
case None => 0
}
assert(selectStar === model, "The number of columns do not match")
}
test("Test table1") {
testTableHasCols(Table1)
compareNumOfCols(Table1)
}
// And on for each table
}
I ended up devising a better solution that uses the following idea. It is more or less the same, and unfortunately I still have to manually create a test for each table, but the method is cleaner, I think. Note, however, that this only works for PostgreSQL because of the information schema, but other database systems have other methods.
class TablesIT extends FunSuite with BeforeAndAfter with ScalaFutures {
var db: Database = _
before{ db = Database.forURL( /* personal details */ )}
def testTableHasCols[A <: Table[_]](table: slick.lifted.TableQuery[A]): Unit = {
whenReady(db.run(table.take(1).result.headOption.asTry)) { case Success(t) => t match {
case Some(r) => logTrace(r.toString)
case None => logTrace("Empty table")
}
case Failure(ex) => fail("Query exception: " + ex.toString)
}
}
def compareNumOfCols[A <: Table[_]](table: slick.lifted.TableQuery[A]) = {
val tableName = table.baseTableRow.tableName
val selectStar = whenReady(db.run(sql"""select column_name from information_schema.columns where table_name='#$tableName'""".as[String])) {
case m: Seq[String] => m.size
case _ => 0
}
val model = table.baseTableRow.create_*.map(_.name).toSeq.size
assert(selectStar === model, "The number of columns do not match")
}
test("Test table1") {
testTableHasCols(Table1)
compareNumOfCols(Table1)
}
// And on for each table
}

Executing a code block even after catching an exception in Scala

In my current method, I am trying to make a series of calls and if any of them fail, I want to be able to continue running the remainder (while capturing the Exception that was thrown). I am having a hard time figuring this out in Scala.
So in this example, I want to kick off each of these calls - RunA, RunB and RunC but if RunB throws an exception, I want to print that and continue kicking off RunC after that.
var result = Try {
new RunA()
new RunB()
new RunC()
} catch {
case e: Throwable => e.printStackTrace()
false
}
Outside of having them all individually wrapped in a Try/Catch, I am sure there are better ways to do this which is why I am hoping someone can help with this.
I looked at the 'Ignoring' exception but it appears to completely ignore the exception which I want to atleast log.
Thanks!
First, don't mix try { ... } catch { ... } up with scala.util.Try{ ... }.
You can
import scala.util._
val runA = Try{ new RunA }
val runB = Try{ new RunB }
val runC = Try{ new RunC }
and then deal with the exceptions as you see fit. For instance, if you want to print and continue, you could deal with the try statements right there:
def getOrPrint[A](f: => A): Option[A] = Try{ f } match {
case Success(x) => Some(x)
case Failure(e) => e.printStackTrace; None
}
getOrPrint{ new RunA }
...
There can be more elegant ways for such things with scalaz (e.g. read an article here for some inspiration: http://johnkurkowski.com/posts/accumulating-multiple-failures-in-a-ValidationNEL/), but with "only" Scala you can do something like this:
import scala.reflect.ClassTag
import scala.util.{Try, Success, Failure}
def tryAndLog[T: ClassTag] = Try {
implicitly[ClassTag[T]].runtimeClass.newInstance.asInstanceOf[T] // new instance
} match {
case Success(_) => true
case Failure(ex) => ex.printStackTrace ; false
}
def tryRunAll = {
val A = tryAndLog[RunA]
val B = tryAndLog[RunB]
val C = tryAndLog[RunC]
A && B && C // returns true if all invocations succeeded, false otherwise
}
You are mixing scala.util.Try with try {} catch {} which are different concepts. Try wraps function into Success(result) or Failure(error) class, and try-catch is like Java try-catch. I suggest you something like this:
class RunA
class RunB
class RunC
class Result(a: RunA, b: RunB, c: RunC)
implicit class LogFailure[T](t: Try[T]) {
def logFailure: Try[T] = t match {
case scala.util.Failure(err) => err.printStackTrace(); t
case _ => t
}
}
val tryA= Try(new RunA())
val tryB= Try(new RunB())
val tryC = Try(new RunC())
val result: Try[Result] = for {
a <- tryA.logFailure
b <- tryB.logFailure
c <- tryC.logFailure
} yield {
// do smth with a, b, c
new Result(a, b, c)
}
If A, B, C will be successful you'll get Success(Result) if one of them failure you'll get Failure with first exception, however all of them will be logged (printed stack trace)