trying to fetch result from database and returning the future resultset. But the issue is while accessing future result i am not getting any response.
below is the code snippnet:
def getAll(): Future[Iterable[Employee]] = {
Future{
fetchEmployees()
}(ec)
}
def fetchEmployees(): Iterable[Employee]={
var empList = ListBuffer[Employee]()
db.withConnection{ conn =>
val statement = conn.createStatement()
val rs = statement.executeQuery("Select * from Employee")
while (rs.next()){
println(rs.getString("EmpCode")+" "+rs.getString("FirstName")+" "+rs.getString("LastName"),rs.getString("Department"))
val emp = Employee(rs.getString("EmpCode"),rs.getString("FirstName"),rs.getString("LastName"),rs.getString("Department"))
empList.appended(emp)
}
}
empList
}
this is where trying to access return future object
def findAll: Future[Iterable[EmployeeResource]] = {
println("Inside resource handler")
repository.getAll().map(iterableEmp => {
iterableEmp.foreach(emp => println(s"Name is $emp.firstName"))
iterableEmp.map(emp=>createResource(emp))
})(ec)
}
Prints nothing.
Look at the doc for the appended method -- and note the term, it is not "append" (like in a command), but "appended", like what if...
def appended[B >: A](elem: B): ListBuffer[B]
A copy of this sequence with an element appended.
Your code:
empList.appended(emp)
creates a new ListBuffer, but you discard its result and your initial list buffer is never actually modified. (It is always a good idea to switch on the -Ywarn-value-discard scalac option!)
You need to use the += operator (or the addOne method).
Related
I wrote simple callback(handler) function which i pass to async api and i want to wait for result:
object Handlers {
val logger: Logger = Logger("Handlers")
implicit val cs: ContextShift[IO] =
IO.contextShift(ExecutionContext.Implicits.global)
class DefaultHandler[A] {
val response: IO[MVar[IO, A]] = MVar.empty[IO, A]
def onResult(obj: Any): Unit = {
obj match {
case obj: A =>
println(response.flatMap(_.tryPut(obj)).unsafeRunSync())
println(response.flatMap(_.isEmpty).unsafeRunSync())
case _ => logger.error("Wrong expected type")
}
}
def getResponse: A = {
response.flatMap(_.take).unsafeRunSync()
}
}
But for some reason both tryPut and isEmpty(when i'd manually call onResult method) returns true, therefore when i calling getResponse it sleeps forever.
This is the my test:
class HandlersTest extends FunSuite {
test("DefaultHandler.test") {
val handler = new DefaultHandler[Int]
handler.onResult(3)
val response = handler.getResponse
assert(response != 0)
}
}
Can somebody explain why tryPut returns true, but nothing puts. And what is the right way to use Mvar/channels in scala?
IO[X] means that you have the recipe to create some X. So on your example, yuo are putting in one MVar and then asking in another.
Here is how I would do it.
object Handlers {
trait DefaultHandler[A] {
def onResult(obj: Any): IO[Unit]
def getResponse: IO[A]
}
object DefaultHandler {
def apply[A : ClassTag]: IO[DefaultHandler[A]] =
MVar.empty[IO, A].map { response =>
new DefaultHandler[A] {
override def onResult(obj: Any): IO[Unit] = obj match {
case obj: A =>
for {
r1 <- response.tryPut(obj)
_ <- IO(println(r1))
r2 <- response.isEmpty
_ <- IO(println(r2))
} yield ()
case _ =>
IO(logger.error("Wrong expected type"))
}
override def getResponse: IO[A] =
response.take
}
}
}
}
The "unsafe" is sort of a hint, but every time you call unsafeRunSync, you should basically think of it as an entire new universe. Before you make the call, you can only describe instructions for what will happen, you can't actually change anything. During the call is when all the changes occur. Once the call completes, that universe is destroyed, and you can read the result but no longer change anything. What happens in one unsafeRunSync universe doesn't affect another.
You need to call it exactly once in your test code. That means your test code needs to look something like:
val test = for {
handler <- TestHandler.DefaultHandler[Int]
_ <- handler.onResult(3)
response <- handler.getResponse
} yield response
assert test.unsafeRunSync() == 3
Note this doesn't really buy you much over just using the MVar directly. I think you're trying to mix side effects inside IO and outside it, but that doesn't work. All the side effects need to be inside.
I am trying to read incremental data from my data source using Scala-Spark. Before hitting the source tables, I am trying to calculate the min & max of partition column that I use in my code in a Future which is present in a class: GetSourceMeta as given below.
def getBounds(keyIdMap:scala.collection.mutable.Map[String, String]): Future[scala.collection.mutable.Map[String, String]] = Future {
var boundsMap = scala.collection.mutable.Map[String, String]()
keyIdMap.keys.foreach(table => if(!keyIdMap(table).contains("Invalid")) {
val minMax = s"select max(insert_tms) maxTms, min(insert_tms) minTms from schema.${table} where source='DB2' and key_id in (${keyIdMap(table)})"
println("MinMax: " + minMax)
val boundsDF = spark.read.format("jdbc").option("url", con.getConUrl()).option("dbtable", s"(${minMax}) as ctids").option("user", con.getUserName()).option("password", con.getPwd()).load()
try {
val maxTms = boundsDF.select("minTms").head.getTimestamp(0).toString + "," + boundsDF.select("maxTms").head.getTimestamp(0).toString
println("Bounds: " + maxTms)
boundsMap += (table -> maxTms)
} catch {
case np: java.lang.NullPointerException => { println("No data found") }
case e: Exception => { println(s"Unknown exception: $e") }
}
}
)
boundsMap.foreach(println)
boundsMap
}
I am calling the above method in my main method as:
object LoadToCopyDB {
val conf = new SparkConf().setAppName("TEST_YEAR").set("some parameters")
def main(args: Array[String]): Unit = {
val spark = SparkSession.builder().config(conf).master("yarn").enableHiveSupport().config("hive.exec.dynamic.partition", "true").config("hive.exec.dynamic.partition.mode", "nonstrict").getOrCreate()
val gsm = new GetSourceMeta()
val minMaxKeyMap = gsm.getBounds(keyIdMap).onComplete {
case Success(values) => values.foreach(println)
case Failure(f) => f.printStackTrace
}
.
.
.
}
Well, the onComplete didn't print any values so I used andThen as below and that didn't help as well.
val bounds: Future[scala.collection.mutable.Map[String, String]] = gpMetaData.getBounds(incrementalIds) andThen {
case Success(outval) => outval.foreach(println)
case Failure(e) => println(e)
}
Earlier the main thread exits without letting the Future: getBounds execute. Hence I couldn't find any println statements from the Future displayed on the terminal. I found out that I need to keep the main thread Await inorder to complete the Future. But when I use Await in main along with onComplete:
Await.result(bounds, Duration.Inf)
The compiler gives an error:
Type mismatch, expected: Awaitable[NotInferedT], actual:Unit
If I declare the val minMaxKeyMap as Future[scala.collection.mutable.Map[String, String] the compiler says: Expression of type Unit doesn't conform to expected type Future[mutable.map[String,String]]
I tried to print the values of bounds after the Await statement but that just prints an empty Map.
I couldn't understand how can to fix this. Could anyone let me know what do I do to make the Future run properly ?
In this kind of cases, is always better to follow the types. The method onComplete only returns Unit, it won´t return a future hence it can´t be passed using Await.
In case you want to return a Future of any type you will have to map or flatmap the value and return an option, for example. In this case, does not matter what you return, you only want Await method to wait for this result and print a trace. You can treat the possible exception in the recover. It would be like that in your code:
val minMaxKeyMap:Future[Option[Any] = gsm.getBounds(keyIdMap).map { values =>
values.foreach(println)
None
}.recover{
case e: Throwable =>
e. printStackTrace
None
}
Note that the recover part has to return an instance of the type.
After that, you can apply the Await to the Future, and you will get the results printed. Is not the prettiest solution but it will work in your case.
I have been struck with a piece on how to obtain a listbuffer of strings in the case where the listbuffer happens to be constructed in a scala future called in a loop.
Here is a kiss example
def INeedThatListBuffer(): ListBuffer[String] = {
var myCollections: ListBuffer[String] = new ListBuffer[String]()
for (day <- daysInaWeek) {
val myFuture: Future[String] = Future {
// use 'day' do some stuff and get me a result
???
}
myFuture.onComplete {
case Success(result) =>
myCollections += result
}
}
myCollections
}
My problem is that sometimes listBuffer is empty list and sometimes the content that I expected. Clearly, this method is complete before the future is evaluated.
Just to Add
I don't want to used future.await
Passing myCollections as Future obj does not work as there is no binding that myFuture must be complete before myCollections is evaluated.
Kindly help me out.
Thanks
This returns a future. If you don't care about waiting for it to be complete, you can always access the underlying value using .value:
def INeedThatListBuffer(): Future[ListBuffer[String]] = {
def buildFutureFromDay(day: String): Future[String] = Future { ??? }
Future
.sequence(daysInAWeek.map(buildFutureFromDay))
.map(_.foldLeft(ListBuffer[String]())(_ += _))
}
You need to await at some point. Either await for each of the futures to get resolved or change the ListBuffer[String] to ListBuffer[Future[String]] and await on the whole buffer later.
val myFutureCollection: ListBuffer[Future[String]] = new ListBuffer[Future[String]]()
val myFuture: Future[String] = Future(???)
myFutureCollection += myFuture
val eventualBuffer: Future[ListBuffer[String]] = Future.sequence(myFutureCollection)
val buffer: ListBuffer[String] = Await.result(eventualBuffer, 2 seconds)
P.S: You can val instead of var for the list buffer as it is already mutable.
I have a Future lazy val that obtains some object and a function which submits operations in the Future.
class C {
def printLn(s: String) = println(s)
}
lazy val futureC: Future[C] = Future{Thread.sleep(3000); new C()}
def func(s: String): Unit = {
futureC.foreach{c => c.printLn(s)}
}
The problem is when Future is completed it executes operations in reverse order than they have been submited. So for example if I execute sequentialy
func("A")
func("B")
func("C")
I get after Future completion
scala> C
B
A
This order is important for me. Is there a way to preserve this order?
Of course I can use an actor who asks for future and stashing strings while future is not ready, but it seems redundant for me.
lazy val futureC: Future[C]
lazy vals in scala will be compiled in to the code which uses a synchronized block for thread safety.
Here when the func(A) is called, it will obtain the lock for the lazy val and that thread will go to sleep.
Therefore func(B) & func(C) will blocked by the lock.
When those blocked threads are run, the order cannot be guaranteed.
If you do it like below, you'll have the order as you expect. This is because the for comprehension creates a flatMap, & map based chain that gets executed sequentially.
lazy val futureC: Future[C] = Future {
Thread.sleep(1000)
new C()
}
def func(s: String) : Future[Unit] = {
futureC.map { c => c.printLn(s) }
}
val x = for {
_ <- func("A")
_ <- func("B")
_ <- func("C")
} yield ()
The order preserves even without the lazy keyword. You can remove the lazy keyword unless it is really necessary.
Hope this helps.
You can use Future.traverse to ensure the order of execution.
Something like this.. Im not sure how your func has a reference to the correct futureC, so I moved it inside.
def func(s: String): Future[Unit] = {
lazy val futureC = Future{Thread.sleep(3000); new C()}
futureC.map{c => c.printLn(s)}
}
def traverse[A,B](xs: Seq[A])(fn: A => Future[B]): Future[Seq[B]] =
xs.foldLeft(Future(Seq[B]())) { (acc, item) =>
acc.flatMap { accValue =>
fn(item).map { itemValue =>
accValue :+ itemValue
}
}
}
traverse(Seq("A","B","C"))(func)
def returnFuture[A](x: A): Future[A] = {
val xFuture = Future { x } // suppose an API call that returns a future
xFuture.flatMap(x => {
println(x) // logging the value of x
xFuture
})
}
This is the way I'm currently doing it. To provide more context:
This function is being called inside an API when a request is made and I'd like the log message to be printed just before the value computed in the request is returned. Which is why, the following is not a good solution for me:
def returnFuture[A](x: A): Future[A] = {
val xFuture = Future { x } // suppose an API call that returns a future
xFuture.map(x => {
println(x) // logging the value of x
})
xFuture
}
Logging is a side-effect, meaning that you don't want the operation to fail if the logging fails for any reason (e.g. a call to toString throwing NPE).
Future#andThen is perfect for this use case. From the docs:
Applies the side-effecting function to the result of this future, and returns a new future with the result of this future.
This method allows one to enforce that the callbacks are executed in a specified order.
Note that if one of the chained andThen callbacks throws an exception, that exception is not propagated to the subsequent andThen callbacks. Instead, the subsequent andThen callbacks are given the original value of this future.
Your example becomes:
def returnFuture[A](x: A): Future[A] = {
Future { x } // suppose an API call that returns a future
.andThen { case Success(v) => println(v) }
}
You can use onComplete callback:
def returnFuture[A](x: A): Future[A] = {
val f = Future { x }
f.onComplete(println)
f
}
A map will work too:
def returnFuture[A](x: A): Future[A] = {
Future { x }.map { v =>
println(v)
v
}
}
Keep in mind that the whole point of using Futures is that you are trying to avoid blocking and that you don't control exactly when the Future will be executed. So, if you want more detailed logs while keeping the asynchronous nature of a Future, do something like this:
def doSomething(param: String): String = {
// log something here
val result = param.toUpperCase
// log something else here
result
}
def asFuture(param: String) = Future {
doSomething(param)
}
In other words, if this is an option, add logs to the x operation instead.