I am confused on how one would conditionally update a document based on a previous query using only futures.
Let say I want to push to some value into an array in a document only if that array has a size less than a given Integer.
I am using this function to get the document, after getting the document I am pushing values - what I am unable to do is do that conditionally.
def joinGroup(actionRequest: GroupRequests.GroupActionRequest): Future[GroupResponse.GroupActionCompleted] = {
//groupisNotFull() is a boolean future
groupIsNotFull(actionRequest.groupId).map(
shouldUpdate => {
if(shouldUpdate){
Logger.info(actionRequest.initiator + " Joining Group: " + actionRequest.groupId)
val selector = BSONDocument("group.groupid" -> BSONDocument("$eq" -> actionRequest.groupId))
val modifier = BSONDocument("$push" -> BSONDocument("group.users" -> "test-user"))
val updateResult = activeGroups.flatMap(_.update(selector, modifier))
.map(res => {
GroupActionCompleted(
actionRequest.groupId,
actionRequest.initiator,
GroupConstants.Actions.JOIN,
res.ok,
GroupConstants.Messages.JOIN_SUCCESS
)
})
.recover {
case e: Throwable => GroupActionCompleted(
actionRequest.groupId,
actionRequest.initiator, GroupConstants.Actions.JOIN,
success = false,
GroupConstants.Messages.JOIN_FAIL
)
}
updateResult
}
else {
val updateResult = Future.successful(
GroupActionCompleted(
actionRequest.groupId,
actionRequest.initiator,
GroupConstants.Actions.JOIN,
success = false,
GroupConstants.Messages.JOIN_FAIL
))
updateResult
}
}
)
}
//returns a Future[Boolean] based on if there is room for another user.
private def groupIsNotFull(groupid: String): Future[Boolean] = {
findGroupByGroupId(groupid)
.map(group => {
if (group.isDefined) {
val fGroup = group.get
fGroup.group.users.size < fGroup.group.groupInformation.maxUsers
} else {
false
}
})
}
I am confused on why I cannot do this. The compilation error is:
Error:type mismatch;
found : scala.concurrent.Future[response.group.GroupResponse.GroupActionCompleted]
required: response.group.GroupResponse.GroupActionCompleted
for both the if and else branch 'updateResult'.
As a side question.. is this the proper way of updating documents conditionally - that is querying for it, doing some logic then executing another query?
Ok got it - you need to flatMap the first Future[Boolean] like this:
groupIsNotFull(actionRequest.groupId).flatMap( ...
Using flatMap, the result will be a Future[T] with map you would get a Future[Future[T]]. The compiler knows you want to return a Future[T] so its expecting the map to return a T and you are trying to return a Future[T] - so it throws the error. Using flatMap will fix this.
Some further clarity on map vs flatmap here: In Scala Akka futures, what is the difference between map and flatMap?
I believe the problem is because the joinGroup2 function return type is Future[Response], yet you are returning just a Response in the else block. If you look at the signature of the mapTo[T] function, it returns a Future[T].
I think you need to wrap the Response object in a Future. Something like this:
else {
Future { Response(false, ERROR_REASON) }
}
Btw you have a typo: Respose -> Response
Related
I am trying to read incremental data from my data source using Scala-Spark. Before hitting the source tables, I am trying to calculate the min & max of partition column that I use in my code in a Future which is present in a class: GetSourceMeta as given below.
def getBounds(keyIdMap:scala.collection.mutable.Map[String, String]): Future[scala.collection.mutable.Map[String, String]] = Future {
var boundsMap = scala.collection.mutable.Map[String, String]()
keyIdMap.keys.foreach(table => if(!keyIdMap(table).contains("Invalid")) {
val minMax = s"select max(insert_tms) maxTms, min(insert_tms) minTms from schema.${table} where source='DB2' and key_id in (${keyIdMap(table)})"
println("MinMax: " + minMax)
val boundsDF = spark.read.format("jdbc").option("url", con.getConUrl()).option("dbtable", s"(${minMax}) as ctids").option("user", con.getUserName()).option("password", con.getPwd()).load()
try {
val maxTms = boundsDF.select("minTms").head.getTimestamp(0).toString + "," + boundsDF.select("maxTms").head.getTimestamp(0).toString
println("Bounds: " + maxTms)
boundsMap += (table -> maxTms)
} catch {
case np: java.lang.NullPointerException => { println("No data found") }
case e: Exception => { println(s"Unknown exception: $e") }
}
}
)
boundsMap.foreach(println)
boundsMap
}
I am calling the above method in my main method as:
object LoadToCopyDB {
val conf = new SparkConf().setAppName("TEST_YEAR").set("some parameters")
def main(args: Array[String]): Unit = {
val spark = SparkSession.builder().config(conf).master("yarn").enableHiveSupport().config("hive.exec.dynamic.partition", "true").config("hive.exec.dynamic.partition.mode", "nonstrict").getOrCreate()
val gsm = new GetSourceMeta()
val minMaxKeyMap = gsm.getBounds(keyIdMap).onComplete {
case Success(values) => values.foreach(println)
case Failure(f) => f.printStackTrace
}
.
.
.
}
Well, the onComplete didn't print any values so I used andThen as below and that didn't help as well.
val bounds: Future[scala.collection.mutable.Map[String, String]] = gpMetaData.getBounds(incrementalIds) andThen {
case Success(outval) => outval.foreach(println)
case Failure(e) => println(e)
}
Earlier the main thread exits without letting the Future: getBounds execute. Hence I couldn't find any println statements from the Future displayed on the terminal. I found out that I need to keep the main thread Await inorder to complete the Future. But when I use Await in main along with onComplete:
Await.result(bounds, Duration.Inf)
The compiler gives an error:
Type mismatch, expected: Awaitable[NotInferedT], actual:Unit
If I declare the val minMaxKeyMap as Future[scala.collection.mutable.Map[String, String] the compiler says: Expression of type Unit doesn't conform to expected type Future[mutable.map[String,String]]
I tried to print the values of bounds after the Await statement but that just prints an empty Map.
I couldn't understand how can to fix this. Could anyone let me know what do I do to make the Future run properly ?
In this kind of cases, is always better to follow the types. The method onComplete only returns Unit, it won´t return a future hence it can´t be passed using Await.
In case you want to return a Future of any type you will have to map or flatmap the value and return an option, for example. In this case, does not matter what you return, you only want Await method to wait for this result and print a trace. You can treat the possible exception in the recover. It would be like that in your code:
val minMaxKeyMap:Future[Option[Any] = gsm.getBounds(keyIdMap).map { values =>
values.foreach(println)
None
}.recover{
case e: Throwable =>
e. printStackTrace
None
}
Note that the recover part has to return an instance of the type.
After that, you can apply the Await to the Future, and you will get the results printed. Is not the prettiest solution but it will work in your case.
I have a method that returns a Future[Boolean] in a Play controller and i want to evaluate that using async but i can't seem to get it to compile.
The following will work:
def health = Action {
logger.info("Endpoint method: health")
val isHealthy = healthCheckService.checkDynamo()
val b: Boolean = Await.result(isHealthy, scala.concurrent.duration.Duration(5, "seconds"))
Ok(Json.toJson(HealthCheckResponse(b.toString)))
}
But i don't think i want that Await in there. So i'm trying things like this with no success:
def health =
Action.async {
Future {
logger.info("Endpoint method: health")
healthCheckService.checkDynamo() match {
case Future.successful(true) => Ok(Json.toJson("false"))
case false => Ok(Json.toJson("true"))
}
val r = healthCheckService.checkDynamo() match {
case true => Ok(Json.toJson("false"))
case false => Ok(Json.toJson("true"))
}
}
}
I can't even get those to compile to test them out.
Any suggestions?
Try this:
def health = Action.async {
healthCheckService.checkDynamo().map {
case true => Ok(Json.toJson("false"))
case false => Ok(Json.toJson("true"))
}
}
Let Play handle the awaiting for you under the hood. That is, Action.async accepts a Future, which checkDynamo() already returns. All you have to do is map it to the appropriate result.
With Futures you have to use combinators like map and flatMap to express the final value. For example:
Action.async {
healthCheckService.checkDynamo()
.map { result => // boolean
HealthCheckResponse(result.toString)
}
.map(Json.toJson(_))
.map(Ok(_))
}
(You can merge maps above to one map and construct the final Ok value there; it is more or less a matter of taste)
If you have, say, two async calls which you want to execute and return a result based on their results, you can use flatMap, which could be easily expressed using a for comprehension:
Action.async {
for {
result1 <- someService.someCall()
result2 <- anotherService.anotherCall(result1.someProperty)
finalResult = SomeFinalResultType(result1, result2)
} yield Ok(Json.toJson(finalResult))
}
If you are not familiar with futures, you might want to read some tutorial which explains their nature, how to combine them and how to get useful results from them, like this one: http://hello-scala.com/920-scala-futures.html
No need to pay attention to the purpose of the function here, it's only for demonstration:
def readAllByPersonOrFail(person: Person, otherPersonId: Long): Future[List[Person]] = {
val personSiblingsFuture: Future[List[Person]] = personSiblingsDomain.readAllByPersonId(person.id)
personSiblingsFuture.map { persons =>
persons.find(_.id == otherPersonId) match {
case Some(person) =>
person.isActive match {
case true => person
case false => throw new IllegalArgumentException("something inactive")
}
case None => throw new IllegalArgumentException("something wrong ehre")
}
}
personSiblingsFuture
}
I would like to return personSiblingsFuture above iff it validates (makes sure correct person is in the list and is active), otherwise throw the exception. I don't think the above code is doing the right thing as it is not existing upon failure.
Take a look at scala.concurrent.Future.map. This creates a new future, whose value is resolved by applying a function to the successful result of this future.
Note that here you're throwing away the resulting future you just created with .map() too.
There are a few areas to solve your problem, though you should question more deeply the use of exceptions with Futures. Scala provides concepts like Future, Option, and Try specifically to avoid throwing exceptions and have a clearer control flow.
Option 1, return the mapped future
In your funciton,
def func(...): Future[List[Person]] {
val personSiblingsFuture = ...;
personSiblingsFuture.map { persons =>
...
}
}
// note we're not returning personSiblingsFuture,
// but the mapped result
When someone actually tries to get the value of the future, e.g. by using .value, they might see an exception intead:
def main() {
val future = func(...); // this is fine
val my_list = future.value; // awaits Future, might throw here
}
Option 2, actually await the list and throw in the function
Returning a future that might throw is strange, it might be a bit easier if the you actually explicitly a had a function that might throw, e.g.
/** jsdoc describing function **/
def funcMightThrow(...): List[Person] {
val personSiblingsFuture = ...;
val personSiblings = personSiblingsFuture.value;
personSiblings.find(_.id == otherPersonId) match {
case Some(person) =>
person.isActive match {
case true => personSiblings
case false => throw new IllegalArgumentException("something inactive")
}
case None => throw new IllegalArgumentException("something wrong ehre")
}
}
Option 3, consider making return types more explicit
def func(...): Future[Try[List[Person]]] {
val personSiblingsFuture = ...;
personSiblingsFuture.map { persons =>
...
// successful case returns 'persons' (List[Person])
// fail cases return Failure(...) instead
}
} // return the mapped future
You can also return Try[List[Person]] rather than a Future[] of that, by using .value, which makes func a blocking function.
Hi I am trying to process data in a file.
This the code I am using below.
I have a list of Futures and trying to get the output from these futures.
Everything is fine but the last line of return is executing before OnSuccess.
How can I change that behaviour without having a blocking operation.
def processRow(rowNumber: Int, row: String, delimiter: String, rules: List[Rule]): RowMessage = {
var cells = row.split(delimiter)
var passedRules = new ListBuffer[RuleResult]()
val failedRules = new ListBuffer[RuleResult]()
val rulesFuture = rules.map {
i => Future {
val cells = row.split(delimiter);
//some processing....
}
}
val f1 = Future.sequence(rulesFuture)
f1 onComplete {
case Success(results) => for (result <- results) (result.map(x => {
if (x.isPassFailed) {
passedRules += x
}
else {
failedRules += x
}
}))
case Failure(t) => println("An error has occured: " + t.getMessage)
}
return new RowMessage(passedRules.toList, failedRules.toList)
}
You can't avoid blocking and return a plain RowMessage. You need to return a Futureas well.
def processRow(rowNumber: Int, row: String, delimiter: String, rules: List[Rule]): Future[RowMessage] = {
val cells = row.split(delimiter)
Future.traverse(rules) { i =>
Future {
//some processing....
}
} map { results =>
val (passed, failed) = results.partition(_.isPassFailed)
new RowMessage(passed, failed)
}
}
Also think about your algorithm to avoid mutable state, especially when you change it from different Futures.
Future.traverse is equivalent of your map + Future.sequence. Then instead of onComplete, just map your Future to modify the list. You can split it easly using partition instead of what you've been doing.
You don't need to use return, in fact you shouldn't unless you know what you are doing.
Btw isPassFailed doesn't sound like a reasonable method name to me, especially considering that when it's true you are adding it to passed rules.
I have a method that does a couple of database look up and performs some logic.
The MyType object that I return from the method is as follows:
case class MyResultType(typeId: Long, type1: Seq[Type1], type2: Seq[Type2])
The method definition is like this:
def myMethod(typeId: Long, timeInterval: Interval) = async {
// 1. check if I can find an entity in the database for typeId
val myTypeOption = await(db.run(findMyTypeById(typeId))) // I'm getting the headOption on this result
if (myTypeOption.isDefined) {
val anotherDbLookUp = await(doSomeDBStuff) // Line A
// the interval gets split and assume that I get a List of thse intervals
val intervalList = splitInterval(interval)
// for each of the interval in the intervalList, I do database look up
val results: Seq[(Future[Seq[Type1], Future[Seq[Type2])] = for {
interval <- intervalList
} yield {
(getType1Entries(interval), getType2Entries(interval))
}
// best way to work with the results so that I can return MyResultType
}
else {
None
}
}
Now the getType1Entries(interval) and getType2Entries(interval) each returns a Future of Seq(Type1) and Seq(Type2) entries!
My problem now is to get the Seq(Type1) and Seq(Type2) out of the Future and stuff that into the MyResultType case class?
You could refer to this question you asked
Scala transforming a Seq with Future
so you get the
val results2: Future[Seq([Iterable[Type1], [Iterable[Type2])] = ???
and then call await on it and you have no Futures at all, you can do what you want.
I hope I understood the question correctly.
Oh and by the way you should map myTypeOption instead of checking if it's defined and returning None if it's not
if (myTypeOption.isDefined) {
Some(x)
} else {
None
}
can be simply replaced with
myTypeOption.map { _ => // ignoring what actually was inside option
x // return whatever you want, without wrapping it in Some
}
If I understood your question correctly, then this should do the trick.
def myMethod(typeId: Long, timeInterval: Interval): Option[Seq[MyResultType]] = async {
// 1. check if I can find an entity in the database for typeId
val myTypeOption = await(db.run(findMyTypeById(typeId))) // I'm getting the headOption on this result
if (myTypeOption.isDefined) {
// the interval gets split and assume that I get a List of thse intervals
val intervalList = splitInterval(interval)
// for each of the interval in the intervalList, I do database look up
val results: Seq[(Future[Seq[Type1]], Future[Seq[Type2]])] = for {
interval <- intervalList
} yield {
(getType1Entries(interval), getType2Entries(interval))
}
// best way to work with the results so that I can return MyResultType
Some(
await(
Future.sequence(
results.map{
case (l, r) =>
l.zip(r).map{
case (vl, vr) => MyResultType(typeId, vl, vr)
}
})))
}
else {
None
}
}
There are two parts to your problem, 1) how to deal with two dependent futures, and 2) how to extract the resulting values.
When dealing with dependent futures, I normally compose them together:
val future1 = Future { 10 }
val future2 = Future { 20 }
// results in a new future with type (Int, Int)
val combined = for {
a <- future1
b <- future2
} yield (a, b)
// then you can use foreach/map, Await, or onComplete to do
// something when your results are ready..
combined.foreach { ((a, b)) =>
// do something with the result here
}
To extract the results I generally use Await if I need to make a synchronous response, use _.onComplete() if I need to deal with potential failure, and use _.foreach()/_.map() for most other circumstances.