Scala Futures with DB - postgresql

Question

I'm writing code in scala/play with anorm/postgres for match generation based on users profiles. The following code works, but I've commented out the section that is causing problems, the while loop. I noticed while running it that the first 3 Futures seem to work synchronously but the problem comes when I'm retrieving the count of rows in the table in the fourth step.
The fourth step returns the count before the above insert's actually happened. As far as I can tell, steps 1-3 are being queued up for postgres synchronously, but the call to retrieve the count seems to return BEFORE the first 3 steps complete, which makes no sense to me. If the first 3 steps get queued up in the correct order, why wouldn't the fourth step wait to return the count until after the inserts happen?
When I uncomment the while loop, the match generation and insert functions are called until memory runs out, as the count returned is continually below the desired threshold.
I know the format itself is subpar, but my question is not about how to write the most elegant scala code, but merely how to get it to work for now.
def matchGeneration(email:String,itNum:Int) = {
var currentIterationNumber = itNum
var numberOfMatches = MatchData.numberOfCurrentMatches(email)
while(numberOfMatches < 150){
Thread.sleep(25000)//delay while loop execution time
generateUsers(email) onComplete {
case(s) => {
print(s">>>>>>>>>>>>>>>>>>>>>>>>>>>STEP 1")
Thread.sleep(5000)//Time for initial user generation to take place
genDemoMatches(email, currentIterationNumber) onComplete {
case (s) => {
print(s">>>>>>>>>>>>>>>>>>>>>>>>>>>STEP 2")
genIntMatches(email,currentIterationNumber) onComplete {
case(s) => {
print(s">>>>>>>>>>>>>>>>>>>>>>>>>>>STEP 3")
genSchoolWorkMatches(email,currentIterationNumber) onComplete {
case(s) => {
Thread.sleep(10000)
print(s">>>>>>>>>>>>>>>>>>>>>>>>>>>STEP 4")
incrementNumberOfMatches(email) onComplete {
case(s) => {
currentIterationNumber+=1
println(s"current number of matches: $numberOfMatches")
println(s"current Iteration: $currentIterationNumber")
}
}
}
}
}
}
}
}
}
}
//}
}
The match functions are defined as futures, such as :
def genSchoolWorkMatches(email:String,currentIterationNumber:Int):Future[Unit]=
Future(genUsersFromSchoolWorkData(email, currentIterationNumber))
genUsersFromSchoolWorkData(email:String) follows the same form as the other two. It is a function that initially gets all the school/work fields that a user has filled out in their profile ( SELECT major FROM school_work where email='$email') and it generates a dummyUser that contains one of those fields in common with this user of email:String. It would take about 30-40 lines of code to print this function so I can explain it further if need be.
I have edited my code, the only way I found so far to get this to work was by hacking it with Thread.sleep(). I think the problem may lie with anorm
as my Future logic constructs did work as I expected, but the problem lies in the inconsistency of when writes occur versus what the read returns. The numberOfCurrentMatches(email:String) function returns the number of matches as it is a simple SELECT count(email) from table where email='$email'. The problem is that sometimes after inserting 23 matches the count returns as 0, then after a second iteration it will return 46. I assumed that the onComplete() would bind to the underlying anorm function defined with DB.withConnection() but apparently it may be too far removed to accomplish this. I am not really sure at this point what to research or look up further to try to get around this problem, rather than writing a separate sort of supervisor function to return at a value closer to 150.
UPDATE
Thanks to the advice of user's here, and trying to understand Scala's documentation at this link: Scala Futures and Promises
I have updated my code to be a bit more readable and scala-esque:
def genMatchOfTypes(email:String,iterationNumber:Int) = {
genDemoMatches(email,iterationNumber)
genIntMatches(email,iterationNumber)
genSchoolWorkMatches(email,iterationNumber)
}
def matchGeneration(email:String) = {
var currentIterationNumber = 0
var numberOfMatches = MatchData.numberOfCurrentMatches(email)
while (numberOfMatches < 150) {
println(s"current number of matches: $numberOfMatches")
Thread.sleep(30000)
generateUsers(email)
.flatMap(users => genMatchOfTypes(email,currentIterationNumber))
.flatMap(matches => incrementNumberOfMatches(email))
.map{
result =>
currentIterationNumber += 1
println(s"current Iteration2: $currentIterationNumber")
numberOfMatches = MatchData.numberOfCurrentMatches(email)
println(s"current number of matches2: $numberOfMatches")
}
}
}
I still am heavily dependent upon the Thread.sleep(30000) to provide enough time to run through the while loop before it tries to loop back again. It's still an unwieldy hack. When I uncomment the Thread.sleep()
my output in bash looks like this:
users for match generation createdcurrent number of matches: 0
[error] c.MatchDataController - here is the list: jnkj
[error] c.MatchDataController - here is the list: hbhjbjjnkjn
current number of matches: 0
current number of matches: 0
current number of matches: 0
current number of matches: 0
current number of matches: 0
This of course is a truncated output. It runs like this over and over until I get errors about too many open files and the JVM/play server crashes entirely.

Answer 1

One solution is to use Future.traverse for known iteration count
Implying
object MatchData {
def numberOfCurrentMatches(email: String) = ???
}
def generateUsers(email: String): Future[Unit] = ???
def incrementNumberOfMatches(email: String): Future[Int] = ???
def genDemoMatches(email: String, it: Int): Future[Unit] = ???
def genIntMatches(email: String, it: Int): Future[Unit] = ???
def genSchoolWorkMatches(email: String, it: Int): Future[Unit] = ???
You can write code like
def matchGeneration(email: String, itNum: Int) = {
val numberOfMatches = MatchData.numberOfCurrentMatches(email)
Future.traverse(Stream.range(itNum, 150 - numberOfMatches + itNum)) { currentIterationNumber => for {
_ <- generateUsers(email)
_ = print(s">>>>>>>>>>>>>>>>>>>>>>>>>>>STEP 1")
_ <- genDemoMatches(email, currentIterationNumber)
_ = print(s">>>>>>>>>>>>>>>>>>>>>>>>>>>STEP 2")
_ <- genIntMatches(email, currentIterationNumber)
_ = print(s">>>>>>>>>>>>>>>>>>>>>>>>>>>STEP 3")
_ <- genSchoolWorkMatches(email, currentIterationNumber)
_ = Thread.sleep(15000)
_ = print(s">>>>>>>>>>>>>>>>>>>>>>>>>>>STEP 4")
numberOfMatches <- incrementNumberOfMatches(email)
_ = println(s"current number of matches: $numberOfMatches")
_ = println(s"current Iteration: $currentIterationNumber")
} yield ()
}
Update
If you urged to check some condition each time, one way is to use cool monadic things from scalaz library. It have definition of monad for scala.Future so we can replace word monadic with asynchronous when we want to
For example StreamT.unfoldM can create conditional monadic(asynchronous) loop, even if we don need elements of resulting collection we still can use it just for iteration.
Lets define your
def generateAll(email: String, iterationNumber: Int): Future[Unit] = for {
_ <- generateUsers(email)
_ <- genDemoMatches(email, iterationNumber)
_ <- genIntMatches(email, iterationNumber)
_ <- genSchoolWorkMatches(email, iterationNumber)
} yield ()
Then iteration step
def generateStep(email: String, limit: Int)(iterationNumber: Int): Future[Option[(Unit, Int)]] =
if (MatchData.numberOfCurrentMatches(email) >= limit) Future(None)
else for {
_ <- generateAll(email, iterationNumber)
_ <- incrementNumberOfMatches(email)
next = iterationNumber + 1
} yield Some((), next)
Now our resulting function simplifies to
import scalaz._
import scalaz.std.scalaFuture._
def matchGeneration(email: String, itNum: Int): Future[Unit] =
StreamT.unfoldM(0)(generateStep(email, 150) _).toStream.map(_.force: Unit)
It looks like synchronous method MatchData.numberOfCurrentMatches is reacting on your asynchronous modification inside the incrementNumberOfMatches. Note that generally it could lead to disastrous results and you probably need to move that state inside some actor or something like that