I am new to Scala and having a hard time to handle Future.
I am calling a method which returns Future[Seq[String]]
def getBookingIds(headers: Seq[(String, String)]): Future[Seq[String]] = {
val responseF = legacyService.get(new URL(Settings.Legacy.getBookingURL), headers)
responseF.map { response =>
response.extract[Seq[String]]
}
}
I am calling above method here and need to loop through this list and check if booking exist.
def verifyUserBooking(bookingId: Int, headers: Seq[(String, String)]): Unit = {
bookedCruisesService.getBookingIds(headers).onComplete {
case Success(s) => println()
case Failure(ex) => println(ex.toString)
case _ => println("what the hell happened")
}
I am getting here JsonParseException. Why?
I have used different methods to loop through future list, but I keep getting Future in debug more.
Can someone help me to solve this issue, please.
Thank you.
It sounds obvious, but you are getting a JSON parsing error because Play can't parse the JSON to the type that you specify. It is nothing to do with how you are handling the Future, which looks OK.
The type for extract is Seq[String], so to match this the JSON would have to be an array of strings, e.g. ["A", "B", "C"]. Check that this is what that URL is returning by printing the response before extracting it inside the map call, or by running it under a debugger and inspecting the values.
Related
I am trying to read incremental data from my data source using Scala-Spark. Before hitting the source tables, I am trying to calculate the min & max of partition column that I use in my code in a Future which is present in a class: GetSourceMeta as given below.
def getBounds(keyIdMap:scala.collection.mutable.Map[String, String]): Future[scala.collection.mutable.Map[String, String]] = Future {
var boundsMap = scala.collection.mutable.Map[String, String]()
keyIdMap.keys.foreach(table => if(!keyIdMap(table).contains("Invalid")) {
val minMax = s"select max(insert_tms) maxTms, min(insert_tms) minTms from schema.${table} where source='DB2' and key_id in (${keyIdMap(table)})"
println("MinMax: " + minMax)
val boundsDF = spark.read.format("jdbc").option("url", con.getConUrl()).option("dbtable", s"(${minMax}) as ctids").option("user", con.getUserName()).option("password", con.getPwd()).load()
try {
val maxTms = boundsDF.select("minTms").head.getTimestamp(0).toString + "," + boundsDF.select("maxTms").head.getTimestamp(0).toString
println("Bounds: " + maxTms)
boundsMap += (table -> maxTms)
} catch {
case np: java.lang.NullPointerException => { println("No data found") }
case e: Exception => { println(s"Unknown exception: $e") }
}
}
)
boundsMap.foreach(println)
boundsMap
}
I am calling the above method in my main method as:
object LoadToCopyDB {
val conf = new SparkConf().setAppName("TEST_YEAR").set("some parameters")
def main(args: Array[String]): Unit = {
val spark = SparkSession.builder().config(conf).master("yarn").enableHiveSupport().config("hive.exec.dynamic.partition", "true").config("hive.exec.dynamic.partition.mode", "nonstrict").getOrCreate()
val gsm = new GetSourceMeta()
val minMaxKeyMap = gsm.getBounds(keyIdMap).onComplete {
case Success(values) => values.foreach(println)
case Failure(f) => f.printStackTrace
}
.
.
.
}
Well, the onComplete didn't print any values so I used andThen as below and that didn't help as well.
val bounds: Future[scala.collection.mutable.Map[String, String]] = gpMetaData.getBounds(incrementalIds) andThen {
case Success(outval) => outval.foreach(println)
case Failure(e) => println(e)
}
Earlier the main thread exits without letting the Future: getBounds execute. Hence I couldn't find any println statements from the Future displayed on the terminal. I found out that I need to keep the main thread Await inorder to complete the Future. But when I use Await in main along with onComplete:
Await.result(bounds, Duration.Inf)
The compiler gives an error:
Type mismatch, expected: Awaitable[NotInferedT], actual:Unit
If I declare the val minMaxKeyMap as Future[scala.collection.mutable.Map[String, String] the compiler says: Expression of type Unit doesn't conform to expected type Future[mutable.map[String,String]]
I tried to print the values of bounds after the Await statement but that just prints an empty Map.
I couldn't understand how can to fix this. Could anyone let me know what do I do to make the Future run properly ?
In this kind of cases, is always better to follow the types. The method onComplete only returns Unit, it won´t return a future hence it can´t be passed using Await.
In case you want to return a Future of any type you will have to map or flatmap the value and return an option, for example. In this case, does not matter what you return, you only want Await method to wait for this result and print a trace. You can treat the possible exception in the recover. It would be like that in your code:
val minMaxKeyMap:Future[Option[Any] = gsm.getBounds(keyIdMap).map { values =>
values.foreach(println)
None
}.recover{
case e: Throwable =>
e. printStackTrace
None
}
Note that the recover part has to return an instance of the type.
After that, you can apply the Await to the Future, and you will get the results printed. Is not the prettiest solution but it will work in your case.
I have the following function:
private def constraintToJson(req: => Request[IO])
: EitherT[IO, Throwable, Unit]
= {
val err: EitherT[IO, Throwable, Unit] = EitherT.fromEither[IO](Left(new Exception("Not JSON format request.")))
req.contentType match {
case Some(s) =>
if (s != `Content-Type`(MediaType.`application/json`))
err
else
EitherT.fromEither[IO](Right(()))
case None =>
err
}
}
The question is, it is wrong to return a Unit if it is right or there is another choice?
I think that returning Unit (wrapped into Either/EitherT) might be OK if this is a kind of the final step of your computation and it actually does not produce any output. In other cases (most probably including yours) you should return some value for the successful case so that you can chain it further. So the main question is: how constraintToJson is supposed to be used? The obvious suspect for your case is returning Request or its body if the Content-Type matches JSON because this is most probably the data that will be used by the next step.
Simplified Question
Let me abstract your problem first by removing IO.
The question is whether a method with a signature like the following is useful:
def validate[A](a: Request[A]): Either[Error,()] =
if(isOk(a.someProperty)) Left(()) else Right("invalid")
Noting that Either[A,()] =:= Option[A], this method checks for the presence of any Error in the value of the request, and returns this Error, if detected.
What could be done with such a method?
Check and exit
We could write a program that checks a Request:
val someRequest: Request[X] = getRequestFrom("somewhere")
validate(someRequest).toLeft.foreach{error =>
println("invalid request: " + error
System.exit(-1)
}
System.exit(0)
In this case validate has a useful signature.
Check and go on
We could also validate a request, and then go on to execute it:
val someRequest: Request[X] = getRequestFrom("somewhere")
validate(someRequest) match {
case Left(error) =>
println("invalid request: " + error)
System.exit(-1)
case Right(_) =>
println("executing request: " + someRequest.execute)
System.exit(0)
}
While this code works perfectly well, it throws away an argument (_) in the second match clause. This is an indicator of some bad design. The code within the case Right(_) takes the value someRequest from outside, and treats it as a valid request because of the way you wrote your code.
If we would call someReuqest.execute within the Left case, we would probably end up with a runtime exception when executing an invalid Request.
Depending on our desired level of rigor, this can be a code smell.
We can improve the code in the following way.
Using a Non-Unit return type
We can circumvent returning Unit by simply returning the checked argument.
This appears to be a bit redundant, and we shall later see how to turn the returned value into something useful.
But let's first look at this option.
def validateTyped[A](a: Request[A]): Either[Error,Request[A]] =
if(isOk(a.someProperty)) Left(a) else Right("invalid")
Then we can write the Check and go on code as
val someRequest: Request[X] = getRequestFrom("somewhere")
validate(someRequestTyped) match {
case Left(error) =>
println("invalid request: " + error)
System.exit(-1)
case Right(validatedRequest) =>
//we now execute the validated request
println("executing request: " + validatedRequest.execute)
System.exit(0)
}
Now this improves the code a bit.
We don't throw away a returned value anymore in the second match clause.
We can execute the validatedRequest, because we've read in the documentation of validateRequest that a returned Right value is well formed, and it may be executed without error.
Looks good, right?
But we can do better.
Prevent execution of malformed requests
We can still improve it further by totally preventing executing a malformed Request by changing our Request type.
case class Request[A](query: String){
def validate: Either[Error,ValidRequest[A]] = ???
}
case class ValidRequest[A](query: String){
def execute: A
}
With this code, it is not possible to call Request.execute anymore, as a validation now becomes mandatory.
Otherwise the code will not compile.
We also note that the Right value returned by Request.validate is now needed.
It has become the only source of obtaining a ValidRequest.
Now it is impossible to call execute a Request that isn't validated, yet:
val someRequest: Request[X] = getRequestFrom("somewhere")
someRequest.validate match {
case Left(error) =>
println("invalid request: " + error)
System.exit(-1)
case Right(validatedRequest) =>
println("executing request: " + validatedRequest.execute)
System.exit(0)
}
Conclusion
In this way we have turned a strange looking method that returned Unit into a method that returns a meaningful and needed value.
In addition it has become impossible to execute a Request that has not been validated successfully.
I am new to Scala and Play!, but have a reasonable amount of experience of building webapps with Django and Python and of programming in general.
I've been doing an exercise of my own to try to improve my understanding - simply pull some records from a database and output them as a JSON array. I'm trying to use the Enumarator/Iteratee functionality to do this.
My code follows:
TestObjectController.scala:
def index = Action {
db.withConnection { conn=>
val stmt = conn.createStatement()
val result = stmt.executeQuery("select * from datatable")
logger.debug(result.toString)
val resultEnum:Enumerator[TestDataObject] = Enumerator.generateM {
logger.debug("called enumerator")
result.next() match {
case true =>
val obj = TestDataObject(result.getString("name"), result.getString("object_type"),
result.getString("quantity").toInt, result.getString("cost").toFloat)
logger.info(obj.toJsonString)
Future(Some(obj))
case false =>
logger.warn("reached end of iteration")
stmt.close()
null
}
}
val consume:Iteratee[TestDataObject,Seq[TestDataObject]] = {
Iteratee.fold[TestDataObject,Seq[TestDataObject]](Seq.empty[TestDataObject]) { (result,chunk) => result :+ chunk }
}
val newIteree = Iteratee.flatten(resultEnum(consume))
val eventuallyResult:Future[Seq[TestDataObject]] = newIteree.run
eventuallyResult.onSuccess { case x=> println(x)}
Ok("")
}
}
TestDataObject.scala:
package models
case class TestDataObject (name: String, objtype: String, quantity: Int, cost: Float){
def toJsonString: String = {
val mapper = new ObjectMapper()
mapper.registerModule(DefaultScalaModule)
mapper.writeValueAsString(this)
}
}
I have two main questions:
How do i signal that the input is complete from the Enumerator callback? The documentation says "this method takes a callback function e: => Future[Option[E]] that will be called each time the iteratee this Enumerator is applied to is ready to take some input." but I am unable to pass any kind of EOF that I've found because it;s the wrong type. Wrapping it in a Future does not help, but instinctively I am not sure that's the right approach.
How do I get the final result out of the Future to return from the controller view? My understanding is that I would effectively need to pause the main thread to wait for the subthreads to complete, but the only examples I've seen and only things i've found in the future class is the onSuccess callback - but how can I then return that from the view? Does Iteratee.run block until all input has been consumed?
A couple of sub-questions as well, to help my understanding:
Why do I need to wrap my object in Some() when it's already in a Future? What exactly does Some() represent?
When I run the code for the first time, I get a single record logged from logger.info and then it reports "reached end of iteration". Subsequent runs in the same session call nothing. I am closing the statement though, so why do I get no results the second time around? I was expecting it to loop indefinitely as I don't know how to signal the correct termination for the loop.
Thanks a lot in advance for any answers, I thought I was getting the hang of this but obviously not yet!
How do i signal that the input is complete from the Enumerator callback?
You return a Future(None).
How do I get the final result out of the Future to return from the controller view?
You can use Action.async (doc):
def index = Action.async {
db.withConnection { conn=>
...
val eventuallyResult:Future[Seq[TestDataObject]] = newIteree.run
eventuallyResult map { data =>
OK(...)
}
}
}
Why do I need to wrap my object in Some() when it's already in a Future? What exactly does Some() represent?
The Future represents the (potentially asynchronous) processing to obtain the next element. The Option represents the availability of the next element: Some(x) if another element is available, None if the enumeration is completed.
I'm using Scala to make HTTP GET requests to an API (Play Framework's WS, to be exact) which responds with a JSON response that looks like;
{
data: [
{text: "Hello there", id: 1},
{text: "Hello there again", id: 2}
],
next_url: 'http://request-this-for-more.com/api?page=2' //optional
}
So, the next_url field in the returned JSON may or may not be present.
What my method needs to do is start with calling the first URL, check if the response has a next_url and then do a GET on that. In the end, I should have all the data fields from the responses combined into one single future of all the data fields. I terminate when the response has no next_url present in it.
Now, doing this in a blocking way is easier, but I don't want to do that. What is the best way do tackle a problem like this?
There's probably a method to do this somewhere in scalaz, but if you don't know a specific solution it's usually possible to construct one with recursion and flatMap. Something like:
//Assume we have an async fetch method that returns Result and Option of next Url
def fetch(url: Url): Future[(Result, Option[Url])] = ...
//Then we can define fetchAll with recursion:
def fetchAll(url: Url): Future[Vector[Result]] =
fetch(url) flatMap {
case (result, None) => Future.successful(Vector(result))
case (result, Some(nextUrl)) =>
fetchAll(nextUrl) map {results => result +: results}
}
(Note that this uses a stack frame for each call - if you want to do thousands of fetches then we need to write it a little more carefully so that it's tail-recursive)
the Future.flatMap method exists exaclty for cases like that
Suppose you have such things:
case class Data(...)
def getContent(url:String):Future[String]
def parseJson(source:String):Try[JsValue]
def getData(value: JsValue):Seq[Data]
and JsValue type have methods inspired by play json library
def \ (fieldName: String): JsValue
def as[T](implicit ...):T //probably throwing exception
you could compose final result like
def innerContent(url:String):Future[Seq[Data]] = for {
first <- getContent(url)
json <- Future.fromTry(parseJson(first))
nextUrlAttempt = Try((json \ "next_url").as[String])
dataAttempt = Try(getData(json \ "data"))
data <- Future.fromTry(dataAttempt)
result <- nextUrlAttempt match {
case Success(nextUrl) => innerContent(nextUrl)
case Failure(_) => Future.successful(Seq())
} yield data ++ result
Also check out libraries that are targeted for complex asynchronous streams like your one:
play iteratees
scalaz iteratees
scalaz stream
I want to implement a BodyParser which parses and validates request.body, it's based on parse.json and currently looks like this:
def parseModel[A](implicit reads: Reads[A]): BodyParser[JsResult[A]] =
parse.json.map(_.validate[A])
The problem is it is currently of type BodyParser[JsResult[A]] while I want it to be of type BodyParser[A]. In case of JsError I want it basically to return 400 Bad Request with validation errors.
In Play API docs I can't find a method which allows me to inspect result of previous body parser and return a result or continue to a controller.
A BodyParser, after parsing the body, produces an Either[SimpleResult, A], where SimpleResult is an error result that should be returned immediately instead of processing the action. The convenience methods on BodyParser don't let you do this, so instead, create a new body parser that delegates to the JSON body parser:
def validateJson[A](implicit reads: Reads[A]) = new BodyParser[A] {
def apply(request: RequestHeader) = parse.json(request).map(_.right.flatMap { json =>
json.validate[A].asEither.left.map(e => BadRequest(JsError.toFlatJson(e)))
})
}
You can see here that we're mapping the result of the parse, and then taking the right value (a successful parse, will be JsValue), and calling flatMap on it. Our flatMap method converts the JsResult from validate to an Either[JsError, A], so we're halfway there with A, after that we map the left JsError to SimpleResult, and we're good to go.
Ok, I've implemented the desired behaviour via a method which produces an Action:
def validateJson[A](implicit reads: Reads[A]) =
parse.json.map(_.validate[A])
def ModelAction[A](action: A => Result)(implicit r: Reads[A]) =
Action(validateJson[A]) { request =>
request.body match {
case JsSuccess(model, _) => action(model)
case JsError(e) => BadRequest(JsError.toFlatJson(e))
}
}
I can use it like this:
def create = ModelAction[MyModel] { model =>
???
}
I'm still interested if it's possible to do the same with BodyParser and if I need to do so or it's better as it is now?