I have to query an API with pagination. But because the API is weird I have to query it until the response is empty (contains no items).
Actually it is done in Java with a do-while :
do {
objects = doQuery(from);
from += 200
} while ( !objects.isEmpty() )
But I would like to convert it in scala. My first idea is to use a stream with a step and takeWhile :
Stream.iterate(0)(_+200)
.map( from => doQuery(from) )
.takeWhile( objects -> !objects.isEmpty )
But another changes make doQuery returns a Future. Thus I cannot do the test in takeWhile and have no best idea on how to do it (maybe a recursive call).
Hopefully this new code will be into an Akka actor that will tell another for each object (no need to return anything)
This gives you Stream[Future[(Boolean, T)]], where the first time will be false as soon as the rest will be empty. Not sure how to do the takeWhile without blocking.
Stream.iterate(0)(_+200)
.scanLeft(Future((true, List[Int]())))({case (prev, from) =>
prev.flatMap({case (cont, _) =>
if(cont) {
doQuery(from).map(res => (res.isEmpty, res.toList))
} else {
Future((false, List()))
}
})
})
Related
I am trying to count rejects that were returned by RejectionHandler
I guess the way I doing this now is not the best one, or even incorrect. I am just trying to invoke the incremental method in my database, in each of the handled cases.
implicit def rejectionHandler: RejectionHandler =
RejectionHandler.newBuilder()
.handle {
case MissingCookieRejection(cookieName) =>
requestInfoEntry.incrementRjectedNum
complete(HttpResponse(BadRequest, entity = "No cookies, no service!!!"))
}
.handle {
case AuthorizationFailedRejection =>
requestInfoEntry.incrementRjectedNum
complete((Forbidden, "You're out of your depth!"))
}
.handle {
case ValidationRejection(msg, _) =>
requestInfoEntry.incrementRjectedNum
complete((InternalServerError, "That wasn't valid! " + msg))
}
.handleAll[MethodRejection] { methodRejections =>
requestInfoEntry.incrementRjectedNum//todo sideeffect ??
val names = methodRejections.map(_.supported.name)
complete((MethodNotAllowed, s"Can't do that! Supported: ${names mkString " or "}!"))
}
.handleNotFound {
requestInfoEntry.incrementRjectedNum
complete((NotFound, "Not here bldghad!"))
}
.result()
While I "visit my unfound page", Akka HTTP returns me a right response: "Not here bldghad!" every time I refresh browser on not existed web-page. But when I check my database, I see the only one increment. Can I do this way as I do at all? (I need to count successes too)
PS Maybe I need to work with status codes and do not use side effects. But what is the best place where I can do it? I have a lot of controllers and do not want to intercept this in every controller)
Can I globally intercept responses somewhere?
I've done it this way, hope it would be useful
def rejectionHandlerWithCounter: RejectionHandler = { (rejections: Seq[Rejection]) =>
requestInfoEntry.incrementRjectedNum
Some(complete((StatusCodes.Forbidden)))
}
I write a custom spark sink. In my addBatch method I use ForEachPartitionAsync which if I'm not wrong only makes the driver work asynchronously, returning a future.
val work: FutureAction[Unit] = rdd.foreachPartitionAsync { rows =>
val sourceInfo: StreamSourceInfo = serializeRowsAsInputStream(schema, rows)
val ackIngestion = Future {
ingestRows(sourceInfo) } andThen {
case Success(ingestion) => ackIngestionDone(partitionId, ingestion)
}
Await.result(ackIngestion, timeOut) // I would like to remove this line..
}
work onSuccess {
case _ => // move data from temporary table, report success of all workers
}
work onFailure{
//delete tmp data
case t => throw t.getCause
}
I can't find a way to run the worker nodes without blocking on the Await call, as if I remove them a success is reported to the work future object although the future didn't really finish.
Is there a way to report to the driver that all the workers finished
their asynchronous jobs?
Note: I looked at the foreachPartitionAsync function and it has only one implementation that expects a function that returns a Unit (i would've expected it to have another one returning a future or maybe a CountDownLatch..)
Consider this scala function that takes a server socket and turns it into a stream of sockets based on the accept method:
def accepting(ss: ServerSocket): Stream[Socket] =
try { Stream.cons(ss.accept(), accepting(ss)) }
catch { Stream.empty }
If this stream is evaluated left-right, it will function properly. But since the source of data is based on side-effects, the proper evaluation of an item depends on all previous items having been evaluated, and only previous items having been evaluated.
For example, if I were to skip the first 10 items in the stream and then take the next item, I would expect it to return the 11th socket to connect to this server socket, but it would actually return the first. So, what I need is a way to ensure that all previous items are evaluated before any given item is evaluated.
The best solution I've came up with is this:
def accepting(ss: ServerSocket): Stream[Socket] = {
def accepting(ss: ServerSocket, previous: Option[Socket]): Stream[Socket] = {
try {
lazy val current = ss.accept()
Stream.cons(current, accepting(ss, Some(current)))
} catch { Stream.empty }
}
accepting(ss, None)
}
But it's pretty ugly, and so my question is, is there a better way do achieve my goal?
Coming from a Java background, I have been trying to teach myself Scala for some time now. As part of that, I am doing a small pet project that exposes a HTTP endpoint that saves the registration numberof a vehicle against the owner and returns the status.
To give more context, I am using Slick as FRM which performs DB operations asynchronously and returns a Future.
Based on the output of this Future, I want to set the status variable to return back to the client.
Here, is the code
def addVehicleOwner(vehicle: Vehicle): String = {
var status = ""
val addFuture = db.run((vehicles returning vehicles.map(_.id)) += vehicle)
addFuture onComplete {
case Success(id) => {
BotLogger.info(LOGTAG, s"Vehicle registered at $id ")
status = String.format("Registration number - '%s' mapped to owner '%s' successfully", vehicle.registration,
vehicle.owner)
println(s"status inside success $status") //--------- (1)
}
case Failure(e: SQLException) if e.getMessage.contains("SQLITE_CONSTRAINT") => {
status = updateVehicleOwner(vehicle)
BotLogger.info(LOGTAG, s"Updated owner='${vehicle.owner}' for '${vehicle.registration}'")
}
case Failure(e) => {
BotLogger.error(LOGTAG, e)
status = "Sorry, unable to add now!"
}
}
exec(addFuture)
println(s"Status=$status") //--------- (2)
status
}
// Helper method for running a query in this example file:
def exec[T](sqlFuture: Future[T]):T = Await.result(sqlFuture, 1 seconds)
This was fairly simple in Java. With Scala, I am facing the following problems:
The expected value gets printed at (1), but (2) always prints empty string and same is what method returns. Can someone explain why?
I even tried marking the var status as #volatile var status, it still evaluates to empty string.
I know, that the above is not the functional way of doing things as I am muting state. What is the clean way of writing code for such cases.
Almost all the examples I could find described how to map the result of Success or handle Failure by doing a println. I want to do more than that.
What are some good references of small projects that I can refer to? Specially, that follow TDD.
Instead of relying on status to complete inside the closure, you can recover over the Future[T] which handle the exception if they occur, and always returns the result you want. This is taking advantage of the nature of expressions in Scala:
val addFuture =
db.run((vehicles returning vehicles.map(_.id)) += vehicle)
.recover {
case e: SQLException if e.getMessage.contains("SQLITE_CONSTRAINT") => {
val status = updateVehicleOwner(vehicle)
BotLogger.info(
LOGTAG,
s"Updated owner='${vehicle.owner}' for '${vehicle.registration}'"
)
status
}
case e => {
BotLogger.error(LOGTAG, e)
val status = "Sorry, unable to add now!"
status
}
}
val result: String = exec(addFuture)
println(s"Status = $result")
result
Note that Await.result should not be used in any production environment as it synchronously blocks on the Future, which is exactly the opposite of what you actually want. If you're already using a Future to delegate work, you want it to complete asynchronously. I'm assuming your exec method was simply for testing purposes.
I was hoping someone can briefly go over the various ways of consuming a service (this one just returns a string, normally it would be JSON but I just want to understand the concepts here).
My service:
def ping = Action {
Ok("pong")
}
Now in my Play (2.3.x) application, I want to call my client and display the response.
When working with Futures, I want to display the value.
I am a bit confused, what are all the ways I could call this method i.e. there are some ways I see that use Success/Failure,
val futureResponse: Future[String] = WS.url(url + "/ping").get().map { response =>
response.body
}
var resp = ""
futureResponse.onComplete {
case Success(str) => {
Logger.trace(s"future success $str")
resp = str
}
case Failure(ex) => {
Logger.trace(s"future failed")
resp = ex.toString
}
}
Ok(resp)
I can see the trace in STDOUT for success/failure, but my controller action just returns "" to my browser.
I understand that this is because it returns a FUTURE and my action finishes before the future returns.
How can I force it to wait?
What options do I have with error handling?
If you really want to block until feature is completed look at the Future.ready() and Future.result() methods. But you shouldn't.
The point about Future is that you can tell it how to use the result once it arrived, and then go on, no blocks required.
Future can be the result of an Action, in this case framework takes care of it:
def index = Action.async {
WS.url(url + "/ping").get()
.map(response => Ok("Got result: " + response.body))
}
Look at the documentation, it describes the topic very well.
As for the error-handling, you can use Future.recover() method. You should tell it what to return in case of error and it gives you new Future that you should return from your action.
def index = Action.async {
WS.url(url + "/ping").get()
.map(response => Ok("Got result: " + response.body))
.recover{ case e: Exception => InternalServerError(e.getMessage) }
}
So the basic way you consume service is to get result Future, transform it in the way you want by using monadic methods(the methods that return new transformed Future, like map, recover, etc..) and return it as a result of an Action.
You may want to look at Play 2.2 -Scala - How to chain Futures in Controller Action and Dealing with failed futures questions.