I am working with a paginated API and would like to read the Json for each page dynamically. I have implemented this using a while loop and would like to break out of it when I have read the last page's Json
var pageNumber = 1
val pageSize = 1000
while (pageNumber >= 1) {
println("Inside the while loop!" + pageNumber)
url = url.concat("OWNER_MI_KEY&dataset_version=v1.1&pageNumber=$pageNumber&pageSize=$pageSize")
val apiResponse = MakePostHttpRestCalls(url) // makes request to external rest endpoint
if (apiResponse != ""){
println("Got api response")
println("apiResponse :" + apiResponse)
val apiJson = parseApiResponse(apiResponse)//parses the json response
println("apiJson :" + apiJson)
if (apiJson.length == 0){
pageNumber = pageNumber + 1
}
}
}
Any ideas on how I could achieve this in my code?
Further to my comment, here is a rough idea of how this might look as a recursive function, which is the functional way of doing this kind of thing that Scala is designed for.
def readResponsePages(pageSize: Int) = {
def loop(pageNumber: Int, responses: List[String]): List[String] = {
println("Inside the loop!" + pageNumber)
val thisUrl = url.concat(
s"OWNER_MI_KEY&dataset_version=v1.1&pageNumber=$pageNumber&pageSize=$pageSize"
)
MakePostHttpRestCalls(thisUrl) match {
case "" =>
responses.reverse
case apiResponse =>
println("Got api response")
println("apiResponse :" + apiResponse)
val apiJson = parseApiResponse(apiResponse) //parses the json response
println("apiJson :" + apiJson)
loop(pageNumber + 1, apiJson +: responses)
}
}
loop(1, Nil)
}
val responsePages = readResponsePages(1000)
Related
Trying to execute a function in a given time frame, but if computation fails by TimeOut get a partial result instead of an empty exception.
The attached code solves it.
The timedRun function is from Computation with time limit
Any better approach?.
package ga
object Ga extends App {
//this is the ugly...
var bestResult = "best result";
try {
val result = timedRun(150)(bestEffort())
} catch {
case e: Exception =>
print ("timed at = ")
}
println(bestResult)
//dummy function
def bestEffort(): String = {
var res = 0
for (i <- 0 until 100000) {
res = i
bestResult = s" $res"
}
" " + res
}
//This is the elegant part from stackoverflow gruenewa
#throws(classOf[java.util.concurrent.TimeoutException])
def timedRun[F](timeout: Long)(f: => F): F = {
import java.util.concurrent.{ Callable, FutureTask, TimeUnit }
val task = new FutureTask(new Callable[F]() {
def call() = f
})
new Thread(task).start()
task.get(timeout, TimeUnit.MILLISECONDS)
}
}
I would introduce a small intermediate class for more explicitly communicating the partial results between threads. That way you don't have to modify non-local state in any surprising ways. Then you can also just catch the exception within the timedRun method:
class Result[A](var result: A)
val result = timedRun(150)("best result")(bestEffort)
println(result)
//dummy function
def bestEffort(r: Result[String]): Unit = {
var res = 0
for (i <- 0 until 100000) {
res = i
r.result = s" $res"
}
r.result = " " + res
}
def timedRun[A](timeout: Long)(initial: A)(f: Result[A] => _): A = {
import java.util.concurrent.{ Callable, FutureTask, TimeUnit }
val result = new Result(initial)
val task = new FutureTask(new Callable[A]() {
def call() = { f(result); result.result }
})
new Thread(task).start()
try {
task.get(timeout, TimeUnit.MILLISECONDS)
} catch {
case e: java.util.concurrent.TimeoutException => result.result
}
}
It's admittedly a bit awkward since you don't usually have the "return value" of a function passed in as a parameter. But I think it's the least-radical modification of your code that makes sense. You could also consider modeling your computation as something that returns a Stream or Iterator of partial results, and then essentially do .takeWhile(notTimedOut).last. But how feasible that is really depends on the actual computation.
First, you need to use one of the solution to recover after the future timed out which are unfortunately not built-in in Scala:
See: Scala Futures - built in timeout?
For example:
def withTimeout[T](fut:Future[T])(implicit ec:ExecutionContext, after:Duration) = {
val prom = Promise[T]()
val timeout = TimeoutScheduler.scheduleTimeout(prom, after)
val combinedFut = Future.firstCompletedOf(List(fut, prom.future))
fut onComplete{case result => timeout.cancel()}
combinedFut
}
Then it is easy:
var bestResult = "best result"
val expensiveFunction = Future {
var res = 0
for (i <- 0 until 10000) {
Thread.sleep(10)
res = i
bestResult = s" $res"
}
" " + res
}
val timeoutFuture = withTimeout(expensiveFunction) recover {
case _: TimeoutException => bestResult
}
println(Await.result(timeoutFuture, 1 seconds))
I want to ban a URL from varnish. What I do first is collect all the healthy IP's from consul.
private def nodeInfo:Future[List[NodeInfo]] = {
val request = HttpRequest(method = HttpMethods.GET, uri = consulUrl)
Future.successful {
Http().singleRequest(request).flatMap(response =>
response.status match {
case OK => Unmarshal(response).to[List[NodeInfo]]
case _ =>
response.entity.toStrict(5.seconds).flatMap { entity =>
val body = entity.data.decodeString("UTF-8")
log.warning(errorMessage(request, response, body))
Future.failed(new IOException(errorMessage(response, body)))
}
})
}.flatMap(value => value)
}
This works as expected.
With the help of for comprehension I want to loop through all of them.
def banFromCache(toBanUrl:String): Future[String] = {
for {
nodes <- nodeInfo
result <- loopNodes(nodes, toBanUrl)
} yield result
}
With a foreach loop I send HttpRequest and get HttpResponses for each. But caused by the Future functionality the result yield is done before the request are completed.
private def loopNodes(nodes:List[NodeInfo], toBanUrl:String):Future[String] = Future {
val banResult = new ListBuffer[String]
nodes.foreach(node => {
banAllHealthy(node, toBanUrl).onComplete {
case Failure(err) =>
banResult += node.Node.Address + " " + err.getMessage
log.error("Request failed: " + node.Node.Address + " " + err.getMessage)
case Success(res) =>
banResult += node.Node.Address + " " + res.toString
log.info("Request success: " + node.Node.Address + " " + res.toString)
}
})
banResult.toList.toString()
}
private def banAllHealthy(nodeInfo:NodeInfo, toBanUrl: String):Future[HttpResponse] = {
def request(): Future[HttpResponse] =
Http().singleRequest(HttpRequest(method = HttpMethods.GET, uri = "http://localhost:9000/healthcheck"))
//Http().singleRequest(HttpRequest(method = HttpMethods.GET, uri = "http://" + nodeInfo.Node.Address + "/" + toBanUrl))
val responseFuture: Future[HttpResponse] = request()
responseFuture
}
The route is quite simple here:
} ~ pathPrefix("ban") {
pathPrefix(Segment) { banpath =>
pathEndOrSingleSlash {
get {
complete(banFromCache(banpath).map(_.asJson))
}
}
}
Is there a way to show all the responses at once ?
To accumulate the result strings, stay within the context of a Future:
private def loopNodes(nodes: List[NodeInfo], toBanUrl: String): Future[String] = {
val futures: List[Future[String]] = nodes.map { node =>
banAllHealthy(node, toBanUrl)
.map(res => s"${node.Node.Address} ${res}")
.recover { case err => s"${node.Node.Address} ${err.getMessage}" }
}
Future.reduceLeft(futures)(_ + "\n" + _)
}
If you put the callbacks to handle the result of each future you will lost the control of the execution. With the foreach, each future lives in its own parallel execution. The parent future yields because it does not to wait for anything. I would suggest to not use the ListBuffer and use a more immutable style. Anyway, try to build the whole computation as just one future which encapsulates the whole computation:
private def loopNodes(nodes:List[NodeInfo], toBanUrl:String):Future[String] = {
nodes.map { node =>
// Creates a tuple to store current node and http result
// (Node, result, HttpResult)
(node, "", banAllHealthy(node, toBanUrl))
}.foldLeft(Future(""))((str, b) =>
b match {
case (node, str ,response) => {
// Each response will be transformed to string
(response map (result => str + " " + node.Node.Address + " " + "Success"))
// In case of node is not available its suppose that the HttpClient will raise an execption
.recover {
case err: Throwable =>
str + " " + node.Node.Address + " " + "Error " + err.getMessage
case _ =>
str + " " + node.Node.Address + " " + "Unknown Error"
}
}
})
}
I've got a question that concerns scoping in scala. I have a function like this:
def getElements(id: Int): Seq[Element] = {
var test = ""
dto.getElementIds(id).map {
elementIds => {
test += " hello "
elementIds.foreach(elementId => dto.getElement(elementId).map {
case Some(element) => test += " hi "
println("ThirdPrint: " + test)
})
println("SecondPrint: " + test)
}
}
println("FirstPrint: " + test)
}
Lets say "elementsIds" has a lenght of 2.
The console says:
FirstPrint:
SecondPrint: hello
ThirdPrint: hello hi
ThirdPrint: hello hi hi
Why is scala doing that? I would assume that third print is executed first. By the time I get to the "FirstPrint" the "hi"'s are gone. Why is the last line of code executed first?
I'm working with slick and Futures, does this have something to do with it? Thanks!
UPDATE
Thanks, works fine.
Is it possible to return a seq instead? Like this:
def getElements(id: Int): Future[Seq[Element]] = {
var mySequence: Seq[Element] = Seq()
val elementsIds: Future[Seq[Int]] = dto.getElementIds(id)
var test = ""
val elementsF = elementsIds.flatMap {
elementIds => {
test += " hello "
val idsAsElements: Seq[Future[Element]] = elementIds.map(elementId => dto.getElement(elementId).collect {
case Some(element) => mySequence = mySequence :+ element
})
val idsAsElementsF: Future[Seq[Element]] = Future.sequence(idsAsElements)
idsAsElementsF.onComplete(_ => println("SecondPrint: " + test))
idsAsElementsF
}
}
elementsF.onComplete(_ => println("FirstPrint: " + test))
elementsF
}
Is it possible to return "mySequence" whenever idsAsElements is "onComplete"?
object X {
import scala.concurrent.ExecutionContext.Implicits.global
case class Element()
object dto{
def getElementIds(i: Int): Future[Seq[Int]] = Future(Seq(1,2,3))
def getElement(i: Int): Future[Option[Element]] = Future(Some(Element()))
}
def main(args: Array[String]): Unit = {
getElements(0)
Thread.sleep(10000) // waiting logs
}
def getElements(id: Int): Future[Seq[Element]] = {
val elementsIds: Future[Seq[Int]] = dto.getElementIds(id)
var test = ""
val elementsF = elementsIds.flatMap {
elementIds => {
test += " hello "
val idsAsElements: Seq[Future[Element]] = elementIds.map(elementId => dto.getElement(elementId).collect {
case Some(element) => test += " hi "
println("ThirdPrint: " + test)
element
})
val idsAsElementsF: Future[Seq[Element]] = Future.sequence(idsAsElements)
idsAsElementsF.onComplete(_ => println("SecondPrint: " + test))
idsAsElementsF
}
}
elementsF.onComplete(_ => println("FirstPrint: " + test))
elementsF
}
}
output:
ThirdPrint: hello hi
ThirdPrint: hello hi hi
ThirdPrint: hello hi hi hi
SecondPrint: hello hi hi hi
FirstPrint: hello hi hi hi
Yes, that's not a good solution. You should avoid blocking in your code, except, maybe at the highest possible level (main method). Also, mutable state is bad, especially, when combined with concurrency (futures).
Your function should return a Future. Something like this will work (I am not sure if I guess the intent of your code correctly - your function was declared to return Seq[Element], but written to return a Unit ... I assume, that what you really wanted to return was the result of getElement calls for every id):
def getElements(id: Int): Future[Seq[Element]] = dto
.getElementIds(id)
.map { ids => ids.map(dto.getElement) }
.flatMap(Future.sequence)
.map(_.flatten)
I removed your printouts, because wasn't sure what was the purpose they serve (since the calls to dto.getElement are also happening in parallel, it is not obvious where and in which order you want those strings printed).
You could simulate your "expected output", by adding another transformation at the end for example:
.andThen { case Success(results) =>
val str = results.foldLeft("hello") { case (a,b) =>
println("ThirdPrint: " + a + " hi")
a + " hi"
}
println("SecondPrint: " + str)
println("FirstPrint: " + str
}
I met a problem that I cannot solve these days. I'm doing multiple http requests and in each response, it should have a Array[DTAnnotation]. I want to accumulate all the resulting list into one (this is not the problem here). My problem is that I cannot return the results from a WSResponse. What I try :
import mymodel.{DTFeatures, DTResponse, DTRequest, DTAnnotations}
def checkForSpike
(
awsKey : String,
collection : JSONCollection,
record : Record,
features : Array[DTFeatures]
) : Unit = {
val url = config.getString("url").getOrElse
val holder = WS.url(url)
val complexHolder =
holder.withHeaders(("Content-Type","application/json"))
// excepting result is List[Array[DTAnnotations]]
val test : List[Array[DTAnnotations]] =
for(feature <- features) yield {
val dtFeature = Json.stringify(Json.toJson(DTRequest(feature)))
val futureResponse = complexHolder.post(dtFeature)
Logger.info("Make the HTTP POST Request in " + (t1 - t0) + " msecs")
futureResponse.map { response =>
Logger.info("Get response in " + (System.currentTimeMillis - t1))
if(response.status == 200) {
response.json.validate[DTResponse].map { dtResponse =>
// I want to return this
dtResponse.annotations
}.recoverTotal { _ =>
Logger.debug("invalid json")
}
} else {
Logger.debug(Json.prettyPrint(Json.obj("status" -> response.status, "body" -> response.body)))
}
}
Await.result(futureResponse, 10.seconds)
}
}
Because the response is a Future, I try to add a Await to get the annotations but I have one error at the typing phase :
[error] found : Array[play.api.libs.ws.WSResponse]
[error] required: List[Array[DTAnnotation]]
How I can fix this ? Thank you !
There are a number of errors that avoid this to work. I add a version that compiles with your expected type and if you have questions I will answer in the comments.
def checkForSpike
(
awsKey: String,
collection: JSONCollection,
record: Record,
features: Array[DTFeatures]
): Unit = {
val url = config.getString("url").getOrElse
val holder = WS.url(url)
val complexHolder =
holder.withHeaders(("Content-Type", "application/json"))
// excepting result is List[Array[DTAnnotations]]
val test: List[Array[DTAnnotations]] =
for (feature <- features.toList) yield {
val dtFeature = Json.stringify(Json.toJson(DTRequest(feature)))
val futureResponse = complexHolder.post(dtFeature)
val futureAnnotations: Future[Array[DTAnnotations]] = futureResponse.map { response =>
if (response.status == 200) {
response.json.validate[DTResponse].map { dtResponse =>
// I want to return this
dtResponse.annotations
}.recoverTotal { _ =>
Logger.debug("invalid json")
??? // An Array response should be expected, maybe empty
}
} else {
Logger.debug(Json.prettyPrint(Json.obj("status" -> response.status, "body" -> response.body)))
??? // An Array response should be expected, maybe empty
}
}
Await.result(futureAnnotations, 10.seconds)
}
}
Issues:
Features must be converted to List if you expect a list to be
returned by the for comprehension
The map on future response returns
another future and this value should be used in the Await
To make sure the type of futureAnnotations is correct in all the branches the type should be valid
I have 3 futures of type Response. The first future returns a JsValue which defines if future 2 and future 3 shall be executed or only future 3 shall be executed.
Pseudocode:
If Future 1 then {future2 and future 3}
else future 3
Iam trying to do this in a play framwork action which means in order to use the result of the futures later I cant use onSuccess, onFailure and onComplete because all of them return Unit and not the actual JsValue from the last future.
I tried to do this with map() and andThen but I am a Scala noob and I guess i wasn't able to do it because I always just missed a little point. Here is my current approach which does not work!
def addSuggestion(indexName: String, suggestion: String) = Action.async {
val indexExistsQuery: IndexExistsQuery = new IndexExistsQuery(indexName);
val addSuggestionQuery: AddSuggestionQuery = new AddSuggestionQuery(indexName, suggestion)
val indexCreationQuery: CreateCompletionsFieldQuery = new CreateCompletionsFieldQuery(indexName)
val futureResponse: Future[Response] = for {
responseOne <- EsClient.execute(indexExistsQuery)
responseTwo <- EsClient.execute(indexCreationQuery) if (indexExistsQuery.getBooleanResult(indexExistsQuery.getResult(responseOne)))
responseThree <- EsClient.execute(addSuggestionQuery)
} yield responseThree
futureResponse.map { response =>
Ok("Feed title: " + (response.json))
}
I created some pseudocode:
checkIndexExists() map {
case true => Future.successful()
case false => createIndex()
} flatMap { _ =>
query()
}.map { response =>
Ok("Feed title: " + (response.json))
}.recover {
case _ => Ok("bla")
}
First you fire up the query if the index exists.
Then you map that future how to work with that Future[Boolean] if it successful. Since you use map, you kind of extract the Boolean. If the index exists, you just create a future that is already complete. If the index not exists you need to fire up the index creation command. Now you have the situation that you have nested Future's (Future[Future[Response]]). Using flatMap you remove one dimension, so that you only have Future[Response]. That can be mapped to a Play result.
Update (the implementation of MeiSign):
EsClient.execute(indexExistsQuery) map { response =>
if (indexExistsQuery.getBooleanResult(indexExistsQuery.getResult(response))) Future.successful(Response)
else EsClient.execute(indexCreationQuery)
} flatMap { _ =>
EsClient.execute(addSuggestionQuery)
} map { response: Response =>
Ok("Feed title: " + (response.json))
}
I found this solution but I dont think that it is a good solution because Iam using Await.result() which is a blocking operation.
If anyone knows how to refactor this code without blocking operations please let me know.
def addSuggestion(indexName: String, suggestion: String) = Action.async {
val indexExistsQuery: IndexExistsQuery = new IndexExistsQuery(indexName);
val addSuggestionQuery: AddSuggestionQuery = new AddSuggestionQuery(indexName, suggestion)
val indexCreationQuery: CreateCompletionsFieldQuery = new CreateCompletionsFieldQuery(indexName)
val indexExists: Boolean = indexExistsQuery.getBooleanResult(indexExistsQuery.getResult(Await.result(EsClient.execute(indexExistsQuery), 5.second)))
if (indexExists) {
EsClient.execute(addSuggestionQuery).map { response => Ok("Feed title: " + (response.json)) }
} else {
val futureResponse: Future[Response] = for {
responseTwo <- EsClient.execute(indexCreationQuery)
responseThree <- EsClient.execute(addSuggestionQuery)
} yield responseThree
futureResponse.map { response =>
{
Ok("Feed title: " + (response.json))
}
}.recover { case _ => Ok("bla") }
}
}