I am trying to consume a paginated REST call, currently doing something like:
def depaginateGetEnvironmentUuids(uri: Uri, filters: Seq[BasicNameValuePair], pageNumber: Int = 1): Future[Seq[UUID]] = {
val paginationFilters = Seq(new BasicNameValuePair("per_page", "1000"), new BasicNameValuePair("page", pageNumber.toString))
serviceManager.getAssignment(uri, filters ++ paginationFilters: _*).flatMap { cP =>
(cP.getStatus, cP.getBody.parseJson) match {
case (HttpStatus.SC_OK, JsArray.empty) =>
Future (Seq.empty[UUID] )
case (HttpStatus.SC_OK, json) =>
depaginateGetEnvironmentUuids (uri, filters, pageNumber + 1).map (_ ++ json.convertTo[Seq[AssignmentView]].map (se => se.environment.uuid).distinct)
case (_, error) =>
Future.failed(new Throwable(s"call to retrieve environment assignment: $error"))
}
}
}
Is there a better way of handling a RESTful service endpoint with pagination?
Related
I have an Akka-Http route where I'm calling Bing Search API. I want to add some extra records to the result, before sending the response to client. I'm using circe to handle json.
Here is the code which doesn't work but illustrates the idea:
val extraData = Map("key1"->"value1","key2"->"val2").asJson
val query = URLEncoder.encode(q, "utf8")
val responseFuture: Future[HttpResponse] = Http().singleRequest(
HttpRequest(uri = "https://api.cognitive.microsoft.com/bing/v7.0/search?q=" + query)
.withHeaders(RawHeader("Ocp-Apim-Subscription-Key", k1)))
val alteredResponse = responseFuture.map{ response => {
response.entity.toStrict(2 seconds) flatMap { e =>
e.dataBytes
.runFold(ByteString.empty) { case (acc, b) => acc ++ b }
.map(k => parse(k.utf8String)
match {
case Left(failure) => "Can't parse"
case Right(json) => Try {
json.hcursor.withFocus{
_.mapObject(x =>
x.add("extraData",extraData)
)
}
}}
)
}
}}
complete(alteredResponse)
Is it a good approach to take ? How can I get it to work ?
I ended up using flatMap and creating HttpResponse object manually:
val responseFuture: Future[HttpResponse] = Http().singleRequest(
HttpRequest(uri = "https://api.cognitive.microsoft.com/bing/v7.0/search?q=" + query)
.withHeaders(RawHeader("Ocp-Apim-Subscription-Key", k1)))
.flatMap{ response => {
response.entity.toStrict(2 seconds) flatMap { e =>
e.dataBytes
.runFold(ByteString.empty) { case (acc, b) => acc ++ b }
.map(k => parse(k.map(_.toChar).mkString)
match {
case Left(failure) => HttpResponse(
StatusCodes.OK,
List(),
HttpEntity("NO RESULTS".map(_.toByte).toArray),
HttpProtocol("HTTP/1.1")
)
case Right(json) => {
json.hcursor.withFocus{
_.mapObject(x =>
x.add("extraData",extraData)
)
}.top match {
case Some(jsn) => {
HttpResponse(
StatusCodes.OK,
List(headers.`Content-Type`(ContentType(MediaTypes.`application/json`,() => HttpCharsets.`UTF-8`))),
HttpEntity(jsn.noSpaces.toCharArray.map(_.toByte)),
HttpProtocol("HTTP/1.1")
)
}
}
}}
)
}
}}
complete(responseFuture)
When converting response Bytes from Bing, i tried to use .utf8String function, but it messed up the Json, so I eneded up parsing bytes with map.(_toChar).mkString.
If there is a better way of writing this code, it would be nice to see it. Thanks.
I'd like to use akka streams in order to pipe some json webservices together. I'd like to know the best approach to make a stream from an http request and stream chunks to another.
Is there a way to define such a graph and run it instead of the code below?
So far I tried to do it this way, not sure if it is actually really streaming yet:
override def receive: Receive = {
case GetTestData(p, id) =>
// Get the data and pipes it to itself through a message as recommended
// https://doc.akka.io/docs/akka-http/current/client-side/request-level.html
http.singleRequest(HttpRequest(uri = uri.format(p, id)))
.pipeTo(self)
case HttpResponse(StatusCodes.OK, _, entity, _) =>
val initialRes = entity.dataBytes.via(JsonFraming.objectScanner(Int.MaxValue)).map(bStr => ChunkStreamPart(bStr.utf8String))
// Forward the response to next job and pipes the request response to dedicated actor
http.singleRequest(HttpRequest(
method = HttpMethods.POST,
uri = "googl.cm/flow",
entity = HttpEntity.Chunked(ContentTypes.`application/json`,
initialRes)
))
case resp # HttpResponse(code, _, _, _) =>
log.error("Request to test job failed, response code: " + code)
// Discard the flow to avoid backpressure
resp.discardEntityBytes()
case _ => log.warning("Unexpected message in TestJobActor")
}
This should be a graph equivalent to your receive:
Http()
.cachedHostConnectionPool[Unit](uri.format(p, id))
.collect {
case (Success(HttpResponse(StatusCodes.OK, _, entity, _)), _) =>
val initialRes = entity.dataBytes
.via(JsonFraming.objectScanner(Int.MaxValue))
.map(bStr => ChunkStreamPart(bStr.utf8String))
Some(initialRes)
case (Success(resp # HttpResponse(code, _, _, _)), _) =>
log.error("Request to test job failed, response code: " + code)
// Discard the flow to avoid backpressure
resp.discardEntityBytes()
None
}
.collect {
case Some(initialRes) => initialRes
}
.map { initialRes =>
(HttpRequest(
method = HttpMethods.POST,
uri = "googl.cm/flow",
entity = HttpEntity.Chunked(ContentTypes.`application/json`, initialRes)
),
())
}
.via(Http().superPool[Unit]())
The type of this is Flow[(HttpRequest, Unit), (Try[HttpResponse], Unit), HostConnectionPool], where the Unit is a correlation ID you can use if you want to know which request corresponds to the response arrived, and HostConnectionPool materialized value can be used to shut down the connection to the host. Only cachedHostConnectionPool gives you back this materialized value, superPool probably handles this on its own (though I haven't checked). Anyway, I recommend you just use Http().shutdownAllConnectionPools() upon shutdown of your application unless you need otherwise for some reason. In my experience, it's much less error prone (e.g. forgetting the shutdown).
You can also use Graph DSL, to express the same graph:
val graph = Flow.fromGraph(GraphDSL.create() { implicit b =>
import GraphDSL.Implicits._
val host1Flow = b.add(Http().cachedHostConnectionPool[Unit](uri.format(p, id)))
val host2Flow = b.add(Http().superPool[Unit]())
val toInitialRes = b.add(
Flow[(Try[HttpResponse], Unit)]
.collect {
case (Success(HttpResponse(StatusCodes.OK, _, entity, _)), _) =>
val initialRes = entity.dataBytes
.via(JsonFraming.objectScanner(Int.MaxValue))
.map(bStr => ChunkStreamPart(bStr.utf8String))
Some(initialRes)
case (Success(resp # HttpResponse(code, _, _, _)), _) =>
log.error("Request to test job failed, response code: " + code)
// Discard the flow to avoid backpressure
resp.discardEntityBytes()
None
}
)
val keepOkStatus = b.add(
Flow[Option[Source[HttpEntity.ChunkStreamPart, Any]]]
.collect {
case Some(initialRes) => initialRes
}
)
val toOtherHost = b.add(
Flow[Source[HttpEntity.ChunkStreamPart, Any]]
.map { initialRes =>
(HttpRequest(
method = HttpMethods.POST,
uri = "googl.cm/flow",
entity = HttpEntity.Chunked(ContentTypes.`application/json`, initialRes)
),
())
}
)
host1Flow ~> toInitialRes ~> keepOkStatus ~> toOtherHost ~> host2Flow
FlowShape(host1Flow.in, host2Flow.out)
})
I'm a scala newbie trying to write a Rest Api using play framework. I have the following 3 data access methods
getDataDict: (dsType:String, name:String) => Future[Option[DatasetDictionary]]
getDatasetData: (DatasetDictionary) => Future[List[DatasetData]]
getMetadata: (DatasetDictionary) => Future[List[Metadata]]
I need to use these 3 methods to get the result of my async action method.
def index(dstype:String, name:String, metadata:Option[Boolean]) = Action.async{
/*
1. val result = getDataDict(type, name)
2. If result is Some(d) call getDatasetData
3.1 if metadata = Some(true)
call getMetadata function
return Ok((dict, result, metadata))
3.2 if metadata is None or Some(false)
return Ok(result)
4. If result is None
return BadRequest("Dataset not found")
*/
}
I got the steps 1 and 2 working as follows
def index1(dsType:String, dsName: String, metadata:Option[Boolean]) = Action.async {
getDataDict(dsType, dsName) flatMap {
case Some(x) => getDatasetData(x) map (x => Ok(Json.toJson(x)))
case None => Future.successful(BadRequest("Dataset not found"))
}
}
I'm stuck at how to get the metadata part working.
First of all, it is not very clear (d, result, x) what you really want to return. Hopefully I guessed it correctly:
def index(dstype:String, name:String, metadata:Option[Boolean]) = Action.async {
getDataDict(dstype, name) flatMap {
case Some(datasetDictionary) =>
getDatasetData(datasetDictionary) flatMap { datasetDataList =>
if (metadata == Some(true)) {
getMetadata(datasetDictionary) map { metadataList =>
Ok(Json.toJson((datasetDictionary, datasetDataList, metadataList)))
}
} else {
Future.successful(Ok(Json.toJson(datasetDataList)))
}
}
case None => Future.successful(BadRequest("Dataset not found"))
}
}
I am trying to create a stream which polls a rest service and unmarshals the json object.
I have created a source.tick which does a http request every 5 seconds. If this succeeds the HttpResponse will contain an OK. If not the service is unavailable. The result will be send to an actor. See the following code:
def poll(pollActor: ActorRef) {
val source = Source.tick(0.seconds, 3.seconds, HttpRequest(uri = Uri(path = Path("/posts/1"))))
val flow = Http().outgoingConnectionHttps("jsonplaceholder1.typicode.com").mapAsync(1) {
case HttpResponse(StatusCodes.OK, _, entity, _) =>
Unmarshal(entity).to[Item]
case resp # HttpResponse(code, _, _, _) =>
log.warning("Request failed, response code: " + code)
Future.failed(new Exception)
}
source.via(flow).runWith(Sink.actorRef[Equals](pollActor,akka.actor.Status.Success(())))
}
An actor will receive the result from the stream as can be seen in the following code:
def receive = {
case k : Item => println(k)
case f : Failure => {
println("We failed: " + f)
}
}
Where and how should I handle the exception thrown by the future?
One way of approaching this is to make failures an explicit part of your stream.
val flow = Http().outgoingConnectionHttps("jsonplaceholder1.typicode.com").mapAsync(1) {
case HttpResponse(StatusCodes.OK, _, entity, _) =>
Unmarshal(entity).to[Item].map(Right(_))
case resp # HttpResponse(code, _, _, _) =>
Future.successful(Left(MyDomainFailure("some meaningful message/data")))
}
Note that now the type of your flow is
Flow[HttpRequest, Either[MyDomainFailure, Item], Future[OutgoingConnection]]
This has the added value of clarity, making downstream stages aware of the failure and forcing them to handle it (well, in this case not really, because you're using an actor. If you stay within the realm of streams, you will be forced to handle them).
def receive = {
case Right(item) => println(item)
case Left(failure) => {
println("We failed: " + failure.msg)
}
}
This was the fix I used, while it does not produce an exception, the Failure contained in the HttpResponse is simply matched in the receive function.
def poll(pollActor: ActorRef) {
val source = Source.tick(0.seconds, 3.seconds, HttpRequest(uri = Uri(path = Path("/posts/1"))))
val flow = Http().outgoingConnectionHttps("jsonplaceholder1.typicode.com").mapAsync(1) {
// Where able to reach the API.
case HttpResponse(StatusCodes.OK, _, entity, _) =>
// Unmarshal the json response.
Unmarshal(entity).to[Item]
// Failed to reach the API.
case HttpResponse(code, _, _, _) =>
Future.successful(code)
}
source.via(flow).runWith(Sink.actorRef[Any](pollActor,akka.actor.Status.Success(())))
}
Here we match the Failure produced by the HttpResponse.
def receive = {
case item: Item => println(item)
case failure: Failure => {
log.warning("Request failed, response code: " + failure)
}
}
I have 3 futures of type Response. The first future returns a JsValue which defines if future 2 and future 3 shall be executed or only future 3 shall be executed.
Pseudocode:
If Future 1 then {future2 and future 3}
else future 3
Iam trying to do this in a play framwork action which means in order to use the result of the futures later I cant use onSuccess, onFailure and onComplete because all of them return Unit and not the actual JsValue from the last future.
I tried to do this with map() and andThen but I am a Scala noob and I guess i wasn't able to do it because I always just missed a little point. Here is my current approach which does not work!
def addSuggestion(indexName: String, suggestion: String) = Action.async {
val indexExistsQuery: IndexExistsQuery = new IndexExistsQuery(indexName);
val addSuggestionQuery: AddSuggestionQuery = new AddSuggestionQuery(indexName, suggestion)
val indexCreationQuery: CreateCompletionsFieldQuery = new CreateCompletionsFieldQuery(indexName)
val futureResponse: Future[Response] = for {
responseOne <- EsClient.execute(indexExistsQuery)
responseTwo <- EsClient.execute(indexCreationQuery) if (indexExistsQuery.getBooleanResult(indexExistsQuery.getResult(responseOne)))
responseThree <- EsClient.execute(addSuggestionQuery)
} yield responseThree
futureResponse.map { response =>
Ok("Feed title: " + (response.json))
}
I created some pseudocode:
checkIndexExists() map {
case true => Future.successful()
case false => createIndex()
} flatMap { _ =>
query()
}.map { response =>
Ok("Feed title: " + (response.json))
}.recover {
case _ => Ok("bla")
}
First you fire up the query if the index exists.
Then you map that future how to work with that Future[Boolean] if it successful. Since you use map, you kind of extract the Boolean. If the index exists, you just create a future that is already complete. If the index not exists you need to fire up the index creation command. Now you have the situation that you have nested Future's (Future[Future[Response]]). Using flatMap you remove one dimension, so that you only have Future[Response]. That can be mapped to a Play result.
Update (the implementation of MeiSign):
EsClient.execute(indexExistsQuery) map { response =>
if (indexExistsQuery.getBooleanResult(indexExistsQuery.getResult(response))) Future.successful(Response)
else EsClient.execute(indexCreationQuery)
} flatMap { _ =>
EsClient.execute(addSuggestionQuery)
} map { response: Response =>
Ok("Feed title: " + (response.json))
}
I found this solution but I dont think that it is a good solution because Iam using Await.result() which is a blocking operation.
If anyone knows how to refactor this code without blocking operations please let me know.
def addSuggestion(indexName: String, suggestion: String) = Action.async {
val indexExistsQuery: IndexExistsQuery = new IndexExistsQuery(indexName);
val addSuggestionQuery: AddSuggestionQuery = new AddSuggestionQuery(indexName, suggestion)
val indexCreationQuery: CreateCompletionsFieldQuery = new CreateCompletionsFieldQuery(indexName)
val indexExists: Boolean = indexExistsQuery.getBooleanResult(indexExistsQuery.getResult(Await.result(EsClient.execute(indexExistsQuery), 5.second)))
if (indexExists) {
EsClient.execute(addSuggestionQuery).map { response => Ok("Feed title: " + (response.json)) }
} else {
val futureResponse: Future[Response] = for {
responseTwo <- EsClient.execute(indexCreationQuery)
responseThree <- EsClient.execute(addSuggestionQuery)
} yield responseThree
futureResponse.map { response =>
{
Ok("Feed title: " + (response.json))
}
}.recover { case _ => Ok("bla") }
}
}