Consecutive conditional Future calls (in Play Scala server) not working - scala

I am using the below code to fetch news from a remote site in my Play Scala application server (2.5.x). I am using the suggested mechanism from "https://www.playframework.com/documentation/2.5.x/ScalaWS".
The use case is in 2 steps:
Call remote API-2 to fetch the news content using a cached token.
If the token is invalid/expired, call remote API-1 to get a token and cache it.
Problem: I am not able to avoid step #2 even if the cached token is valid. Therefore I am fetching the token for each call of this Action in listRemoteNews.
If I use if-else conditions, then I get compile errors as below. Note that I have tried for comprehensions, too, but I need to still use if conditions there. Then I get the same compile errors.
[error] /sandbox/git/play-scala-app.repo/app/controllers/remoteapi/ListRemoteNewsController.scala:26: overloaded method value async with alternatives:
[error] [A](bodyParser: play.api.mvc.BodyParser[A])(block: play.api.mvc.Request[A] => scala.concurrent.Future[play.api.mvc.Result])play.api.mvc.Action[A] <and>
[error] (block: play.api.mvc.Request[play.api.mvc.AnyContent] => scala.concurrent.Future[play.api.mvc.Result])play.api.mvc.Action[play.api.mvc.AnyContent] <and>
[error] (block: => scala.concurrent.Future[play.api.mvc.Result])play.api.mvc.Action[play.api.mvc.AnyContent]
[error] cannot be applied to (scala.concurrent.Future[Any])
[error] def listRemoteNews = Action.async {
Code section below (requestOne and requestTwo are methods forming WSRequest and returning Future[WSResponse]):
#Singleton
class ListRemoteNewsController #Inject()(ws: WSClient)(implicit context: ExecutionContext) extends Controller {
def listRemoteNews = Action.async {
println("<--- Stored Token 1: " + remoteToken)
val responseTwo: Future[WSResponse] = requestTwo.withQueryString("token" -> remoteToken).get()
val resp: Future[JsValue] = responseTwo.map {
response =>
response.json
}
resp.map {
response => {
Ok(response)
}
}
println("<-- Get token...")
val responseOne: Future[WSResponse] = requestOne
val resp2: Future[String] = responseOne.map {
response => {
remoteToken = (response.json \ "token_list" \ "token").as[String]
println("<--- Obtained token 2: " + remoteToken)
remoteToken
}
}
val responseThree: Future[WSResponse] = requestTwo.withQueryString("token" -> remoteToken).get()
responseThree.map {
response => {
println("<--- status: " + response.status)
Ok(response.json)
}
}
}

Futures are things that must be composed. Right now, you're making a number of different requests, which each return futures, but you're not returning those futures or combining them together in any way. Mapping a future is pointless if you don't do anything with the new future returned by the map function.
What you need to do is compose your futures, and this is done using methods like flatMap.
So here's roughly what I'm guessing your code should look like:
requestTwo.withQueryString("token" -> remoteToken).get().flatMap { response =>
if (tokenWasValid(response)) {
Future.successful(Ok(response.json))
} else {
requestOne.get().flatMap { tokenResponse =>
val remoteToken = (response.json \ "token_list" \ "token").as[String]
requestTwo.withQueryString("token" -> remoteToken).get()
}.map { response =>
Ok(response.json)
}
}
}
If the above makes no sense whatsoever, then I recommend you stop and read up on Scala futures before doing anything else.
Here's a selection of tutorials I found by doing a quick Google search:
http://code.hootsuite.com/introduction-to-futures-in-scala/
https://danielasfregola.com/2015/04/01/how-to-compose-scala-futures/
https://doc.akka.io/docs/akka/current/scala/futures.html

Related

Facing issue testing akka http cache

I am using Akka HTTP cache to cache my result. But i am facing issue to test it.
class GoogleAnalyticsController #Inject()(cache: Cache[String, HttpResponse],
googleAnalyticsApi: GoogleAnalyticsTrait,
googleAnalyticsHelper: GoogleAnalyticsHelper)
(implicit system: ActorSystem, materializer: ActorMaterializer) {
def routes: Route =
post {
pathPrefix("pageviews") {
path("clients" / Segment) { accountsClientId =>
entity(as[GoogleAnalyticsMetricsRequest]) { googleAnalyticsMetricsRequest =>
val googleAnalyticsMetricsKey = "key"
complete(
cache.getOrLoad(googleAnalyticsMetricsKey, _ => getGoogleAnalyticsMetricsData(accountsClientId, googleAnalyticsMetricsRequest))
)
}
}
}
}
private def getGoogleAnalyticsMetricsData(accountsClientId: String,
request: GoogleAnalyticsMetricsRequest) = {
val payload = generate(request)
val response = googleAnalyticsApi.googleAnalyticsMetricResponseHandler(accountsClientId, payload) // response from another microservice
googleAnalyticsHelper.googleAnalyticsMetricResponseHandler(
googleAnalyticsMetricsRequest.metricName, response)
}
}
class GoogleAnalyticsHelper extends LoggingHelper {
def googleAnalyticsMetricResponseHandler(metricName: String, response: Either[Throwable, Long]): Future[HttpResponse] =
response.fold({ error =>
logger.error(s"An exception has occurred while getting $metricName from behavior service and error is ${error.getMessage}")
Marshal(FailureResponse(error.getMessage)).to[HttpResponse].map(httpResponse => httpResponse.copy(status = StatusCodes.InternalServerError))
}, value =>
Marshal(MetricResponse(metricName, value)).to[HttpResponse].map(httpResponse => httpResponse.copy(status = StatusCodes.OK))
)
}
Test case: Sharing only the relevant part
"get success metric response for " + pageviews + " metric of given accounts client id" in { fixture =>
import fixture._
val metricResponse = MetricResponse(pageviews, 1)
val eventualHttpResponse = Marshal(metricResponse).to[HttpResponse].map(httpResponse => httpResponse.copy(status = StatusCodes.OK))
when(cache.getOrLoad(anyString, any[String => Future[HttpResponse]].apply)).thenReturn(eventualHttpResponse)
when(googleAnalyticsApi.getDataFromGoogleAnalytics(accountsClientId, generate(GoogleAnalyticsRequest(startDate, endDate, pageviews))))
.thenReturn(ApiResult[Long](Some("1"), None))
when(googleAnalyticsHelper.googleAnalyticsMetricResponseHandler(pageviews, Right(1))).thenReturn(eventualHttpResponse)
Post(s"/pageviews/clients/$accountsClientId").withEntity(requestEntity) ~>
googleAnalyticsController.routes ~> check {
status shouldEqual StatusCodes.OK
responseAs[String] shouldEqual generate(metricResponse)
}
}
By doing this, I am best to test if the cache has the key but not able to test if cache misses the hit. In code coverage, it misses following highlighted part
cache.getOrLoad(googleAnalyticsMetricsKey, _ =>
getGoogleAnalyticsMetricsData(accountsClientId,
googleAnalyticsMetricsRequest))
If there is a design issue, please feel free to guide me on how can I make my design testable.
Thanks in advance.
I think you don't need to mock the cache. You should create an actual object for cache instead of mocked one.
What you have done is, you have mocked the cache, in this case, the highlighted part will be not called as you are providing the mocked value. In the following stubbing, whenever cache.getOrLoad is found, eventualHttpResponse is returned:
when(cache.getOrLoad(anyString, any[String => Future[HttpResponse]].apply)).thenReturn(eventualHttpResponse)
and hence the function getGoogleAnalyticsMetricsData(accountsClientId, googleAnalyticsMetricsRequest) is never called.

required: play.api.mvc.Request[?] => play.api.mvc.Result

I am migrating to Play 2.6 and have the following API wrapper functions that used to work:
trait API {
self: Controller =>
def api(businessLogic: Request[AnyContent] => Any): Action[AnyContent] = apiWithBody(parse.anyContent)(businessLogic)
def apiWithBody[A](bodyParser: BodyParser[A])(businessLogic: Request[A] => Any): Action[A] = Action(bodyParser) {
implicit request =>
{
val apiResult = businessLogic(request)
val start = new java.util.Date().getTime
val actionDuration = (new java.util.Date().getTime - start)
val response = resultFrom(apiResult, request, actionDuration) // Returns a Result
response
}
}
}
Called by Controller functions like:
object Accounts extends Controller with API {
def all = superUser {
implicit principal =>
api {
request =>
models.Account.all
}
}
}
Where superUser is the principal (user) type "admin".
And get the following compiler error:
[error] type mismatch;
[error] found : play.api.mvc.Action[play.api.mvc.AnyContent]
[error] required: play.api.mvc.Request[?] => play.api.mvc.Result
[error] api {
[error] ^
I'm building with sbt 1.1.5 and Scala 2.11.8.
I am guessing the [?] means the compiler doesn't know what type is required but I don't understand what is wrong. I have searched for this issue but not found the specific answer for this problem.
In addition I'm getting an error:
[error] could not find implicit value for parameter parser: play.api.mvc.BodyParser[Any]
[error] def all = superUser {
[error] ^
that I posted as a separate issue (see could not find implicit value for parameter parser: play.api.mvc.BodyParser[Any]) but might be relevant here?
def superUser[A](f: => Principal => Request[A] => Result)(implicit parser: BodyParser[A]): SecureAction[A] = {
_superUser {
user =>
implicit val principal = data.Principal(user)
Action(parser)(request => f(principal)(request))
}
}
private def _superUser[A](action: String => Action[A]) = {
play.api.mvc.Security.Authenticated(getSuperUser, onUnauthorized)(action)
}
Any help would be appreciated.
Sorry I'm little bit confused here about the architecture as:
Where is the API call? Is it within the model? Are you calling outside the current Play app? Why don't you use the Future API for it? Because then you can recover it. Which then help you with logging and error handling.
The all method get the request, and then it does not return an HTTP response. Why don't you pass on what you need from the request (e.g., the Cookie).
I think the answer to your question is to use your action composition with action. Something like:
def myMethod(queryParam: String) = myDefinedAction compose Action { ??? }
//In case you want to use the Future API, then you need to be async
def myMethod(queryParam: String) = (myDefinedAction compose Action).async {
implicit request =>
apiCall.map{
case _ => Ok("good request")
}.recover{case _ => BadRequest}
}

Scala/Play: how to write JSON result from async db call to http Ok

Q: Where is the correct place to be calling Ok() to send http response from an async database call?
I have taken the very basic Scala Play framework tutorial play-scala-starter-example as a starting point and adding some additional basic Controller / Service classes that make use of ReactiveCouchbase for database access.
The application successfully:
Connects to Couchbase
Stores a JSON document in Couchbase
Retrieves the stored JSON document from Couchbase
Logs the contents of the JSON to the console
I am new to Scala/Play and can't work out the correct way to succesfully write the JSON back to the http response using Ok(), when the async db call completes.
Inside the Controller class is the following function:
def storeAndRead() = Action {
testBucket
.insert[JsValue]("key1", Json.obj("message" -> "Hello World", "type" -> "doc"))
val res = testBucket
.get("key1")
.map(i => Json.toJson(i.get))
.map(j => Ok(j)) // PROBLEM LINE
}
Looking at the "//PROBLEM LINE", having the Ok() call inside the map leads to a compile error:
CouchbaseController.scala:30:19: Cannot write an instance of Unit to HTTP response. Try to define a Writeable[Unit]
Placing the call to Ok() later on, fails with a different compile error:
def storeAndRead() = Action {
testBucket
.insert[JsValue]("key1", Json.obj("message" -> "Hello World", "type" -> "doc"))
val res = testBucket
.get("key1")
.map(i => Json.toJson(i.get))
Ok(res)
}
Compile error:
CouchbaseController.scala:35:7: Cannot write an instance of scala.concurrent.Future[play.api.libs.json.JsValue] to HTTP response. Try to define a Writeable[scala.concurrent.Future[play.api.libs.json.JsValue]]
In this second case, I believe the issue is that the Future may not yet have completed when Ok() is called?
Lastly, I have tried placing the call to Ok() inside an onSuccess() function, in an effort to ensure it is called after the async function has completed:
def storeAndRead() = Action {
testBucket
.insert[JsValue]("key1", Json.obj("message" -> "Hello World", "type" -> "doc"))
val res = testBucket
.get("key1")
.map(i => Json.toJson(i.get))
.onSuccess {
//case doc => Console.println("completed: " + doc)
case doc => Ok(doc)
}
}
Again...compilation error:
CouchbaseController.scala:22:24: overloaded method value apply with alternatives:
[error] (block: => play.api.mvc.Result)play.api.mvc.Action[play.api.mvc.AnyContent] <and>
[error] (block: play.api.mvc.Request[play.api.mvc.AnyContent] => play.api.mvc.Result)play.api.mvc.Action[play.api.mvc.AnyContent] <and>
[error] [A](bodyParser: play.api.mvc.BodyParser[A])play.api.mvc.ActionBuilder[play.api.mvc.Request,A]
[error] cannot be applied to (Unit)
[error] def storeAndRead() = Action {
Question:
I am clearly missing something fairly fundamental:
Where should Ok() be called in this kind of basic scenario? I assume it needs to be called as a result of a callback when the async db request has completed?
What's the correct and appropriate way to structure this in an async way for Scala/Play?
Handle future results using Action.async
Play knows how to handle Future (async call). You have to use Action.async.
For instance :
def myAction = Action.async {
// ...
myFuture.map(resp => Ok(Json.toJson(resp)))
}
In your case :
def storeAndRead() = Action.async {
// by the way, the expression behind probably returns a future, you should handle it
testBucket
.insert[JsValue]("key1", Json.obj("message" -> "Hello World", "type" -> "doc"))
testBucket
.get("key1")
.map(i => Json.toJson(i.get))
.map(j => Ok(j))
}
You have to return a Result (or a Future[Result])
You get the error CouchbaseController.scala:30:19: Cannot write an instance of Unit to HTTP response. Try to define a Writeable[Unit] because you don't return anything. A Result is expected here.
Handle futures chain
Futhermore, you should handle the call of several Futures. If you don't, you will get silent errors even if the client received the http response.
For instance :
def storeAndRead() = Action.async {
for {
_ <- testBucket.insert[JsValue]("key1", Json.obj("message" -> "Hello World", "type" -> "doc"))
value <- testBucket.get("key1")
} yield Ok(Json.toJson(value))
}

Basic akka http client post not working

I am trying to make a basic http client posting some data to a REST API with Akka HTTP but I cannot make the following code work :
def sendData(e : GenericEvent): Future[Either[(String,StatusCode),(GenericEvent,StatusCode)]] = {
val request = Marshal(e).to[RequestEntity]
val responseFuture: Future[HttpResponse] = request map { req =>
Source.single(HttpRequest(method = HttpMethods.POST, uri = s"/data-ingest", headers = List(auth), entity = req))
.via(dataIngestFlow)
.runWith(Sink.head)
}
responseFuture.flatMap { response =>
response.status match {
case OK => Unmarshal(response.entity).to[GenericEvent].map(Right(_, response.status))
case BadRequest => Future.successful(Left(s"$e.data: incorrect data", response.status))
case _ => Unmarshal(response.entity).to[String].flatMap { entity =>
val error = s"generic event ingest failed with status code ${response.status} and entity $entity"
logger.error(error)
Future.failed(new IOException(error))
}
}
}
I got the following error
polymorphic expression cannot be instantiated to expected type;
[error] found :
[T]akka.stream.scaladsl.Sink[T,scala.concurrent.Future[T]]
[error] required:
akka.stream.Graph[akka.stream.SinkShape[akka.http.scaladsl.model.HttpResponse],akka.http.scaladsl.model.HttpResponse]
[error] .runWith(Sink.head)
Here is the code for the dataIngestFlow
val dataIngestFlow = Http().outgoingConnection(config.endpointUrl,config.endpointPort)
Here is the code on server side :
val routes = {
logRequestResult("akka-http-microservice") {
path("event-ingest") {
post {
entity(as[GenericEvent]) { eventIngest =>
log.info(s"Ingesting {} and publishing event to Kafka topic {}.", eventIngest.eventType,config.kafkaTopic)
kafka ! eventIngest
complete {
eventIngest
}
}~
entity(as[List[GenericEvent]]) { eventIngestList =>
eventIngestList.foreach{ eventIngest=>
log.info(s"Ingesting {} and publishing event List to Kafka topic {}.", eventIngest.eventType,config.kafkaTopic)
kafka ! eventIngest
}
complete {
eventIngestList
}
}
}
}
}
}
I tried another simple client, it builds well but the ingestion stop after 160 events, the server doesn't receive anymore events.
The first problem I see with your example is that RequestEntity does not have a map function. Therefore, the following line
val responseFuture: Future[HttpResponse] = request map { ...
should not compile.
Further, if request is actually a Future (which I infer from the map) then responseFuture is actually of type Future[Future[HttpResponse]] because the stream materializes into its own Future. To solve this problem you can use Future.flatMap instead of map. Namely:
val responseFuture: Future[HttpResponse] = request flatMap { req =>
This is the monadic bind operation within Futures.

Sending POST Request to Twitter API with Play Framework 2.0

How do I send a POST request to twitter API using Play Framework 2.0 (with Scala)? The API I'm trying to call work with both GET and POST, and I've successfully called it using GET with this code:
val followersURL = "http://api.twitter.com/1/users/lookup.json?user_id=" + listOfFollowers.mkString(",")
WS.url(followersURL)
.sign(OAuthCalculator(Twitter.KEY, tokens))
.get()
.map{ response =>
val screenName: Seq[String] = response.json match {
case res: JsArray => res.value.map{ value => (value \ "name").toString }
case _ => Seq("")
}
}
Then I tried to call the API using POST like this:
WS.url("http://api.twitter.com/1/users/lookup.json")
.sign(OAuthCalculator(Twitter.KEY, tokens))
.post(Map("user_id"->listOfFollowers))
.map { response =>
val screenName: Seq[String] = response.json match {
case res: JsArray => res.value.map{ value => (value \ "name").toString }
case _ => Seq("")
}
}
It didn't work and I get this exception:
[error] play - Waiting for a promise, but got an error: null
java.lang.NullPointerException: null
at java.io.Reader.<init>(Unknown Source) ~[na:1.7.0_01]
at java.io.InputStreamReader.<init>(Unknown Source) ~[na:1.7.0_01]
at oauth.signpost.OAuth.decodeForm(OAuth.java:157) ~[signpost-core.jar:na]
at oauth.signpost.AbstractOAuthConsumer.collectBodyParameters(AbstractOAuthConsumer.java:236) ~[signpost-core.jar:na]
at oauth.signpost.AbstractOAuthConsumer.sign(AbstractOAuthConsumer.java:96) ~[signpost-core.jar:na]
at play.api.libs.oauth.OAuthCalculator.sign(OAuth.scala:106) ~[play_2.9.1.jar:2.0.1]
Since it says that the exception occurs on the OAuthCalculator, I try to comment out the .sign call, and it didn't throw any exception, but of course I didn't get the right result.
Am I doing something wrong? What am I doing wrong, and why? How could I fix the problem?
Thanks before.
I have found this to work:
WS.url("http://api.twitter.com/1/users/lookup.json?user_id="+listOfFollowers)
.sign(OAuthCalculator(Twitter.KEY, tokens))
.post("ignored")
.map { response =>
val screenName: Seq[String] = response.json match {
case res: JsArray => res.value.map{ value => (value \ "name").toString }
case _ => Seq("")
}
}
I've also made notes to revisit my code with every major upgrade of Play! to check if the above gets fixed, because this is obviously not right.
Once you use the Play Framework tools for getting your user's token and secret, you can then use the twitter4j ("org.twitter4j" % "twitter4j-core" % "3.0.3") library to do your posting like this.
import twitter4j.conf.ConfigurationBuilder
import twitter4j.{StatusUpdate, TwitterFactory}
val config = new ConfigurationBuilder()
.setOAuthConsumerKey(twitterKey)
.setOAuthConsumerSecret(twitterSecret)
.setOAuthAccessToken(token)
.setOAuthAccessTokenSecret(secret)
.build()
val twitter = new TwitterFactory(config).getInstance()
val status = new StatusUpdate(tweet)
status.media(photoName, stream)
val twitResp = twitter.updateStatus(status)
Annoying to have to use two libraries, and twitter4j isn't async so it's slightly less resource-efficient, but it lets you do real posts which are necessary sometimes.